photo of Antonios Chalkiopoulos
Antonios Chalkiopoulos

7 Aug 2016 Read in about 3 min

Hey there,
You are looking at an older version. Go to Lenses to view data in Kafka.

Kafka is now the de-facto platform for streaming architectures, and it’s eco-system is maturing, but is not just yet as Enterprise Ready as many people in Big | Fast Data would like it to be. Lenses.io is a London based start-up that wants to drive Kafka faster to the future, and thus..

We are announcing the kafka-topics-ui a User Interface that allows browsing data from Kafka Topics and a lot more

About Kafka-Topics-UI

The kafka-topics-ui is a user interface that interacts with the Kafka rest-proxy to allow browsing data from Kafka Topics. At the same time it brings visibility by providing a single entry point to explore i) Kafka data, ii) Kafka Schemas, iii) Kafka connectors and a lot more, such as partitions per topic, replication factor per topic and topic configurations.

Kafka REST Proxy

The Kafka REST Proxy - is an excellent state-less micro-service by Confluent that allows anyone to easily:

  • Create a consumer or producer via a POST request
  • Consume or Produce Avro | JSON | Binary data on a Kafka Topic
  • Delete a consumer

Utilizing it’s capabilities for building a UI was a straight-forward processes after overcoming the initial difficulties. First and foremost we had to tackle the fact you can’t tell a priory what type of data a particular topic contains. Once we figured out that, we can

  1. hard-code the type of particular config-topics
  2. automatically detect Avro type by fetching the _schemas topic
  3. in case type is still unknown iterate through Avro->JSON->Binary consumers till one succeeds

Then things got so much more interesting. Another hidden gem was the flag - that allows us to efficiently stream just enough data to feed the UI and achieve the above process in just 2-3 seconds.

?max_bytes=50000

Kafka-Topics-UI features

Currently

  • Identifies and visualizes Kafka topics and config overrides
  • Automatically detects data types (Avro|JSON|Binary)
  • Base64 decodes and parses of binary topics
  • Provides Table and JSON views
  • Allows downloading data from Kafka topics
  • Displays number of partitions and replication factor per topic
  • Docker container

Using it

Check https://github.com/landoop/kafka-topics-ui and follow instructions (Tip: Remember to enable CORS). The easiest way to get it running is using docker:

docker pull landoop/kafka-topics-ui

And run the container, by specifying the url of the kafka-rest service:

docker run --rm -it -p 8000:8000 \
       -e "KAFKA_REST_PROXY_URL=http://localhost:8082" \
       landoop/kafka-topics-ui

Quick preview

Conclusions

Hey there,
This is an old version.
Check Lenses to view data in Kafka or the new version for Kafka Topic UI here.

Lenses.io - similarly to you continuously brings to light best practices while thriving for operational excellence. Together with the open-source community we are aiming to enhance kafka-topics-ui with new features and capabilities:

  • Stream from beginning or real-time capturing
  • DC/OS universe components for your Mesos cluster
  • Real-time monitoring

Find out more Tools

Lenses for Kafka

Lenses Box, Kafka development environment (topics, schemas, connectors & Kafka all-in-one!)

Tutorials

- Kafka Connect Pipelines, sink to Elasticsearch
- Time-Series with Kafka, Kafka Connect & InfluxDB
- From MQTT to Kafka with Connect and Stream Reactor


Did you like this article?