How to Explore Data in Kafka Topics with Lenses

Lenses is a tool  with a powerful user interface for Apache Kafka used for  exploring historical or in-motion Apache Kafka data. This lets you easily access data for debugging, analyzing or reporting.

In this post, we explain how Lenses can help you find and explore Kafka data in any compatible system, on-premise or in the cloud.

Lenses has two query engines; SQL Studioand SQL Processing. Let's focus on the former, Lenses SQL Studio. This helps you quickly access data for debugging, analysis or reporting, without the need for a platform team ticket.

In addition, Lenses runs in your infrastructure - on prem or in the cloud. Lenses HQ acts as the Lenses control plane and provides the user interface. The other component: Lenses Agent runs alongside your Kafka ecosystem and facilities connectivity between HQ and Kafka processes. 

Let's get started.

Understanding Kafka topics

A Kafka topic is a named channel for a specific type of message, such as user activity, transactions, or logs. Kafka topic configurations control how data is stored and processed. This can include settings like the number of partitions, data retention policies, and replication factors.

With Lenses, you can easily list topics in Kafka, describe a topic’s structure, explore and show topics, or even delete topics (with the right permissions). 

Unlike traditional command-line tools, Lenses provides a user-friendly way to explore Kafka topics. That’s without needing to run terminal commands like Kafka list topics, Kafka delete topics, or Kafka describe topics.

Whether you're a developer, data scientist, or platform engineer navigating Kafka topic configurations, understanding partitions and retention settings, and viewing real-time data streams effortlessly is crucial.

Kafka data types

Lenses identifies your data types for the Key and the Value in every Kafka topic, even if they are different types. Out-of-the-box support is provided for any schema registry that is compatible with Apache Kafka’s APIs. So, Avro and Protobuf are automatically taken care of. 

However, Lenses goes beyond this. Even without schema registry support, it automatically detects your JSON schemas in your Kafka topics to determine the schema and format. For example, maybe it’s plain JSON, JSON with a Schema or even good old XML. This information is then used by the query engines to deserialize the messages.

You can see the detected type from the topics lists:

Even though auto detection kicks in, Lenses allows the user to specify the accurate types for keys and values by the correct format.

H2: Explore real-time data

Once you have searched for and found the stream you are interested in, you can explore this data with Lenses SQL Studio. 

The SQL Studio is powerful. It's designed for quick access, debugging, and exploring. It supports:

1. Filtering

2. Joins

3. Aggregates

4. Functions

5. User-defined functions

6. Access to key, value, headers and message metadata (offsets, timestamps and partitions)

Here is an example where we apply a simple filter for a specific partition within the Kafka topic:

Seeing consumers

Want to see consumer groups alongside your query results? Select “View Consumers” from the topic actions, and split the tabs right or left:

Pre-Canned

Don’t want to write SQL? Lenses will automatically display the most recent 

By default, Lenses will bring up to 200 events In this view, you can either query data by executing SQL queries in Apache Kafka topics or by using the data controls provided by the Filters function..

By using the filter function, you can specify specific fields, data and time ranges, and offsets ranges easily. 

The filter will apply the changes to your displayed data immediately:

Time travelling, functions, metadata

Using Lenses SQL Studio, you can access the Key and the Value of the message, but also its metadata. We saw above how to specify the partition. 

Here is an example of how to apply a time range: show me data from the last 10 days based on the message timestamp:

And of course, Lenses SQL editor includes auto-complete to help you further.

Data views

Lenses provides two different ways to explore your Kafka data: Grid and List. You can switch among the different views, and each view will display the current data set.

In Grid, Lenses flattens the data in an Excel-like format. This means you can sort data sets, and move and reorder the columns. From there, you can and  either copy the whole data set or part of it to your spreadsheets or any other relevant tool.

This is the default view in Global SQL Studio. However, you can also view it in a raw JSON-like format. With the list view.