Unified data observability across Kafka & Elasticsearch

In Lenses 3.1 we introduced the ability to explore and discover data and metadata in Elasticsearch indices. All in the same self-service fashion that you currently use Lenses to explore data in your Kafka topics via a UI or CLI with SQL.

The unified visibility and data discovery is important to meet data governance requirements.

It's also very powerful for developers or data engineers to investigate whilst building or debugging a flow or microservice without all the back n fourths between different tools and teams.

In this blog post we are going to walk your through how to connect Lenses to an existing Elasticsearch server and explore data.

Pre-requisites

In order to be able to follow this blog, you will need the following:

• Lenses 3.1 (or a newer version when available) up and running. You can use our free Box instance if you want.

• An Elasticsearch server up and running.

• A working network connection between these two machines.

Creating the connection

In order for Lenses to be able to look at Elasticsearch indexes, you only need to create a connection using the relevant connection template from the Connection Manager.

First, you should go to the ADMIN panel and select Connections. Then, you should click on the New Connection button that will get you to the screen that allows you to select the service that you are going to use, which in this case is Elasticsearch.

Click on Elasticsearch box in order to type the details of the connection. This is illustrated in the next screenshot.

In this case, the hostname of the machine that runs Elasticsearch is Elasticsearch. You should change that to match your configuration. Elasticsearch servers usually listen to TCP port number 9200, which is what is used here.

Now that you have given the required information, press the Save Connection button to create the connection and you are done. Lenses automatically generates the following screen:

Your connection will be now listed on the Connections tab of the ADMIN panel.

After that you should go to the DASHBOARD screen and press the Elasticsearch Indexes option on the left column of Lenses UI. The output depends on the Elasticsearch installation – in our case the output will be as follows:

Clicking on the names index will make Lenses to display its data:

Not only does this give you visibility over indexes, metadata and entities in your Elasticsearch environment but also provides data observability, enabling you to explore data using the same SQL engine that our customers love for Kafka.

Next Steps

Now that you know how easy it is to view Elasticsearch indexes with Lenses 3.1, you should try it for yourself in your own environment.

Other Links

• Lenses Box 5 min tour

• Elasticsearch

• Lenses Features

• Lenses 3.1 full release notes