Secure layer to insert data into Apache Kafka

Lenses provides a secure access layer to streaming data for any Apache Kafka (self-managed, AWS MSK, HDInsight etc.).

As well as being able to query data via SQL, you can also produce data into a topic. This adds an extra level of security and auditing since Lenses works at the application layer and is protected by fine-grained security via namespaces.

Let’s take a look at four different ways of injecting data into a Kafka Topic via Lenses.

Pre-requisites

To make it easier to follow this guide please download the free Lenses Docker “Box” image from here.

Additionally, you should get the Lenses CLI here.

As Lenses comes with sample Kafka topics populated with data, this tutorial will use an existing Kafka topic.

Using Lenses UI

There exist two distinct ways to insert data into a Kafka topic from Lenses UI. By inserting your own events or by inserting data that is automatically generated by Lenses.

Inserting your own data

From the SQL Studio page, simply write the following SQL code:

One way of verifying data has been inserted is via the Topics page and selecting cc_data topic.

Alternatively, you can execute the following SELECT query in SQL Studio:

SELECT * FROM cc_data where _key='Using-sql-studio'

Using Auto Generated Data

From the Topics page, select cc_data topic.

From the bottom on upper-right side of page, select Insert Records.

By clicking Autogenerate button Lenses will create an event based on random data that conforms to the schema such as:

The key for the record can be either specified by you or by Lenses and Kafka. In this case it is specified by us and will be using-auto-generate. After that you will need to press the + Insert Records button and you are done.

The following SELECT query executed in SQL Studio will verify that the data is in cc_data:

SELECT * FROM cc_data where _key='using-auto-generate'

Using Lenses Data API

The remaining three ways will use Lenses Data API to insert data into an existing Kafka topic.

Using Lenses CLI

Execute the lenses-cli shell command that will put you in the interactive lenses-cli shell. The following output shows the interaction that allows you to insert data into a Kafka topic:

You can verify that the record is there by executing the following SQL statement:

Using the Command Line

This method uses a command line utility like curl(1) to access Lenses API and insert data into a Kafka topic. You will need to have your data stored in JSON format.

In order to connect to Lenses using curl(1), you will need to have a authentication token that is connected to your current connection. In order to get that authentication token you will need to execute the following command first:

The output of that command will be the authentication token, which will have a format similar to 16a8b2dd-b2d0-449e-8203-9dd51d7950cf. As this functionality is embedded in the presented bash(1), you will not need to remember the authentication token.

Lenses also provides the ability to create service account tokens.

The data will be in a JSON file named /tmp/myData.json with the following contents:

A bash(1) shell script named insertData.sh will automate the process and make it more user friendly:

The insertData.sh script accepts one command line parameter, which is the name the of the JSON file. Executing insertData.sh will create the following kind of output:

./insertData.sh /tmp/myData.json

The request has been fulfilled and resulted in a new resource being created.

You can verify that the insert operation was successful using lenses-cli shell:

Creating a Client in Python 3

Here we are going to develop a Lenses client. We will use Python 3 but you can use any programming language you want provided that it supports the needed functionality.

The name of the Python 3 utility is insertData.py and contains the following code:

Do not be put off by the code, it is used for illustrating the capabilities of Lenses and how easy it is to insert data to a Kafka topic.

The script has two main parts. The first one has to do with connecting to Lenses and uses the connect() function whereas the second one is about executing the desired SQL statement with the help of the execSQL() function.

Note that you might need to install some extra Python 3 package for insertData.py to work on your local machine.

Executing insertData.py will generate the following output:

The output of insertData.py verifies that the INSERT statement was successful.

Additionally, the image that follows will also verify that the data is inserted in the desired Kafka topic.

Conclusions

Using Lenses as a secure data access layer is a great way to safely open up Apache Kafka to more developers, users and applications. Learn more about Lenses from the different usecases or get access to a free trial.

Useful Links

• Lenses Quick Tour

• Lenses CLI documentation