Lenses provides a secure access layer to streaming data for any Apache Kafka (self-managed, AWS MSK, HDInsight etc.).
As well as being able to query data via SQL, you can also produce data into a topic. This adds an extra level of security and auditing since Lenses works at the application layer and is protected by fine-grained security via namespaces.
Let’s take a look at four different ways of injecting data into a Kafka Topic via Lenses.
To make it easier to follow this guide please download the free Lenses Docker “Box” image from here.
Additionally, you should get the Lenses CLI here.
As Lenses comes with sample Kafka topics populated with data, this tutorial will use an existing Kafka topic.
There exist two distinct ways to insert data into a Kafka topic from Lenses UI. By inserting your own events or by inserting data that is automatically generated by Lenses.
From the SQL Studio page, simply write the following SQL code:
One way of verifying data has been inserted is via the
Topics page and selecting
Alternatively, you can execute the following
SELECT query in SQL Studio:
SELECT * FROM cc_data where _key='Using-sql-studio'
Topics page, select
From the bottom on upper-right side of page, select
Autogenerate button Lenses will create an event based on random data that conforms to the schema such as:
The key for the record can be either specified by you or by Lenses and Kafka. In
this case it is specified by us and will be
using-auto-generate. After that you
will need to press the
+ Insert Records button and you are done.
SELECT query executed in SQL Studio will verify that the data is in
SELECT * FROM cc_data where _key='using-auto-generate'
The remaining three ways will use Lenses Data API to insert data into an existing Kafka topic.
lenses-cli shell command that will put you in the interactive
lenses-cli shell. The following output shows the interaction that allows you to insert data into a Kafka topic:
You can verify that the record is there by executing the following SQL statement:
This method uses a command line utility like
curl(1) to access Lenses API and
insert data into a Kafka topic. You will need to have your data stored in JSON format.
In order to connect to Lenses using
curl(1), you will need to have a authentication token
that is connected to your current connection. In order to get that authentication token you
will need to execute the following command first:
The output of that command will be the authentication token, which will have a format
16a8b2dd-b2d0-449e-8203-9dd51d7950cf. As this functionality is embedded
in the presented
bash(1), you will not need to remember the authentication token.
Lenses also provides the ability to create service account tokens.
The data will be in a JSON file named
/tmp/myData.json with the following contents:
bash(1) shell script named
insertData.sh will automate the process and make it
more user friendly:
insertData.sh script accepts one command line parameter, which is the name the
of the JSON file. Executing
insertData.sh will create the following kind of output:
The request has been fulfilled and resulted in a new resource being created.
You can verify that the insert operation was successful using
Here we are going to develop a Lenses client. We will use Python 3 but you can use any programming language you want provided that it supports the needed functionality.
The name of the Python 3 utility is
insertData.py and contains the following code:
Do not be put off by the code, it is used for illustrating the capabilities of Lenses and how easy it is to insert data to a Kafka topic.
The script has two main parts. The first one has to do with connecting to Lenses and uses the
connect() function whereas the second one is about executing the desired SQL statement
with the help of the
Note that you might need to install some extra Python 3 package for
to work on your local machine.
insertData.py will generate the following output:
The output of
insertData.py verifies that the
INSERT statement was successful.
Additionally, the image that follows will also verify that the data is inserted in the desired Kafka topic.
Using Lenses as a secure data access layer is a great way to safely open up Apache Kafka to more developers, users and applications. Learn more about Lenses from the different usecases or get access to a free trial.