How to integrate Apache Kafka flows into your CI/CD with Jenkins and Lenses
This tutorial will illustrate how you can integrate Lenses into your Jenkins CI/CD using the
The Lenses CLI allows you to manage your Apache Kafka environment using a single unified and secure API across all Kafka components (Kafka Brokers, Zookeepers, Kafka Connect, Schema Registry etc.).
Integrating Lenses with Jenkins will simplify automating the deployment of real time applications across your different Apache Kafka environments.
To demonstrate the integration, we will simply walk you through a very basic example of triggering a job that uses the
lenses-cli to list the available topics in your Apache Kafka cluster. See this blog for more details about managing your Apache Kafka flows via GitOps.
In order to be able to follow the steps of this blog, you will need the following:
Lenses up and running. You can use our free Box instance if necessary.
A Jenkins server up and running and accessible from the Internet. Additionally, the Jenkins server should have the required plugins for GitHub support installed.
A working network connection between these two machines.
A GitHub repository with administrative privileges on it.
The Jenkins job that will be implemented will download
lenses-cli and query a Lenses instance for information using the
We are going to use a public GitHub repository in order to avoid having to authenticate to GitHub from Jenkins, which will make the scenario more complex. The URL for the GitHub repository is https://github.com/mactsouk/jenkins-blog – you should use your own GitHub repository or any other repository as long as you can create a Webhook for it.
This section will illustrate the steps needed for implementing the scenario.
Note that both Jenkins and Lenses servers will be exposed to the Internet using the
ngrok utility, which explains the hostnames of the servers. The Jenkins server hostname is
15789550.ngrok.io whereas the Lenses server hostname is
In order to get the Webhook, you should go to the home page of your repository and then select the
Settings tab. Then, select the
Webhooks option on the left column and then click on the
Add Webhook button. The values that were used for the webhook are shown in the next screenshot:
After we are done, we will press the
Add Webhook button or the
Update Webhook if the webhook was already created. Note that you will need to add the
/github-webhook/ string at the end of the Jenkins hostname – this is really important for the scenario to work.
This step was needed in order for GitHub to inform Jenkins about changes to the GitHub repository.
Creating the Jenkins Job
This is the most critical part of the process as you will need to create a Jenkins job and correctly associate it with the appropriate GitHub repository.
Go to Jenkins UI and click on
create new jobs link. You will presented with the next screen:
Lenses as the name of the item and select the
Freestyle project. After that click on the
OK button at the bottom of the screen.
The next string is pretty long so it will be presented in two steps. The first part is about specifying the GitHub project URL and selecting the
Source Code Management tool, which will be Git and filling in the correct information – see the next screenshot for more information:
The second part is about selecting the
GitHub hook trigger for GITScm polling box and defining the job under the
Build title. On the
Add Build step menu select the
Execute shell item. After that, insert the desired UNIX commands in the box. The final configuration is presented in the following screenshot.
Save button and you are done with the definition of the job. After that, each time someone makes a push to the associated GitHub repository, the Jenkins job will be triggered.
bash shell commands that will be executed in the Jenkins job are the following:
tar xzvf lenses-cli-linux-386.tar.gz
mv lenses-cli-linux-386 lenses-cli
./lenses-cli --host http://a00474c0.ngrok.io --pass admin --user admin topics
uname informs us about the characteristics of the UNIX machine and is not necessary to be executed – it just gives you a good idea of the Linux system you will be working on.
wget command downloads the
lenses-cli archive, which is extracted using
tar and renamed to
lenses-cli connects to the desired Lenses server
(http://a00474c0.ngrok.io), using the specified username and password information and gets a list of the available Kafka topics. This will download the client each time the job runs, you may want to place the client in your Jenkins workspace in order to avoid unnecessary bandwidth use and improve execution time.
You can also authenticate using a service account if you do not want to use a regular account:
lenses-cli --host http://a00474c0.ngrok.io --token <service account name>:<service token> topics
This is a simple job but you can perform any job you want. Look at the GitHub Actions + Lenses blog post for learning how to insert data into Kafka through Lenses using shell commands and at lenses-cli documentation.
Pushing data to the GitHub repository
Now all you have to do is perform a push to the GitHub repository and the Jenkins job will be automatically triggered.
Watching the results
Click on the
#1 link, or on any other link that exists there, in order to see more information about the execution of the job.
Click on the
Console Output link of the left in order to see the output of the shell commands of the job.
You will see lots of output related to the execution of the
wget command - if you want less output from
wget, please use the
--quiet option. The bottom of the screen will look as follows:
The last output verifies that
lenses-cli was successfully downloaded and connected to the desired Lenses server in order to list the available Kafka topics!
Now that you know how to use GitHub actions to communicate with Lenses, you should start automating as many things as possible. Learn how to apply GitOps to your real time pipelines in our blog post.