Integrating Apache Kafka into your CI/CD with Jenkins & Lenses

This tutorial will illustrate how you can integrate Lenses into your Jenkins CI/CD using the lenses-cli. 

The Lenses CLI allows you to manage your Apache Kafka environment using a single unified and secure API across all Kafka components (Kafka Brokers, Zookeepers, Kafka Connect, Schema Registry etc.).

Integrating Lenses with Jenkins will simplify automating the deployment of real time applications across your different Apache Kafka environments.

To demonstrate the integration, we will simply walk you through a very basic example of triggering a job that uses the lenses-cli to list the available topics in your Apache Kafka cluster. See this blog for more details about managing your Apache Kafka flows via GitOps.

Pre-requisites

In order to be able to follow the steps of this blog, you will need the following:

• Lenses up and running. You can use our free Box instance if necessary.

• A Jenkins server up and running and accessible from the Internet. Additionally, the Jenkins server should have the required plugins for GitHub support installed.

• A working network connection between these two machines.

• A GitHub repository with administrative privileges on it.

The Scenario

The Jenkins job that will be implemented will download lenses-cli and query a Lenses instance for information using the admin account.

We are going to use a public GitHub repository in order to avoid having to authenticate to GitHub from Jenkins, which will make the scenario more complex. The URL for the GitHub repository is https://github.com/mactsouk/jenkins-blog – you should use your own GitHub repository or any other repository as long as you can create a Webhook for it.

The Implementation

This section will illustrate the steps needed for implementing the scenario.

Note that both Jenkins and Lenses servers will be exposed to the Internet using the ngrok utility, which explains the hostnames of the servers. The Jenkins server hostname is 15789550.ngrok.io whereas the Lenses server hostname is a00474c0.ngrok.io.

GitHub Webhook

In order to get the Webhook, you should go to the home page of your repository and then select the Settings tab. Then, select the Webhooks option on the left column and then click on the Add Webhook button. The values that were used for the webhook are shown in the next screenshot:

After we are done, we will press the Add Webhook button or the Update Webhook if the webhook was already created. Note that you will need to add the /github-webhook/ string at the end of the Jenkins hostname – this is really important for the scenario to work.

This step was needed in order for GitHub to inform Jenkins about changes to the GitHub repository.

Creating the Jenkins Job

This is the most critical part of the process as you will need to create a Jenkins job and correctly associate it with the appropriate GitHub repository.

Go to Jenkins UI and click on create new jobs link. You will presented with the next screen:

Enter Lenses as the name of the item and select the Freestyle project. After that click on the OK button at the bottom of the screen.

The next string is pretty long so it will be presented in two steps. The first part is about specifying the GitHub project URL and selecting the Source Code Management tool, which will be Git and filling in the correct information – see the next screenshot for more information:

The second part is about selecting the GitHub hook trigger for GITScm polling box and defining the job under the Build title. On the Add Build step menu select the Execute shell item. After that, insert the desired UNIX commands in the box. The final configuration is presented in the following screenshot.

Click on Save button and you are done with the definition of the job. After that, each time someone makes a push to the associated GitHub repository, the Jenkins job will be triggered.

The bash shell commands that will be executed in the Jenkins job are the following:

uname informs us about the characteristics of the UNIX machine and is not necessary to be executed – it just gives you a good idea of the Linux system you will be working on.

The wget command downloads the lenses-cli archive, which is extracted using tar and renamed to lenses-cli. Last, lenses-cli connects to the desired Lenses server (http://a00474c0.ngrok.io), using the specified username and password information and gets a list of the available Kafka topics. This will download the client each time the job runs, you may want to place the client in your Jenkins workspace in order to avoid unnecessary bandwidth use and improve execution time.

You can also authenticate using a service account if you do not want to use a regular account:

This is a simple job but you can perform any job you want. Look at the GitHub Actions + Lenses blog post for learning how to insert data into Kafka through Lenses using shell commands and at lenses-cli documentation.

Pushing data to the GitHub repository

Now all you have to do is perform a push to the GitHub repository and the Jenkins job will be automatically triggered.

Watching the results

Click on the #1 link, or on any other link that exists there, in order to see more information about the execution of the job.

Click on the Console Output link of the left in order to see the output of the shell commands of the job.

You will see lots of output related to the execution of the wget command - if you want less output from wget, please use the --quiet option. The bottom of the screen will look as follows:

The last output verifies that lenses-cli was successfully downloaded and connected to the desired Lenses server in order to list the available Kafka topics!

Next Steps

Now that you know how to use GitHub actions to communicate with Lenses, you should start automating as many things as possible. Learn how to apply GitOps to your real time pipelines in our blog post.

Other Links

• Lenses Box 5 min tour

• Lenses CLI

• Jenkins

• Download Lenses CLI

• ngrok