Manage and govern Kafka through GitOps - Part 2

In this blog post you are going to see how you can declare your entire Kafka environment and data flows as just declarative configuration in a GitOps fashion.

This means you can deploy flows and Kafka in a repeatable and standardised fashion across your different environments.

Read Andrew’s part 1 of our blog of GitOps for Kafka with Lenses: Part 1

Pre-requisites.

You’re going to need Lenses and the Lenses CLI client.

As a quick reminder, Lenses provides a powerful, secure UI and APIs to explore, manage, monitor, troubleshoot Kafka and your data streams. It’s just a small 4GB JVM that you point to your Kafka and Kubernetes (if you have one) clusters.

But to make it easier to follow this guide just download the best Kafka development tool as a docker container

You can get the Lenses CLI from here.

Getting started with Lenses CLI

Two appropriate places to put the lenses-cli binary on a UNIX machine are ~/bin and /usr/local/bin. Please make sure that the directory you put lenses-cli is in the PATH environment variable in order for the lenses-cli executable to be accessible from everywhere on your UNIX system.

As this blog post is going to use a macOS machine, the following commands were executed for installing Lenses CLI in /usr/local/bin:

If you are using lenses-cli for the first time, you should execute lenses-cli configure in order to allow lenses-cli to connect to Lenses.

Basic lenses-cli usage

First, execute lenses-cli configure:

Notice that the information you gave to lenses-cli configure will be written in ~/.lenses/lenses-cli.yml.

Executing lenses-cli version will show the version of lenses-cli that you are using:

Adding resources

The lenses-cli utility allows you to create new resources without having to connect to Lenses GUI. Although you will still need to visit Lenses GUI in order to see the generated topology, lenses-cli can perform almost all tasks.

For the purposes of this blog post, we are going to execute the following lenses-cli commands:

The contents of the connector configuration file, which is saved in log_bloker.yaml, are the following:

Notice that the default clusterName for a Lenses Box is dev. If you are using a different Lenses installation, you should change the value of clusterName.

You can verify that the desired connector is created by executing the following command:

Alternatively, you can visit Lenses GUI and see the available connectors.

If you try to execute the same command again, it will fail with the next error message:

The last line of the output states the cause of the error - a connector with the same name already exists (connector file-connector already exists).

As logs_broker.yaml is already available, there will be no need to export the generated resource again. However, in the next section you will learn how to export existing resources in order to import them elsewhere.

Exporting

The lenses-cli utility allows you to export the desired resources from Lenses so that they can be version controlled and imported into other Lenses environments.

In this case we are going to export resources using the following lenses-cli commands:

This command exports all available topics, processors, quotas, connectors, schemas and policies and puts them under the export directory. If the directory does not already exist, it will be created by lenses-cli.

The next command will export the logs-broker connector only:

The output of the tree(1) utility can help us view the directory structure of the export directory as generated by the previous lenses-cli export commands:

Notice that the name of the directory that the data will be saved is defined as the value of the --dir option in the lenses-cli export command.

Pushing using git

Provided that we have a GitHub repository, we can clone it using git clone and put the contents of the export directory, including the directory itself, into the directory of the GitHub repository. Then we will have to execute the following commands:

Pulling from git

First, we will need to get the data from GitHub and store in on our local machine. This can be done by executing the following command:

Now, that we have the configuration on our local machine, we are ready to transfer that information to an empty Lenses Box, which will be illustrated in the next section.

Notice that the name of the directory that contains the Lenses configuration data will be given as a command line parameter to all lenses-cli import commands using the --dir option.

Importing

In this section we are going to use a Lenses Box. The main advantage we get from using a Lenses Box is that it is a Docker image that we can experiment with without the fear of destroying a working Lenses installation.

After exporting the desired configuration data, we will be able to import all of it using multiple lenses-cli import commands. In this case we are going to execute the following commands:

If you execute the lenses import policies command more than once, Lenses will understand that the policies already exist and will update the existing policies.

Conclusions

The lenses-cli command line tool will help you work with Lenses without having to connect to Lenses GUI all the time. Additionally, lenses-cli helps you perform GitOps related tasks efficiently.

Download the best DataOps development tool which comes with a free for developers version of Lenses and a Kafka environment as a single docker container.

Useful Links

• Development box

• Lenses CLI documentation