Manage and govern Kafka through GitOps - Part 2
In this blog post you are going to see how you can declare your entire Kafka environment and data flows as just declarative configuration in a GitOps fashion.
This means you can deploy flows and Kafka in a repeatable and standardised fashion across your different environments.
Read Andrew’s part 1 of our blog of GitOps for Kafka with Lenses: Part 1
You’re going to need Lenses and the Lenses CLI client.
As a quick reminder, Lenses provides a powerful, secure UI and APIs to explore, manage, monitor, troubleshoot Kafka and your data streams. It’s just a small 4GB JVM that you point to your Kafka and Kubernetes (if you have one) clusters.
But to make it easier to follow this guide just download the free “all-in-one” Lenses+Kafka container
You can get the Lenses CLI from here.
Getting started with Lenses CLI
Two appropriate places to put the
lenses-cli binary on a UNIX machine are
/usr/local/bin. Please make sure that the directory you put
PATH environment variable in order for the
lenses-cli executable to be
accessible from everywhere on your UNIX system.
As this blog post is going to use a macOS machine, the following commands were
executed for installing Lenses CLI in
$ wget https://github.com/Landoop/lenses-go/releases/download/2.3.4/lenses-cli-darwin-amd64.tar.gz $ tar zxvf lenses-cli-darwin-amd64.tar.gz x lenses-cli-darwin-amd64 $ mv lenses-cli-darwin-amd64 /usr/local/bin/lenses-cli
If you are using
lenses-cli for the first time, you should execute
lenses-cli configure in order to allow
lenses-cli to connect to Lenses.
Basic lenses-cli usage
__ ________ ____ / / ___ ____ ________ _____ / ____/ / / _/ / / / _ \/ __ \/ ___/ _ \/ ___/ / / / / / / / /___/ __/ / / (__ ) __(__ ) / /___/ /____/ / /_____/\___/_/ /_/____/\___/____/ \____/_____/___/ Docs at https://docs.lenses.io ? Enable debug mode? No ? Enable insecure https connections? Yes ? Host http://localhost:3030 ? How would you like to be authenticated? lenses BASIC auth or LDAP (default) ? Username admin ? Password [? for help] *****
Notice that the information you gave to
lenses-cli configure will be
lenses-cli version will show the version of
lenses-cli that you are using:
$ lenses-cli version lenses-cli 2.3.4 >>>> build revision 04647909dbcdaa19a7c98720a6a9a918dad2187b datetime Fri Jul 12 11:34:03 EEST 2019 go go1.12
lenses-cli utility allows you to create new resources without having to
connect to Lenses GUI. Although you will still need to visit Lenses GUI in order
to see the generated topology,
lenses-cli can perform almost all tasks.
For the purposes of this blog post, we are going to execute the following
$ lenses-cli connector create log_bloker.yaml Loading from file 'log_bloker.yaml' Connector [file-connector] created
The contents of the connector configuration file, which is saved in
are the following:
clusterName: dev config: name: file-connector connector.class: org.apache.kafka.connect.file.FileStreamSourceConnector tasks.max: 1 config.action.reload: RESTART errors.log.include.messages: false file: /var/log/broker.log topic: var_log_broker batch.size: 2000
Notice that the default
clusterName for a Lenses Box is
dev. If you are using
a different Lenses installation, you should change the value of
You can verify that the desired connector is created by executing the following command:
$ lenses-cli connectors --cluster-name=dev CLUSTER NAME CONFIGS TASKS --------- ---------------- --------- ------- dev file-connector 8 1 dev logs-broker 5 1 dev nullsink 5 4
Alternatively, you can visit Lenses GUI and see the available connectors.
If you try to execute the same command again, it will fail with the next error message:
$ lenses-cli connector create logs_broker.yaml Loading from file 'logs_broker.yaml' connector file-connector already exists
The last line of the output states the cause of the error - a connector with the same
name already exists (
connector file-connector already exists).
logs_broker.yaml is already available, there will be no need to export the generated
resource again. However, in the next section you will learn how to export existing
resources in order to import them elsewhere.
lenses-cli utility allows you to export the desired resources from Lenses
so that they can be version controlled and imported into other Lenses environments.
In this case we are going to export resources using the following
$ lenses-cli export topics --dir export $ lenses-cli export processors --dir export $ lenses-cli export quotas --dir export $ lenses-cli export connectors --dir export $ lenses-cli export schemas --dir export $ lenses-cli export policies --dir export
This command exports all available topics, processors, quotas, connectors, schemas and
policies and puts them under the
export directory. If the directory does not already
exist, it will be created by
The next command will export the
logs-broker connector only:
$ lenses-cli export connectors --resource-name=logs-broker --cluster-name dev --dir export
The output of the
tree(1) utility can help us view the directory structure of the
export directory as generated by the previous
lenses-cli export commands:
$ tree -d export export ├── apps │ ├── connectors │ └── sql ├── kafka │ ├── quotas │ └── topics ├── policies └── schemas 8 directories
Notice that the name of the directory that the data will be saved is defined as
the value of the
--dir option in the
lenses-cli export command.
Pushing using git
Provided that we have a GitHub repository, we can clone it using
git clone and put
the contents of the
export directory, including the directory itself, into the directory
of the GitHub repository. Then we will have to execute the following commands:
$ git add . $ git commit -a -m "Storing export directory" $ git push
Pulling from git
First, we will need to get the data from GitHub and store in on our local machine. This can be done by executing the following command:
$ git clone <URL of GitHub repository> $ cd <directory of GitHub repository>
Now, that we have the configuration on our local machine, we are ready to transfer that information to an empty Lenses Box, which will be illustrated in the next section.
Notice that the name of the directory that contains the Lenses configuration data
will be given as a command line parameter to all
lenses-cli import commands
In this section we are going to use a Lenses Box. The main advantage we get from using a Lenses Box is that it is a Docker image that we can experiment with without the fear of destroying a working Lenses installation.
After exporting the desired configuration data, we will be able to import all of it
lenses-cli import commands. In this case we are going to execute
the following commands:
$ lenses-cli import processors --dir export $ lenses-cli import topics --dir export $ lenses-cli import schemas --dir export $ lenses-cli import policies --dir export $ lenses-cli import connectors --dir export $ lenses-cli import quotas --dir export
If you execute the
lenses import policies command more than once, Lenses will understand that
the policies already exist and will update the existing policies.
lenses-cli command line tool will help you work with Lenses without having to
connect to Lenses GUI all the time. Additionally,
lenses-cli helps you perform GitOps
related tasks efficiently.
Download the free Lenses Box which comes with instance of Lenses and a Kafka environment as a single container to test out.