Manage and govern Kafka through GitOps - Part 2

In this blog post you are going to see how you can declare your entire Kafka environment and data flows as just declarative configuration in a GitOps fashion.

This means you can deploy flows and Kafka in a repeatable and standardised fashion across your different environments.

Read Andrew’s part 1 of our blog of GitOps for Kafka with Lenses: Part 1

Pre-requisites.

You’re going to need Lenses and the Lenses CLI client.

As a quick reminder, Lenses provides a powerful, secure UI and APIs to explore, manage, monitor, troubleshoot Kafka and your data streams. It’s just a small 4GB JVM that you point to your Kafka and Kubernetes (if you have one) clusters.

But to make it easier to follow this guide just download the best Kafka development tool as a docker container

You can get the Lenses CLI from here.

Getting started with Lenses CLI

Two appropriate places to put the 

lenses-cli

~/bin

/usr/local/bin

lenses-cli

PATH

lenses-cli

As this blog post is going to use a macOS machine, the following commands were
executed for installing Lenses CLI in 

/usr/local/bin

```
$ wget https://github.com/Landoop/lenses-go/releases/download/2.3.4/lenses-cli-darwin-amd64.tar.gz
$ tar zxvf lenses-cli-darwin-amd64.tar.gz
x lenses-cli-darwin-amd64
$ mv lenses-cli-darwin-amd64 /usr/local/bin/lenses-cli
```

If you are using 

lenses-cli

lenses-cli configure

lenses-cli

Basic lenses-cli usage

First, execute 

lenses-cli configure

```
    __                                 ________    ____
   / /   ___  ____  ________  _____   / ____/ /   /  _/
  / /   / _ \/ __ \/ ___/ _ \/ ___/  / /   / /    / /
 / /___/  __/ / / (__  )  __(__  )  / /___/ /____/ /
/_____/\___/_/ /_/____/\___/____/   \____/_____/___/
Docs at https://docs.lenses.io
? Enable debug mode? No
? Enable insecure https connections? Yes
? Host http://localhost:3030
? How would you like to be authenticated? lenses BASIC auth or LDAP (default)
? Username admin
? Password [? for help] *****
```

Notice that the information you gave to 

lenses-cli configure

~/.lenses/lenses-cli.yml

Executing 

lenses-cli version

lenses-cli

```
$ lenses-cli version
lenses-cli 2.3.4
>>>> build
           revision 04647909dbcdaa19a7c98720a6a9a918dad2187b
           datetime Fri Jul 12 11:34:03 EEST 2019
           go       go1.12
```

Adding resources

The 

lenses-cli

lenses-cli

For the purposes of this blog post, we are going to execute the following

lenses-cli

```
$ lenses-cli connector create log_bloker.yaml
Loading from file 'log_bloker.yaml'
Connector [file-connector] created
```

The contents of the connector configuration file, which is saved in 

log_bloker.yaml

```
clusterName: dev
config:
    name: file-connector
    connector.class: org.apache.kafka.connect.file.FileStreamSourceConnector
    tasks.max: 1
    config.action.reload: RESTART
    errors.log.include.messages: false
    file: /var/log/broker.log
    topic: var_log_broker
    batch.size: 2000
```

Notice that the default 

clusterName

dev

clusterName

You can verify that the desired connector is created by executing the following command:

```
$ lenses-cli connectors --cluster-name=dev
  CLUSTER   NAME             CONFIGS   TASKS
 --------- ---------------- --------- -------
  dev       file-connector         8       1
  dev       logs-broker            5       1
  dev       nullsink               5       4
```

Alternatively, you can visit Lenses GUI and see the available connectors.

If you try to execute the same command again, it will fail with the next error message:

```
$ lenses-cli connector create logs_broker.yaml
Loading from file 'logs_broker.yaml'
connector file-connector already exists
```

The last line of the output states the cause of the error - a connector with the same
name already exists (

connector file-connector already exists

As 

logs_broker.yaml

Exporting

The 

lenses-cli

In this case we are going to export resources using the following 

lenses-cli

```
$ lenses-cli export topics --dir export
$ lenses-cli export processors --dir export
$ lenses-cli export quotas --dir export
$ lenses-cli export connectors --dir export
$ lenses-cli export schemas --dir export
$ lenses-cli export policies --dir export
```

This command exports all available topics, processors, quotas, connectors, schemas and
policies and puts them under the 

export

lenses-cli

The next command will export the 

logs-broker

```
$ lenses-cli export connectors --resource-name=logs-broker --cluster-name 
dev --dir export
```

The output of the 

tree(1)

export

lenses-cli export

```
$ tree -d export
export
├── apps
│   ├── connectors
│   └── sql
├── kafka
│   ├── quotas
│   └── topics
├── policies
└── schemas

8 directories
```

Notice that the name of the directory that the data will be saved is defined as
the value of the 

--dir

lenses-cli export

Pushing using git

Provided that we have a GitHub repository, we can clone it using 

git clone

export

```
$ git add .
$ git commit -a -m "Storing export directory"
$ git push
```

Pulling from git

First, we will need to get the data from GitHub and store in on our local machine.
This can be done by executing the following command:

```
$ git clone <URL of GitHub repository>
$ cd <directory of GitHub repository>
```

Now, that we have the configuration on our local machine, we are ready to transfer
that information to an empty Lenses Box, which will be illustrated in the next
section.

Notice that the name of the directory that contains the Lenses configuration data
will be given as a command line parameter to all 

lenses-cli import

--dir

Importing

In this section we are going to use a Lenses Box. The main advantage we get from
using a Lenses Box is that it is a Docker image that we can experiment with without
the fear of destroying a working Lenses installation.

After exporting the desired configuration data, we will be able to import all of it
using multiple 

lenses-cli import

```
$ lenses-cli import processors --dir export
$ lenses-cli import topics --dir export
$ lenses-cli import schemas --dir export
$ lenses-cli import policies --dir export
$ lenses-cli import connectors --dir export
$ lenses-cli import quotas --dir export
```

If you execute the 

lenses import policies

Conclusions

The 

lenses-cli

lenses-cli

Download the best DataOps development tool which comes with a free for developers version of Lenses and a Kafka environment as a single docker container.

Useful Links

• Development box
• Lenses CLI documentation