GitHub Actions into Apache Kafka

This tutorial will illustrate how you can integrate GitHub Actions with Apache Kafka via Lenses.io CLI. GitHub Actions offers a powerful way of triggering workflows and allowing you to communicate with lots of systems using command line utilities.

Two reasons for sending events into Kafka through lenses.io using a CI/CD system would be to be consumed by a microservice or be processed by the powerful SQL Engine of lenses.io.

Pre-requisites

In order to follow this tutorial, you are going to need to have Lenses installed.

You can download the free Lenses Box from here, which comes with an instance of Lenses and a complete Kafka environment as a single Docker image.

The scenario also assumes the following:

You have Lenses up and running and accessible from the Internet
You already have created and cloned the desired GitHub repository on your local machine
You have the necessary privileges for making changes to your GitHub repository

The Scenario

You want to log to Kafka certain kinds of events that happen on a GitHub repository that you own using GitHub actions. These events can then be consumed by a microservice.

Implementing the scenario

This section will illustrate the steps needed for implementing the scenario, starting with the creation of a new service account in Lenses.

Creating a Lenses Group

Before creating a new service account, you should have at least one Group that this service account will belong to. For the purposes of this blog post, we are going to create a new group using Lenses UI. For creating a new group you should go the

Admin

panel and click on the + sign under the GROUPS text. The github-actions group will be created as follows.

Creating a Lenses Service Account

After logging in to Lenses, you should go to the

Admin

panel and select the Service Accounts option. After that click on the New Service Account button and add the necessary information.

The creation of the new service account is illustrated in the following screenshot.

The name of the service account will be

github

. After clicking on Create New Service Account button, the service account will be created and a security token will be given to you, as illustrated by the next screenshot.

This security token will be displayed only once and there is no other way to be displayed again in Lenses. This means that you will have to save it in order to be able to put it in a GitHub Secret variable or use it elsewhere.

Creating a Kafka Topic

We will create a new Kafka Topic (

Dashboard > Topics

) in Lenses to store the records coming from GitHub Actions. The name of the Kafka topic will be github-logs and is going to be created as follows:

After that go drill down to the created topic and change the serialisation type of the topic key and values as

STRING

and the value as JSON followed by clicking Update.

Note that you will need to create the Kafka Topic only once, which is the main reason that we do not automate the creation of the Kafka topic. Additionally, you can also create a Kafka topic in Lenses using

lenses-cli

and the lenses-cli topic create command or with a SQL command in SQL Studio section.

Last, note that it is not necessary to define the structure of the JSON records that will be stored in

github-logs

Storing Secrets in GitHub

The value of the token that was generated by Lenses when creating the service account is what is going to be stored as a GitHub secret.

In your GitHub repository go to the

Setting

tab and select Secrets from the left column. You will see existing secrets, if any, and a Add new secret link that you will need to click on.

The next screenshot shows the creation of the new secret in GitHub UI.

The name of the secret is

SERVICE_ACCOUNT

and its value is the token from the Lenses service account – the name of the secret can be anything you want.

Creating GitHub Actions

You can create a new GitHub action in an existing GitHub repository by going to the

Actions

tab and selecting the type of action you want to create. In our case, the type of action will be Simple workflow, which means that you should click on the Set up this workflow button under Simple workflow. This will allow you to create a new configuration file. As we will make the necessary changes afterwards, we will press on the Start commit button on the top right corner and continue with the default configuration.

All GitHub actions are defined and implemented under the

.github/workflows

directory of your GitHub repository. In this case the name of the YAML file with the definition of the GitHub action will be lenses.yml.

GitHub Environment Variables

When a GitHub Action is triggered and before it gets executed, GitHub automatically creates and sets a number of environment variables, including the following:

```
GITHUB_EVENT_NAME
```
: This environment variable holds the name of the GitHub event that took place.
```
GITHUB_REPOSITORY
```
: This environment variable holds the name of the GitHub repository.
```
GITHUB_REF
```
: This environment variable holds the name of the GitHub branch.
```
GITHUB_ACTOR
```
: This environment variable holds the name of the GitHub user.

Viewing the GitHub Actions configuration

The contents of the

lenses.yml

file are as follows:

```
name: Lenses CLI

on: [push, pull_request, create, issue_comment, delete]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2

    - name: Download lenses-cli
      env:
         SERVICE_ACCOUNT: ${{ secrets.SERVICE_ACCOUNT }}
      run: |
        wget https://github.com/lensesio/lenses-go/releases/download/3.0.10/lenses-cli-linux-386.tar.gz
        tar xzvf lenses-cli-linux-386.tar.gz
        mv lenses-cli-linux-386 lenses-cli
        chmod 755 insert.sh
        ./insert.sh
```

It is really important to understand how you can access the GitHub secret that we defined earlier because this has to be done through the use of an environment variable. So, you access the secret as

secrets.SERVICE_ACCOUNT

in your YAML file and you assign its value to an environment variable named SERVICE_ACCOUNT that will be accessible in the GitHub job.

lenses.yml

also shows how you can download and extract lenses-cli in case you need it for querying Lenses or performing other administrative tasks.

The values of the

on

key define when this job will be triggered. The last command is lenses.yml is a bash script that will be shown in the next subsection.

The insert.sh bash script

The last step in the job is the execution of a script named

insert.sh

. Although it is not mandatory to use a script, a script is very convenient for grouping and executing a set of commands.

The contents of

insert.sh

are the following:

```
{
	"value": {
		"VendorID": 2,
		"tpep_pickup_datetime": "2016-01-01 00:00:00",
		"tpep_dropoff_datetime": "2016-01-01 00:00:00",
		"passenger_count": 1,
		"trip_distance": 10.54,
		"pickup_longitude": "-73.98455047607422",
		"pickup_latitude": "40.6795654296875",
		"RateCodeID": 1,
		"store_and_fwd_flag": "N",
		"dropoff_longitude": "-73.95027160644531",
		"dropoff_latitude": "40.78892517089844",
		"payment_type": 1,
		"fare_amount": 33,
		"extra": 0.5,
		"mta_tax": 0.5,
		"improvement_surcharge": 0.3,
		"tip_amount": 0,
		"tolls_amount": 0,
		"total_amount": 34.3
	},
	"metadata": {
		"offset": 0,
		"partition": 0,
		"timestamp": 1587447010296,
		"__keysize": 0,
		"__valsize": 147
	}
},
```

The

insert.sh

script is using curl(1) for inserting data into Lenses.

The definition of the

endpoint

variable includes the name of the Kafka topic as well as the type of its key (kt=STRING) and its value (vt=JSON) and is really important for the successful execution of insert.sh.

Additionally, the value of

server

is the hostname or the IP address of the Lenses server and should be accessible from the Internet. If you do not have a Lenses server available on the Internet, you might find the ngrok utility really helpful.

Viewing the results

Once you start pushing, pulling, raising issues, branches and adding comments to your repository, the GitHub action we defined will start being executed and write data to Kafka topic through Lenses.

We can visit the

github-actions

Kafka topic and see its entries.

Additionally, you can check the results of the GitHub action in GitHub UI.

If there exist any errors in the definition of the GitHub Action, they will be shown in GitHub UI.

Next Steps

Now that you know how to use GitHub actions to communicate with Lenses, you should start automating as many things as possible. Learn how to apply GitOps to your real time pipelines in our blog.

Pre-requisites

In order to follow this tutorial, you are going to need to have Lenses installed.

You can download the free Lenses Box from here, which comes with an instance of Lenses and a complete Kafka environment as a single Docker image.

The scenario also assumes the following:

You have Lenses up and running and accessible from the Internet
You already have created and cloned the desired GitHub repository on your local machine
You have the necessary privileges for making changes to your GitHub repository

The Scenario

You want to log to Kafka certain kinds of events that happen on a GitHub repository that you own using GitHub actions. These events can then be consumed by a microservice.

Implementing the scenario

This section will illustrate the steps needed for implementing the scenario, starting with the creation of a new service account in Lenses.

Creating a Lenses Group

Admin

panel and click on the + sign under the GROUPS text. The github-actions group will be created as follows.

Creating a Lenses Service Account

After logging in to Lenses, you should go to the

Admin

panel and select the Service Accounts option. After that click on the New Service Account button and add the necessary information.

The creation of the new service account is illustrated in the following screenshot.

The name of the service account will be

github

. After clicking on Create New Service Account button, the service account will be created and a security token will be given to you, as illustrated by the next screenshot.

Creating a Kafka Topic

We will create a new Kafka Topic (

Dashboard > Topics

) in Lenses to store the records coming from GitHub Actions. The name of the Kafka topic will be github-logs and is going to be created as follows:

After that go drill down to the created topic and change the serialisation type of the topic key and values as

STRING

and the value as JSON followed by clicking Update.

lenses-cli

and the lenses-cli topic create command or with a SQL command in SQL Studio section.

Last, note that it is not necessary to define the structure of the JSON records that will be stored in

github-logs

Storing Secrets in GitHub

The value of the token that was generated by Lenses when creating the service account is what is going to be stored as a GitHub secret.

In your GitHub repository go to the

Setting

tab and select Secrets from the left column. You will see existing secrets, if any, and a Add new secret link that you will need to click on.

The next screenshot shows the creation of the new secret in GitHub UI.

The name of the secret is

SERVICE_ACCOUNT

and its value is the token from the Lenses service account – the name of the secret can be anything you want.

Creating GitHub Actions

You can create a new GitHub action in an existing GitHub repository by going to the

Actions

All GitHub actions are defined and implemented under the

.github/workflows

directory of your GitHub repository. In this case the name of the YAML file with the definition of the GitHub action will be lenses.yml.

GitHub Environment Variables

When a GitHub Action is triggered and before it gets executed, GitHub automatically creates and sets a number of environment variables, including the following:

```
GITHUB_EVENT_NAME
```
: This environment variable holds the name of the GitHub event that took place.
```
GITHUB_REPOSITORY
```
: This environment variable holds the name of the GitHub repository.
```
GITHUB_REF
```
: This environment variable holds the name of the GitHub branch.
```
GITHUB_ACTOR
```
: This environment variable holds the name of the GitHub user.

Viewing the GitHub Actions configuration

The contents of the

lenses.yml

file are as follows:

```
name: Lenses CLI

on: [push, pull_request, create, issue_comment, delete]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
    - uses: actions/checkout@v2

    - name: Download lenses-cli
      env:
         SERVICE_ACCOUNT: ${{ secrets.SERVICE_ACCOUNT }}
      run: |
        wget https://github.com/lensesio/lenses-go/releases/download/3.0.10/lenses-cli-linux-386.tar.gz
        tar xzvf lenses-cli-linux-386.tar.gz
        mv lenses-cli-linux-386 lenses-cli
        chmod 755 insert.sh
        ./insert.sh
```

It is really important to understand how you can access the GitHub secret that we defined earlier because this has to be done through the use of an environment variable. So, you access the secret as

secrets.SERVICE_ACCOUNT

in your YAML file and you assign its value to an environment variable named SERVICE_ACCOUNT that will be accessible in the GitHub job.

lenses.yml

also shows how you can download and extract lenses-cli in case you need it for querying Lenses or performing other administrative tasks.

The values of the

on

key define when this job will be triggered. The last command is lenses.yml is a bash script that will be shown in the next subsection.

The insert.sh bash script

The last step in the job is the execution of a script named

insert.sh

. Although it is not mandatory to use a script, a script is very convenient for grouping and executing a set of commands.

The contents of

insert.sh

are the following:

```
{
	"value": {
		"VendorID": 2,
		"tpep_pickup_datetime": "2016-01-01 00:00:00",
		"tpep_dropoff_datetime": "2016-01-01 00:00:00",
		"passenger_count": 1,
		"trip_distance": 10.54,
		"pickup_longitude": "-73.98455047607422",
		"pickup_latitude": "40.6795654296875",
		"RateCodeID": 1,
		"store_and_fwd_flag": "N",
		"dropoff_longitude": "-73.95027160644531",
		"dropoff_latitude": "40.78892517089844",
		"payment_type": 1,
		"fare_amount": 33,
		"extra": 0.5,
		"mta_tax": 0.5,
		"improvement_surcharge": 0.3,
		"tip_amount": 0,
		"tolls_amount": 0,
		"total_amount": 34.3
	},
	"metadata": {
		"offset": 0,
		"partition": 0,
		"timestamp": 1587447010296,
		"__keysize": 0,
		"__valsize": 147
	}
},
```

The

insert.sh

script is using curl(1) for inserting data into Lenses.

The definition of the

endpoint

variable includes the name of the Kafka topic as well as the type of its key (kt=STRING) and its value (vt=JSON) and is really important for the successful execution of insert.sh.

Additionally, the value of

server

Viewing the results

Once you start pushing, pulling, raising issues, branches and adding comments to your repository, the GitHub action we defined will start being executed and write data to Kafka topic through Lenses.

We can visit the

github-actions

Kafka topic and see its entries.

Additionally, you can check the results of the GitHub action in GitHub UI.

If there exist any errors in the definition of the GitHub Action, they will be shown in GitHub UI.

Next Steps

Now that you know how to use GitHub actions to communicate with Lenses, you should start automating as many things as possible. Learn how to apply GitOps to your real time pipelines in our blog.

GitHub Actions into Apache Kafka

Pre-requisites

The Scenario

Implementing the scenario

Creating a Lenses Group

Creating a Lenses Service Account

Creating a Kafka Topic

Storing Secrets in GitHub

Creating GitHub Actions

GitHub Environment Variables

Viewing the GitHub Actions configuration

The insert.sh bash script

Viewing the results

Next Steps

Other Links

Related Blogs

Lenses 6.2 - Trusting Agents to build & operate event-driven applications

Kafka Migrations Need More Than a Replicator

Self-Service Data Replication with K2K - part 1

GitHub Actions into Apache Kafka

Pre-requisites

The Scenario

Implementing the scenario

Creating a Lenses Group

Creating a Lenses Service Account

Creating a Kafka Topic

Storing Secrets in GitHub

Creating GitHub Actions

GitHub Environment Variables

Viewing the GitHub Actions configuration

The insert.sh bash script

Viewing the results

Next Steps

Other Links

Related Blogs

Lenses 6.2 - Trusting Agents to build & operate event-driven applications

Kafka Migrations Need More Than a Replicator

Self-Service Data Replication with K2K - part 1