This tutorial will illustrate how you can integrate GitHub Actions with Apache Kafka via Lenses.io CLI. GitHub Actions offers a powerful way of triggering workflows and allowing you to communicate with lots of systems using command line utilities.
Two reasons for sending events into Kafka through lenses.io using a CI/CD system would be to be consumed by a microservice or be processed by the powerful SQL Engine of lenses.io.
In order to follow this tutorial, you are going to need to have Lenses installed.
You can download the free Lenses Box from here, which comes with an instance of Lenses and a complete Kafka environment as a single Docker image.
The scenario also assumes the following:
You have Lenses up and running and accessible from the Internet
You already have created and cloned the desired GitHub repository on your local machine
You have the necessary privileges for making changes to your GitHub repository
You want to log to Kafka certain kinds of events that happen on a GitHub repository that you own using GitHub actions. These events can then be consumed by a microservice.
This section will illustrate the steps needed for implementing the scenario, starting with the creation of a new service account in Lenses.
Before creating a new service account, you should have at least one Group that this service account will belong to. For the purposes of this blog post, we are going to create a new group using Lenses UI. For creating a new group you should go the
Admin panel and click on the
+ sign under the
GROUPS text. The
github-actions group will be created as follows.
After logging in to Lenses, you should go to the
Admin panel and select the
Service Accounts option. After that click on the
New Service Account button and add the necessary information.
The creation of the new service account is illustrated in the following screenshot.
The name of the service account will be
github. After clicking on
Create New Service Account button, the service account will be created and a security token will be given to you, as illustrated by the next screenshot.
This security token will be displayed only once and there is no other way to be displayed again in Lenses. This means that you will have to save it in order to be able to put it in a GitHub Secret variable or use it elsewhere.
We will create a new Kafka Topic (
Dashboard > Topics) in Lenses to store the records coming from GitHub Actions. The name of the Kafka topic will be
github-logs and is going to be created as follows:
After that go drill down to the created topic and change the serialisation type of the topic key and values as
STRING and the
JSON followed by clicking
Note that you will need to create the Kafka Topic only once, which is the main reason that we do not automate the creation of the Kafka topic. Additionally, you can also create a Kafka topic in Lenses using
lenses-cli and the
lenses-cli topic create command or with a SQL command in
SQL Studio section.
Last, note that it is not necessary to define the structure of the JSON records that will be stored in
The value of the token that was generated by Lenses when creating the service account is what is going to be stored as a GitHub secret.
In your GitHub repository go to the
Setting tab and select
Secrets from the left column. You will see existing secrets, if any, and a
Add new secret link that you will need to click on.
The next screenshot shows the creation of the new secret in GitHub UI.
The name of the secret is
SERVICE_ACCOUNT and its value is the token from the Lenses service account – the name of the secret can be anything you want.
You can create a new GitHub action in an existing GitHub repository by going to the
Actions tab and selecting the type of action you want to create. In our case, the type of action will be
Simple workflow, which means that you should click on the
Set up this workflow button under
Simple workflow. This will allow you to create a new configuration file. As we will make the necessary changes afterwards, we will press on the
Start commit button on the top right corner and continue with the default configuration.
All GitHub actions are defined and implemented under the
.github/workflows directory of your GitHub repository. In this case the name of the YAML file with the definition of the GitHub action will be
When a GitHub Action is triggered and before it gets executed, GitHub automatically creates and sets a number of environment variables, including the following:
GITHUB_EVENT_NAME: This environment variable holds the name of the GitHub event that took place.
GITHUB_REPOSITORY: This environment variable holds the name of the GitHub repository.
GITHUB_REF: This environment variable holds the name of the GitHub branch.
GITHUB_ACTOR: This environment variable holds the name of the GitHub user.
The contents of the
lenses.yml file are as follows:
It is really important to understand how you can access the GitHub secret that we defined earlier because this has to be done through the use of an environment variable. So, you access the secret as
secrets.SERVICE_ACCOUNT in your YAML file and you assign its value to an environment variable named
SERVICE_ACCOUNT that will be accessible in the GitHub job.
lenses.yml also shows how you can download and extract
lenses-cli in case you need it for querying Lenses or performing other administrative tasks.
The values of the
on key define when this job will be triggered. The last command is
lenses.yml is a bash script that will be shown in the next subsection.
The last step in the job is the execution of a script named
insert.sh. Although it is not mandatory to use a script, a script is very convenient for grouping and executing a set of commands.
The contents of
insert.sh are the following:
insert.sh script is using
curl(1) for inserting data into Lenses.
The definition of the
endpoint variable includes the name of the Kafka topic as well as the type of its key (
kt=STRING) and its value (
vt=JSON) and is really important for the successful execution of
Additionally, the value of
server is the hostname or the IP address of the Lenses server and should be accessible from the Internet. If you do not have a Lenses server available on the Internet, you might find the
ngrok utility really helpful.
Once you start pushing, pulling, raising issues, branches and adding comments to your repository, the GitHub action we defined will start being executed and write data to Kafka topic through Lenses.
We can visit the
github-actions Kafka topic and see its entries.
Additionally, you can check the results of the GitHub action in GitHub UI.
If there exist any errors in the definition of the GitHub Action, they will be shown in GitHub UI.
Now that you know how to use GitHub actions to communicate with Lenses, you should start automating as many things as possible. Learn how to apply GitOps to your real time pipelines in our blog.