Staying on top of your Kafka with email alerts from Lenses.io

Imagine being out on a Saturday morning, drinking coffee and catching some sun when a message on Slack from a colleague tells you about an incident on your Kafka infrastructure. Fortunately, you are able to identify the problem and correct it from where you are but wouldn't it be good to know about that Kafka broker going down at the time that this has happened? Is there a way to prevent that from happening again?

Yes, there is: Lenses alerts.

This tutorial will illustrate how to integrate Lenses and Lenses Alerts with Zapier in order to get email notifications when a Kafka broker is down.

Pre-requisites

In order to be able to follow the steps of this tutorial, you will need the following:

Lenses up and running. You can use our free Lenses Box instance if necessary.
An Internet connection, an email address and a Zapier account that allows you to create webhooks and use Zapier email.

The Scenario

We want to get email notifications for certain types of Lenses Alerts, which is this case will be a Kafka broker going offline.

The Implementation

This section will present the steps needed for implementing the described scenario beginning from Lenses.

Note that we are going to connect Lenses and Zapier using Webhooks.

Creating a Zap in Zapier

The first thing that you should do is create a new Zap in Zapier. The Zap will have two components:

A Zap Webhook that will be used by Lenses.
A Zap Email that will send an email each time the Webhook is visited.

After logging in to Zapier, select

Create Zap

or MAKE A ZAP from your Zapier screen.

After that you will have to configure your Zap using existing Zapier components. In this case we will use

Webhooks by Zapier

and Email by Zapier in that order for creating our pipeline. At this point it would also be a good idea to name you Zap by putting some text in the upper left corner. We will name our Zap Lenses Alert to email - Tsoukalos. Although you can use any description you want, it is good to give a descriptive name followed by your name or surname in order to be able to find your Zap easily.

After that, select

Webhooks by Zapier

, which will take you to the next screen. The name of the App will automatically appear on the screen – for that app you will have to select the Trigger Event, which in this case will be Catch Raw Hook.

Press on the CONTINUE text in order for Zapier to generate the URL of the webhook, which in this case will be

https://hooks.zapier.com/hooks/catch/6433676/ort3dym/

After making sure that you copy and paste the webhook URL for easy access, press on

CONTINUE

. As it would be difficult to test the webhook at this phase, press SKIP TEST on the screen in order to continue, select and configure the next component of your pipeline.

From the next screen, select

Email by Zapier

from Built-In Apps. As the Action Event is correctly set to Send Outbound Email, click on CONTINUE. The next screen is about setting up the email message that will be sent. Use the destination email address you want as well as a good email message subject and a descriptive message body.

Although you can set many fields for the email, only the first three are compulsory.

After setting the desired values, click on

CONTINUE

at the end of the page. You are now given the opportunity to test the configuration of your email – please do so if you want, otherwise press the SKIP TEST link on the right side of the screen.

You are done creating your Zap. The last action you should perform is turning on that Zap by pressing on the relevant switch that appears at the end of your screen and at the upper right corner. After that, press the

DONE EDITING

click and select GO TO MY ZAPS.

Your newly created Zap should appear on the Zapier list. Press the down sign on the right side of your Zap and select

Task History

from the menu that appears.

The

Task History

screen will look similar to the following – it will currently be empty:

Keep that screen open for easy reference. We are done with Zapier – we will now continue with Lenses.

Creating a Lenses Connection

You will use the Zapier webhook to create a new Lenses Connection from Lenses UI. Go to the Lenses

ADMIN

screen and select Connections. Then click on + New Connection where you should select the Webhook kind of connection.

Based on the address (

https://hooks.zapier.com/hooks/catch/6433676/ort3dym/

), of the Zapier webhook, you should type hooks.zapier.com on the Host box and turn on the Use HTTPS switch. The final configuration is shown in the next screenshot:

Please make sure that you do not put

HTTP

or HTTPS in front of the value of the Host box. The name of the connection can be whatever you want provided that is an alphanumeric string up to 127 characters long that does not contain any special symbol or whitespaces. For this tutorial, the name of the connection is zapier-broker. Click on the Save Connection box and you are done. The next screen will present summary information about your connection.

Creating a Lenses Channel

Now select

Channels

, which is under the ALERTS group, and then select New Alert Channel. The type of the new channel will be Webhook, so click on the relevant button.

In the next screen, you will need to fill in the required information, including the

Request path

of the channel, which you will discover from the Zapier webhook address – in this case the value will be /hooks/catch/6433676/ort3dym/. The configuration of the channel appears on the next screenshot.

After that press the

Create Alert Channel

button and you are done! The name of the new channel is zapier-webhook-broker.

Using a Channel for getting Alerts

The last step is about connecting that channel with the desired types of alerts – in this case the rule of interest is

Kafka Broker is down

. For that, you should go the Rules screen and edit the Kafka Broker is down Rule as follows:

You are done!

Seeing the results

If a Kafka broker goes down or becomes unreachable, Lenses will use the Zapier webhook as a trigger for sending the mail. The same will happen when a Kafka broker goes up again!

The email message that will be sent, which depends on your configuration, will look as follows:

Additionally, you can visit the

Task History

screen of your Zap to see the number of emails that have been sent for that particular Zap.

For more technical people

The kind of log messages that you will get for webhooks are as follows:

```
2020-05-01 14:55:41,315 INFO  [i.l.c.a.c.w.WebHookAlertPublisher:42] [pool-
12-thread-1] Issuing HTTP request for web hook: 
io.lenses.core.alerting.channels.webhook.WebHookChannel@7cc67bd5. Http 
Request: HttpRequest(HttpMethod(POST),https://hooks.zapier.com/hooks/catch/6433676/or
t3dym/,List(),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.
1))
```

Next steps

Now that you know how to generate email alerts from Lenses, you should begin designing your strategy in order to get notified when something happens to your Kafka infrastructure.

Useful links