How to create an Alert Channel with Prometheus Alertmanager

Mihalis Tsoukalos
By Mihalis TsoukalosMarch 25, 2020
Kafka Alerts to AlertManager

Introduction

Lenses 3.1 is finally here with lots of improvements including a better and faster way to send alerts to alert management solutions through different Channels. In this blog post you are going to see how to setup Lenses 3.1 to send alerts to Prometheus Alertmanager.

To play this video, accept functional cookies in your privacy settings.

Vimeo video

Pre-requisites

In order to be able to follow the steps of this blog, you will need the following:

  • Lenses 3.1 (or a newer version when available) up and running. You can use our free Box instance if necessary.
  • Prometheus Alertmanager up and running
  • A working network connection between these two machines

The Implementation

The process requires performing the following three steps:

  • Creating a Connection
  • Creating a Channel
  • Associating that channel to one or more existing alerts

Creating a Connection

This step is about pointing Lenses to the desired Alertmanager instance using a connection template. First, you should go to the

ADMIN
panel and select
Connections
. Then, you should click on the
New Connection
button that will get you to the screen that allows you to select the service that you are going to use, which in this case is Prometheus Alertmanager.

The next screenshot shows the available templates at the time of writing this.

Create an alert connection - available templates

Click on

PrometheusAlertmanager
box in order to give the details of the connection. This is illustrated in the next screenshot.

Creating a new Alertmanager connection

Note that in this case the hostname of the machine that runs Alertmanager is

alertmanager
. You should change that to match your configuration.

Now that you have given the required information, press on the

Save Connection
button to create the connection and you are done. Your connection will be now listed on the
Connections
tab of the
ADMIN
panel.

Creating a Channel

You should now go to the

ADMIN
panel and select the
Channels
tab. Then, you should click on the
New Alert Channel
button in order to create your channel. First, you will need to select the Channel Type from a list that looks as follows:

List of Channel Types

Click on the button for Prometheus Alertmanager. After that, you will be asked to fill in some information.

Configure Alert Channel

You can put any name you want on the

Channel
text box. However, on the
Select connection
box you should select the appropriate connection from the presented list, which is this case will be
AlertManagerConnection
. The
Source
box should have a descriptive name for your channel and the
Generator URL
box should have the URL of your Lenses server.

Click on the

Create Alert Channel
button and you are done. The only thing left is associating your new channel with one or more alerts.

Associating a channel with an existing alert

The last step requires that you associate a channel with one or more existing alerts. Go to the

ADMIN
panel and select
Rules
from
ALERTS
. Press the
Edit
icon on the right of the
Infrastructure
rules that you want to associate with one or more Channels.

Edit rule

Note that you should click on the empty box on the left of a channel name in order to associate that channel with a rule.

The configuration used in this blog post is presented in the next screenshot.

Rules configuration

You should now wait for an event to happen in order for Lenses to send an alert to Alertmanager.

Visiting the Alertmanager UI

Prometheus Alertmanager usually uses port number

9093
, which means that we should visit the
http://alertmanager:9093
URL to see its UI, which will look similar to the following:

Alertmanager UI

If a Kafka broker becomes unavailable or if the Zookeeper server is down, you will get the following kind of output on Alertmanager.

Prometheus Alertmanager alerts

The previous output verifies that Lenses 3.1 can successfully communication with the Prometheus Alertmanager server. Additionally, it shows that there exist two

CRITICAL
alerts. The first alert is for Zookeeper whereas the second alert is for a Kafka broker.

Next Steps

Now that you know how easy it is to set up alerts with Lenses 3.1, you should start using that handy feature as soon as possible.