01.

Understand
Lenses Box

Lenses Box is an all-in-one instance of Lenses, a Kafka Broker, Schema Registry, Kafka Connect and sample data streams. Here are some of the things you’ll be able to do:

Monitor and alert on health of Kafka
Administer Kafka such as creating Topics
Query data in Kafka streams using SQL syntax
Explore Kafka including Schemas, Consumers and more
Deploy Kafka-Connect compatible connectors
Build stream processing applications with SQL

screenshots3

02.

Run Lenses

You will need to have Docker installed in your machine. the following command from the Docker host:

docker run -e ADV_HOST=127.0.0.1 \
    -e EULA="https://dl.lenses.stream/d/?id=REGISTER_FOR_KEY" \
    --rm -p 3030:3030 -p 9092:9092 landoop/kafka-lenses-dev
docker run -e ADV_HOST=127.0.0.1 \
-e EULA="https://dl.lenses.stream/d/?id=
REGISTER_FOR_KEY" \
--rm -p 3030:3030 
-p 9092:9092 landoop/kafka-lenses-dev

TIP
If you want to send data from your own application to Kafka, you can change the ADV_HOST IP to be the IP your application will use to communicate to Kafka. This is the same as advertised.host.name on a Kafka Broker.

03.

Access Lenses from your browser

You will now be able to access Lenses User Interface from your browser at http://localhost:3030

The default username and password is admin / admin.

TIP
It takes about 45 seconds for Lenses Box to spin up all services and the UI to become available.

screenshots3

04.

Write your first query

1. Navigate to the Topics
2. Search for sea_vessel_position_reports
3. Filter data by running the query below:

SELECT Speed, Latitude, Longitude, _meta.partition
FROM sea_vessel_position_reports
WHERE Speed > 10
SELECT Speed, Latitude,
Longitude, _meta.partition
FROM sea_vessel_position_reports
WHERE Speed > 10

05.

Create New Topics

1. Navigate to the Topics
2. Click the New Topic button and fill the form.

Feel free to add custom configuration. But remember, don’t set the replication to more than one because this Kafka instance only has one Broker.

TIP

TIP
You can also create topics using SQL with the CREATE TABLE

screenshots3

06.

Produce Data

A simple way to create data is to stream a log file into a Kafka Topic:
1. Navigate to Connectors
2. Click the New Connector and choose the File Source Connector
3. Copy and paste the configuration below and create:

connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
topic=logs_broker
file=/var/log/broker.log
tasks.max=1
connector.class=
org.apache.kafka.connect
.file.FileStreamSourceConnector
topic=logs_broker
file=/var/log/broker.log
tasks.max=1

The above config takes the broker logs from /var/log/broker.log file. Verify you get data in the logs_broker topic.

USE CODE USE SQL CONNECTORS

USE SQL
You can produce data with SQL using INSERT INTO statement

USE CODE
To produce data from your code just set the bootstrap broker to the host of the Lenses Box and port 9092. Ensure you set the correct ADV_HOST in Docker

CONNECTORS
You can add any Kafka Connector plugin to Lenses Box

07.

Process data

You can use SQL to create queries with streaming semantics.

1. Navigate to SQL Processors
2. Click New Processor, put a name, set to 1 runner and copy the query below:

SET autocreate=true;
INSERT INTO speed_boats
     SELECT Speed, Latitude, Longitude
     FROM sea_vessel_position_reports
     WHERE Speed > 10
SET autocreate=true;
INSERT INTO speed_boats
SELECT Speed, Latitude, Longitude
FROM sea_vessel_position_reports
WHERE Speed > 10

Once the query is registered will continuously filter and produce data to speed_boats when the Speed is greater than 10. The topic records will just contain Speed, Latitude, Longitude fields.

TIP

TIP
When run in production scale the SQL Processor via Kafka or Kubernetes modes

08.

Check Consumers

The SQL Processor in the previous step, is a streaming App which consumes data from a topic and produces to another.

Navigate to the Consumers and select the consumer created. You can monitor the Consumer Lag and status per instance.

TIP

TIP
When a Consumer is not running you can change the commit offset per partition

consumer-lag-lensesio

09.

Set Alert Rules

Lenses comes with pre-configured Alert Rules for your infrastructure. You can also create custom rules for your Consumers by navigating to Alert settings. Lenses will monitor the Consumer Lags and trigger an alert event when exceeds the threshold.

TIP

TIP
You can view the alerts in Lenses or forward them onto Slack or your Alert Management solution.

alert

10.

Monitor flows

Navigate to Topology, for identifying the connections between your Apps, Connectors and Processors with your Topics.

Click on each node to see the relevant details, metrics and healthcheck of the problem.

TIP

TIP
You can connect any custom JVM based microservice to the topology graph. For example, custom producers and consumers, Kafka streams, Spark, Akka etc by adding the Lenses lib to your code. You can visualise even the internal topology of each app in order to have full visibility of your data processing layer.

screenshots3

Explore Advanced Options

Data Policies, to mask data before sharing
Manage Kafka ACLs & Quotas
Create namespaces to limit access via whitelisting
Automate Flows with Lenses CLI and GitOps
Integrate data, metrics and admin via Lenses API

See the Docs →

contact
screenshots3

Want to connect to your cluster?

screenshots3

Frequently Asked Questions

Yes.

Lenses Box is optimized for development and includes a single broker Kafka environment. However, you can evaluate Lenses against your existing setup. Lenses is compatible with all popular Kafka distributions and clouds. Contact our team to help you

Lenses Box is Free for ALL and forever.

In order to use the Docker of Lenses Box you require a security (EULA) key. You may need to refresh your key from time to time for security reasons.

You will need a paid License to connect Lenses to your own cluster or production.

See pricing

It's not recommended.

Lenses Box is a single-node setup. For production environmnets it is recommended to have a multi-node setup for scalability and fail-over. Also, Lenses Box allows up to 25M records on the cluster. However, if you want to use Lenses Box with over 25M records or for a production deployment you will require a License.

For production or multi-node setups of your Kafka cluster contact our team

Yes. Lenses works with AWS MSK, Azure HDIside, Aiven, Confluent Cloud and other managed cloud services.

Lenses also provides deployment templates tailored made for each Cloud provider.

Check Deployment Options

Do you have more questions?
Let us help →