Apache Kafka monitoring & observability to drive engineering productivity

Confidence with Apache Kafka depends on engineers having unified visibility of real-time data & applications. Or you end up with whiplash from swivelling between so many screens and Kafka monitoring tools.

Kafka Monitoring

Monitoring any distributed environment is hard; then there’s Kafka monitoring!

Infinite failure modes can impact the performance & availability of a Kafka streaming app.

Monitoring your Kafka infrastructure by looking into Replica Fetcher Threads and Disk Write wait times won’t save you if there’s schema drift, incorrect ACLs, maxed-out Quotas or poor partitioning.

When you have to troubleshoot a Kafka flow, where do you start to figure out what the heck is going on?

What is Kafka monitoring? What are the benefits?

Apache Kafka is a complex a black box, requiring monitoring for many services including Schema Registry, Kafka Connect and real-time flows.

Higher engineering productivity

Higher engineering productivity

A Kafka UI to monitor consumer lag, partitioning, schemas & more, saving engineering from maintaining multiple tools.

Better quality code

Better quality code

Inspect data streams with SQL to understand how your microservice is behaving.

Improving Service Levels

Improving service levels

Metrics & real-time alerts on Kafka performance & streaming flows.

Playtika

“Lenses is critical for us in making our teams productive with Kafka and giving confidence to hundreds of developers.”

VP of IT Engineering at Playtika - Ella Vidra

Best practices for Apache Kafka monitoring

Are my real-time data platform, streaming applications and data healthy?

Monitor Kafka infrastructure & application performance from a single role-based and secured Kafka UI. Spinning up full-service visibility helps stay on top of your data and app health.

Are my applications healty?
Alerted for Kafka problems

How can I be alerted of any problems in Kafka?

Consumer lag? Under-replicated partitions? Or is your Kafka cluster down? Trigger alerts in real time to your incident management tools including AWS Cloudwatch, Prometheus Alert Manager, Slack, PagerDuty, Splunk or DataDog .

My Kafka applications are all over the place! How can I see how my streams are connected?

View & monitor streaming data pipelines via a Topology view that dynamically shows deployed Kafka Connect connectors and custom microservices connected to your topics.

Observe Streams connection
View events inside Kafka Topics

How can I view events inside my Kafka topics?

Troubleshoot by exploring events using SQL via a Kafka GUI or API. Lenses intelligently understands your data regardless of serialization: Avro, Protobuf, JSON, CSV and more.

How can I explore Schemas, Quotas and ACLs to troubleshoot my application?

Get to that root-cause by viewing schemas, Quotas and ACLs all within a secured & unified Kafka UI and API.

Explore Schemas

Kafka monitoring architecture

Monitoring architecture

What Kafka components should I monitor?

business_service

Business Service

application

Application

flows

Flows

data

Data

brokers

Brokers

zookeeper

Zookeeper

schema_registry

Schema Registry

connect

Connect

system

System

network

Network

Case Study

Babylon

Kafka monitoring for a healthtech unicorn

Exploring a universe of events using a simple SQL-like syntax helps Babylon Health engineers to troubleshoot their microservices.

GDPR Compliance Blog

Guide

The ultimate guide to

Kafka monitoring

Download now
Kafka Connect Blog

Case study

Vortexa delivers real-time insights on

AWS MSK

with Lenses.io

Read Case Study
Kafka to Splunk Blog

Checklist

Building an event streaming platform for

Apache Kafka

Download now

Get your DataOps workspace for Apache Kafka

Let's go