Home Lenses

Apache Kafka monitoring & administration to drive engineering productivity

Confidence with Apache Kafka starts with visibility. This goes beyond being able to monitor your Kafka infrastructure.

Kafka Monitoring

“How do I monitor Kafka performance?”

Apache Kafka is a complex a black box, requiring monitoring for many services including Schema Registry, Kafka Connect and real-time flows.

For productive data operations & avoiding incidents, teams need complete visibility over the health of Kafka infrastructure and data flows.

Engineers need self-service data access to monitor and heal the applications they deploy, without constant ticketing and frenzied Slack messages.

What if you had Kafka monitoring tooling that allowed your entire streaming platform to be observed and navigated by everyone in ops, not only the Kafka literate?

Playtika

"Lenses has been critical for us making our teams productive with Kafka and having production-ready confidence across hundreds of developers."

VP of IT Engineering at Playtika - Ella Vidra

What is Kafka Monitoring?


It’s important to monitor the health of your Kafka deployment to maintain reliable performance from the applications that depend on it. Some best practices for Kafka monitoring:

  • Use a Kafka monitoring UI to keep track of key metrics
  • Monitor Kafka clusters with alerting on issues as they emerge
  • Inspect and debug data in seconds without a command line

Kafka Monitoring

What components should I monitor?

business_service

Business Service

application

Application

flows

Flows

data

Data

brokers

Brokers

zookeeper

Zookeper

schema_registry

Schema Registry

connect

Connect

system

System

network

Network

These components should help you answer the following questions.

Lenses Dashboard - Kafka metrics

Are my real-time data platform, applications and data healthy?

Monitor the performance and view the state of your Kafka projects from a single role-based and secured UI. Spinning up full-service visibility helps stay on top of your data and app health.

How can I be notified of any problems in my data flows?

If your consumer, producer or any part of your platform is under pressure, channel smart notifications to different teams of your data platform including AWS Cloudwatch, Prometheus Alert Manager, Slack, PagerDuty or DataDog.

Kafka monitoring & alerting
Kafka topics - Partition map

Are my topics well-replicated and partitioned evenly?

See a global view of all topics and their configuration through your Kafka monitoring tools. Drilldown to inspect the data or zoom into consumer performance, messages and partitions, without a command line.

How can I allow users to create Topics, ACLs & Quotas?

View, create, edit and delete Kafka Topics, Quotas and ACLs from a single unified UI and API with full role-based access controls and audits. Tenants can then export as configuration files and port across different environments using GitOps practices.

Add new quota
Babylon Health

Case Study

Flattening the curve with Kafka monitoring

Exploring a universe of events using a simple SQL-like syntax helps Babylon Health find and act on patterns in patient data.

Enjoy operational excellence with Apache Kafka monitoring

Start

Or speak with a Kafka expert