All about Lenses 6.0 for multi-Kafka

Lenses 6.0 (code-named Panoptes after the Greek mythical character with many eyes) is generally available. It brings a multi-Kafka developer experience and complete autonomy to streaming data.

What’s new in Lenses 6.0?

The biggest change: Lenses now lets you access multiple Kafka clusters across different environments, at the same time. This reflects how the biggest enterprises work with Kafka today. We shifted to a global perspective that:

• Unifies Identity and Access Management (IAM)

• Brings a global data catalog to explore and find topics

• Centralized SQL experience for exploring data across topics.

This works across all clusters and all environments, no matter the vendor. 

With Lenses 6.0, you get this, plus everything else Lenses offers: Kafka Connectors and their self-service deployment, SQL Processors, consumer and producer alerting, data lineage, and governance.

Why multi-Kafka?

As AI grows, real-time services powered by streaming technologies like Apache Kafka have become essential. Consumers now expect instant responses, pushing more companies to expand their Kafka infrastructure. 

This expansion has created a multi-Kafka reality. Companies now commonly run hundreds of Kafka clusters across different clouds and environments for better scaling, cost control, workload isolation, and regulatory compliance. 

Lenses 6 lets you use any streaming technology while giving engineers a unified, well-governed experience. It’s part of our journey to create a Data Operating Fabric for streaming data. 

Lenses Data Operating Fabric is an intelligent platform that oversees and controls your complete ecosystem of real-time data and apps across any streaming infrastructure. Think of automated operations to build and deploy applications, including AI Agents over a streaming backbone of your choice. 

New features

Global SQL Studio

Our most loved feature has received major improvements.

Invariably, you spend a lot of time in SQL Studio to explore data. But you also need to monitor consumer groups in the same workflow. Or validate a schema. And if you’re working across multiple clusters, you want to compare one environment with another.

So not only is SQL Studio now multi-Kafka native, it also has:

• Split-screen to directly compare topics in different clusters

• Better insights into query performance, bad records and tuning

• Consumer group monitoring in a new IDE experience

• Multi-Kafka native support 

We’re continuing to evolve SQL Studio into an IDE for streaming data, helping you build and deploy faster.

Global access controls (IAM)

Our permission model now supports multi-Kafka environments with more granular controls and role-based access. For example, you can create operator roles that can only restart connectors or SQL Processors in production – without providing access to data or configuration. 

The new model is inspired by AWS IAM, using YAML with IDE-style support such as intellisense, autocompletion and schema validation.

Global Data Catalog

The Data Catalog now collects metadata across your entire streaming ecosystem. It infers schemas even for non-schema sources (CSV, JSON, even XML) by analyzing sample events, and supports custom tagging and classification. 

Now operating at a global level, it’s the launchpad to find, grant access and share real-time data across your multi-Kafka environment. 

Architectural change

We had to think carefully about how to architect version 6 to scale to the growing number of Kafka clusters within a business, what security requirements they may have, and what clouds and environments they run in:

• Lenses 5 becomes a headless agent installed next to each Kafka “environment” (including the Kafka cluster and surrounding services such as Schema Registry, Kafka Connect, etc)

• The new HQ component manages the governance & provides the developer experience

HQ federates all requests to agents. No raw data is transferred to HQ. This makes the architecture secure and scalable to hundreds Kafka clusters that may live in totally different regions and environments.

What’s next?

Kafka to Kafka replication (K2K)

It’s no secret that Lenses has developed a Kafka replicator, called K2K. 

Lenses 6.0 lays an amazing foundation for us to offer a single-click experience to replicate data from one Kafka to another, for data sharing, disaster recovery, and more.

Our new Kafka replicator builds on our SQL Processors technology instead of Kafka Connect (like MirrorMaker2). It deploys directly to Kubernetes, requires no additional infrastructure, and uses Lenses’ existing authentication and governance features. 

This joins Kafka Connectors and SQL Processors as our third managed application type.

AI integration

We’re also building AI capabilities to:

• Deploy agents in your workflows

• Boost productivity within Lenses

• Identify issues in data flows

• Simplify Environment & IAM configuration

• Query data in Kafka with Lenses SQL via AI interaction

• Configure pipelines, Connectors, SQL Processors and more.

Introducing Lenses Community Edition

Many of you would have known Lenses Box - a free and self-contained Docker of Lenses and a single-broker Kafka. We’ve replaced Lenses Box with Lenses Community Edition. It includes: 

• Lenses 6.0 (HQ and 1 agent)

• Single broker Kafka with supporting services

• Ability to connect to a second Kafka cluster

• No license key required

It’s completely free and ideal for: 

• Developers: Building and testing data-driven applications.

• Data Engineers: Managing and monitoring Kafka environments.

• Data Analysts: Exploring and visualizing data streams.

• AI Engineers: Discovering and preparing data for RAG pipelines and more

• Students and Educators: Learning about data streaming and Kafka.

Get started

Download Lenses Community Edition from our website and follow the installation instructions. We've prepared detailed documentation and tutorials to help you get up and running quickly.

Feedback?

We are committed to making Lenses an operating fabric with the best developer experience for streaming data. We want your feedback! Join our Slack community to share your thoughts and experiences with us.

We’re excited to see what you will achieve with Lenses 6 and Community Edition. Start today and see where it takes you.