New Lenses Multi-Kafka Developer Experience

Download

Introducing Lenses 6.0 Panoptes

Unifying data streaming across modern data architectures.

Andrew Stevenson

Nov 19, 2024

Organizations today face complex data challenges as they scale, with more distributed data architectures and a growing number of teams building streaming applications. They will need to implement Data Mesh principles for sharing data across business domains, ensure data sovereignty across different jurisdictions and clouds, and maintain real-time operations.

To address these demands, we've completely rebuilt Lenses for version 6.0, with a unified set of tooling that simplifies the experience for developers, and how they work with distributed streaming infrastructure.

New Lenses architecture: Breaking down data silos

Managing a sprawling data landscape can be a real challenge. As organizations scale their streaming infrastructure, they often end up with dozens or even hundreds of Kafka clusters spread across clouds, deployments, and vendors. This segregation helps address compliance needs, control costs, and minimize risk – or it may stem from an acquisition or a strategic shift.

But this distributed approach has traditionally made it hard for platform teams to optimize their data operations. Meanwhile, developers are similarly scattered, with little to no direct access to the streaming data they need to explore or manage. Instead, they must rely on the platform team.

Based on this reality, here is what we’re building towards:

An operating fabric to unify developer experience, bring intelligence and automation to real-time data in a multi-Kafka world.

Lenses 6.0, codenamed Panoptes (after the Greek mythological figure with many eyes), unifies the developer experience when working with any technology that is compatible with the Apache Kafka API. This includes:

On-premise deployments
Managed services like Confluent Cloud and Azure Event Hubs
Self-managed solutions like Confluent Platform and Redpanda
Edge computing solutions

New Architecture

We have adapted Lenses 6 to this new multi-Kafka reality. Lenses now consists of a centralized control plane we've labeled HQ, and individual Agents that sit next to the Kafka clusters themselves.

The Agents are much like the previous Lenses, but headless, and can only be accessed through HQ.

These Agents let Lenses scale to hundreds of clusters across global networks, and help to unify data management, while respecting organizational boundaries and compliance requirements:

https://docs.lenses.io/latest

The architecture will support an increasing number of cross-Kafka operations and automation that will be introduced into the product in the coming releases.

Enhanced governance with more granular IAM

One place to manage all streaming platform access and permissions

Secure access is a big challenge with distributed streaming setups across vendors and deployments. How can you enable self-service for developers, whilst keeping compliant?

We’ve rebuilt our Identity and Access Management to support this modern, distributed model. The new IAM framework is closely aligned with cloud solutions such as AWS IAM. This allows organizations to delegate more responsibility to engineers, whilst meeting compliance requirements.

Platform teams can now:

Define more granular permissions across multiple environments
Support decentralized operations while maintaining compliance
Manage access patterns through as-code-first policies

For example, a simple policy can ensure developers access only development topics across all environments:

Or giving the support team the ability to restart connectors, without allowing them to view the underlying data or connector configurations.

Global topic discovery for data sharing

Explore the metadata of all streams across all clusters, from a single Global Data Catalog

The new concept of a Global Topic Catalog allows engineers to discover streaming data across different domains and Kafka clusters of their business – a core Data Mesh principle.

Instead of relying on tribal knowledge to locate data in different clusters, teams can now use Lenses 6 to search metadata across all their streaming infrastructure.

A Global Topic Catalog like Lenses 6 allows developers to find data streams across the organization. It supports data democratization while maintaining security through granular access controls. Developers can discover topic metadata even without direct access, eventually gaining access through your governance processes.

Global SQL Studio for exploring data enterprise-wide

Global Topic Catalog is coupled with the much loved Lenses SQL capabilities for data exploration, now brought to the global level. Just one SQL workspace to explore data across any cluster.

And what makes our SQL exploration capabilities so special? It is intentionally designed for exploration, it never needs to move the data out of the cluster, cloud or topic. The engine explores data where it lives: in the topic.

There are big further developments coming for SQL Studio, so stay tuned over the coming months.

Lenses 6.0 Community Edition: Free forever

Lenses 6.0 is now available as a Community Edition for you to use for free, and it is evolving every day. It is designed to be run locally on your laptop using Docker, and contains: