Lenses upgrade: powerful SQL, Connect multi-tenancy & message headers

We’re continuously releasing new capabilities to enable more people to be productive and compliant when working with Apache Kafka.

Our engineers have been hard at work on a great Christmas present for our amazing community of Kafka users. 

Before unwrapping our latest and greatest release, here’s a sneak peak of what’s inside.

Streaming SQL workloads on Kafka Connect

In July, we revamped our Streaming SQL engine for powerful data processing minus the typical operational and development complexities. 

This came with a new deployment framework (“DAD” as we call it).

You’ve got enough distributed technologies to manage. So we built the framework to simplify deploying streaming SQL workloads on your existing infrastructure alongside wherever you run your other applications.  

For our first DAD Release, we included support for Kubernetes. And now we’re extending the framework to deploy your workloads over your Kafka Connect infrastructure too!

If you want to understand how it works, checkout our recent CNCF webinar.

Powered-up Streaming SQL for complex data structures

Speaking of Streaming SQL, Lenses 4.1 comes packed with new features that make life even easier when processing data with complex data structures. We’ve introduced 6 new functions to work with arrays, better support for array literals, and lateral joins.

All these features are available both for our Snapshot and Streaming SQL engines.

Lateral joins for example is a new kind of join that allows you to extract and work with the single elements found inside an array, as if the array was a normal table.

As a basic example converting the structure:

To single-value structure such as:

This means you can define even more types of data processing workloads with SQL rather than on complex stream-processing frameworks. 

Read Nicolò’s blog about Lateral Joins for more detail.

Explore Kafka Message Headers with SQL

Apache Kafka 0.11 introduced headers to messages in Kafka. A very welcome feature.

Kafka Headers are useful for annotating, auditing, monitoring and routing of events as they flow through Kafka. 

This opens up a number of new use cases, including for monitoring data lineage and distributed tracing, that help make Kafka more governable and allow you to offer higher service-levels to your users.

Our new release now allows you to query headers alongside Kafka message payload and key with the Lenses snapshot SQL engine. This will radically accelerate the time to investigate incidents and problems.

See our message headers blog for more information. 

Data Policies namespaces

Data privacy has been a first-class citizen of Lenses since the beginning.  Policies allow you to discover and anonymize data within your streaming data. This means data can be socialized across your business whilst maintaining top notch compliance.

Data Policies were applied globally across all matching Kafka streams and Elasticsearch indexes. This wasn’t always what everyone needed. As an example, an emailAddress field should be anonymized for production topics and not across dev topics. 

So whilst you're defining Data Policies you can now apply policies based on specific topic namespaces. 

Read more about the latest Data Policies features and how to protect your data in our blog life of PII for Apache Kafka.

Kafka Connect multi-tenancy

We see many of you are managing dozens if not hundreds of Kafka Connect clusters. Often in a multi-tenant fashion. Operating so many clusters puts a strain on good governance and adds risk.

Not only should you control what topics users and connectors can access but also potentially completely isolate tenants to your platform for compliance. Avoiding them even seeing other tenants clusters let alone having access to them.

The Lenses security model has protected which Kafka topics users can access (on top of what you may have already with ACLs). We’ve now extended this security model so users can set privileges over which Connect clusters can be accessed by whom.

To understand more about applying compliance and governance to Kafka Connect, some further reading here.

App Catalog Drilldown

The Topology view is one of our most popular features. It provides a view of the lineage of real-time data flows. Very useful for impact analysis and troubleshooting for example. 

We’re now simplifying the workflow so you can understand the relationships between data and applications with an instant drill down from a data flow to an enhanced view of an App’s metadata. 

This shows detailed information about an application generating data including state, health and business metadata.

Take Lenses 4.1 for a spin?

The release is packed with more features including support for recursive schemas, new Kafka Connect Alerting and improvements to how you can manage your license keys.  Full release notes can be found here: https://docs.lenses.io/4.1/release-notes

Experience the latest version in a cloud demo environment or as an all-in-one Kafka & Lenses docker now!

Merry Christmas and a thanks to everyone in the community for their wonderful support.