New Kafka governance: approval flows & app topology

It feels like only yesterday that we announced Lenses 3.1. The release was focused around helping developers be more productive when developing real-time data applications. 

Since then, our customers are onboarding new applications and new users onto their data platform at a faster rate than ever.

And with more apps and users come stricter and tougher requirements for compliance and governance. 

We don’t want to have any great features sitting on the shelf longer than they have to in order to ensure you get as much value as quickly as possible from Lenses.io. So brace yourselves for great new features from now on as soon as they leave the factory.

3.2 is our first release on this cadence and we’re thrilled with it.  This release is packed with features to help our customers meet those compliance and governance requirements that so often slow organizations down in adopting real-time data platforms. 

Let’s take a look.

Approval Workflows

We know Kafka can be sensitive to poor configuration. So after popular demand, we’ve introduced new approval workflows for change management. This removes the need to rely on external systems (such as ServiceNow or Git) to manage workflows. 

Approval workflows for Topic creation means you can put in place segregation of duties to increase compliance and reduce risk. 

We’ve also added a new permission capability “Create Topic Request” to our security model. A manager or colleague can approve (or reject) the request having reviewed all the details and capacity planning requirements (we ask the submitter to provide metadata such as associated business unit, description etc.)

All Kafka App Topology for Observability

Our Topology view is one of our most appreciated features. Imagine having been locked in a dark room all your life and then coming out to see sunlight for the first time. That’s the experience you may have when seeing how your deployed real-time applications and flows are connected in an interactive map.  

Flows and applications deployed through Lenses (such as SQL Processors or Connectors) have always been automatically mapped in the topology view. For external Apps, we provided a Topology Client library that could be included in your JVM based code. 

But what about non JVM-based apps? Now in 3.2, we’re adding topology support across any application: Python, .NET, Go or via a simple HTTP request.

Having an application topology and understanding the performance and interdependencies between different apps, flows and data processed will help ensure good hygiene of your data platform, assist operations to troubleshoot problems (or avoid problems), increase developer productivity and ensure overall data governance such as data provenance and data lineage.

SSO Support

With security threats increasing, organizations really don’t want to manage access to business applications across multiple identities. They also want to cover off two-factor authentication and password compliance. 

Connecting users with a single identity provider is one of the first things a security team will impose to reduce risk. Your data platform is a business application like any other and shouldn’t be treated any differently. 

With 3.2, we have extended support for Okta, Keycloak and OneLogin SSO adding to the Azure AD we included in 3.1. And don’t forget, we also support management across multiple different authentication strategies including LDAP AND Kerberos simultaneously!

Integrate audits into Splunk

Customers love that Lenses.io provides the building blocks to operate a data platform. Our centralized security layer with auditing reduces risk. 

When it comes to audits though, no one wanted logs held in a silo or there to be a risk that audits are tampered. Compliance and security teams will ask that they have real-time monitoring of user actions from within their tools. 

So now in 3.2 we have a native integration to send audit logs directly into Splunk, often used as a SIEM. The audits will also meet Splunk’s Common Information Model to ensure it can be easily integrated into any CIM-conforming Splunk Apps such as Enterprise Security. 

Last but not least

We want to make Lenses.io more accessible to everyone. This includes those just wanting to learn Kafka or DataOps practices. Our Lenses.io Box docker container, packaged with Lenses.io and a single-broker Kafka environment has been downloaded millions of times by developers and Kafka enthusiasts over the years.  

We’re happy to announce that we’ve now brought the Box experience to the cloud. Access a Lenses.io Cloud Workspace for free to help you develop real-time applications on Kafka. Sign up now!

All our new features can be tracked at features.lenses.io. 

Enjoy the release!