Lenses 2.0 Release

Today, we are very pleased to announce the release of Lenses v.2.0.

Lenses is the streaming management platform for Apache Kafka. This release focuses on improvements based on the feedback we’ve received and introduces a ton of exciting new features.

Here’s a quick overview:

• Lenses SQL Engine supports JDBC and introduces a Kafka JDBC driver

• Lenses platform now ships with Lenses CLI

• New Clients for Python & Go

• Notification Gateway and native integration with Alert Manager

• New Features for Data Governance and Multi-tenancy management

• Enhanced Topology with Metrics for Processors, Connectors & Topics

• Cloud Native capabilities

The release notes are available here. Let’s explore them bit by bit:

Lenses JDBC for Apache Kafka

Lenses 2.0 offers a JDBC 4.0 compliant driver, suitable for any application that uses the JDBC interface, to communicate with Apache Kafka.

Apart from SELECT statements users of the library can INSERT data into topics, use Prepared statements and batched inserts and also fetch metadata around topics and messages.

The JDBC driver opens new use-cases, as you can now interact with Kafka like you would do with a traditional database, or pull or push data from other third-party tools like Apache Spark.

You can download the JDBC driver from here (docs, source)

Lenses-cli

Lenses now ships with a Lenses CLI (Command Line Interface) that allows you to streamline your ops, simplify many actions and integrate with the CI/CD processes but at the same time keep aligned with the security and governance Lenses provides.

All the capabilities to interact with your data, in live stream or batch mode, administrative features to interact with various components, for example register a stream processor and manage its lifecycle, create and configure topics, quotas or ACLs, spin up connectors and so on. All those actions in Lenses that are audited and stored to ensure your data governance are also now exposed via CLI.

By using this CLI you can perform all the major CRUD actions, similarly to the Lenses web interface, execute SQL queries and automate several tasks. Download now and explore with:

  $ lenses-cli

You can download the cli driver here (docs, source)

Lenses Clients

Beyond the lenses-redux client to interact with SQL queries straight from your browser app, Lenses is now introducing the lenses-python and lenses-go libraries for native apps. The Go library was used to build and deliver the CLI.

The Python library that you can explore more here (docs and source code) is widely used in the space of data science for its rich ecosystem around machine learning. Via Lenses you can now integrate and get data directly to pandas, Jupyter notebooks, plotly etc. Stay tuned for additional blogs and articles over the next few weeks.

Lenses Platform

Lenses is more scalable, provides additional Data Governance capabilities, supports better multitenancy, allows managing Quotas and setting custom alerts and at the same time is more Kubernetes and Cloud ready. Read below additional details:

Data Governance

With the new regulations around data privacy and companies handling private data making efforts to minimize their risks, Lenses treats Data Governance as a first-class citizen for your data in motion. Apart from: security, role based access, authentication (LDAP), authorization and auditing features that come out of the box with Lenses platform, features over data security are now also available.

If you are leveraging Apache Avro as a data container, you can now annotate any sensitive field of your Schemas with:

“obfuscate”:“true”

Once a field is marked, Lenses will obfuscate the contents and make them invisible to any user accessing via any Lenses channel (web, cli, endpoints, clients etc). Here is an example:

In addition to the above, Lenses introduces the ANONYMIZE function for both browsing and processing data, making it a useful tool to allow data to be shared, whilst preserving privacy:

Similarly to user actions, all SQL queries can (optionally) be tracked to ensure the detection of unauthorized access if needed.

Multi-tenancy

A multi-tenant Apache Kafka cluster always brings challenges for operations. With Lenses you can now manage such aspects via the web UI or the CLI such as ACLs. In this newest version you will also find Quotas management but also as an admin you can set-up fine grained level (read/write) access via applying blacklist and whitelist to users to better control access to topic data.

Topology

Topology is one of the main features in Lenses that visualizes the Apache Kafka based data flows with interactive nodes. It includes the elements that are used in the currently supported flows like source and sink connectors and Lenses SQL processors as well as the involved topics. This view is now enhanced with the state of each node and the relevant metrics to easily identify how the node performs or if any inconsistency occurs.

Both Connectors and Processor views have been enhanced with metrics even in their granular level ie. at a task level. With a single look you can now identify any failures or performance issues.

Monitoring & Alerting

The alerting mechanism has been re-designed, to be more meaningful in detecting any unexpected behavior or anomaly and allow you to set up custom Kafka domain specific alerts. You can now control cluster infrastructure alerts and set custom alert rules around consumer lags.

The integration with Alert Manager fully decouples the alert generation and the notification routing. This allows notifications to be pushed into different channels with full support for email, sms, pager duty, slack and many other channels.

Data Browsing

Data browsing has been enhanced with new capabilities to make it easier to execute long running queries and perform thorough look ups. Every query can now be bounded by max bytes / max time or max records. Queries can also be short-circuited if numerous consecutive polls do not fetch any data, for better resource utilization.

All the default values can be overridden, but you can also manage and even stop/cancel any live queries. This functionality

Here is an example of the syntax:

Of course you may run the Lenses SQL via the web UI, the CLI and Lenses Clients.

Cloud native capabilities

Lenses brings Cloud Service discovery capabilities for widely used cloud providers, such as Amazon AWS, Google Cloud, Microsoft Azure, DigitalOcean, OpenStack, Aliyun Cloud, Scaleway and SoftLayer.

Kubernetes

New Lenses Helm Chart. The Lenses chart now supports services, ingress and RBAC in Kubernetes with improved configurations.

Lenses Box!

Lenses Box is the free version of Lenses for developers. It is a single docker image, built for all Kafka developers. If you haven’t tried it yet, just go ahead at https://docs.lenses.io/dev

Release notes

To review the full release notes check

https://docs.lenses.io/2.0/release-notes/