Lenses 2.3 Release

Lenses 2.3 is out and it’s cloud ready!

As it happens with every Lenses release, new features, improvements and fixes take place. A big focus of this release was bringing Lenses and DataOps into multiple major cloud providers and managed services of Apache Kafka. SQL capabilities have been enriched, and many performance (integrating with large clusters with up to 20000 topics!) and security aspects have been addressed.

Also with regards to version updates, Lenses is now also compatible with Kafka 2.1.

Lenses on Cloud

Visibility and control over your streaming data and flows with enterprise-ready features are what make Lenses essential for your DataOps platform. Today, the underlying infrastructure is available both by managed services, cloud providers as well as on-premise. Lenses provides an infrastructure agnostic environment for its end users, making the experience seamless whether you go cloud, on premises, or even hybrid environments. Lenses offers a common interface and an easy way to operate over multiple setups.

With the new release, Lenses has simplified the deployment in cloud providers using its own provisioning and management cloud templates tailored to the particular cloud. (AWS, Azure). Lenses is more deeply integrated with cloud resources:

AWS:

• Compatible with Amazon Managed Streaming for Kafka (Amazon MSK)

• Service discovery of brokers, zookeeper and workers (schema registry and connect cluster)

• CloudWatch logging

• CloudFormation templates (VM, VM Docker, ECS, Reference Architecture)

• Run as Linux Service with systemd

Azure:

• Compatible with HDInsight >= 3.6

• Deployment as part of HDInsight cluster as Edge Node

• ARM Templates (VM, VM Docker, HDInsight)

• Boot Diagnostics for VMs starts

• Run as Linux Service with systemd

Google Cloud:

Lenses can be installed on GCE near your Apacke Kafka cluster. In the near future we will bring increase the automation process for it

Discover Lenses on clouds

Secure JMX metrics & Jolokia

JMX (Java Management Extensions) metrics are used for infrastructure and data flow observability and although an optional feature, it greatly enhances the level of detail in regards to monitoring. In a typical Kafka setup, most of the services like the Brokers, Connect workers, Zookeeper etc expose JMX metrics.

We advise everyone to treat security as a high priority, so we recommend when using JMX to retrieve monitoring information to setup security. This can be done either via credentials or alternatively via a Joolokia integration. Using credentials you can now secure the access to the metrics to tighten up security.

An alternative to JMX is now Jolokia, that is a JMX-HTTP bridge. Jolokia provides simple access to JMX beans via a RESTful API. Typically an agent exposes the metrics via HTTP and provides APIs serving either GET or POST requests.

Find out more about Kafka Broker Metrics

Lenses SQL

Lenses SQL engine, gets constantly updated and enhanced with new features in order to allow users to operate their data with ease. Here is what’s new in this version.

User Defined Aggregate Functions

Lenses SQL has a great variety of built-in functions available. User Defined Functions (UDF) is a supported feature in order to extend Lenses SQL whilst manipulating data on Kafka.

User-Defined Aggregate Functions (aka UDAF) is a new feature of Lenses SQL that enables defining new functions and extending the SQL vocabulary for data aggregations.

Lenses provides a simple API in order to implement the UDAFs. Here is an example:

Find out more about UDAF

Views & Synonyms

Views & Synonyms support has been added to the supported features.

A view is a virtual table, generated dynamically based on the results of a SELECT statement, whereas a synonym gives you an alias for a table(topic) which can be very handy when you deal with long names for your topics.

Find out more about Views & Synonyms

DateMath expressions

Working with date and time data types in programming languages, whether this is converting from timestamps to human readable formats, adjusting for different time zones, or handling intervals and durations is brought with tedious conversions and edge cases in both SQL and general purpose programming languages.

This release offers direct support for easier manipulation of dates, times and timestamps with DateMath expressions. DateMath expressions can be used in any SQL statement where a general purpose expression is allowed - for example as part of a where filter, in a group by clause, a select projection, and so on.

The basic types supported by DateMath are: Date, Time, DateTime, Timestamp

Find out more about DateMath expressions

Aggregations live updates

Doing aggregations on the table based engine (ad-hoc queries) can take a while before the result is given. There is now a new option to trigger intermediate results; for example, if you do a simple SELECT COUNT(*) total FROM smart_meters_temperature you will get to see the total value increasing as the engine processes the Kafka records.

Conclusions

While the next version (which will bring new and unique functionality) is already well underway, we hope you enjoy the new features of our 2.3 version.

We would like to thank our customers for providing constant feedback and ideas to make the Lenses DataOps platform even better.

Get the new version now!

If you are running the Lenses Box, don’t forget to docker pull to get the latest updates:

docker pull landoop/kafka-lenses-dev:latest

You can go through the 2.3 Release Notes