How Vortexa makes Apache Kafka smooth sailing

Vortexa is an energy analytics provider. They offer the most complete dataset for global refined products and crude oil movements, supporting traders to formulate trading decisions on the market.

Their advanced real-time analytics service is built around an Apache Kafka streaming data platform; made up of two clusters and over 250 streams in each cluster. The uninterrupted flow of data and fast delivery of new data services to their customers is critical to their business.

But to start with, Apache Kafka was a black box. And the team struggled to stabilize the self-managed Kafka infrastructure.

• Debugging incidents took days

• The smallest mistake could bring down a whole cluster

• Slowdowns getting into production

• Specialist skills were difficult to come by These fragilities and complexities in an all-important Kafka infrastructure cost the team their weekends and ate into their release schedule.

While Kafka and Kafka Streams are extremely powerful technologies, they are black-boxes. Lenses created unprecedented visibility into the real-time data flows in Vortexa’s Kafka infrastructure, helping to reduce development cycle times and build confidence in R&D.

Maksym Schipka, CTO

From minutes to seconds to inspect messages

To address this, the Vortexa team evaluated a number of managed solutions, including Amazon Managed Streaming for Apache Kafka (MSK), and found that MSK best met their requirements and budget. Amazon MSK continuously monitors cluster performance, is self-healing, and automatically replaces unhealthy nodes without the need for downtime. Vortexa used Lenses.io as their data operations portal, giving them data observability and governance control for their real-time applications that run across their MSK and Kubernetes platform.

Today, both engineering and operations use Lenses for insights into the Kafka environment.

Everyday tasks performed by developers that are essential for creating high-quality services, such as inspecting messages on a stream, understanding the profile of a topic, or verifying topics have the right replication factor are now easy.

Previously, such ad-hoc tasks took around five-to-ten minutes and were achieved using a combination of different CLI tools and open source UIs. Having a one-stop-shop for prototyping, debugging and operations now brings that time down to seconds.

I’ve found the Lenses UI absolutely brilliant for creating Kafka connectors. Lenses makes experimenting with connectors almost trivial, it’s so fast to change config and redeploy.

Kit Burgess, Data Scientist

Open monitoring with Amazon MSK & Lenses.io

The health of the MSK infrastructure can be monitored within external tools using the AWS Open Monitoring framework. Vortexa uses Open Monitoring with Prometheus to access JMX metrics generated by the MSK Brokers.

Investigating consumer problems in minutes

The availability of their Kafka platform is critical as Vortexa’s customers make decisions worth millions of Euros on the confidence of having correct and up-to-date information.

Consumer problems, and in particular identifying the root cause of performance degradation in a consumer group, was a difficult challenge. With Lenses, they track consumer performance and can instantly alert when a consumer has high lag.

Prior to Lenses, Vortexa had tried Confluent Control Center but found it placed too big load on their brokers. They also attempted to use various open-source Kafka tools but struggled with limited functionality. Vortexa estimates that troubleshooting a problem on an application can be done within one minute using Lenses compared to hours using previous tools.

We had problems with Confluent Control Center putting significant strain on our Brokers and other open source tools, such as Kafka Manager, lacked the rich features. Lenses is one of the lightest on resources and gives us the single tool that we need to make us productive.

Jakub Korzeniowski, Head of Data Services

Empowering engineers to build data flows

Deploying new data pipelines that integrate data across their different technologies, such as Elastic Search and Redis, can now be built with no coding and deployed in minutes with Lenses open source connectors and helm charts.

Lenses allows Vortexa to first prototype a new data pipeline: this empowers engineers to test their configuration in a UI and explore and validate the data created in the stream. Before, the team felt that the Kafka-Connect framework lacked enterprise-readiness, so all the data integration connectors were in-house developed applications.

Today Vortexa has the enterprise-features on Lenses to adopt Kafka Connect. They estimate that through their real-time data portal, a new data flow can be developed and deployed into production in minutes compared to one day of effort using Kafka Connect connector minus Lenses and several days of effort when maintaining their own connectors.

Lenses has streamlined prototyping and deploying Kafka connectors to S3, Redis and Elasticsearch, by removing the need to context-switch between configuration and monitoring, and by feeding back error messages due to misconfiguration. This has greatly reduced the development time needed to complete these tasks.

Richard Mathie, Platform Engineer

Monitoring as a map for Developers & SREs

Many of Vortexa’s apps are based on the Kafka Streams framework. By integrating the Lenses topology client into their Kafka Streams applications, Vortexa is able to see a real-time topological view of not only their data landscape (including their flows to Redis and Elastic Search) but also inside their Kafka Streams applications.

This allows their Developers and SREs to better understand the streams and identify where there are bottlenecks, such as low message rate throughput as well as drilling-down to see consumer group lag.

Energy market players rely on Vortexa to provide reliable real-time data so they can make smarter shipping and trading decisions. This all hinges not only on possessing the streaming power of Apache Kafka; but in harnessing it by making their data available to the right people - thereby shortening the development cycle and speeding up time-to-market.