New Lenses Multi-Kafka Developer Experience

SaaS company increases AWS MSK productivity by 200%

How Vortexa sped up and modernized their data platform with DataOps + AWS Managed Streaming for Apache Kafka.

Vortexa tracks trillions of dollars of sea-freight moving in real-time

Vortexa uses state-of-the-start data science and engineering to process data at a massive scale, tracking more than $7 trillion per year of seaborne energy flows globally and instantly.

The service is built around an Apache Kafka streaming data platform, where an uninterrupted flow of data and fast delivery of new data products is critical to their business.

Challenge

Kafka was powerful, but it was also a black box difficult to stabilize & self-manage

Debugging incidents took days

One small mistake could bring down an entire cluster

Getting to production was painful

Solution

Adding DataOps to Apache Kafka

“The combination of Lenses.io and Amazon MSK allows us to focus on business logic instead of SLAs.” - Jakub Korzeniowski, Head of Data Services at Vortexa.

Open monitoring: Vortexa views their application topology and exposes consumer lag metrics, visualized in Lenses before forwarding to third-party tools such as Slack and PagerDuty.

Building streaming apps: Lenses provides full management capabilities for streaming applications in Apache Kafka clusters, making testing and ops painless and more intuitive.

Self-service for Kafka & Kubernetes: Vortexa uses Lenses governance controls for real-time applications that run across their MSK and Kubernetes platform.

Vortexa uses open monitoring with Prometheus to access JMX metrics that Amazon MSK brokers generate. See the architecture above.

Results

Fast-tracking data to production in minutes, not days

“Amazon MSK and Lenses.io have been pivotal technologies for Vortexa, enabling us to shift significant efforts from maintaining and stabilizing a complex and fragile Kafka infrastructure to focusing more on the quality of analytics and market insights that directly impact the value we deliver to our customers.”