Get started with Lenses for AWS MSK in minutes.Start now →

SaaS company increases AWS MSK productivity by 200%

How Vortexa sped up and modernized their data platform with DataOps + AWS Managed Streaming for Apache Kafka.

Kafka Governance

Vortexa tracks trillions of dollars of sea-freight moving in real-time

Vortexa uses state-of-the-start data science and engineering to process data at a massive scale, tracking more than $7 trillion per year of seaborne energy flows globally and instantly.

The service is built around an Apache Kafka streaming data platform, where an uninterrupted flow of data and fast delivery of new data products is critical to their business.

Challenge

Kafka was powerful, but it was also a black box difficult to stabilize & self-manage

Debugging

Debugging incidents took days

Cluster

One small mistake could bring down an entire cluster

Production

Getting to production was painful

Solution

Adding DataOps to Apache Kafka

“The combination of Lenses.io and Amazon MSK allows us to focus on business logic instead of SLAs.” - Jakub Korzeniowski, Head of Data Services at Vortexa.

  • Open monitoring: Vortexa views their application topology and exposes consumer lag metrics, visualized in Lenses before forwarding to third-party tools such as Slack and PagerDuty.

  • Building streaming apps: Lenses provides full management capabilities for streaming applications in Apache Kafka clusters, making testing and ops painless and more intuitive.

  • Self-service for Kafka & Kubernetes: Vortexa uses Lenses governance controls for real-time applications that run across their MSK and Kubernetes platform.
Architecture

Vortexa uses open monitoring with Prometheus to access JMX metrics that Amazon MSK brokers generate. See the architecture above.

Results

Fast-tracking data to production in minutes, not days

Vortexa

“Amazon MSK and Lenses.io have been pivotal technologies for Vortexa, enabling us to shift significant efforts from maintaining and stabilizing a complex and fragile Kafka infrastructure to focusing more on the quality of analytics and market insights that directly impact the value we deliver to our customers.”

Maksym Schipka

Maksym Schipka, CTO - Vortexa

15%

Working hours saved on Apache Kafka management

Fast

Takes minutes (not days) to deploy to production

200%

Increase in Kafka users (both operators and application developers)

90%

Reduction in AIS signal noise aided by QA done exclusively in Lenses.io

How to

Get started with Lenses for AWS MSK in minutes

Start