SaaS company increases AWS MSK productivity by 200%
How Vortexa sped up and modernized their data platform with DataOps + AWS Managed Streaming for Apache Kafka.
Vortexa tracks trillions of dollars of sea-freight moving in real-time
Vortexa uses state-of-the-start data science and engineering to process data at a massive scale, tracking more than $7 trillion per year of seaborne energy flows globally and instantly.
The service is built around an Apache Kafka streaming data platform, where an uninterrupted flow of data and fast delivery of new data products is critical to their business.
Challenge
Kafka was powerful, but it was also a black box difficult to stabilize & self-manage
Debugging incidents took days
One small mistake could bring down an entire cluster
Getting to production was painful
Solution
Adding DataOps to Apache Kafka
“The combination of Lenses.io and Amazon MSK allows us to focus on business logic instead of SLAs.” - Jakub Korzeniowski, Head of Data Services at Vortexa.
- Open monitoring: Vortexa views their application topology and exposes consumer lag metrics, visualized in Lenses before forwarding to third-party tools such as Slack and PagerDuty.
- Building streaming apps: Lenses provides full management capabilities for streaming applications in Apache Kafka clusters, making testing and ops painless and more intuitive.
- Self-service for Kafka & Kubernetes: Vortexa uses Lenses governance controls for real-time applications that run across their MSK and Kubernetes platform.
Vortexa uses open monitoring with Prometheus to access JMX metrics that Amazon MSK brokers generate. See the architecture above.
Results
Fast-tracking data to production in minutes, not days
“Amazon MSK and Lenses.io have been pivotal technologies for Vortexa, enabling us to shift significant efforts from maintaining and stabilizing a complex and fragile Kafka infrastructure to focusing more on the quality of analytics and market insights that directly impact the value we deliver to our customers.”
Maksym Schipka, CTO - Vortexa
15%
Working hours saved on Apache Kafka management
Fast
Takes minutes (not days) to deploy to production
200%
Increase in Kafka users (both operators and application developers)
90%
Reduction in AIS signal noise aided by QA done exclusively in Lenses.io