How Vortexa sped up and modernized their data platform with DataOps + AWS Managed Streaming for Apache Kafka.
Vortexa tracks trillions of dollars of sea-freight moving in real-time
Vortexa uses state-of-the-start data science and engineering to process data at a massive scale, tracking more than $7 trillion per year of seaborne energy flows globally and instantly.
The service is built around an Apache Kafka streaming data platform, where an uninterrupted flow of data and fast delivery of new data products is critical to their business.
Kafka was powerful, but it was also a black box difficult to stabilize & self-manage
Debugging incidents took days
One small mistake could bring down an entire cluster
Getting to production was painful
Adding DataOps to Apache Kafka
“The combination of Lenses.io and Amazon MSK allows us to focus on business logic instead of SLAs.” - Jakub Korzeniowski, Head of Data Services at Vortexa.
Vortexa uses open monitoring with Prometheus to access JMX metrics that Amazon MSK brokers generate. See the architecture above.
“Amazon MSK and Lenses.io have been pivotal technologies for Vortexa, enabling us to shift significant efforts from maintaining and stabilizing a complex and fragile Kafka infrastructure to focusing more on the quality of analytics and market insights that directly impact the value we deliver to our customers.”
Maksym Schipka, CTO - Vortexa
Working hours saved on Apache Kafka management
Takes minutes (not days) to deploy to production
Increase in Kafka users (both operators and application developers)
Reduction in AIS signal noise aided by QA done exclusively in Lenses.io