MarineTraffic maps shipping fleet data with Apache Kafka

To achieve this, they enabled their platform team with Lenses.io - an easy, fast way to query real-time data on Kafka.

The Lenses.io portal helps us to harness all of Kafka’s powerful complexity and make it more available and actionable for us. It would have taken us 3x as long to develop our own tools with tape and sticks - so working on Lenses.io helped us move much faster to fulfil analytics requirements. It has also been much easier to onboard newcomers to Kafka. The technology seems infinitely less intimidating which means that more team members can master its benefits.

Christos Mantzouranis, Back-end Team Leader

“It didn’t end well for Kafka the author, so watch out.”

MarineTraffic helps the marine industry to track vessel positions. Real-time data is a mission-critical propeller for all their products - from predictive models for fuel consumption and changing ship routes, to live maps for customers to view their fleet in relation to the weather.

This was why the Engineering team decided to switch to Kafka for its decentralization and horizontal scalability. The team was partway in a necessary transition from self-hosted bare-metal infrastructure to the cloud. This involved rebuilding their monolithic .NET, PHP and Node.JS application to a new cloud-native microservice architecture based on streaming data. The team's bigger goal was to make smaller deployments on cloud services like Amazon and migrate from self-hosted Kafka to Kafka-as-a-service with AIVEN.

But although Christos and his team wanted to get better-acquainted with Kafka’s complexities, the resources required to make the data accessible and comprehensible across analytics and product teams was too big of an ask.

From batch to stream, sink or swim?

“We needed connectors and ACLs - an easy way to query data in Kafka across the company without investing the manpower and hours needed, and with Lenses.io, these needs were met.”

MarineTraffic uses Lenses.io to achieve the following overview for their real-time data:

• Visibility of data flows to get production-ready: providing a full overview of the health and security of applications
• Setting alerts: tracking consumer lag and configuring alert conditions allows the team to give the right critical information to the right data consumers
• Easily creating topics, ACLs & Quotas: viewing, creating, edit or deleting from a UI or API with full role-based access controls and audits.

Simplifying Kafka commands with SQL has increased productivity by 50%

“We now have the capacity to ask everything with Kafka; to find messages which were discarded. We are using LSQL to query our data which is very familiar to both our software and data engineers. Also, the ease of set-up for the various connectors has allowed fast ETLs.

Now 10-12 developers log in every day.

Being able to horizontally process messages allows us to generate notifications that would otherwise be complex to calculate. With Lenses.io on Kafka, we can easily monitor the topology and scale up when needed.

It’s increased our productivity by 50%.

If it weren’t for Lenses.io the decentralization process would have been a real struggle. Keeping a monolith running on a single humongous server is a huge investment and a constant liability.”explains Christos.