Creating your managed Kafka shortlist

You’ve been handed the not-so-easy task of scoping a managed Kafka for your team.

How do you start the shortlist? Post something on Reddit? Skim read a gazillion review blogs? Crash Google Chrome opening a thousand tabs to compare feature lists?

If you’re going to run a Kafka POC with two or three vendors, or you’re trying to find the best Kafka for your business, how can you narrow down your selection?

Let’s get to it.

The challenge is three-fold:

• Before moving to a managed Kafka, what should you consider? 

• How do you begin to know what factors should be overlooked and what will make all the difference? 

• Pressure to pick the right vendor from an exploding market 

From big players like AWS MSK to new entrants like Aiven, an increasingly crowded Kafka-verse means more vendors popping up. 

Which vendor meets your chosen considerations? Cost, vendor lock-in, security, performance. 

Save your sanity with this comparison tool: ta daaa

The managed Kafka comparison tool helps you shortlist a few vendors saving you weeks of collecting disparate information. 

Lenses DataOps and productivity tooling works with any flavor of Kafka (it acts just as a Consumer & Producer would). We have also worked with each service provider to collect their latest capabilities, and will continue to update these. 

A few notes on why we made this, and how best to use it: 

Filter by… Cloud(s)

Whether you’re an AWS-only shop - or you plan to host Kafka in a multi-cloud environment over the coming years, choose up to three clouds that can fulfil your requirements. 

Filter by… Required features 

Next, you may need supporting services. Some vendors will provide these as part of their offering, others won’t:

• Replicating your Kafka data - MirrorMaker?

• Handling the distribution of Schemas - Schema Registry?

• Integrating data in and out of Kafka - Kafka Connect? 

• More easily handle platform admin actions - Kafka Rest Proxy?

Grouped by… General capabilities

Your recommended Kafkas are then highlighted and added to the tables below and on the right, so you can remove or compare services as you scroll through the five tables:

• Overview: A summary of the Kafka’s strengths and superpowers  

What do they each do best and what makes them stand apart?

• Hosted by: Available clouds side-by-side

As you filtered above, but it’s useful to see in a full view to help multi-cloud companies.

• Deployment: Where or how do you need to deploy your Kafka? 

• Core service features: Which extra capabilities do you need to plug or bake into your Kafka? How much manual tuning and access control do you need over performance metrics, Zookeepers and costs? And what can you easily automate?

• Compliance: What ISO certifications do they have? Can audit trails be extracted on who has made infrastructure changes? How can you authenticate users?

We hope it helps you skip a few stages of research, close a few tabs and clock off slightly earlier.