Apache Kafka - When the honeymoon is over

We have a large number of really talented developers using Lenses Open Source and commercial solutions. Many of whom have built and deployed Apache Kafka in various organizations the wrong way.  

“I’ve created a monster”, “I should have never started this”, “When am I going to find time to do my day job?”, “What have I done?”, “My Kafka honeymoon is over”. 

We all speak of the same deep regret.. 

A new project requires streaming data. 

You’ve worked with Apache Kafka in a previous role, you know it works and it’s Open Source. 

Feeling confident, you tell your team Kafka makes a lot of sense. Besides, it’s free and supported by a huge community. 

What could go wrong?

The project is high profile, your team is in a hurry and they don’t want to go through the slow and painful processes of working with the Infrastructure team to provide this service. 

You know Infrastructure will add too many constraints. They'll limit it to specific client versions, impose strict ACLs and lengthy onboarding processes. It won’t be as good as what we can do ourselves. And the chargeback will be expensive.

Either way, the faster path is going around them and standing up Kafka on our own.

And besides, it’s only to support a pilot initiative so I'm not even sure if it’s worth getting IT involved.

So I take it on myself to deploy and run Kafka. Yeserie.

Me and my team come out looking like rock stars, the project goes well and Kafka is popular - very popular.  

In fact, it’s making me such a star that other teams are now coming to me to ask questions about whether they can onboard onto Kafka too. I'm wheeled-in and out of senior management meetings to present what we've done. 

So other teams onboard. First one. Then another. 

Before I know it, I'm spending 85% of my non-day job caring for and feeding this baby. Well? After a while, this baby ain't looking quite as cute as it used to be.

What was simple to manage at the beginning is now not so easy. Kafka is starting to go south. And requiring constant tuning to cater for different workloads. 

I didn’t ask or need this headache

At first, I was doing the other teams a favor, but now they're complaining their service is down because I need to bring down the cluster for rebalancing. 

IT hear about this and are pissed because people have onboarded critical services and the Service teams aren’t getting alerts to support a 24x7 SLA.

And then...I'm really in hot water. The security audit. 

You have *whaaaat* data in there?? (I didn’t know either, I just found out some dev decided to send PII data).  Wait, no audit trail of who’s accessing that data either? No identity management? People logging into servers with root like it’s 1999. Oops!

Now unless your Kafka went to prod but was a bigger flop than Cats (the movie), running it without governance and visibility will always end in tears. And not happy tears. 

And if your Kafka never even made it to prod, that too was probably because you didn’t have the right tools. 

What if you could be the rockstar and still have your weekends to yourself?

Imagine if Kafka wasn't such a black box. That it could be operated in a secure and self-service fashion by anyone or any department without needing you at all. Or if devs could explore their data, debug their microservices and see what flows they have deployed by themselves.

And you never had to explain again that “Kafka needs some time alone” to a few dozen developers and execs shouting at you just because someone "fat-fingered" deleting a topic which brought down the cluster.

Think this sounds too good to be true?

We don't.