Kafka Total Cost of Ownership: What are you missing?

“We’ve seen two years’ worth of digital transformation in two months” said Microsoft’s Satya Nadella. Due to COVID-19, digital transformation roadmaps have been deleted, redrafted, doubled down and accelerated by up to a decade. Traditional companies are moving by osmosis towards streaming technologies such as Apache Kafka to kick off new digital services. 

But how much should it cost to experience 2030 in 2021?  

Only a couple of months prior, Everest Group found that 73% of digital transformations fail to provide any business value whatsoever. 

Why the big gulf between streaming technology and business value? What can we learn from this? 

How can you measure the less obvious parts of your Total Cost of Ownership of Kafka, beyond infrastructural or managed service costs?

With great streaming power comes increased complexity

Kafka has gained popularity with application developers and data management experts because it can greatly simplify working with data streams... If it is scaled in the right way. 

We see many new users of Kafka report that initial results are outstanding. Their cluster handles any amount of data they throw at it; showing incredible fault tolerance and horizontal scaling.

Our community reports that growing from a pilot to a small budgeted Kafka project solo or with your small team, for example, is relatively straightforward.

Increased exposure to risk

Their Kafka implementation is often so stable that they almost forget about it - until one day, when disaster strikes, and their internal customers are left in the dark. 

Teams scanning through billions of messages in a topic from a command prompt when you have no time to waste can cost you millions of dollars in downtime.

Low data visibility

It isn’t only important to be able to see data, streams and applications on every level and across time during a catastrophic incident. Developers need access to all kinds of data in and around Kafka on a daily basis, to answer the following questions (beyond the performance of Kafka itself): 

• Are my applications and data healthy? 

• If not, why not? 

• What events are published to this topic? 

• Which applications are consuming that topic?

• What’s the provenance, lineage and access level for this data in Kafka? 

These blind spots create inevitable slowdowns. Lacking line of sight into the critical business components touched by Kafka means developers can’t see far enough to ship new features to market.

Reduced product velocity

These slowdowns result in reduced time-to-market of your critical features and applications. As your platform team becomes the gatekeeper of all that is Kafka - and Kafka becomes the backbone of your business - these slowdowns can mean the difference between delivering that recommendation engine or that Customer 360 initiative ahead of your competitors; or not. 

Here Swiss cashless payment solution Viseca shows how the Kafka business value curve steepens as projects become more strategic at a certain stage of maturity.

2020 taught us many things, one being that consumer habits change fast. We are living on a knife-edge where a matter of hours for a single release becomes the difference between a company’s making, or its downfall. 

So where do you want your team’s focus to be?

Is Kafka your product, or is data your product? 

It reads like a rhetorical question, but actually it’s a question that companies often forget the answer to. 

As a technologist and business leader, it’s a question of whether you want to hire more Kafka specialists, or more business domain experts. Data practitioners that can drive real value, whether that’s the latest investment analytics product, the fastest fraud detection algorithm or the greatest mobile banking app.

These dimensions offer a high-level insight into how Kafka costs can quickly impact your development productivity and even your business’s bottom-line. 

Even so, it’s hard to measure where you are in your Kafka adoption in relation to each of them. How much is Kafka costing you currently compared to other real-time businesses? What do you need to help mitigate the unnecessary expense?

That’s where our new Kafka ROI calculator comes in. 

It’s our very own pilot project that we will continue to build on, adding more dimensions as we get feedback from our community. 

We hope it helps you get a grasp of your Kafka TCO from a productivity perspective.