Confluent Current 2024 - the good, the bad, and the tacos

Austin, Texas is one of those North American cities that truly stands out from its peers. It’s a strange mix of music, state government, higher education, and a major tech hub. Oh and did I mention tacos? Soooo many tacos!

All of these factors make it an ideal place to hold a Kafka conference, and Current 2024 did not disappoint. At key times during the conference, from day 1 check in to final drinks on Thursday, live bluegrass and country-western bands serenaded us.

Local food trucks fed us tacos and bacon-covered donuts in that oh-so-Austin style. 

Confluent Current Keynote Tacos

Like most tech conferences these days, the main theme was focused around AI – which, not gonna lie, did elicit a tiny eye roll from your author as he was registering. However, my eyes rolled quickly back into focus when I realized this wasn’t going to just be empty AI hype.

Starting from the day 1 keynote, the integral role that Kafka and streaming will play in the coming AI revolution was placed front and center. Indeed, multiple speakers convincingly demonstrated that data streaming will be what brings Large Language Models (LLMs) and AI out of the “new and cool way to cheat on your high school history term paper” into something that is actually useful to businesses and society as a whole.

The keynote and many of the breakout sessions I attended underlined how moving AI out of the realm of “ain’t it cool” into something actually useful will require rapid and reliable data streams, both incoming and outgoing from the AI agent. Over the course of the two days at Confluent Current 2024, I attended many sessions detailing AI agents and processes that can do real work in real time, using Kafka streams behind the scenes to ferry the massive data flows necessary.

Left unsaid in all the keynote excitement was the fact that we are just at the beginning of this process. Most companies are dipping their toes into the world of AI agents, and the like. To get to the goal of “actually useful,” developers and platform engineers have their work cut out for them.

To enable this AI revolution, engineers will need to build and manage multiple Kafka clusters, connectors, and databases to move and host all this data. Then, developers will have to build the actual applications on top of these streams and databases. And yes of course, we want our revolutionary AI agents as soon as possible, please!

Real-time AI Example Tacos

A simple but well known example: asking an AI agent to rebook you after a flight cancellation. That AI agent will need to know about you: your seating preferences, your status with the airline, and the kind of ticket you purchased. Then it will need the most up-to-date flight info, available seating, and even weather information streamed into it before it can even offer you alternatives.

Once you’ve made your choice it will need to be able to stream back out those updated reservations, and seating assignments. Finally it will need to stream back the updated reservation, free meal certificates, as well as good restaurant recommendations in your particular terminal. That was just one such example among many I saw during the conference. 

Lurking underneath this example is the major issue of data security and governance. This AI agent is going to have access to my airline account, my ID or passport, not to mention my credit card information. Making sure this data is secure and kept within the proper organization and geopolitical silos will be of the utmost importance.

Platform engineers will need to keep these issues at the forefront as they design and implement these complex systems. Developers will need an easy and secure way to understand both what’s in the data streams, and how and where it can be used.

I saw another exciting example when I attended a session outlining the complex tech stacks required to power things like AI news agents. Corbett Waddingham’s talk “Live Context: Powering RAG with Real-Time News Integration” gave a fascinating demo of a static AI that had ingested the BBC news site but didn’t have a method to stream in live and breaking news from the site. This static LLM was displayed side-by-side with the same AI agent, but with a live and almost real-time news feed. Asking about current events – both sporting and news – gave very different results from the two models. Live streamed data is a requirement for true AI agent utility.

One unexpected takeaway from attending multiple of these types of talks was the many different database and AI technologies that could be used with the common streaming framework of Kafka. Kafka data streaming is truly poised to become the “circulation system” of AI. These systems weren’t simple, though. AI that can perform complex tasks will require complex stacks. Developers will need tools to help them navigate these complex stacks of clusters, topics, and connectors to build AI agents and applications. 

Kafka Maturity Tacos

Another more unspoken theme of the conference was the overall maturity of the Kafka streaming ecosystem – it has grown to impressive proportions in the past decade. Loads of vendors of all shapes and sizes lined the aisles of the expo hall. Day 2’s keynote “The Rise of the Data Streaming Engineer” really underlined the maturity of the ecosystem.

Streaming data has moved from a nice-to-have, to an integral part of most modern data stacks. Naturally Kafka brings the ability to do so at scale, both reliably and securely.

Useful and revolutionary AI will be powered by the Kafka ecosystem going forward, and all of us “data streaming engineers” will be working tirelessly behind the scenes to make it all happen. Moving even further into the future “non-IT, non-developer” users will also want access to the data flowing through our streams - tools and products that allow non-technical folks access will be integral.

Several other sessions I attended underlined the overall maturity of the ecosystem. One memorable one was also one of the shorter ones – just a 10 minute mini session from Simon Aubury - a contractor who has been implementing Kafka from its earliest days until now. He was able to pack in a decade’s worth of Kafka wisdom into just 10 mins.

Another notable session that traced an evolving implementation of Kafka along with an interesting use case: streaming auction sales from “deep in the wilds of Saskatchewan” with a non-reliable cellular uplink.

In A Tale of Two Platforms: Routing Events at the Edge, Nic Pegg detailed his team’s years-long journey to turn a paper-based auction system into a modern near real-time data system for these remote auctions. His team did all this while dealing with the real issues that crop up when you’re a 10+ hour drive from “civilization.” Another recording to check out!

Constructive Criticism Tacos

Overall I was genuinely impressed and sometimes amazed by the tech and integrations being explained and often demoed in many of the talks. Some of the sessions, while being impressive in their technical techniques or vendor offerings, fell into the all-too-common trap of being cool but contextless.

A good rule of thumb for any conference talk is to include the business problem that needs to be solved. Then tell the story of how you came to solve it and show the techniques you used. Just giving me the cool techniques without this context makes the session way less memorable. Remember, why you do something is just as important in the long run as how.

Lenses Multi-Kafka Tacos

Back at the Lenses booth, our headline and theme was “Lenses 6 - Multi-Kafka Developer Experience.” While it may seem that we are targeting developers and the developer experience with our newest release, the key phrase “Multi-Kafka” enticed many platform engineers to come over and ask questions as well. They, more than anyone else, have watched the number of Kafka implementations increase throughout their organizations, while the number of people on their teams have stayed the same, or decreased. 

We were only too happy to explain how Lenses 6 could take their developers’ Kafka needs off the platform team’s hands and out of their ticketing system. Enabling developers helps both sides of the Dev/Ops equation. While it can be said that Dev/Ops and platform folks aren’t easily pleased – many seemed genuinely enthused with the idea of empowered developers who don’t file tickets just to see a few events from a stream. 

Speaking of developers, we spoke with many of them as well. I could feel a palpable pressure on them to deliver on the promise of real-time streaming in their applications. Many spoke of running up against the complexity of their company’s Kafka systems - multiple clusters across dev / staging / prod in multiple business units across the organization.

One developer team told me they did an overall Kafka audit and found that multiple teams had recreated similar topics and streams, simply because they didn’t know that other teams had already created them. 

Of course we would be remiss if we didn’t mention our management visitors. They too saw the value of a “single pane of glass” for their ever growing Kafka implementations. As Seema Acharya from Mercedes-Benz presented in the keynote:

“We need observability into our whole fleet, what our customers like, and what they do not... Not every team is lucky enough to have access to a data engineer, so we need a way for those working with the data to get closer to it all.”

I can honestly say that it felt good to offer these folks a tool designed to make their lives as data streaming engineers easier. Seeing people smiling when they realize Lenses 6 can actually help make their jobs easier. 

Taco-nclusion

After two days of Kafka, bluegrass, and tacos, I came away really excited about the future of Kafka. Seeing the actual utility of AI for genuine business needs and services that can improve our lives and make things more rapid and intelligent was a nice change of pace from the gloom and doom or utopian AI that dominates the mainstream press.

As for Kafka itself, It was really fascinating to see a technology I’ve been working with tangentially since 2015 really move into the mainstream of technology. I’m genuinely excited to see what all you data streaming engineers have in store for us in the years to come. Can’t wait for Current 2025 in yet another food mecca: New Orleans!