Top takeaways from Kafka Summit and industry trends
The Lenses team share their top takeaways from Kafka Summit and industry trends
Oct 09, 2019
We’re back from Kafka Summit San Francisco 2019. Our leadership team, always with an ear to the ground, sat down over a cup of coffee to distill; and agree on the top trends and takeaways from the event!
DataOps resonates
Among hundreds of people that visited our stand, it was impressive how many people now understand DataOps and the growing momentum behind it.
We have been championing this DataOps movement since our birth; helping modern organisations to finally focus on people and business value, the application layer and develop with efficiency rather than struggle with infrastructure.
Clearly the message is strong and we all are excited leading DataOps to the future!
One of the early adopters of Lenses, Martin Kess working at Namely boldly presented on stage “no one cares about Kafka for the sake of Kafka. We all care about having a positive impact on your customers”. It fully resonates with us and is a strong statement to say in a room full of Kafka enthusiasts!
Skill shortage
When asking engineering managers “What is your biggest challenge?”, the answer was almost unanimous: “The lack of skills in the market”. Apache Kafka can no longer be just for the elite data engineers. It’s now mainstream. Either the number of skilled engineers needs to increase, or we need to make it easier for everyone.
We prefer the latter, and this is what we call a DataOps approach:
The power of SQL
There were a few announcements at Summit regarding SQL including push based queries for applications. We can’t agree more, and this is why Lenses supports this for almost 3-years now! It was one of the founding principles of Lenses and only a matter of time others thought about it. Check out the three ways we use SQL on Kafka here:
Apache Kafka as a commodity & growth in managed Kafka
The open source community has done the hard work. The foundations are there and a powerful streaming technology such as Kafka is now widely adopted. But now people are tired and it’s time to hand it to others to manage it. We sensed an exceptional amount of momentum to leverage managed Kafka and in particular AWS MSK and Azure HDInsight.
But infrastructure is just part of the puzzle. How you monitor, debug, secure, build and develop with real-time data is a whole new challenge which is where we come in!
Everyone trying to build DIY self service
To increase developer productivity whilst imposing governance and security, many organisations are developing Kafka self-service portals - to create topics, manage ACLs, edit schemas, etc. Many are even thinking about self-service access to data!
There is so much wasted energy and valuable time here! One can easily add up the cost. But it’s not just about that! It’s more about not focusing on delivering new digital experiences for customers!
That’s why we have a team of 50+ engineers working for almost 3-years to deliver a unified Kafka and streaming data experience with APIs, GitOps, a UX/UI with multiple security, data privacy, governance and auditing layers. So that organisations can focus their engineering efforts on delivering those digital experiences.
Read about Self Service Adminstration for Kafka Read about Security for Kafka
Data privacy & Kafka security challenges
Multiple product teams building on vast amounts of streaming data opens quite rightly significant security, data privacy challenges and risks. This is one of the biggest areas slowing down adoption we’re hearing: plugging security risks with Kafka and getting sign off from compliance teams.
Thankfully, organisations need not have such concerns to take risks. Lenses has a whole suite of features to secure platform and data and meet compliance.
Related Blogs
