DataOps on Kafka - AWS MSK with Lenses.io

Hi everyone, I’m sharing this from the Kafka Summit San Francisco.

Apache Kafka has become a central component of the modern data platform or microservice-based architecture. Nothing new there.

Kafka is also known to be painful for operational teams to manage. Everyone wants NoOps. Nothing new there either.

Even after ten years. To be slightly fair to the community, expectations are rising higher than the technology is able to develop.

• More developers needing visibility into streams

• Higher criticality of overlayed services

• Increased security, compliance & data privacy requirements

• A mixture of technologies lacking governance and controls

Lenses has a proven record as a DataOps layer over data infrastructures such as Kafka and Kubernetes. This empowers developers to build code-free streaming apps with confidence and deploy via GitOps.

To be successful, it’s not just Kafka

But that still leaves the infrastructure challenges of Kafka. This has now got to a tipping point and a race-to-the-door towards a hosted managed Kafka service.

But before everyone rushes, it’s important to remember that Kafka is just one of many infrastructure components that are required as part of a successful data platform architecture.

You need a compute platform for your workloads (for example via Kubernetes), secret management, identity management, a range of data stores or applications as your data sources and sinks and more.

Lenses as DataOps layer for MSK

There’s no better partner in the market with such a rich set of mature services than AWS. We’ve been collaborating with their product teams for a while and have helped many customers such as Vortexa successfully migrate from self-managed to Amazon MSK.

Our joint value proposition is clear: turn-key, secure infrastructure services that scale up and down cost effectively with AWS. Topped by a secure data access layer with audits, governance, role based access, monitoring and SQL to quickly access data and Streaming SQL to build streaming applications via Kubernetes and manage with GitOps from Lenses.

Today we are formally announcing our partnership and our drive to reduce the time and effort required to build new and secure data experiences for our customers.

This partnership fully aligns with our mission: “To enable a small team, to deliver in a few days what currently large teams usually take months to deliver, and to do that in a future-proof way”.

More info, getting started

• You want to give Lenses & MSK a try in just a few minutes ? Follow the 3 steps here

• Read more into Lenses with AWS MSK in Spiros’ blog

• And read our press release