There are two major challenges with building and operating streaming data platforms.
Managing the infrastructure itself, a cost centre: Installing, patching, fixing.
And managing the app & data landscape, the value generator for business. That’s building, deploying, administering, governing, securing, monitoring.
The former has been commoditized. AWS have been trail blazers and provide a vast array of infrastructure services.
Take AWS Managed Streaming for Apache Kafka (MSK) as an example, delivering a successful streaming infrastructure such as Kafka requires dozens of supporting services (identity, storage, secrets, Kubernetes etc.). Services that are unmatched with other cloud Kafka providers.
It’s the latter, the app & data landscape, that remains the battleground for businesses. This is where Lenses comes in to help.
For joint Lenses/AWS customers such as Vortexa, AWS data infrastructure with a DataOps layer to build, deploy and manage streaming applications is a powerful combination.
We have worked closely with the AWS MSK Managed Kafka product team to coordinate our engineering efforts and make migrating to MSK or simply starting with MSK, an amazing experience with value delivered in hours and days.
Here are just a few areas that help in your delivery on MSK.
Monitoring
Visibility into the infrastructure and flow performance is critical to getting to production and giving stakeholders confidence to adopt Kafka further.
Today at Re:Invent 2019, we’re announcing with AWS being an official launch partner of MSK Open Monitoring with Prometheus.
It exposes JMX metrics from the MSK brokers and collected by Lenses.
This adds to the existing metrics that Lenses provides - including performance of flows and microservices (such as consumer lag, throughout etc) and other Kafka infrastructure components (Kafka Connect, Schema Registry etc.).
Self-service administration
To ensure developer productivity, it is essential to provide a secure admin & governance portal for engineers to build and operate an application landscape in a self-service fashion.
At one end, we have a control-deck of sorts across Kafka & Kubernetes. This allows everything from creating Topics and evolving schemas to deploying apps & flows from a single UI and API.
Then for devs that want to see inside their streams (ie. for debugging), this portal provides data access layer allowing them to explore, insert or delete data via SQL.
The data access and the self-service administration is consistent across any Kafka, this is especially useful to provide consistency when migrating from one Kafka to another and of course supporing multi-cloud Kafka. And it is backed by fine-grained security controls, audits and support for multiple different authentication strategies (LDAP, Kerberos etc.).
Migrating to MSK
If you’re migrating from a self-managed or any other Kafka to MSK, the Lenses CLI client allows you to manage your application state as configuration (Topics, ACLs, Quotas, pipelines etc.) and port across different environments. With a GitOps approach, this reduces risk, accelerates and simplifies the migration to MSK.
Building Apps
Building a streaming application that joins, filters or wrangles data or creating a data integration from, for example, Snowflake to MSK, can be done in seconds and be deployed over your existing infrastructure such as AWS ECS or Kafka Connect. All this with a layer of security, auditing and monitoring.
This was just a teaser. Explore all the benefits of Lenses for AWS MSK and get access to a free trial.
Thanks for reading.
