UI-driven GitOps: Opening up Kafka without giving up governance

As Kafka evolves in your business, adopting best practices becomes a must. 

The GitOps methodology makes sure deployments match intended outcomes, anchored by a single source of truth. 

When integrating Apache Kafka with GitOps, many will think of Strimzi. Strimzi uses the Operator pattern for synchronization. This approach, whilst effective, primarily caters to Kubernetes-based Kafka resources (e.g. Topics). But this isn’t ideal.  

Firstly, the Kafka market includes self-managed and fully-managed deployments — some with some with Kubernetes, others not — and many operating in a range of Kafka environments. 

Secondly, even with Kubernetes, a streaming data landscape covers more than just Kafka entities like Topics and ACLs:

• Alerts for application and platform health

• Metadata and topology of your applications

• Data policies for protection and data masking

• User permissions and accounts

• SQL-based streaming applications

Furthermore, GitOps is best delivered through a high-quality Developer Experience (DevX), and robust governance including auditing and security.

Why Developer Experience matters for Kafka

Kafka’s rise to a mainstream technology brings new challenges:

• How do I make Kafka as simple to adopt as possible?

• How do I meet corporate data security and governance?

• How to avoid the platform team being the bottleneck for integration, and innovation by product and application teams?

• How can Kafka development fit within corporate engineering, CI/CD practices, and tools?

Adopting a DevX strategy for Kafka needs to address these points.

This DevX will allow teams to efficiently deploy data pipelines and streaming applications, work around an engineering team’s existing tooling, and importantly adhere to CI/CD practices.

All are designed to ensure maximum engineering productivity and to open up Kafka to 100s, perhaps 1000s of engineers or users sitting outside IT and Engineering.

Kafka-as-code, from a UI

Lenses provides all you need to visualize and build real-time data streaming applications and pipelines on any Apache Kafka; from Topic creation and alerting to Kafka Connector lifecycle management, and deploying SQL-based data processing apps. 

Calling the API or clicking in our much-loved UI has a direct result: As long as you have the Lenses permission, Lenses will make a change directly on your infrastructure; from deploying a connector, to creating a Topic, to evolving a schema.

This may have been fine early in a Kafka journey. But as the workloads onboarded to Kafka become more critical, deployment should go through your corporate CI/CD practices. 

Here’s the problem. The users that require access to deploy data pipelines increasingly don’t sit with IT or Engineering, and are not familiar with CI/CD practices or Git. Even if they're acquainted with software delivery practices, they may consider a UI to be more productive.

Self-service access whilst conforming to CI/CD practices

Through the Lenses CLI you can export all Kafka and Lenses resources. Everything from Topics and Connectors to Alerts & Data Policies. Simply check these files into your Git and promote them in different environments using the Lenses import CLI function. 

However, while the CLI works, we can do better. For Developer Experience, productivity is key. 

Lenses 5.5 is the first release in our follow-up 5.* releases to offer native UI-driven GitOps capabilities to manage your Kafka environment. 

We’re starting with “View as code” for Kafka Connectors, allowing users to configure a Connector and download the definition to add to their Git, and trigger their own CI/CD process instead of deploying it in Lenses. 

Lenses now allows you to:

• Deploy connectors from the configuration files for Connectors, directly in the UI

• Configure new connectors and download the configuration as YAML to your file system, which can be checked into your Git repository.

The road ahead for GitOps and Kafka

Future Lenses releases will include an expanded GitOps integration covering all aspects, from Topic Management to SQL Processors. This will enable direct Git interactions for pull requests and state reconciliation within Lenses, moving towards a fully automated Git-driven workflow.

Integration with CI/CD workflow is a wider theme. Imagine the ability to import and export Lenses resources via CLI, checking these environments into Git, and promoting them in different environments. 

The ultimate goal is a holistic, policy-as-code approach that combines flexibility with software governance. Every environment owner should have the ability to apply their granular permissions and role-based access controls, paving the way for policy-as-code and emergency overrides, without compromising on CI/CD principles.

______

Interested to learn more about Kafka and GitOps? Watch the guided tour: