Summary
- K2K 2.0 introduces continuous offset replication: consumer group offsets are translated to a standby cluster in near real time, so applications can fail over and resume from the latest offset.
- This unlocks credible disaster recovery for any Kafka, cloud or on-prem, with no broker changes, no client changes, and no Kafka Connect.
- Available today in standalone K2K; integration into the Lenses DevX comes next.
Kafka has become the backbone of the real-time enterprise. The streams it carries are not only time, but business critical: a fraud event isn't processed, a sales order not fulfilled, a trade not settled.
Yet we heard a recurring theme from Kafka teams: their business is running critical streaming applications without proper Kafka resiliency.
Among organizations that have a DR solution in place, operational complexity is a recurring concern, and it drives cost: replication pipelines commonly require several dedicated engineers to manage. In a recent research paper from Stratola 43% of respondents said their business assigns three or more dedicated engineers to managing replication pipelines.
The pressure is only going to grow. Resilience now tops the priority list of security and platform leaders, and regulations like DORA turn it from good practice into legal obligation.
Kafka is widely recognised as lacking choice when it comes to DR - IBM/Confluent (means being locked into their Kafka stack) and an open-source option with MirrorMaker2 (not fit for true enterprise environments).
Until now.
Today we are giving enterprises more choice and higher resilience with K2K 2.0: enterprise-grade data replication with continuous offset replication, delivering disaster recovery for Kafka, independent of vendor.
Continuous offset replication, the missing piece of Kafka DR
Like for any infrastructure, good DR for Kafka requires
- Synchronisation of infrastructure configuration & state (topics, ACLs, Quotas …)
- Replication of topic data
- Synchronisation of app state (consumer groups)
- Practices for failing over (and back)
- Monitoring of DR health posture
Synchronisation of configuration has been solved via the likes of Kubernetes Operators and GitOps practices.
Lenses K2K, offered standalone or as part of Lenses platform, has provided cross-cluster data replication for some of the biggest enterprises. This was especially powerful offering self-service to engineers for data sharing and Kafka migrations.
For migrations, Synchronisation of app state came by a manually-invoked action to replicate and translate the offsets for consumer groups. It assumes the source cluster is online of course.
In the event of a disaster, that might not always be the case.
So K2K 2.0 now provides continuous offset replication & translation, designed for situations where the source cluster and its consumer group state are no longer available.
It’s made up of two cooperating components. The K2K replicator publishes source-to-target topic data as well as offset mappings to a control topic as it replicates. The Offset Mapper continuously reads those mappings and translates committed consumer group offsets onto the target, in near real time.
The result: your consumer groups exist on the target with correctly translated offsets at all times. Same topic names, same group names. Planned migration or unplanned outage, applications connect and resume from their last committed offset.
For a deeper look at the architecture and engineering behind it, check out our technical deep dive.
Enterprise Kafka Disaster Recovery but without the vendor lock-in
Despite Kafka's diverse vendor ecosystem, enterprise DR was a gap: many businesses ended up choosing a vendor not on cost or capabilities, but simply because it was the only one that credibly covered disaster recovery.
This lack of choice also made it difficult for those with a multi-vendor Kafka estate (MSK and Confluent for example). This meant they needed to operate an open-source (specifically MirrorMaker2) or fragmented DR solution.
At Lenses we believe in enterprises having the choice of best-of-breed data streaming infrastructure managed through a unified and enterprise-grade operating fabric.
With K2K 2.0, the management of data across a multi-Kafka environment now extends to the replication of data across that infrastructure for DR, independent of the underlying infrastructure vendors.
Designed for enterprise footprints
The Lenses Developer Experience solution has been trusted by some of the largest companies building and operating critical streaming apps for almost ten years.
K2K was also built for these same environments - to address the enterprise compliance, assurance, cost, scale and operational simplicity requirements of large businesses.
How much enterprise teams struggle with those requirements shows in the Stratola's survey: almost half assign three or more dedicated engineers to managing replication pipelines, and of those, 84% take a week or more to provision a single new replication job.
Much of that burden traces back to one architectural decision: MirrorMaker 2 runs on Kafka Connect, so every pipeline means standing up and babysitting a separate Connect cluster. K2K removes that dependency entirely.
But also performance compares favorably: In our cross-region MSK benchmark, K2K matched MirrorMaker 2's 100% reliability while delivering 14–32% lower end-to-end latency and 51–78% faster producer writes, with 5x better batching under exactly-once. And it does so from a smaller footprint: one file of configuration, no Connect cluster, and, together with Lenses, governance and observability across all of your clusters.
Getting started
K2K is available as standalone or as part of Lenses DevX.
The standalone K2K Community Edition is free - it replicates up to 5 topic-partitions per pipeline; offers at-least-once processing, schema mapping and the offset replication.
Get up and running in minutes: pull the Docker image or install the Helm chart, point one YAML file at source and target clusters, and replication is running.
To scale beyond that, a commercial K2K license unlocks enterprise features like exactly-once and much more.
For the full DevX experience, including governance, guided deployment and observability, pair K2K with the Lenses Community Edition.
Last but not least, don’t forget to join our community Slack to talk through your DR strategy.
