The new K2K Offset Mapper replicates and translates Consumer Groups between any clusters.
If you replicate data between two distinct Kafka clusters, you already know the payloads can match while the offsets might not.
This post is about how K2K 2.0 now also keeps consumer committed offsets in sync between the source and the target so consumer groups can fail over in a Disaster Recovery situation, avoiding large re-reading of data or row skips.
This offers the community more choice for DR than just MirrorMaker2 and Confluent solutions have until now.
We will explain the new offset mapper service for K2K, why offsets diverge, what that costs you at runtime, and the offset-mapping path that makes the two clusters agree on progress.
1. The problem
Each Kafka cluster assigns its own offsets when a message is written. Replication copies messages; it does not copy the source broker’s offset. Fail over a consumer group to the target cluster and its source commits are just integers, not consistent with the original offsets at source and must be translated in order to avoid applications for the consumer groups missing or replaying data.
2. Why source and target offsets differ
Retention
Kafka drops log segments when data retention or topic size limits are hit.
After that, those offsets cannot be read back.
As shown below, the deleted record cannot be read causing the source and target offsets to differ.
Compaction
On a compacted topic, the broker throws away older rows when a newer row with the same key arrives. Kafka will compact its own log on its own schedule.
If the replicator reads the segment after the compaction has taken place, it won’t see the compacted record nor will it be informed that it took place.
Offsets will therefore differ when copied as shown below.
Transactions on the source
Transactional writes weave control records into the log. A consumer that only reads committed data will see gaps (Kafka will not forward those internal or aborted slices as application messages).
When replicating records, consumers follow that view: they ship application messages and skip control batches and aborted segments causing offsets to diverge once more.
Transactions on the target
On the target, the replicator runs its own transactions, its own batch sizes and commit cadence. Kafka still writes transactional metadata, but its layout in the log will not match the source. Transactions can abort on the target too (think retries, failures). So you can get transactional machinery on the target that never happened the same way on the source, while the source still has holes the pipeline never shipped.
In summary the cases above are all different scenarios of the same underlying property: replication moves application data, it is not a replay of everything the source broker ever did. Retention deletes, compaction, transactional control batches, aborts are all part of how the source log got to its current shape.
The K2K application reads what consumers actually see and appends fresh writes on the target. After replication, the same number almost never names the same record on both sides, which is why committed offsets must be mapped between source and target clusters.
3. Challenges
K2K optimizes a few things at once:
Control topic size. Mappings land in a dedicated control topic on the target. Mapping information should take as little memory as possible.
In-memory footprint. The offset-mapping application keeps a queryable structure while it translates commits. It cannot pretend RAM is infinite and store all the individual mappings from source.
Lookup speed. Every time a commit needs translation, the lookup needs to be fast.
Older mappings still count. Someone can spin up a new consumer and commit the offset 1 in the source. For that committed offset to be correctly replicated, you need older mappings on hand - not just the recently mapped pairs.
Offsets can rewind. Consumer commits do not only march forward. At any point, operators can set a consumer group backwards on the source cluster in order to reprocess records. The Mapper has to translate whatever commit the broker reports now, not only positions newer than the last high-water ever saw. If you threw away every mapping “below” the previous commit, a rewind would have nothing to look up. K2K needs to keep history, not a single trailing pair after the latest offsets.
So the problem is not “translate a mapping” once. It is “store enough history for late joiners and for rewinds”, “keep the control topic and the heap under control”, and still answer lookups quickly every time commits move. The next section is how K2K and the K2K Offset Mapper split that work.
4. Solution
Two things run at the same time: K2K (replication + mapping records) and K2K Offset Mapper (read mappings, follow source commits, write target commits).
Replication path (K2K)
The K2K application starts and runs the replication pipeline.
While it copies data from source to target topics, it also produces control records that carry the offset mappings. Those records are written to the control topic on the target cluster. They are the durable log of “what we already translated,” not an optional side channel.
In parallel: K2K Offset Mapper
The K2K Offset Mapper process reads from the control topic as offset mappings are created by the K2K Application. It builds its internal representation of the mappings.
In parallel with that ingest path, it continuously reads consumer groups from the source cluster: which groups exist, which partitions they use, and what offsets are committed. Whenever a committed offset changes, the mapper does a lookup in its internal structure and commits the corresponding offset on the target cluster for the same consumer group and corresponding partition.
The result is a target cluster that contains the same data as the source cluster but also equivalent consumer group information.
5. Ranges, compaction on the wire, and the index in memory
To store offset mappings efficiently, K2K cannot keep every single (source, target) data point forever. Throughput would bury the control topic, and the mapper would balloon in RAM if it materialized one row per message with no structure.
So the app has to do two things at once: cut how many records get written to the control topic, and keep the in-memory picture small enough to query quickly while still answering commits anywhere in the replicated history.
Data model: ranges
K2K does not only store (source, target) pairs.
It store ranges:
- source: range start on the source partition,
- target: range start on the target partition,
- length: how many consecutive source offsets step in lockstep with consecutive target offsets.
Producer side: merge within a batch
When a produce batch flushes to the control topic, consecutive 1:1 pairs for the same mapping key (same source partition, same target partition) get merged: if both sides advance by one each time, we bump "length" instead of appending another row. That is the cheap win for control-topic volume.
Consumer side: interval tree
The service loads ranges into an interval tree (backed by an ordered map). Lookups are O(log n) in the number of intervals. Adjacent intervals that still describe one continuous translation get compacted again so the tree does not sprawl.
6. Conclusion
We built K2K to address many of the challenges widely recognised by the community (and by Lenses customers) when replicating data across Kafka clusters. K2K scales easily, avoids Kafka Connect, remains vendor-agnostic, and provides an enterprise-grade solution that the current landscape still does not offer.
With K2K Offset Mapping now available, K2K can support disaster recovery use cases. By maintaining consistent offset translation across clusters, it enables seamless failover in the event of a failure.
Learn more about K2K by watching this video about Enterprise Disaster Recovery with Andrew and Patrick or use the Community Edition for free.
