This article compares five key solutions to help platform teams and developers navigate disaster recovery, geo-replication, and data sharing.

TL;DR: Key Takeaways

• Lenses K2K Replicator: Advanced data manipulation, Kubernetes-native, platform-neutral.

• Apache MirrorMaker2 (MM2): Free, open-source, vendor-agnostic for basic replication. Complex to configure & operate.

• Confluent Replicator: Enterprise support, part of Confluent ecosystem, but inherits MM2 complexity.

• Confluent Cluster Linking: Native broker-level, excellent exactly-once, but only works in Confluent Cloud.

• AWS MSK Replicator: Fully managed SaaS, vendor-specific (MSK only), no exactly-once or transformation & replication limited to single AWS account.

The Kafka replication matrix: How existing solutions stack up

Kafka replication is essential for use cases like disaster recovery, geo-replication, and data sharing between teams. However, the diverse and often complex solutions available present a significant challenge for platform teams and developers. This article compares five prominent solutions, each with its own strengths and limitations, to help you make an informed choice. Comparing Kafka Replication Solutions.

Let's look at what's already out there. We developed a simple "matrix" to evaluate replication solutions across nine factors, color-coded from great (orange) to okay (peach) to not-so-great (grey).

Lenses K2K Replicator

Designed to be a universal, platform-neutral solution.

The Good:

• Vendor Agnostic: Works with any Kafka distribution, choose your infrastructure, whether on-premise or in the cloud, helping you avoid vendor lock-in.

• Kubernetes-First Design: Built for native deployment and easy scaling on Kubernetes, rather than relying on Kafka Connect.

• Smart Schema Registry Handling: Replicates data between clusters with different Schema Registries, supporting data formats like Avro and Protobuf.

• Flexible Data Routing: Provides the ability to rename, merge, or split topics during replication.

• Data Filtering and Masking: Allows for the selective shipping of production data for testing and masking of sensitive fields for compliance, a critical feature for data governance.

• Exactly-Once Delivery: Guarantees that replicated data arrives without duplicates, ensuring data integrity and consistency.

• Intuitive Control Plane: Features an intuitive UI with full observability, making management straightforward.

• Self-Serve & GitOps: Enables teams to replicate topics easily through a permissions-based self-serve UI and supports a fully declarative GitOps workflow using YAML.

• Auto-Scaling: Automatically scales up or down based on data volume to optimize compute costs.

Deployment Options:

• Standalone container, to be deployed & managed through your standard CI/CD practices.

• Through Lenses (from version 6.1) onto your Kubernetes directly or via CI/CD.

Lenses K2K considerations:

• New in the market.

• It is offered as a managed package not as a SaaS.

Apache MirrorMaker (MM2)

The open-source standard built into Apache Kafka.

The Good:

• Free and open-source.

• Vendor-agnostic, working with any Kafka flavor.

• Exactly-once delivery support.

MirrorMaker2 considerations:

• Complex to set up and use (unless you're a Kafka Connect expert).

• Not Kubernetes-native (because it’s based on Kafka Connect).

• No commercial support.

• Can be difficult to isolate workloads effectively, leading to resource contention and operational challenges.

• Limitations in workload management and modern deployment patterns.

• Limited transformation and routing capabilities. You'd need to create your own SMTs to try and filter or obfuscate messages and even write custom code to rename topics.

• No data governance, information can be viewed by everyone with platform access.

Confluent Replicator

Confluent's commercial take on MM2.

The Good:

• Provides enterprise support.

• Works with any Kafka flavor.

Confluent Replicator considerations:

• Still based on MM2, inheriting its complexity.

• Lacks exactly-once semantics.

• Expensive (consumption-based pricing).

• Not Kubernetes-native (because it’s based on Kafka Connect).

• Limited on transformations and routing.

• Works only for migration between Confluent Kafka environments.

Confluent Cluster Linking:

Confluent's alternative native approach.

The Good:

• Native replication built into Kafka (not MM2-based).

• Excellent at exactly-once replication.

• Includes consumer offset translation.

• Easy to use.

• Fully managed.

• Good support.

Confluent Cluster Linking considerations:

• Vendor-specific (target must be a Confluent Kafka), leading to potential vendor lock-in.

• No transformation capabilities.

• No filtering capabilities.

• No routing capabilities.

• Consumption-based pricing.

AWS MSK Replicator:

The Good:

• Fully managed SaaS (less operational overhead).

• Provides some support through AWS.

• Offers basic observability and operations, standard with AWS.

• Use through AWS console.

AWS MSK considerations:

• Vendor-specific (target must be AWS MSK).

• Target cluster must be in the same AWS account.

• No exactly-once semantics.

• Zero transformation capabilities.

• Zero routing capabilities.

• Locked to AWS ecosystem.

How to make your choice?

Choosing a Kafka replicator comes down to balancing your organization's needs for functionality, operational simplicity, and vendor flexibility. While some solutions are ideal for basic, in-ecosystem replication, others are built for more complex scenarios.

Lenses K2K Replicator excels in situations that require advanced data transformation, cross-vendor compatibility, and a simplified, self-serve operational mode and easy to use operational mode, that can be set up in less than half hour. This allows teams to handle complex tasks like data filtering and governance with a Kubernetes-native solution, without the operational overhead often associated with other tools. Take a look at Lenses K2K or book a call with us.