What is Apache Kafka?

Apache Kafka is a distributed messaging system used for stream processing, and is used by thousands of organizations to power real-time applications.

Let's look at the benefits of Apache Kafka, how it differs from other related data technologies, how it works and how it's used.

Why use Apache Kafka?

Kafka helps to modernize business processes by providing the infrastructure to deliver real-time data and timely experiences to consumers - with use cases ranging from reducing mainframe costs and order fulfilment to fraud detection and risk calculation.

Data integration

Data integration

  • Reducing mainframe costs
  • Integrating market data
  • Security analytics pipelines
AI & data products

AI & data products

  • Risk calculation engine
  • Payment fraud detection
  • Churn prediction
Software applications

Software applications

  • Mobile banking
  • Financial roboadvisor chatbot
  • Digital wallet

Apache Kafka vs. RabbitMQ

CharacteristicApache KafkaRabbitMQ
ArchitectureKafka uses a hybrid approach of messaging queue and publish subscribeRabbitMQ uses a messaging queue approach
ScalabilityKafka allows distributed partitions across different servers Increase the number of consumers to the queue to scale out processing across competing consumers
Message retentionPolicy-based, with the retention window configurable by the user - for example messages can be stored for two days.Acknowledgement-based: messages are deleted as they are consumed
Multiple consumersMultiple consumers can subscribe to the same topic, because Kafka allows the same message to be replayed for a given time window.Messages are deleted as they are consumed, so multiple consumers can't receive the same message.
Apache Kafka Architecture

Benefits of Apache Kafka for organizations

Apache Kafka is a powerful streaming technology when harnessed in the right way, but it also presents a steep learning curve and productivity difficulties for engineering teams when they start to drive adoption beyond a POC.

This means developers need well-governed self-service and data access, observability, security and building data pipelines (stream processing).

Learn Apache Kafka through Lenses

Get started