On the importance of load testing Kafka

Load testing your Apache Kafka cluster and consumer applications.

Evan Rosenstein
Oct 05, 2020
On the importance of load testing Kafka

Socrates preached, “To know thyself is the beginning of wisdom.”

This ancient Greek anecdote applies to your modern Apache Kafka project: developers, go forth and load test your real-time application to understand the capacity and limitations of your project before deployment.

Failure to do so will cost you time and money (e.g. Robinhood’s outage on a historic trading day). Load testing your real-time applications has three main objectives:

  • Determine the system’s maximum capacity

  • Determine the breaking points in your system

  • Observe how the system will recover without manual intervention 

You can test these measures in three different ways. A stress puts the system under a heavy load to observe its reaction; a spike or a surge tests the system’s reaction to a sudden spike in traffic, which is relevant for any business that faces seasonal traffic differences; finally, a soak checks the endurance of your stream.

Load tests focus on the brokers and consumers, not producers. You want to monitor how the broker handles saturation and then computes resources; on the other end, you monitor how the consumer reads the inputs and computes the resources.

We are very lucky to have friends like David Esposito, a Solutions Architect at Aiven, who took the time to present on load testing at our Lenses.io Kafka Office Hours. A lover of all things open-source, David explains in depth how to set up load testing for Apache Kafka and uses real-world examples showcasing the pitfalls of ignoring this important process.

Check out the recording of his presentation on load testing with Apache Kafka:

And download our pre-production Box to practice some load testing of your own.

Ready to get started with Lenses?

Try now for free