On the importance of load testing Kafka

Socrates preached, “To know thyself is the beginning of wisdom.”

This ancient Greek anecdote applies to your modern Apache Kafka project: developers, go forth and load test your real-time application to understand the capacity and limitations of your project before deployment.

Failure to do so will cost you time and money (e.g. Robinhood’s outage on a historic trading day). Load testing your real-time applications has three main objectives:

• Determine the system’s maximum capacity

• Determine the breaking points in your system

• Observe how the system will recover without manual intervention 

You can test these measures in three different ways. A stress puts the system under a heavy load to observe its reaction; a spike or a surge tests the system’s reaction to a sudden spike in traffic, which is relevant for any business that faces seasonal traffic differences; finally, a soak checks the endurance of your stream.

Load tests focus on the brokers and consumers, not producers. You want to monitor how the broker handles saturation and then computes resources; on the other end, you monitor how the consumer reads the inputs and computes the resources.

We are very lucky to have friends like David Esposito, a Solutions Architect at Aiven, who took the time to present on load testing at our Lenses.io Kafka Office Hours. A lover of all things open-source, David explains in depth how to set up load testing for Apache Kafka and uses real-world examples showcasing the pitfalls of ignoring this important process.

Check out the recording of his presentation on load testing with Apache Kafka:

And download our pre-production Box to practice some load testing of your own.