On the importance of load testing Kafka
Load testing your Apache Kafka cluster and consumer applications.
Socrates preached, “To know thyself is the beginning of wisdom.”
This ancient Greek anecdote applies to your modern Apache Kafka project: developers, go forth and load test your real-time application to understand the capacity and limitations of your project before deployment.
Failure to do so will cost you time and money (e.g. Robinhood’s outage on a historic trading day). Load testing your real-time applications has three main objectives:
Determine the system’s maximum capacity
Determine the breaking points in your system
Observe how the system will recover without manual intervention
You can test these measures in three different ways. A stress puts the system under a heavy load to observe its reaction; a spike or a surge tests the system’s reaction to a sudden spike in traffic, which is relevant for any business that faces seasonal traffic differences; finally, a soak checks the endurance of your stream.
Load tests focus on the brokers and consumers, not producers. You want to monitor how the broker handles saturation and then computes resources; on the other end, you monitor how the consumer reads the inputs and computes the resources.
We are very lucky to have friends like David Esposito, a Solutions Architect at Aiven, who took the time to present on load testing at our Lenses.io Kafka Office Hours. A lover of all things open-source, David explains in depth how to set up load testing for Apache Kafka and uses real-world examples showcasing the pitfalls of ignoring this important process.
Check out the recording of his presentation on load testing with Apache Kafka:
And download our pre-production Box to practice some load testing of your own.