Socrates preached, “To know thyself is the beginning of wisdom.”
This ancient Greek anecdote applies to your modern Apache Kafka project: developers, go forth and load test your real-time application to understand the capacity and limitations of your project before deployment.
Failure to do so will cost you time and money (e.g. Robinhood’s outage on a historic trading day).
Load testing your real-time applications has three main objectives:
- Determine the system’s maximum capacity
- Determine the breaking points in your system
- Observe how the system will recover without manual intervention
You can test these measures in three different ways. A stress puts the system under a heavy load to observe its reaction; a spike or a surge tests the system’s reaction to a sudden spike in traffic, which is relevant for any business that faces seasonal traffic differences; finally, a soak checks the endurance of your stream.
Load tests focus on the brokers and consumers, not producers. You want to monitor how the broker handles saturation and then computes resources; on the other end, you monitor how the consumer reads the inputs and computes the resources.
We are very lucky to have friends like David Esposito, a Solutions Architect at Aiven, who took the time to present on load testing at our Lenses.io Kafka Office Hours. A lover of all things open-source, David explains in depth how to set up load testing for Apache Kafka and uses real-world examples showcasing the pitfalls of ignoring this important process.
Check out the recording of his presentation on load testing with Apache Kafka:
And download our pre-production Box to practice some load testing of your own.
