Comoyo provides infrastructure for moving pictures, both movies and TV, as well as live television. On March 23, 2012, the Norwegian national soccer league started up, and we were in no way certain that we would be able to handle the load we expected all the football fans in Norway to generate. The movie delivery subsystem is mostly, but not entirely, consisting of components hosted in Amazon's Elastic Compute Cloud (EC2), and mostly (but not entirely) written in Java. In this talk we describe how we used multiple tools to generate what we believed to be realistic loads against our systems, and how we used this to tune the system to actually perform.
This may seem like a simple task, but it wasn't:
The system is made of components, but the component developers were not aware of the full system architecture.
Some components thought to "just work" (such as load balancers) were discovered to be a bottleneck. Initially, we didn't even think about load balancers, since they were effectively invisible for the developers.
Writing test scripts for load testing is a skill separate from writing other types of tests, and the test tools themselves are much less well standardized or developed and much easier to use for some tasks than others.
We did all of this, we learned how to harness the tools, we delivered the Tippeliga, and we did it all in just a few weeks. In this talk we share some of the hard won lessons on how to write tests, interpret their results, and how to actually tune a system for high performance.