Types of load tests

I got into a bit of an animated discussion at work last week about Load Testing, and what it means. The other people involved were senior QA and delivery folk. We couldn't agree on a common understanding of the concepts, forget terminology. A search onWikipedia also failed to provide a clear explanation.

Here's what I think about it. To me, there are three concepts, and between them, they cover all aspects of testing a system under load. Everything else is a combination of these. The terms I use for them may not be the same as the terms other people use, or perhaps they use other terms for these concepts, but let's focus on the concepts. (I'm using web based application terminology here, but I guess it should be similar in other situations.)

• Performance or Throughput: You load your server to various number of users, and measure the throughput, or response. Again, normally done in pages served per second, or average page response time.
  
• Scalability: Define the lowest throughput that is acceptable to you. Then load the server (for instance, by increasing the number of users) till the throughput starts going below the defined one. The number of users logged on at this point define the scalability of your system in terms of users.
  
• Reliability: Let the system run under extreme load ( same as above), for a large number of computations. In the webapp world, use a scenario that visits all your pages, and let it run for a long long time: time really meaning pages served in this case. Look at two different measures:
• number of faults occurred. This gives you a reliability measure by the standard definition.
  
• amount of server resources consumed. ( RAM, disk space, DB connections, file handles etc.) This should stabilise after some time, and stay there. If any of these keep growing, there's a resource leak.
  

Here are a couple of (horribly drawn) pictures. But I hope they are allowing me to put my point across:-)





These are the basic concepts. A number of variables vary based on your goals and your application, but the concepts remain the same. Here are some examples of things that will vary:

• number of pages in reliability: for an intranet performance review app for a 200 people company, 50,000 requests would probably be a lot. For an amazon.com type of application, we're probably talking hundreds of millions or billions.
• the server resources you measure.
• the metric you use to measure througput.
• How do you measure 'concurrent users'. For instance, you might have your users wait 1 minute between actions, or not wait at all.
• Your acceptable throughput threshold.
• Number of concurrent users you want to measure throughput for.
• Server configuration
  

When executing these tests, typically you will do multiple iterations, across multiple installations, and do all of these at the same time.

One of the item the others did not agree on was the y-axis in the reliability test. They didn't agree that number of pages served was the right number there. I believe that number of pages is the only term one can use there - the x-axis is upto you. And here's why:

Ultimately, all your measures are relative to the agent driving the application, because the agent is responsible for getting your functions called. In a webapp where everything is done via a web page, this agent is your end user, who only makes page requests. Hence the only true measure of 'life' of the application is the number of pages served.

I find quite a few people talking about load testing, including our customers, without really being clear about what they need. In this particular case, they started saying 'load test'. I put across this definition, ( or rather, had the on-site guy do it) and they came out with: well, we can do those as well, as long as they are useful! I doubt he had any better idea of what he wanted even then.

At one of my previous jobs, at an enterprise product development organisation, we did all three - for reliability, we ran through 500,000 pages, which took us a couple of days. For throughput, we wanted at least 0.1 seconds of page response time. Scalability we just measured in throughput terms.

Does this sound like a reasonable definition of load testing? Am I missing something?