19 Performance Testing

Performance is a slippery concept.
Service oriented
	user's perspective: how well is the service provided?
	Availability
	response time
efficiency oriented
	how well are computer resources handled?
	Throughput
	utilization

peformance targets (threshholds)
dependent on application, which aspect are more important
KPI key performance indicators (decide before design of the system)

response time
	real time
	user time (execution of user code)
	system time (in kernel)
	total =user + system 
users care about real time

< 100 ms: Response time required to feel that system is instantaneous
< 1 s: Response time required for flow of thought to not be interrupted
< 10 s: Response time required for user to stay focused on application 

Web pages that load slower than 400 milliseconds has drop off in visitors (decreasing)

availability: rule of 9s (two nines 99%)
What is the downside if the system is not available?
How much effort is the team willing and able to put into availability? 
How does a focus on availability hamper other aspects of the software? 

Can model the system to estimate
mean time between failures (MTBF) 
mean time to repair (MTTR)

bathtub curve: many failures at first, better as problems are fixed, more failures as systems wear out

baseline test—system with no load
stress test—system with heavy load
stability test—realistic load

Efficiency
	Throughput
		what load slows response time?
	Utilization
		OS has tools
		packet analyzer
General advice for performance measurement:
Use a tool for any non-trivial performance measurement. 
Compare apples to apples; keep variables the same between runs of the same test! 
Be mindful of startup and/or shutdown costs. (e.g. V.M.)
Be mindful of caching. 
Have control over the testing system. 
Have good input data. 
Don’t trust a single test run. 
Keep track of your test runs, and store the data. 
Finally, consider how much performance testing is necessary and that you have to do.