Chapter 9: Evaluation techniques
How do you test usability, functionality, acceptability?
Lab or field (controlled or in practice)
Analytic review model
	Experimental
	Observational
	Query
	Choose suitable

9.1 what is evaluation?
	Does it actually work as expected?
	Should be done through the development process
	Think of prototypes
	Who? 
		Designers
		Actual users
9.2 goals
	Assess functionality (requirements)
	Assess user experience
	Identify problems
9.3 evaluation by expert analysts
	Evaluate design early
	Hard to do user testing when system is incomplete

	Cognitive walkthrough
		Low budget version: explain it to your dog
		How easy is it for the system to learn
		Check usability
		Users often prefer learning by exploration
		Need:
			Specification
			Description of task
			List actions to complete task
			User background and experience
		Ask:
			Is effect of an operation the same as the user’s goal?
			Does user see what action is available?
			Does user see that it is desired action?
			Will user understand feedback?
		Document, to aid in tracking and changing

		Example: VCR

Heuristic evaluation
	Several evaluators independently look over (75%)
	Principles and guidelines (7)
	Rate for each principle (0..4) scale
		Visibility of system status
		Match between system and real world
		User control and freedom (emergency exit or backup)
		Consistency and standards
		Error prevention
		Recognition not recall
		Flexibility and efficiency
			Allow tailoring
			Accelerators for expert (short cut)
		Aesthetics and minimalist design
		Help users recognize, diagnose, recover errors
		Help and documentation
	Compare ratings of evaluators

Model based evaluation
	(see chapters 6, 12, 16)

Using previous studies
	General knowledge from research
		Menu types
		Recall commands
		Choice of icons
		Consider both similarities and differences 
with research situation

9.4 	evaluation through user participation
	Tends to be later in process
	Laboratory (ideal) or realistic (natural)
(Research on situated cognition suggests some benefit 
in having users tested in their future environment)
laboratory
	sophisticated observations
	interruption free
	can compare alternative designs
	BUT
	Lack of context
Field
	Distractions
	May see interactions not obvious in lab
	More realistic
	Harder to do detailed observations
	But still being watched
	Experimental evaluation
		Often time consuming and expensive
		Do you get enough value to justify costs?
		Test hypothesis
			Null hypothesis
			Alternative hypothesis
		Participants
			Match end users
			Sample size
				(1 to 5 for observational)
				(more for statistical tests)
				The average of a small group of values is 
					Almost always a NORMAL distribution
					95% of values are within 2 standard deviations of the mean
			Variables
				Independent variable
					What you control/change
				Dependent variable
					What you want
		Approaches:
Variation between subjects
	(randomly assign to different groups)
Repeated measures
	Each subject goes through each treatment
	(randomize order in which you see items)
Need more subjects to do variation between subjects
	Because each subject only sees one treatment

Consult a statistician
	Don’t just use EXCEL
Save raw data
Check outliers
Find standard deviation
Compute sample size

Do you have discrete vs continuous variables
	(treatment A or B) discrete
	Height,weight, average of score continuous
Depending on distribution, and what you seek:
	Anova, ancova, T-test, ranking, 
correlation/regression, goodness of fit test
studies of groups of users harder to evaluate!

Observational Techniques
	Think aloud
	Cooperative evaluation
		Protocol analysis
			Pencil, audio, video, logging(keystrokes), user notebooks
			Analysis can be time consuming
			Can have experimenter punch buttons live to aide analysis
		Post task walkthroughs
		Query
			May be subjective
			Hard to compare alternatives
		   Interview
			Prepare questions
		   Questionnaire
			Less flexibile
			Can miss what user knows
			Need to know what you want
			General
			Open ended
			Scalar (Likert)
			Multiple choice
			Ranked
			(closed questions easier on user)
			Do a pilot study with a few
			Distribution
				Return rate low
				Return bias
		Physiological response
			Eye movements
				Number of fixations
				Duration of fixations
				What is looked at
				Scan path
			Physiological tracking (polygraph)
				Heart rate
				Breathing
				Skin secretions
				EEG
				Stress, anger, involvement, decision making, attention, motivation

9.5 Choosing evaluation method
	Stage at which evaluation is carried out
		Design
			Quick and analytic, biggest payoff
		Implementation
			Comprehensive,users
	Style
		Lab vs field
	Subjective/objective
		Evaluator bias
		User subjectivity
		Repeatable?
	Measures
		Qualitative
			More detail but harder to analyze
		Quantitative
			statistics
	Information
		Low level-one aspect
		Highlevel-overview
	Immediacy
		Immediate (less memory; more chance of interference)
		Recall bias and forgetting
	Level of interference
		Intrusiveness
		Obvious methods subjects influenced
	Resources
		Equipment, time, money, users, experts, context
		May need simulations (space station)

What evaluation is done varies.
Evaluation, when done appropriately, can help produce a much better system.