Guidelines for user testing with thinking aloud
Modified from CSE 510 Spring 2005.
Practical study design
- Reflect on the participants' backgrounds and how they might affect the
study
- Be aware of problems that arise when experimenters know the users
personally
- Prepare for the study carefully (avoid last minute panic)
- Select the tasks carefully to be representative and to fit the allotted
time
- In general, start with an easier (but not frivolous) task
- Write down features of the system that are not being tested as
well as those that are!
- Define the start-up state for the study precisely
- Define precise rules for when and how users can be helped during the
study
- Plan the timing and cut-off procedure (if subject gets stuck) for each
part of the study
- Include reasonable provisions for data collection (e.g., notes,
tape or video recorder, keystroke capture where appropriate)
- Plan data analysis techniques in advance
- Carry out a pilot study (important but often overlooked)
Carrying out the study
- Let users know that complete anonymity will be preserved
- Let them know that they may quit at any time
- Stress that the system is being tested, not the participant
- Indicate that you are only interested in their thoughts relevant to the
system
- Demonstrate the thinking-aloud method by acting it out for a simple
task, such as figuring out how to load a stapler, and a computer-related
task
- Hand out instructions for each part of the study individually, not all
at once
- Maintain a relaxed environment free of interruptions
- Encourage users to keep talking using unobtrusive
comments that don't point the user in a particular direction, e.g.
"uh huh", or
"weird?" (after the user says "This is really weird!"),
or
"What do you think is going on?"
- Debrief each user after the experiment