Guidelines for user testing with thinking aloud

Practical study design

Reflect on the participants' backgrounds and how they might affect the study
Be aware of problems that arise when experimenters know the users personally
Prepare for the study carefully (avoid last minute panic)
Select the tasks carefully to be representative and to fit the allotted time
In general, start with an easier (but not frivolous) task
Write down features of the system that are not being tested as well as those that are!
Define the start-up state for the study precisely
Define precise rules for when and how users can be helped during the study
Plan the timing and cut-off procedure (if subject gets stuck) for each part of the study
Include reasonable provisions for data collection (e.g., notes, tape or video recorder, keystroke capture where appropriate)
Plan data analysis techniques in advance
Carry out a pilot study (important but often overlooked)

Let users know that complete anonymity will be preserved
Let them know that they may quit at any time
Stress that the system is being tested, not the participant
Indicate that you are only interested in their thoughts relevant to the system
Demonstrate the thinking-aloud method by acting it out for a simple task, such as figuring out how to load a stapler, and a computer-related task
Hand out instructions for each part of the study individually, not all at once
Maintain a relaxed environment free of interruptions
Encourage users to keep talking using unobtrusive comments that don't point the user in a particular direction, e.g. "uh huh", or "weird?" (after the user says "This is really weird!"), or "What do you think is going on?" (Remember Judy Ramey's lecture.)
Debrief each user after the experiment

The pilot study should "debug" the study. This minimizes changes during the study, allowing quantitative data analysis. But improvements may be warranted.
Experimenters' role can be improved
Tasks given to participant can be improved
Written materials can be improved