CSE 403 project: Usability

This assignment is lighter than usual, because of the midterm exam.

Overview

Test the usability of your beta release with a small “hallway testing” user study.

When a team member uses your system, they already knows how it works. You need to learn from people who don’t know it. You will learn where they get stuck, what they misunderstand, and what they like. This will guide your team’s work on the final release.

In this assignment, you will:

  1. Address TA comments from your previous assignment. This likely involves editing your living document and your implementation.
  2. Plan a short user study.
  3. Run it with someone who is not familiar with your project and file issues for the problems you found. Triage those issues, addressing as many as you can.
  4. Repeat the preceding step with a new user and the improved system.
  5. Report on what you learned and how your plans changed.

Participants must not be members of your team nor the course staff, and they should not have seen your project before. Ideally, they match the users described in your requirements.

Instructions

1. Plan your study (20%)

Before your first session, write down:

  • Participants. Describe your two participants in general terms (for example, “a UW student who uses a budgeting app weekly”), and explain why they are reasonable stand-ins for your users. Do not include names.

  • Tasks. Choose 3-5 concrete tasks for participants to complete, based on the use case(s) that are operational in your beta release. Phrase each task as a goal, not as instructions: “Create a study group for CSE 403 and invite a friend”, not “Click the + button, then type a name, then…”.

  • Setup. How participants will access the system: for example, your public URL, a laptop you provide, or installing it themselves by following your README.

  • What you will observe. For each task, how you will judge success?

2. Run two sessions (30%)

Each session should take about 20-30 minutes, including the debrief. It is helpful to have one team member facilitate and another take notes. Sessions may be in person or over Zoom with the participant sharing their screen.

Do not run the two sessions on the same day. Between them, review what you observed in the first session, fix what you can, and revise your plan (for example, reword a task that confused the first participant) before running the second session. If the first participant has problems logging in, the second participant is very likely to experience the same issue, and you won’t learn much new from the second user.

During each session:

  • Get consent. Explain that you are testing the software, not the participant, and that they may stop at any time. Ask before recording audio, video, or the screen. (The assignment does not require nor forbid you to make a recording.)

  • Provide the software. For example, point the user at the homepage of your application on the web. That might link to a web app or to an app download.

  • Do not explain your app any further. You want to simulate the experience of a user who has just learned about your app.

  • Give the tasks one at a time and ask the participant to think aloud: say what they are looking for, what they expect to happen, and what surprises them.

  • Take copious notes on what the participant does and says, not on your interpretation. Direct quotes are valuable.

  • Do not talk and do not help. This is difficult to do. Your role is primarily to observe. If the user gets completely stuck and cannot make progress for a minute, then help them to proceed with the task. If you do help, note where and how. Do not explain or defend your design during the session.

  • Debrief after the study. After the participant completes both tasks, ask them their impression of your system. Let them talk without guiding their narrative. When they are done, ask followup questions so that you have information on at least the following points:

    • What was the most difficult part of using the software?
    • What was confusing to you as a user?
    • What was the best part of the software – what did you particularly like?
    • How should the software be changed or improved?
    • Would you use this software in your day-to-day life? Why or why not?

3. File issues and plan next steps (20%)

  • Open a GitHub issue for each problem identified by the users, with the label user-study. Each issue should address only one problem. It should include evidence, including how to reproduce the problem.

4. Report your findings (30%)

Write a report, as a separate PDF. (You won’t add it to your living document, though you may make other edits to your living document.) The report should contain the following parts.

  1. Study plan.

    • Your original user study plan.
    • If you made changes to it between the two sessions, a brief overview of those changes, and the revised plan that you used for the second user study.
    • What would you differently the next time you run a user study?

  2. Results. For each participant, how each task went (completed without help, completed with help, or did not complete) and notable observations or quotes. Refer to participants as P1 and P2, not with real names.

  3. Issues. The titles of the GitHub issues you opened after each session, grouped by participant. Each title should link to the issue on GitHub.

  4. Issues and fixes.

    • A brief discussion of which issues you fixed after the first session. A justification for the issues you did not fix; for each one, do you plan to fix it eventually, or not?
    • The same for the second session.

  5. Changes to your plan. Briefly discuss how the study changed your plans, if at all. For example, did you re-prioritize features or add a new one suggested by a user? Update your living document to reflect these changes.

What to submit (to Canvas)

  • PDF of your user study report.

  • PDF of your living document.

Your repository is public. Its issues and its commits should not identify a participant (no recordings, names, etc.).

FAQ

Can our participants be other CSE 403 students?

Yes, as long as they have not seen your project. However, participants who fit your target users will give you more useful feedback.

What if a participant cannot install or start the system?

That is a finding, and probably your most important one. Note exactly where they got stuck, help them past it so the session can continue, and file an issue.

What if a participant tries to do something that our system does not support?

Note it. This may indicate a missing feature or a feature that users cannot discover.

Is there anything to commit to the repository for this assignment?

  • Improvements based on TA comments on the previous assignment.
  • New issues added as a result of the user study. (These are not strictly part of the repository; they are only on GitHub.)
  • Improvements that address as many of the new issues as possible.
  • If your living document is in your repository, edits that reflect changes in your plan.