A Usability Study of Adobe® Photoshop®

Malia Ansberry
Steve Sjoquist
Kris Solem

3/17/98

Abstract

This paper describes a usability study which the authors performed on Adobe Photoshop, version 4.0.1, running on Microsoft Windows 32-bit platforms. The study consisted of two types of evaluations, a heuristic evaluation based on industry standard and platform usability rules, and a usability test involving four test subjects who were unfamiliar with the Photoshop application. Goals for the study were to discover usability problems with the application, and to compare and contrast the results obtained by the heuristic evaluation with those obtained by the usability test. Results obtained showed that Photoshop contains a variety of usability problems for casual users and that Photoshop violates several platform usability standards which affects the user's ability to accomplish tasks with the product. In addition, a comparison of the two evaluation methods used in the study showed that many of the observations discovered in the heuristic evaluation were also usability problems for the subjects in the usability test. Therefore, we concluded that a heuristic evaluation is a lower cost, simpler means of finding severe usability problems than the more resource-intensive usability test involving novice users.

Table of Contents

Introduction

In this paper we describe a usability study of Adobe Photoshop version 4.0.1 on Microsoft Windows 32-bit platforms. In this study we used two different evaluation methods: a heuristic evaluation performed by the authors and a usability test using participants who were unfamiliar with the Photoshop application.

Photoshop is a software application for processing images, typically digitized photographs or original artwork created on a computer. Included in Photoshop are many features for creating artwork, modifying images, and preparing images for electronic or print publication. It is widely used by graphics designers, web page developers, and photographers to prepare professional quality images for use in many domains.

The main goal of our study was to gather usability observations about Photoshop in the context of non-expert users. Our target user was one who was familiar with using the Windows operating system and applications, and who intended to use Photoshop as a casual user to occasionally process pictures or artwork for personal and/or professional use. Another important goal of our study was to compare and contrast the usability results from the heuristic evaluation with those of the usability test.

In the sections which follow, we first describe the methods used to carry out the heuristic evaluation and the usability test of Photoshop, we then present and discuss the results obtained, and finally we draw conclusions based on the results, paying particular attention to comparing the heuristic evaluation results with those of the usability test.

Method

Our evaluation of Photoshop was executed in two phases. In the first phase, each author performed a heuristic evaluation. In the second phase, we conducted a usability test using four subjects. The results of the heuristic evaluation were used in combination with interviews of current users and expected common usage to develop the scenarios for the usability test.

Heuristic Evaluation

Design

In this phase, we looked at how well Photoshop adheres to well known guidelines. We used two sets of guidelines:

Materials

To perform the heuristic evaluation, we used the following items:

Measures

For each of the usability principles we created a list of the ways in which the application violated the principle, as well as the ways in which the application adhered to the principle.

Procedure

To perform the heuristic evaluation, we each executed several tutorials which came with the Photoshop product. Several of the authors also performed systematic reviews of the dialog boxes associated with each of the tools and menu items. During the evaluation each author made a list of the usability problems they encountered, along with the ways in which Photoshop performed particularly well against the guidelines. The team then met to discuss and compile their findings. The details of our findings can be found in the results section.

Usability Test

Design

In this phase, we created a set of tasks and recruited five subjects to perform these tasks in a usability lab. The tasks were grouped into three scenarios. These scenarios were specifically designed to test whether the usability problems discovered during the heuristic evaluation presented problems for the subjects when performing tasks commonly tried by the casual user of Photoshop.

The first scenario involves opening a file based on a description of its contents, and then selecting and modifying the colors of several objects. These tasks were designed to test the following:

The second scenario involves opening an image file, and using a variety of painting tools to colors in portions of the image. These tasks were designed to test the following:

The third scenario involves opening a photograph and perform some simple retouching tasks. These tasks were designed to test the following:

A copy of the complete task list used in the study can be found in the appendix.

Subjects

For this evaluation, we recruited five coworkers who fit the following profile:

Materials

The tests were recorded in a usability lab. The usability test lab consists of two rooms. One room is the test room and has a PC which runs the application and allows the test subject to perform the task list in an isolated manner. There is a video camera and microphone in this room which feeds video and audio signals into the control room. The second room is the control room, and it is next door to the test room. The control room contains a video monitor which can show both the camera video and the PC screen, so that the observers can see any aspect of tests underway. The control room also contains an 8 mm video tape recorder to capture the video and audio from the test room. The test helper can talk to the test subject using a microphone. The evaluators use this room to observe the test as it is carried out by the test subject.

The following materials were used during each test:

Measures

During this evaluation we were made a list of usability observations for each test subject. We looked for general problems as well as specific failures to adhere to the guidelines. In addition, we noted instances in which the adherence to the guidelines made it easy for the test subjects to figure out how to complete the tasks.

Procedure

First, we created a set of tasks that represent common uses of Photoshop for our target user. To create the task lists, we began by talking to both expert and casual users of Photoshop to find out what kinds of tasks they typically perform. We used this information to choose the three tutorials which most closely matched the types of uses we expected our target audience would want to perform. Using these three tutorials as a starting point, we created three scenarios by removing the steps that did not reflect common usage, and by modifying the remaining steps to specifically target problems found during the heuristic evaluation.

After creating the task list, we conducted a pilot test to determine how well the task list met our goals. A few modifications were made, and the final version was used by the remaining four subjects. The results of the pilot test were not used in the final evaluation.

Before each test, the test room was returned to the same initial state. The initial state for each subject consisted of a PC running Microsoft Windows 95 that was logged on and functioning. Photoshop was installed, but not running. The image files were available in the directory specified in the task list. The Photoshop tools and palettes were reset to their default condition. Any files created by previous subjects were deleted.

Each subject was given one hour to perform the three scenarios. Prior to the test, one of the team members asked a few questions about the subject's background, and read an introduction script that explained the purpose of the study and the subject's role in it. In the introduction, the subject was also asked to talk out loud during the test to describe what they were doing and why. After the introduction, the subject was given an opportunity to ask any questions they had with regard to the test, and then the evaluator left the testing room and the user began the scenarios. A copy of the pre-test questionnaire and introduction script are included the appendix.

During the test, one evaluator monitored the progress of the subject and provided help or assistance if necessary. The rules for helping the subjects are included in the appendix. A second evaluator took notes on what the subject did and said, and noted the tape counter at points where the subject did or said something of interest. The third evaluator jotted down questions to ask during the debriefing. These questions were designed to get the subject to elaborate on what they were thinking at various points during the test, and to get an overall impression of the application.

After the test, an evaluator asked the subject a few questions based on actions the subject took during the test. The subject was given their choice of chocolates as a "thanks" for their participation.

To ensure consistency between tests, each evaluator carried out the same tasks for each of the four tests.

Results

Heuristic Evaluation Based on Usability Principles

For each of the ten usability principles described by Jakob Nielsen in Usability Inspection Methods[1], we list both the ways in which Photoshop fails the heuristic, and the ways in which it addresses the heuristic well. A description of the usability principles can be found in the appendix.

  1. Visibility of system status

    Problems

    Kudos

  2. Match between system and the real world

    Problems

    Kudos

  3. User control and freedom

    Problems

    Kudos

  4. Consistency and standards

    This heuristic is addressed in more detail by the next section, which describes a heuristic evaluation based on the Microsoft User Interface Guidelines.

    Problems

    Kudos

  5. Error prevention

    Problems

  6. Recognition rather than recall

    Problems

    Kudos

  7. Flexibility and efficiency of use

    Kudos

  8. Aesthetic and minimalist design

    Problems

    Kudos

  9. Help users recognize, diagnose, and recover from errors

    Problems

    "Could not complete request because of program error."

    Kudos

  10. Help and documentation

    Problems

    Kudos

Heuristic Evaluation Based on Microsoft User Interface Guidelines

Microsoft has published a set of user interface guidelines for applications that run on their 32-bit Windows operating systems. The principles cover every aspect of the user-interface, from the exact behavior that is expected when certain keys are pressed, to the more general issues involved in window appearance and behavior. Here we list the ways that we found Photoshop to comply with and to violate the guidelines set forth by Microsoft. As with the guidelines, the results are separated into three sections: General guidelines, Mouse interactions and Keyboard interactions.

Mouse Interactions

These guidelines cover the basic mouse interface, including selection and direct manipulation.

Action on Target Expected Action Photoshop result
Press on unselected object Clears any active selection and selects object Usually (not always) the selection is cleared. Another selection is not made.
Press on selected object None Sometimes unselects the object
Press on white space (background) Clears any active selection and initiates a marquee (region) selection Depends on tool mode. Correct in marquee mode, not in other tools.
Click on unselected object Clears any active selection and selects object. Display popup for button2. Usually (not always) the selection is cleared. Another selection is not made.
Click on selected object None Selection is changed or cleared.
Click on white space (background) Clears selection, display popup for button2. Depends on tool mode. Correct in marquee mode, not in other tools.
Drag unselected object Clears any active selection, selects the object and initiates drag. Usually (not always) the selection is cleared. Another selection is not made. No drag initiated.
Drag selected object Carries out transfer operation to destination. Depends on tools mode. In selection mode, the location of selection is changed, not the location of the object.
Drag on whitespace Clears any active selection, selects everything logically included from anchor to drag end. Depends on tools mode. In most (not all) selection modes it works correctly.
Double click on unselected object Selects the object and carries out default operation Depends on mode. Correct for brushes, not correct for some tools.
Double click on selected object Carries out default operation Usually no effect
Double click on whitespace Carries out default operation No effect
SHIFT + press or click on unselected object Extend the current selection Correct, when in selection modes
SHIFT + press or click on selected object Clear active selection, extend current selection Correct, when in selection modes
SHIFT + press or click on whitespace Clear active selection, extend selection from anchor point Correct, when in selection modes
SHIFT + double click on unselected object Extends selection state to click location No effect
SHIFT + double click on selected object Extends selection state to click location No effect
SHIFT + double click on white space Logically extends selection from anchor point No effect
CNTRL + press Selects the object Cntrl is ignored, behaves as press on image. On tools, it acts as delete.
CNTRL + click on unselected object Selects the object Cntrl is ignored, behaves as press.
CNTRL + drag on unselected object Selects the object and initiates drag Moves entire image
CNTRL + drag on selected object Copies entire selection to button up location Moves selection to new layer
CNTRL + drag on whitespace Toggles selection state of region selection Moves selection

Keyboard Interactions

These guidelines cover the basic keyboard interface, including recommended shortcut keys. Since these specific guidelines are relatively simple to determine if an application is in compliance, only violations will be stated here.

Key Expected Action Photoshop result
F1 Display contextual help window No effect in most cases. Some dialogs do respond to this.
SHIFT - F1 Activate context sensitive help No effect
SHIFT - F10 Display pop-up menu No effect
ALT-Space Display pop-up menu No effect
ALT-F6 Switch to next window within the application Hide/Show color palette

General User Interface

These guidelines cover general issues involved in window appearance and behavior. For more information about each category, see the Microsoft User Interface Guidelines Summary in the appendix.

  1. General Design

    Problems

    Kudos

  2. Input and Interaction

    Problems

    Kudos

  3. Windows (display, appearance, behavior)

    Problems

    Kudos

  4. Control Usage

    Problems

    Kudos

  5. Integration

    Problems

    Kudos

  6. User Assistance

    Problems

    Kudos

  7. Visual design

    Problems

  8. Accessibility

    As some of these guidelines concern development process and a program's adaptability to specific hardware, several of these guidelines could not be determined solely through a heuristic evaluation.

    Kudos

  9. International Users

    These guidelines could not be evaluated without access to at least one international version of the software.

  10. Networked Users

    Problems

Usability Test

Results of the usability test are organized by the tasks each scenario was designed to test. We have repeated these tasks from the Methods section here. After each task is a list of the observations we made for that task. The majority of our findings in the usability test had previously been identified in the heuristic evaluation. Those findings that are unique to the usability test are noted in italics.

User Background Responses

Subject Gender Age Windows
Experience
Macintosh
Experience
PhotoProcessing Experience "Paint"
Experience
1 M 51+ moderate skill none none minimal
2 M 31-35 highly skilled highly skilled highly skilled moderate
3 F 31-35 moderate skill none none minimal
4 F 41-45 moderate skill moderate skill moderate skill moderate

Scenario 1 Results

Additional usability observations for this scenario:

Scenario 2 Results

Additional usability observations for this scenario:

Scenario 3 Results

Note: only two of the four subjects made it to this scenario due to time constraints. The two subjects who began scenario 3 only had time to complete the first tasks.

Additional usability observations for this scenario:

Discussion

We performed a heuristic evaluation of Photoshop using two sets of guidelines: Nielson's general usability principles and Microsoft's guidelines for user interfaces of applications that run on 32-bit Windows platforms. The two sets of guidelines differ in their approach to user interface design because the authors have made different assumptions about the software written to the guidelines. However, they both stress that the importance of the guidelines is in creating software that is as easy to learn and use as possible. As we showed in our results, Photoshop violated almost every guideline that we evaluated it against. Some of these violations caused users to be confused and frustrated during the usability test, while others seemed to truly have no effect on usability in our tests. We also found the opposite to be true. For those areas where Photoshop was in accord with sections of the guidelines, users were sometimes able to make assumptions about behavior of the application and thus accomplish significant tasks.

One major area in which Photoshop violates Neilson's principles is in its implementation of on-line help. During the heuristic evaluation, this was noted as a major area of concern. Its impact on usability was clearly illustrated during the subsequent usability test. In most cases during the usability test, when the subjects attempted to find help to accomplish a particular task, they could not find anything appropriate or could not understand the instructions given. One good example of this was when subject 2 attempted to find out how to use the crop tool. Even though he found the correct topic in on-line help, the text was not sufficient to help him find the crop tool hidden under the marquee selection tool in the tools palette. Since finding this tool is especially difficult, the application should have given a complete description for finding this tool.

In addition to the on-line help problems, one function completely missing from Photoshop is context sensitive help. The Microsoft interface guidelines recommend that help be available for every dialog and that a popup window which contains specific help for each item in the user interface appears by selecting the '?' icon in the title bar. This is known as context sensitive or What's This? help. In Photoshop, this function does not exist in any of its many dialogs and windows. Each of our test subjects noticed this missing feature and could have used it many times while trying to figure out how to use the various dialogs and windows. This feature is especially useful for an application like Photoshop which has a large number of dialogs and windows which contain controls and terminology that may be unfamiliar to new users.

During the heuristic evaluation we found that another area in which Photoshop falls short of Neilson's guidelines is providing feedback and preventing errors. We expected that because Photoshop does not always give the user information as to what it is doing, or what the user should do next, the user would often end up making errors or getting stuck in a mode with no obvious way out. This turned out to be an accurate prediction. During the usability tests we found that several test subjects could not figure out how to unselect a region. Several of the test subjects eventually found a Select > None menu option and used this to undo their entire selection and start over. One subject accidentally inverted their selection, and couldn't figure out what was going on, so she reloaded the image and started from scratch.

It is interesting to note that Photoshop deviated significantly from the Microsoft guidelines for mouse interactions, which meant that to correctly select and manipulate parts of an image users had to learn a new mode of selecting and unselecting objects. It was perhaps the combination of violating both Neilson's and Microsoft's guidelines in this area that resulted in so many problems with selections during the usability tests. In contrast, however, there was one specific type of selection where Photoshop adhered to the Microsoft guidelines for selections, and that is in the area of adding two selections together by pressing the SHIFT key. It is this function that the test subjects were able to use to eventually correctly select complex objects.

Another way in which Photoshop's adherence to Neilson's guidelines really helped some users was in the area of matching the system to the real world. Those users that had some photographic processing experience were excited to see the dodge and burn tools and commented that they understood what those tools were to do because of their experience with the physical tools.

During the usability test, the most severely missed feature of the application was that of multi-level undo. All of our test subjects made mistakes in modifying images which could not be completely corrected with the simple single undo function that Photoshop provides. This was especially true when the subjects were using the painting tools to add new colors to an image. In this situation, each separate paint operation (mouse click-drag-unclick) is a single un-doable sequence. With this system it is very easy to paint too much and then want to undo, only to find that very little of it can be undone. This problem was severe enough to cause one of the subjects to completely discard his work and reload the original image from disk in order to start over.

During the usability tests, another major usability issue for Adobe Photoshop was the difficulty the subjects had with using non-orthogonal tools. We noticed this most prominently with the selection and zoom tools. For these tools it was rather straight forward for the subject to perform the action in one way but not the other. For example, all four of our subjects found it relatively easy to select a region of an image, but several subjects had difficulty clearing their selections. Similarly, most of the subjects found the Zoom In tool easy to operate, but could not find the Zoom Out tool. Only one subject discovered that pressing the Alt key changed the Zoom In tool to the Zoom Out tool. The other subjects reverted to using the Zoom Out item on the View menu.

As noted in the heuristic evaluation, Photoshop violated a significant number of Microsoft guidelines with respect to the visual design of the user interface. Most of the controls were not "correct" in appearance, layout, and relative size. In many cases this did not seem to affect user's interpretation of the interface. For instance, Photoshop does not use the correct (as defined by Microsoft) control for the slider. However, the test subjects had no problem with the slider that Photoshop provided, and when asked about it during the debriefing session, they all indicated that it looked like a slider and behaved like a slider, even though they knew it was not exactly like sliders in other Windows applications.

In contrast, there were definitely areas where Photoshop's use of non-standard controls significantly impacted users' ability to perform a task. For instance, on the tool palettes, there are multiple tab dialogs, and a small triangle in the top right corner of the palette that provided access to a hidden menu of options relevant to the palette. A few of the test subjects found the hidden menu by pressing on the small triangle. When asked why they clicked on the triangle, the test subjects said that they thought the small triangle was a control that would cycle through the tab dialogs because it appears very similar to such a control normally found in Windows applications. The one user who accurately predicted that this control might be hiding a menu explained that the control has that behavior on the Apple Macintosh platform, so he thought he would try it. Although this platform inconsistency may not cause severe errors, it did cause unexpected behavior and required the test subjects to learn new conventions for familiar aspects of a user interface.

In addition, we found in both the heuristic evaluation and the usability tests that the tools palette had several non-standard features which caused usability problems for our subjects. The most important issue here is the fact that a user can expand several of the tool icons into a multi-selection menu by clicking and dragging on one of these tools in the palette. This is such a non-standard practice that none of the subjects even thought to try such a procedure. In addition, one subject couldn't even understand this concept after reading about how to do it in the on-line help. There were many such non-standard practices which led to the interface being difficult or confusing to understand and use.

The use of non-standard controls, even when they provided additional useful functionality, did not appear to help usability of the product. For instance, it was noted during the heuristic evaluation that Photoshop had extended the common Windows control for opening a file to provide a preview of the currently selected file. During the heuristic evaluation, this was noted as a positive way for Photoshop to extend outside of the platform guidelines to provide valuable functionality and was not technically a violation of Microsoft's guidelines, as the guidelines allow for extending the behavior of system controls. However, during the usability test, none of the subjects took advantage of this functionality. They were all so accustomed to opening a file by double-clicking on it that they failed to notice that they could single-click to see a preview of each file and then choose to only open the file they were actually looking for.

Conclusions

We conducted a heuristic evaluation and a usability test of Adobe Photoshop to determine how these different means of evaluation would uncover usability issues with the application. We found that the heuristic evaluation method of comparing the application's behavior to established guidelines for user interfaces in general and the Window's platform in particular discovered a significant number of usability problems that were also found during the usability test. In fact, the usability test only lead to three usability issues that had not been previously identified during the the heuristic evaluation.

We also found that although a heuristic evaluation is a valuable means of identifying potential problems, it can also uncover a significant number of "violations" that appear to have no effect on the usability of the product during a usability test. It seems that although there may be good reasons for abiding by the platform guidelines, those reasons may not be solely usability issues or, alternatively, they may not be usability issues for novice users of an application.

In addition, a heuristic evaluation does not provide the sense of how severely a violation of the guidelines will affect usability. Our heuristic evaluation results showed that both the failure to provide a multi-level undo and the use of non-standard system controls were problems. However, they did not indicate that the lack of multi-level undo created a uniformly frustrating experience for the users while the non-standard controls created usability problems that were sometimes not as frustrating and occasionally did not appear to matter at all.

Finally, it seems clear that Photoshop is a very complicated and powerful application. As such, it could benefit greatly from a greater adherence to platform and design standards in order to assist the user in their trek up its very steep learning curve.

Acknowledgments

We would like to thank our company for the use of the usability lab. Thank you also to the usability engineers who worked with us (you know who you are!), for their training and assistance in this study.

References

  1. Usability Inspection Methods. Nielsen, J., and Mack, R.L. (Eds.), John Wiley & Sons, New York, NY, 1994.
  2. The Windows Interface Guidelines for Software Design published by Microsoft Press, 1995.
  3. HANDBOOK OF USABILITY TESTING: How to Plan, Design, and Conduct Effective Tests, Jeffrey Rubin, John Wiley & Sons, Inc., New York, NY,1994.
  4. Readings in Human-Computer Interaction: Toward the Year 2000, Second Edition. Baecker, Ronald M., et. al. (Eds.), Morgan Kaufman Publishers, Inc., San Francisco, CA, 1995.

Appendix

This appendix contains copies of the following documents:


List of Usability Principles

This list was copied from http:/ /www.useit.com/papers/heuristic/heuristic_list.html.

Visibility of system status
The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
Match between system and the real world
The system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
User control and freedom
Users often choose system functions by mistake and will need a clearly marked "emergency exit" to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
Consistency and standards
Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
Error prevention
Even better than good error messages is a careful design which prevents a problem from occurring in the first place.
Recognition rather than recall
Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
Flexibility and efficiency of use
Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
Aesthetic and minimalist design
Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
Help users recognize, diagnose, and recover from errors
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
Help and documentation
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

Summary of Microsoft User Interface Guidelines

The following checklist summarizes the guidelines covered in The Windows Interface Guidelines for Software Design. It is based on appendix C of the Microsoft Developer Network Library Edition, January 1998.

  1. General Design
  2. Design Process
  3. Input and Interaction
  4. Windows
  5. Control Usage
  6. Integration
  7. User Assistance
  8. Visual Design
  9. Sound
  10. Accessibility
  11. International Users
  12. Network Users

Rules for Helping the User During Usability Test

This documnent describes rules for the type of help that we will give users during the execution of a usability test. Here we describe two aspects of help: 1) the type of help that we will give, and 2) the types of problems that we anticipate the user can encounter during the test. The second section also shows which type of help that we will give for each type of anticipated problem.

Type of Help to Be Given

There are two types of responses that we will make to the user during the usability test: direct help and indirect help.

Direct Help

Direct help is for those problems or user questions which arise that are outside of the scope of the usability test. In most cases this means that the topic of the help is not related to anything we are testing. In this case, we will answer the question completing and help the user in whatever way is necessary to help the user continue the test

Indirect Help

We will give indirect help to the users whenever the problem or question relates an aspect of what we are trying to evaluate. For this case we will use a three step approach as follows:

  1. The first time the user requires help during the test, we will ask the user to tell us how they think the problem can be solved and to try any ideas they have for answering their question or performing the task. For example, the user might ask: "How do I paint on an image?" We would respond with something like "How do you think that you can discover how the application intends for you to do that? Please talk about any ideas you have and try out any that seem likely".
  2. If the user does not solve the issue within a few minutes, then we will give some kind of hint to point them in the right direction without actually telling them the complete answer. For example, the user says "OK, I'm really stuck now. I just can't figure out to paint something onto my image." We might respond, "Why don't you try reading the helpful information at the top of the task list and then looking at each item in the palette of tools on the left side of the screen".
  3. If, after a few more minutes, the user still doesn't make progress on the same issue, we will tell them the complete answer to allow them to move on in the test.

Types of Problems Anticipated

The following table describes the type of problems we anticipate could happen while the user is performing the test and which type of help we will provide for an occurrence.

Type of Problem Type of Help to Give
Failure of operating system or PC Direct
Questions about the operating system or PC Direct
Questions about the application Indirect if related to tasks to perform, otherwise direct
Problems with application - unexpected behavior Direct (application bug)
Questions about the semantics of a task in the task list Direct
Questions about how to perform a task in the task list Indirect
Stuck on a task - can't figure out what to do Indirect
Questions about documentation or on-line help Indirect


Usability Test Checklist for Experimenters

Before each participant arrives

When each participant arrives

During the test

After the test


Timing and Cut-off Document

Each user has committed to spending 1 hour for the usability test. We will plan the timing of the study as follows:

We will provide three tasks for each user, with the difficulty of the tasks progressively increasing. If a user gets stuck on an individual item within a task, the testers will encourage them to continue to search for the answer. However, if a user spends more than 10 minutes on any single item, we will prompt them in the direction of the solution.


Introduction Script for Usability Test

Kris, Steve and I are conducting this usability test of Adobe Photoshop as part of a Human-Computer interface class at the University of Washington. The purpose of the study is to investigate how well or poorly certain aspects of the user interface of Photoshop are designed with respect to their usability.

We will give you a list of tasks to complete. Please complete them in order. If you get stuck or if we seem to be running long on some tasks, we may ask you to skip to the next task. We will be in the observation room next door. Although you wont be able to see us, there is an audio feed so you can ask questions if you need to.

Please remember that we are not testing you. Although at times we may ask you to do something that you do not know how to do or may not be able to figure out, that is a reflection on the design of the software, not on you. During the study, we may ask you to search a few more minutes for something that seems hidden or not intuitive.

One of the things that will really help us in this study is if you would be as vocal about what you are doing as possible. I know it may seem very awkward, but since we don't read minds, it will help us a lot to hear you. Please read each step as you encounter it. Then, as you are thinking about how to approach the problem, it's really helpful for us if you would think out loud as you proceed. For instance, if you had a task that said "load the stapler", you might <fill in thoughts here>. Does that make sense?

It's also helpful for us to get an idea of your background with this software. So, we have this questionnaire to fill out. <Do it now>

Thanks so much for helping us with this! Do you have any questions?


Pre-test Questionnaire

1. Gender (circle one) : Male Female
2. Age (circle one): under 21 36-40
21-25 41-45
26-30 46-50
31-35 51 and over
3. Experience with Microsoft Windows:
4. Experience with Macintosh:
5. Photograph processing expertise:
6. Experience with "paint" style programs:

Usability Test Task Lists

Scenario 1 : Selecting Objects

You have found an image that you like, but you don't like some of the colors of the objects and some of them seem a little dim. You will use Photoshop, along with your great eye for color, to fix these minor problems.

Useful Information

Photoshop provides tools to you to help you select areas of your photograph in order to target your actions (such as coloring, moving or copying) to a specific region. In particular, you have tools that will select

Once a region is selected, it can be added or subtracted from the next selected region.

Task list

  1. First you have to find the image that you saved yesterday. You know that it is a picture of a set of children's toys and that you saved it in the c:\adobe_test folder, but you don't remember the name of the file. Find the file and open it in Photoshop.
  2. In this image, there is a set of blocks that have a variety of colors. Since the green "Z" clashes with the rest of the page, you want to change it to a faded shade of red.
  3. Next you see that the wheels on the train are not the right color either. Select the wheels and change their color to blue.
  4. Finally, the color of the word on the train seems a little dim, so you'd like to brighten it. Select the words, including the diamond blocks surrounding each letter (but not the white part of the train), and brighten the colors.
  5. Save the image you've just created under a different name.

You're done!


Scenario 2 : Painting Images

You have been working on a drawing for your web page and now need add the finishing touches with Photoshop.

Useful Information

Photoshop provides tools for you to paint areas of your drawing. In particular, you have tools that will paint using:

Some of the painting tools allow you to select and use a specific paint brush which has several different attributes.

Opacity is a measure of the transparency of the paint. 100% opacity means that you can't see through it at all, while 0% means that the paint is completely transparent.

Task list

  1. Open the image file car_dealer.psd in the c:\adobe_test folder.
  2. First of all you want to finish coloring in the remaining white portions of the person figure in the image.
  3. Now you want to add a few blue lines to the sign that the figure is holding.
  4. Save your finished image.

You're done!


Scenario 3 : Retouching an Image

You have just picked up the photos from your trip to Venice and want to scan in one of your pictures to put on your web page. Unfortunately, the picture quality is not as good as you would like. So, you decide to use the copy of Photoshop that you received for Christmas to touch up your photograph.

Useful Information

Task List

  1. Open the file Retouch1.psd in the c:\adobe_test folder, which contains the photograph that you have scanned in.
  2. Upon opening the file, you discover that the photograph was scanned in crooked and has ragged edges. Crop the ragged edges from the image and straighten it at the same time so that it is upright.
  3. Next you want to remove the small boat near the middle of the picture by painting over it with a copy of the water.
  4. Lastly, you want to sharpen up the details in the picture. You recall from a graphics arts class that you took in college that you can do this by adjusting the image levels of the highlights and shadows.

That's it--you're done!