Assessment

# Assessment
Strategies for assessing user needs and access

CSE443: Winter 2026; Jennifer Mankoff (Last Edited: 2026-01-21). <a href="/courses/cse443/26wi/slides/assessment.html">Live View: /slides/assessment.html</a>

Important Reminder: check zoom & captioning

---

<div class="slide-outline" style="margin: 1em 0; padding: 1em; background: #f5f5f5; border-radius: 5px;">
<h2 style="margin-top: 0;">
  
  
    Outline
  
  
</h2>
    
  <ul style="list-style-type: none; padding-left: 0;">
  
  
  
  
  
  
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
      
        
        
      
    
      
        
        
          
            
            
              
              
                
                  
                  <li><a href="#8">Assessing Accessibility Compliance</a> (Slide 8)</li>
                
                
              
            
          
        
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
    
    
    
    
  
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
      
        
        
      
    
      
        
        
          
            
            
              
              
                
                  
                  <li><a href="#22">(P)OUR: Perceivable</a> (Slide 22)</li>
                
                
              
            
          
        
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
    
    
    
    
  
  </ul>
</div>

---
class: center, middle, inverse
# Announcements

---
## What’s Happening This Week
<ul>
  <li>Jen is likely out/remote this whole week, so their Office Hours are cancelled.</li>
  <li>Please remember that if you have content/questions you want to discuss with the course staff, you can post about it on Ed, send us a Chat message on Ed, come to TA Office Hours (Jesse: Fri 2:30; Yumeng: Mon 2:30), or post on Ed to find a separate time to meet if those options do not work for you.</li>
</ul>

<p>Course Content:</p>
<ul>
  <li>Today’s Lecture: Intro to Assessment & UARs</li>
  <li>Section Tomorrow: Getting started with Assessment assignment</li>
  <li>Friday’s Lecture: More Assessment</li>
</ul>

---
## Due Today (1/21): AT Trial
<ul>
  <li>
    <p>Be sure to submit no later than EOD on Friday</p>
  </li>
  <li>
    <p>As a reminder, First Person Accounts is an <em>individual</em> competency for this assignment– you can work with your groupmates to find an account, but you must each independently write your responses to the reflection questions.</p>
  </li>
</ul>

---
## Released Today (1/21): Assessment
<ul>
  <li>This is a two-week assignment!</li>
  <li>Groups & website assignments will be posted on Ed by the end of the day</li>
  <li>Due next week (1/28): completed UARs</li>
  <li>Due in two weeks (2/4): completed report</li>
</ul>

---
## Section Tomorrow: Getting started with Assessment
<ul>
  <li>Section will be back in person tomorrow! As a reminder, Section B is in a NEW ROOM (ECE 042); Section A is in the same location as before.</li>
  <li>You will be working with your groups on the Assessment assignment, which includes using a screen reader. You may want to review the screen reader basics/resources shared last week.</li>
</ul>

---
class: center, middle, inverse
# Assessing Accessibility Compliance

---
## Automated Tools

Automated Tools ([Review of many options](https://medium.com/@OPTASY.com/what-are-some-of-the-best-web-accessibility-testing-tools-to-evaluate-your-website-with-69def25a386))
- Web: WebAIM's [WAVE](https://wave.webaim.org/);
- Browser Extensions ([Comparison Article](https://medium.com/@OPTASY.com/what-are-some-of-the-best-web-accessibility-testing-tools-to-evaluate-your-website-with-69def25a386)): [WAVE](https://chrome.google.com/webstore/detail/wave-evaluation-tool/jbbplnpkjmmeebjpijfedlgcdilocofh?hl=en-US); [Axe](https://chrome.google.com/webstore/detail/axe-devtools-web-accessib/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US); [Lighthouse](https://developer.chrome.com/docs/lighthouse/overview/); [Siteimprove](https://chrome.google.com/webstore/detail/siteimprove-accessibility/efcfolpjihicnikpmhnmphjhhpiclljc/related);  & other browser extensions

Phone scanners
- [Google Accessibility Scanner](https://support.google.com/accessibility/android/answer/6376570?hl=en&ref_topic=6376582)
- [iOS](https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/OSXAXTestingApps.html) requires source code!

---
## Other assessment methods

- Proprietary: 
  - [Evinced](https://www.evinced.com/) does iOS, web and more.  
  - [Grackle](https://www.grackledocs.com/en/) does google docs
  - Lots of pay for help shops
- Design Guidelines (i.e. know WCAG and apply it heuristically)
- Simulation (Try it yourself using accessibility technologies or simulators such as [aDesigner](https://www.eclipse.org/actf/downloads/tools/aDesigner/))
- User Testing

---
## Wednesday's [Assessment Homework](/courses/cse443/26wi/assignments/website.html)

Identify 3 tasks

(Install and) run an automated Accessibility Checker

Also test it yourself with **two** accessibility technologies

Capture problems using a Usability Assessment Report

---
## Field Trip

Try out [WebAIM for the UW Library](https://wave.webaim.org/report#/https://www.lib.washington.edu/)

???
Talk about how the type of errors found relates to the concepts we've discussed so far

---
## Usability Assessment Report

- You may be familiar with this concept. Also called
  - Usability Problem Report (UIM Ch11)
  - Usability Aspect Report (CMU)
  - Bug/Issue Report (Bugzilla, JIRA, Rational)
- Audience: primarily developers
- Content should be
  - Specific and convincing
  - Accessible

---
## Example from the UW Library

- **Name** as "Missing ALT Text";
- **Evidence** Guideline violated: 1.1 ([Text Alternatives](https://www.w3.org/WAI/WCAG22/Understanding/text-alternatives)); 
- **Screen Shot** as the image and URL ([lib.washington.edu](https://www.lib.washington.edu/)); 
![:img Screenshot of an image without appropriate alt text in WebAIM's automated accessibility checker,80%, width](img/testing/library_evaluation.png)

---
## Example from the UW Library

- **Name** as "Missing ALT Text";
- **Evidence** Guideline violated: 1.1 ([Text Alternatives](https://www.w3.org/WAI/WCAG22/Understanding/text-alternatives)); 
- **Explanation** ALT text does not describe image

--
- **Severity**

Justify severity in terms of

- *Frequency* Is Problem common or rare? For which types of users?
- *Impact* Is it hard or easy to overcome this?
- *Persistence* Is there a way to avoid this problem?

???
What do you think? Discuss with your neighbor  and post  [on Ed](https://edstem.org/us/courses/90089/discussion//5426947)
- The frequency with which the problem occurs: Is it common or rare?
- The impact of the problem if it occurs: Will it be easy or difficult for the users to overcome?
- The persistence of the problem: Is it a one-time problem that users can overcome once they know about it or will users repeatedly be bothered by the problem?

This is debatable, but frequency is low (it only occurs once on this site. If you are writing up all missing image alt text as a group, you might increase your estimate of frequency, but this site doesn't appear to have a lot of undescribed images); impact is low (it is possible to determine the purpose of this image by either clicking on it to see what it links to, or inferring some things from the external link and image file name (both unpleasant alternatives for a screen reader user); and persistence is high (it's not going to go away).

---
## Example from the UW Library

- **Name** as "Missing Image ALT Text";
- **Evidence** Guideline violated: 1.1 ([Text Alternatives](https://www.w3.org/WAI/WCAG22/Understanding/text-alternatives)); 
- **Explanation** A screen reader won't be able to describe this image 
- **Severity** 2. High persistence, middling frequency, impact is low (you can click through to learn more)
- **Possible Solution** Add ALT text and improve practices to add better alt text consistently. A good example of correct ALT text for this  would be something like "pictures of #vote buttons in the background fading to the text '2024 Election Guide'"
- **Relationship to other problems** (TBD, probably other images with similar issues)

---
## WebAIM is really about WCAG

We'll be covering that in the next two classes

It can be hard to understand the interface if you don't understand WCAG

---
## Introduction to Accessibility Standards

The Web Accessibility Initiative ([WAI](http://www.w3.org/wai/)),(from the World Wide Web Consortium (W3C)) 
- Makes recommendations for Web authors, browsers and servers: **Web Content Accessibility Guidelines (WCAG)**
- WCAG is an ongoing project 
- There is no *official* equivalent for non-web programming, <BR> but WCAG can easily be applied to apps as well 
- Lots of ways to learn WCAG <BR> (e.g. this [certificate program](https://de.torontomu.ca/wa/); this [textbook](https://pressbooks.library.torontomu.ca/pwaa/); and [WebAIM](https://webaim.org))

???
We could spend a whole quarter on this... but we're going to limit it to one or two weeks

"live" (regularly being worked on and updated, with input from the disability community).

---
## The [POUR](https://webaim.org/articles/pour/) standard

- Perceivable: Web content is made available to the senses - sight, hearing, and/or touch
- Operable: Interface forms, controls, and navigation are operable
- Understandable: Information and the operation of user interface must be understandable.
- Robust: Content must be robust enough that it can be interpreted by a wide variety of user agents, including assistive technologies

.footnote[Note: There is a 5th thing, Conformance, which we are not covering]

???
This is appropriate for *all* disabilities -- don't think access is only an issue for blind and low vision (BLV) people

Obviously there is some overlap between these, and they build on each other

---
## Three levels of compliance

- *Some* users with disabilities can access and use web content (A)
- *Removal of significant barriers* overall to accessing content (AA)
- *Enhancements to web accessibility* for more users with disabilities (AAA)

Most apps and websites today only meet  *part* of (A) level compliance!

---
class: center, middle, inverse
# (P)OUR: Perceivable

(Guideline 1.1)

---
## Guideline 1.1
Text Alternatives: Provide *electronic text* alternatives for any non-text content.
.left-column50[
- [Decorative and branding](https://dl.acm.org/doi/pdf/10.1145/3308558.3313605) 
- Formatting and text styling
- Images as links
- [Diagrams](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9028522&casa_token=zZw_rYBgu1AAAAAA:eozpbJ-vvMZjQNt8p6WU91X4uFumPs-yVuMn4PTPRjyMhtsVrprdIEe1JfYOCUdv8SFP_TGd9s965Q&tag=1)
- [Visualizations](https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9555469)
- [GUIS](https://dl.acm.org/doi/10.1145/3411764.3445040)
]
.right-column50[
- Animations/Videos 
- AR/VR ([Accessibility, Disabilities, and VR](https://educatorsinvr.com/2019/05/31/accessibility-disabilities-and-virtual-reality-solutions/))
- [Comparison of IoS and Android Rich Interactions](https://dl.acm.org/doi/pdf/10.1145/2851613.2851680?casa_token=dOz4huS0TUkAAAAA:zv0PjZk3-T8Bb4X2SfNpdZFuqO2u9v1jpWn5fq0hKZ0se6t5g0oMKLfrAmhlyufcw_3AuJ-ABZ2yWQ)
- CAPTCHAS (will discuss later)
- [Memes](https://dl.acm.org/doi/10.1145/3308561.3353792)
]

???
Why non text?
- Can be rendered visually, auditorially, tactilely, or by any combination. 
- Can also be easily enlarged

All of these require different strategies to describe them well.

Read up on some of these links when you are faced with specific description needs

---
exclude: true
## Small Group Activity

Break into small groups and take turns.

- Find inaccessible non-photograph you posted on Ed but don't show it. 
- **Read** the group your description. 
- Then look at it.

What did you learn from this? [post](https://edstem.org/us/courses/90089/discussion//5427179)

---
## Diagrams (1 of 2)
.left-column40[
![:img stateDiagram-v2: * --> Still; Still --> *; Still --> Moving; Moving --> Still; Moving --> Crash; Crash --> *: https://mermaid-js.github.io/mermaid-live-editor/edit#pako:eNpdj7sKwzAMRX_FaCzx0tFDl3btlLHuIGKRGPwojhwIIf9e16a0RNPh6EpIGwzRECiYGZluFseEXi5nHUSpx-kppLyInq1zTVWssjSP6h4XG8ZmGx_H_-w14Tw1W_G7FDrwlDxaU87aPgENPJEnDapgoMwJnQYd9hLFzLFfwwCKU6YO8sv8HmlyfwPe-kcg, 120%, width](img/assessment/mermaid.png)
]
.right-column60[
When direct exploration isn't possible, consider descriptions that are *language based*
]
---
## Diagrams (2 of 2)
<tt>
stateDiagram-v2 <BR>
      [*] --> Still<BR>
      Still --> [*]<BR>
      Still --> Moving<BR>
      Moving --> Still<BR>
      Moving --> Crash<BR>
      Crash --> [*]<BR>
</tt>

---
## MathML standard for accessibility

Math has a hierarchy just like other systems (i.e. fractions, parentheses)

Can support with MathML

Can generate MathML using pandoc; MS Word; etc

Capturing an image of an equation and describing it much worse for screen reader users

---
## What about GUIs?

How do BLV technology users understand and access visual semantics?

![:img P9 reconstructing the Google mobile web homepage. The participant is placing Wiki Stix and play dough on the canvas which is shown next to their smartphone., 60%, width](img/assessment/reconstruction.jpg)

.footnote[[Investigating Visual Semantic Understanding of Blind and Low-Vision Technology Users](https://dl.acm.org/doi/abs/10.1145/3411764.3445040)]
???
Interviews;
Screen reader tasks; 
Reconstruction

---
## Results (1/2)

Participants were aware of the overall structure of *phone apps*

- They developed this understanding using screen readers
- Associated size and location and function
- Layouts were understood in terms of absolute, relative, and corner positions
---
## Results (2/2)
<BR>
.quote[
The way I think about this is on the top is my email, to the right is my phone number. Below that [is] essentially a 2 by 2 kind of thing, which has my social profile.
]

---
## Image Description: GUIs

.left-column40[

![:img Two screens of the Uber Eats app. The home page to the left and a map view to the right,120%, width](img/assessment/gui.png) 
]

.right-column60[
- **DO** describe visual attributes / semantics (aesthetics and usability)
- **DO NOT** repeat screen readers (control)
]

---
## My description of the leftmost GUI

.left-column[
![:img Two screens of the Uber Eats app. The home page to the left and a map view to the right,120%, width](img/assessment/gui-left.png) 
]
--
.right-column[
- App has two tabs at top center: Delivery and Pickup. Below is a search bar with address and  time and an advertisement for The Burger Joint (25% of screen)
- Next is a scrolling set of tabs for Pickup; Grocery; Prescription; Top Sites; ...
- The bottom 30% of the screen shows the title Hidden Gems (Up and coming spots you'd like) with a 2d-scrollable list of restaurants. Each row in the list shows an image, restaurant name, rating, and more.  Tsuki Ramen and Chocolate Co. are visible.
]
???

This is very hard to describe without knowing what is accessible; and whether the user is more interested in content or layout.

---
## Describing GUIs is rarely necessary

GUI description best supported dynamically through exploration. Critical needs for this
- Accessibility information available for interface
- Touch screen phone interaction techniques

Don't describe GUIs, explore them.

---
.left-column40[
![:img A picture of Facebook with a like button visible at bottom left (a thumbs up followed by the word like),80%, width](img/assessment/facebook1.png) 
]

.right-column60[
### Developer Responsibility
Expose GUI structure

Provide good ALT text
- What is a good name for the "Like" Button?

- Enable the user to understand the name of the control they have navigated to, what type of control it is, what value it has, what state it has.
]

---
.left-column40[
![:img A picture of Facebook with a like button visible at bottom left (a thumbs up followed by the word like),80%, width](img/assessment/facebook1.png) 
]

.right-column60[
## Proper ALT text

Screen reader will read out name, role, and state. Don't repeat these.

Good alt text: Name ("Like")

API knows: Role ("Button")

API knows: State ("Not selected")
]

---
exclude: true

.left-column40[
![:img A picture of Facebook with a like button visible at bottom left (a thumbs up followed by the word like),80%, width](img/assessment/facebook1.png) 
]

.right-column60[
## Exception: Your UARs
Focus on the relevant stuff

Example:  alt text for the like button

A picture of Facebook with a like button visible at bottom left (a thumbs up followed by the word like)
]

???
 (assuming it's the subject of my UAR, slide or etc)
 
---
## Guideline 1.1 is not just about images
- Controls, Input
- Time-Based Media
- Visual Test or Exercise
- Sensory Experience 
- CAPTCHA
Make this ignorable
- Decoration, Formatting, Invisible