Formative and Summative Studies

# Formative and Summative Studies
Disability inclusion in study design

CSE443: Winter 2026; Jennifer Mankoff (Last Edited: 2025-08-07). <a href="/courses/cse443/26wi/slides/studies.html">Live View: /slides/studies.html</a>

Important Reminder: check zoom & captioning

---

<div class="slide-outline" style="margin: 1em 0; padding: 1em; background: #f5f5f5; border-radius: 5px;">
<h2 style="margin-top: 0;">
  
  
    Outline
  
  
</h2>
    
  <ul style="list-style-type: none; padding-left: 0;">
  
  
  
  
  
  
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
      
        
        
      
    
      
        
        
          
            
            
              
              
                
                  
                  <li><a href="#3">Participatory Design</a> (Slide 3)</li>
                
                
              
            
          
        
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
    
    
    
    
  
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
      
        
        
      
    
      
        
        
          
            
            
              
              
                
                  
                  <li><a href="#12">Disability Inclusion in Formative Studies</a> (Slide 12)</li>
                
                
              
            
          
        
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
    
    
    
    
  
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
      
        
        
      
    
      
        
        
          
            
            
              
              
                
                  
                  <li><a href="#26">Running an inclusive summative study</a> (Slide 26)</li>
                
                
              
            
          
        
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
    
    
    
    
  
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
      
        
        
      
    
      
        
        
          
            
            
              
              
                
                  
                  <li><a href="#30">Storyboarding</a> (Slide 30)</li>
                
                
              
            
          
        
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
    
    
    
    
  
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
      
        
        
      
    
    
    
    
    
  
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
      
        
        
      
    
      
        
        
          
            
            
              
              
                
                  
                  <li><a href="#35">Lofi Prototyping</a> (Slide 35)</li>
                
                
              
            
          
        
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
      
    
    
    
    
    
  
  </ul>
</div>

---
<h2 id="feasibility">
  
  
    <a href="#feasibility" class="anchor-heading" aria-labelledby="feasibility"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Feasibility
  
  
</h2>

<p>Is the idea viable <em>within the constraints of the class</em></p>

<ul>
  <li>Technically</li>
  <li>Financially</li>
  <li>Socially appropriate</li>
  <li>Operationally</li>
  <li>Timeline</li>
</ul><hr />
<h2 id="technical-feasibility">
  
  
    <a href="#technical-feasibility" class="anchor-heading" aria-labelledby="technical-feasibility"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Technical Feasibility
  
  
</h2>

<p>Can be built within 4 weeks</p>

<p>Built –> publically deployed</p>

<ul>
  <li>What technologies will be used</li>
  <li>What is your backup if those don’t work</li>
  <li>How well do your group members know them?</li>
</ul><hr />
<h2 id="financial-feasibility">
  
  
    <a href="#financial-feasibility" class="anchor-heading" aria-labelledby="financial-feasibility"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Financial Feasibility
  
  
</h2>

<p>We have close to $0. Let us know if you need something though and we’ll discuss</p><hr />
<h2 id="social-appropriateness">
  
  
    <a href="#social-appropriateness" class="anchor-heading" aria-labelledby="social-appropriateness"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Social Appropriateness
  
  
</h2>

<p>“Will it be allowed and accepted?”</p>

<ul>
  <li>Non ableist</li>
  <li>Something people will accept</li>
  <li>Other ethical concerns?</li>
</ul><hr />
<h2 id="operational-feasibility">
  
  
    <a href="#operational-feasibility" class="anchor-heading" aria-labelledby="operational-feasibility"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Operational Feasibility
  
  
</h2>

<p>“Will it be possible to maintain and support long-term?” (not a requirement)</p>

<ul>
  <li>What skills would be needed?</li>
  <li>What tools would be needed?</li>
  <li>What documentation would be needed?</li>
</ul><hr />
<h2 id="timeline-feasibility">
  
  
    <a href="#timeline-feasibility" class="anchor-heading" aria-labelledby="timeline-feasibility"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Timeline feasibility
  
  
</h2>

<p>“How does the team’s availability match the needs?”</p>

<ul>
  <li>How many hours per week can each person put in?</li>
  <li>What are the major milestones and decision points?</li>
  <li>What are the preliminary deliverables and deliverable dates?</li>
</ul><hr />
<h2 id="difficulties-of-feasibility-studies">
  
  
    <a href="#difficulties-of-feasibility-studies" class="anchor-heading" aria-labelledby="difficulties-of-feasibility-studies"><svg viewBox="0 0 16 16" aria-hidden="true"><use xlink:href="#svg-link"></use></svg></a> Difficulties of feasibility studies
  
  
</h2>
    
<ul>
  <li>Scope may be uncertain</li>
  <li>Approach may be ill defined (e.g. resource estimates too rough)</li>
  <li>Organizational changes may be needed</li>
  <li>Enthusiasm may cause you to downplay risks</li>
</ul>

---
class: center, middle, inverse
# Participatory Design

---
exclude: true
## What is the alternative to disability dongles?

> Disability Dongle: A well intended elegant, yet useless solution to a problem we never knew we had. Disability Dongles are most often conceived of and created in design schools and at IDEO.] [Liz Jackson](https://twitter.com/elizejackson/status/1110629818234818570)

(reminder)

Alternative: Ethical need-finding: Complicated, especially in industry settings

---
## Participatory Design/Research (1/3)
- Emanates from design & technology field
- Has been specifically used in Assistive Technology & HCI research; 
- Applied in education and healthcare settings; children & older adults

---
## Participatory Design/Research (2/3)

- Working directly with users (& other stakeholders) in the design of systems

- Users are actively involved in setting design goals and planning prototypes
  - Contrasts with methods where user input is sought only after initial concepts and prototypes have been produced (i.e. PD is more than user-testing)

---
## Translating to Need-finding  (3/3)

- Working directly with users (& other stakeholders) in the design of systems

- Users are actively involved in setting design goals and planning prototypes

- Early and continual participation of intended users to produce better technologies that better suit the needs of users

.footnote[Dr Jane Seale-- School of Education, University of Southampton]

---
## Participatory Methods in A11y (1/3)

- Aims to engage participants in the design, conduct and evaluation of products/research with the construction of non-hierarchical  relations  
  - Participants encouraged to own the outcome by setting the goals and sharing in decisions about processes 
  - “Nothing about me, without me”

.footnote[Dr Jane Seale-- School of Education, University of Southampton]

---

## Participatory Methods in A11y (2/3)

- Aims to engage participants in the design, conduct and evaluation of products/research with the construction of non-hierarchical  relations  
- Ensures research topic is one that people with disabilities consider worthy of investigation
  - Asking people with disabilities to act as consultants or advisors to projects
  - Provision of support, training and payment so that people with disabilities can undertake their own research

---
## Participatory Methods in A11y (3/3)

- Aims to engage participants in the design, conduct and evaluation of products/research with the construction of non-hierarchical  relations  
- Ensures research topic is one that people with disabilities consider worthy of investigation
- Other methods 
  - Narrative research: Life history, biography, oral history
  - Focus groups, interviews
  - Action Research

---
## *Accessible* Participatory Methods

- Accessibility doesn't come by accident when planning studies
- All research should be accessible research (regardless of if it is accessibility research)
- You have to make your system accessible (using inspection techniques) *before* doing this 
- We will discuss accessibility *for evaluators* and *for participants* today

---
class: center, middle, inverse
# Disability Inclusion in Formative Studies

---
## Accessible Study Planning Workflow

<div class="mermaid">
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor': '#4CAF50', 'tertiaryColor': '#009688', 'fontSize': '16px', 'textMargin': '0px', 'text-align': 'left' }}%%

flowchart LR

A(Identify stakeholders)
    B(Identify tasks)
    
    A <--> C
    B <--> C
    C <--> D
    D <--> E
	
    subgraph Assign tasks
    C(Access needs
      Effects of familiarity
      Personal preferences
      Institutional constraints)
    end

subgraph Plan Accomm.
      D(Communication 
             Materials 
             Time
             Space)
    end

subgraph Reflect
       E(Access Synergies
         Access conflicts
         Power Dynamics)
    end

classDef default fill:#009688,stroke:#333,stroke-width:2px, color:white;
classDef reflect fill:#4CAF50,stroke:#333,stroke-width:2px, color:white;
class A,B,C,D default
class A,B,C,D,E reflect
linkStyle default stroke: black,stroke-width:8px

</div>

---
## An Example

Supposed you want to co-design a new directional control system for
 increased independence among paddlers who are blind.

---
exclude: true

## Identify Stakeholders

.left-column[you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.]

.right-column[
Who are the stakeholders who you need to take into account for this study?

- Disabled participants (BVI, DHH)
  - Research team
  	- Lua -- lead researcher, has dyslexia
  	- Jay -- project team member, Deaf person who uses sign language.
  	- El -- project team member, non-disabled
]
---
exclude: true

## Design Study

.left-column[you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.]
.right-column[
- Who are the stakeholders who you need to take into account for this study?
- What is the right structure for the study?
]
---
exclude: true
## Design Study (1 of 5)

This is a usability study, so it should include the same tasks as tested for users without disabilities

Metrics should be similar as well -- for example the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) might be useful at the end

---
exclude: true
## Design Study (2 of 5)

This is also a study of accessibility, with disabled users. The study design needs to consider

- Choice of method 
  - e.g. think aloud may be more difficult for a nonspeaking individual, or someone with fatigue
  - Check in on study measures and statistical design. Does user heterogeneity impact this?

---
exclude: true
## Design Study (3 of 5)

This is also a study of accessibility, with disabled users. The study design needs to consider

- Ensure that participants' hardware/software also addresses study needs (e.g. do participants have a working monitor?)
- Study length (estimate before and after piloting)
   - Study length may increase for some disabled participants
   - Study length may negatively impact participants with fatigue-related concerns

---
exclude: true
## Design Study (4 of 5)

Study metrics may also need revision.
.left-column50[
Consider adding
- Click errors
- Success criteria & ratings
- Accessibility errors
]
.left-column50[
Consider changing
- Any metrics that include ableist assumptions
- Approach to time based measurements
]

---
exclude: true
## Design Study (5 of 5)

It is also important to consider ethical issues

- What responsibility do you have to participants in terms of providing skills and help in & out of study?
- What responsibility do you have to give participants a participatory role in design?
- How do you ensure the integrity of your record of their experience?
- Are you compensating them appropriately and addressing costly transportation needs?

???
These changes may in turn impact
- Expense 
- Scalability

---
exclude: true

## Plan Study Tasks

Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities.
- Who are the stakeholders who you need to take into account for this study?
- What is the right structure for the study?
- What study tasks should this team prepare for?

---
## Tasks List 
1. Assign Tasks
2. Train study team 
3. Handle recruitment 
4. Pilot with people with disabilities
5. Run the Study
6. Analyze the data

.footnote[from CHI 2022 paper [Anticipate and Adjust](https://dl.acm.org/doi/10.1145/3491102.3501882)]

---
## Assigning tasks: considerations
- access needs
- effects of familiarity with the task
- personal preferences
- other institutional constraints

---
exclude: true

## 1. Assign Tasks
- Consider:
	- access needs
	- effects of familiarity with the task
	- personal preferences
	- other institutional constraints
- For this scenario:
	- Lua has significant experience running studies; Lua needs a study script in large print with a specific font
	- El wants to gain more experience running studies
	- Jay has significant experience with ASL and Deaf Space; Jay prefers to have interpreters

---
exclude: true

## Task Assignment List
1. ~~Assign Tasks~~
2. **Lua**: Train study team 
3. **Jay**: Handle recruitment 
4. **Jay**: Pilot with people with disabilities
5. **Lua**: Run the Study
    - **Lua**: Prepare study materials (e.g., print out protocols)
    - **Jay**: Hire access support staff
    - **Participants**: Participate in the study
    - **El**: Compensate participants
6. **All**: Analyze the data

---
exclude: true

## 2. Train study team (1 of 2)

As study lead, and a person with both disability and study related experience, Lua will train the study team.

Her goal is to ensure that El and Jay are aware of best practices and considerations that might impact study design before they begin interacting with participants.

---
## Training: considerations
- Ensuring that they address access needs and communication support from recruitment onward
- Providing basic DEIA training as needed so that all those with participant contact know basics. 
  - Don’t be overly sensitive (don’t overcompensate)
  - Don’t rely on useless cues (audio/visual) to convey encouragement
  - Monitor participant fatigue carefully
  - When to help and when to end a task if participants have trouble
---

## Recruitment: considerations
As a person already connected to the disability community, the team determines that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process.

Recruitment typically raises a number of access issues
- Are your recruitment materials accessible
- How will you find participants (next slide)
- How will you compensate participants?

---

## Recruitment: More considerations

Finding participants may require careful work. In this case, a lot of those problems are solved by partnering with a particular community organization. Some considerations:
- How much do you know about the group your are targeting and what to expect from/of them?
- Will they see you as trustworthy? 
- Are you giving them enough agency in the study process?
- Maybe start with gatekeepers

???
- Negotiating access often requires working with gatekeepers
- Participants may see researchers as “just another professional, conducting professional surveillance”

---
## Piloting: Considerations

- First, revisit accommodations plan
- Next, have sufficient prep time
  - Hire interpreters (at least 2 weeks in advance)
  - Print out accessible study materials, consent form, etc
  - Understand (and possibly prepare) the space that the team will work in
  - Have compensation ready

---
## Piloting: Considerations

- First, revisit accommodations plan
- Next, have sufficient prep time
- Finally, reflect and iterate to address any new concerns

---
exclude: true

## Update Design Based on Reflection

- Lua gets tired from reading overtime:  Institute a protocol for monitoring fatigue
- Study length was problematic for one Blind participant who also had a chronic illness: Add a cutoff for tasks that take a long time
- Sometimes sightlines made it hard for Jay to see the interpreter: Rearrange the room to better support communication with interpreters

---
exclude: true
## Update Tasks List Based on Reflection

1. ~~Assign Tasks~~
2. ~~Lua: Train study team~~ (done)
3. **Jay**: Handle recruitment (ongoing)
4. ~~Jay: Pilot with people with disabilities~~ (done)
5. ~~Lua~~ **Jay & El**: Run the Study
    - **Lua**: Prepare study materials (e.g., print out protocols)
    - **Jay**: Hire access support staff
    - **Participants**: Participate in the study
    - **El**: Compensate participants
6. **All**: Analyze the data

---
## Running the study: Considerations

The team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise.

Similar to piloting a study

---
## Analyze the Data: Considerations
- Data analysis tools may not be accessible
- Team members may have different needs with respect to written versus audio transcripts
- **Member Checking**

---
exclude: true

## Special Considerations

Children -- Assent versus Consent

Nonspeaking individuals
- Can be a tendency for support workers to speak on behalf of the person with a 
disability

Find ways to support participants with disabilities to become researchers/designers in their own capacity

Address power dynamics

---
exclude: true
## Other Concerns to Address

- Accountability and ownership
  - Designer/Researcher is accountable to the funder- who owns the research agenda?
  
---
## Moving toward Emancipatory design
  - Find ways to support participants with disabilities to become researchers/designers in their own capacity
  - Address power dynamics
  - Designer/Researcher is accountable to participants with disabilities. Their skills are at the disposal of the people with disabilities
  
.footnome[Mike Oliver: [Emancipatory Research](https://disability-studies.leeds.ac.uk/wp-content/uploads/sites/40/library/Oliver-Mikes-paper.pdf)]

---
## My approach to disability inclusion

1. Advocate for the inclusion of people with disabilities in higher education ad research
2. Seek out and support students who have first person experience with disability (and are also excellent students)
3. Work to be an ally to them so that they can succeed
4. Engage with the disability community in selecting problems and sharing results
5. Include disabled community members who contribute strongly to projects

???
In my own work, I try to do the following. Can any of this translate?

---
exclude: true
## Revisiting COOP and discussion (if time)

Post again on [Ed](https://edstem.org/us/courses/90089/discussion//4882320)

---
class: center, middle, inverse
# Running an inclusive summative study

---

## Summative User Testing

So you have an app and you think it's accessible. How do you check that?

.quote[...getting the big picture and assessing the overall experience of a finished product...] - Nielsen Norman Group

---
## This is a topic for another class!

Lots of variables here,
- typically checks that an app works as expected on standard tasks when used by the target population
- may compare features or apps quantitatively 
- may involve think aloud or other qualitative data

---

## *Accessible* Summative User Testing

- Accessibility doesn't come by accident when planning studies
- All research should be accessible research (regardless of if it is accessibility research)
- You have to make your system accessible (using inspection techniques) *before* doing this 
- Can borrow accessibility strategies from the Anticipate & Adjust here too

Other good guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibility/testing/); [Anticipate and Adjust](https://a11ykelly.medium.com/anticipate-and-adjust-cultivating-access-in-human-centered-methods-1e46c6845e34)

---
class: center, middle, inverse
# Storyboarding

---
## Storyboard
 
![:img "Sketches of five blank rectangles", 60%, height](img/studies/blank-storyboard.png)

---
## Just sketches
 
![:img "Five frames. Each shows an image that captures one of the following sentences: A person passes by an advertisement board. The notice an announcement that says 'New!'. They hold up a flip phone and snap a photo of a barcode. The phone loads a webpage with information about the product. They put away the phone and leave. ", 60%, height](img/studies/sample-storyboard.png)

.footnote[from Sketching User Experiences: The Workbook. Chapter 4.4: Narrative Storyboards. by Greenberg, Carpendale, Marquardt, and Buxton]

---
## Storyboard

- Show the whole interaction
- Show the beginning and end 
- Ideally Create 2 or 3 storyboard flows that show different ways of interacting with your interface (e.g. different task flows; different goals; etc.).

---
class: center, middle, inverse
# Lofi Prototyping

---
## Example accessible lo-fi prototype

![:img "A piece of cardboard is decorated with wiki sticks and play dough to create tactile indicators of an interface for a blind person", 60%, height](img/studies/smartphone.jpg)

---
## Low-fi prototyping definition

Turn design ideas into _testable_ and _tangible_
artifacts for collecting and analyzing the user
_demands_ at the early stage.

---
## Physical Materials

- Cardboard
- Paper
- Legos
- Play dough
- Wiki sticks
- Foam
- Clay
...

---
## Digital Prototyping

- Figma
- HTML ("rapid web prototype")
- Screen shots
- Videos
- Word document

Be wary of "overdesigning" --> more difficult to critique

---
## Creating an experience 
Wizard-of-Oz Prototyping

- Modular interface
- Update it as the user interacts

---
## Basic Interface

![:img "A cardboard interface sits on a table. Surrounding it are screens (drawings on paper) that can rapidly be swapped out" 60%,height](img/studies/woz-interface.png)

---
## Ar interface (1/2)

![:img "A closeup of a person looking through a see-through screen made of cardboard. An arrow and text on the screen are pointing at a building in the background" 60%,height](img/studies/woz-ar1.png)

---
## Ar interface (2/2)

![:img "A different person is interacting with the cardboard. It is clear from this viewpoint that the screen is actually a roll of semi-transparent plastic that a wizard is sliding through the screen to update the image " 60%,height](img/studies/woz-ar2.png)

---
## Kiosk experience

![:img "Two panels. The first shows a person walking up to a kiosk made of paper and cardboard and interacting it. The second shows a wizard modifying the kiosk to make it react to the user" 60%,height](img/studies/wizard-experience.png)

---
## Bike exercise interface

![:img "A set of four panels shows a person riding a bike in a cardboard kiosk. a cardboard screen is visible with an image on it. A wizard updates and points at things to react to the users actions" 60%,height](img/studies/woz-bike.png)

---
## How to Wizard-of-oz Virtually
- share on zoom, swapping out screen shots
- use something like Figma (if you can do it accessibly)
- Use a survey (with images and ALT text, or survey pages that have all the key content)
- Test with a similar system if you need to