background-image: url(img/people.png) .left-column50[ # Week 4: Accessible Evaluation , Spring 2023 ] --- name: normal layout: true class: --- # Important Reminder ## This is an important reminder ## Make sure zoom is running and recording!!! --- [//]: # (Outline Slide) # Learning Goals for Today - Presenting Acessibly - How to build inclusive experiment designs - Running an inclusive summative study --- # Presenting Accessibly - First, make your slides accessible (and share them ahead of time) --- # Slideshow Rules of Thumb (1 of 6) .left-column50[ - image descriptions - use headers & styles - color contrast - san serif fonts - plain language ] -- .right-column50[ - unique slide titles - makes navigation easier - put (x of y) in the title if repeating ] --- # Slideshow Rules of Thumb (2 of 6) .left-column50[ - image descriptions - use headers & styles - color contrast - san serif fonts - plain language ] .right-column50[ - unique slide titles - avoid clutter ] --- # What is clutter? Please, for the love of all that is good on this fine planet we call home, do not do this to your poor audience members They don’t deserve this! What did they do to you? They probably flew hundreds of miles and of all talks and things to do in this new place came to YOUR presentation. And what do you do? You greet them with this GIANT wall of text! How rude. It’s ugly to look at. It’s hard to read. It’s annoying as he ck for me to type out this thing just to make a point! So please, don’t do this to your audience members. Be a responsible presenter. Practice your talks so you don’t have to read off the slide (or use speaker notes! also okay!). Break up your content so looking at your slides isn’t like getting smacked in the face with a wall of text. **But there are exceptions! (e.g., if you have a thick accent)** --- # Slideshow Rules of Thumb (3 of 6) .left-column50[ - image descriptions - use headers & styles - color contrast - san serif fonts - plain language ] .right-column50[ - unique slide titles - avoid clutter - don't use color to convey meaning ] --- # Slideshow Rules of Thumb (4 of 6) .left-column50[ - image descriptions - use headers & styles - color contrast - san serif fonts - plain language ] .right-column50[ - unique slide titles - avoid clutter - don't use color to convey meaning - 9/10, you don’t need sound effects ] --- # Slideshow Rules of Thumb (5 of 6) .left-column50[ - image descriptions - use headers & styles - color contrast - san serif fonts - plain language ] .right-column50[ - unique slide titles - avoid clutter - don't use color to convey meaning - 9/10, you don’t need sound effects - you rarely ever NEED that movement-based slide transition ] --- # Slideshow Rules of Thumb (6 of 6) .left-column50[ - image descriptions - use headers & styles - color contrast - san serif fonts - plain language ] .right-column50[ - unique slide titles - avoid clutter - don't use color to convey meaning - 9/10, you don’t need sound effects - you rarely ever NEED that movement-based slide transition - check slide order (like DOM order) ] --- # If you have a video in your slides - Make sure it is captioned - Make sure it has audio descriptions if non-verbal content is important --- # When presenting - Speak slowly and clearly - Read entire quotes - describe images and videos --- [//]: # (Outline Slide) # Learning Goals for Today - Presenting Acessibly - **How to build inclusive experiment designs** - **Running an inclusive summative study** --- # Summative User Testing So you have an app and you think it's accessible. How do you check that? .quote[Summative evaluations describe how well a design performs, often compared to a benchmark such as a prior version of the design or a competitor. Unlike formative evaluations, whose goals is to inform the design process, summative evaluations involve getting the big picture and assessing the overall experience of a finished product. Summative evaluations occur less frequently than formative evaluations, usually right before or right after a redesign.] - Nielsen Norman Group --- # What is Summative User Testing Lots of variables here, but - typically checks that an app works as expected on standard tasks when used by the target population - may compare features or apps quantitatively - may involve think aloud or other qualitative data --- # Why not Start with Formative User Testing? - Many of the apps we deploy are designed for people in general - Disabled people need to be able to use those apps too - Summative testing is the gold standard in assessing accessibility Of course the entire design process *should not just include majority class people* and much of this presentation applies to formative testing took We'll also cover that later today when we talk about designing apps *for* and *with* people with disabilities --- # *Accessible* Summative User Testing - Accessibility doesn't come by accident when planning studies - All research should be accessible research (regardless of if it is accessibility research) - You have to make your system accessible (using inspection techniques) *before* doing this - We will discuss accessibility *for evaluators* and *for participants* today Best guidance: [Nielsen-Norman Group](http://www.nngroup.com/reports/accessibility/testing/); [Anticipate and Adjust](https://a11ykelly.medium.com/anticipate-and-adjust-cultivating-access-in-human-centered-methods-1e46c6845e34) --- # Accessible Study Planning Workflow  --- # An Example Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities. --- # Identify Stakeholders Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities. - Who are the stakeholders who you need to take into account for this study? --- # Stakeholders List - Disabled participants - Blind people - Deaf people - Hard-of-hearing people - Research team - Lua -- lead researcher, has dyslexia - Jay -- project team member, Deaf person who uses sign language. - El -- project team member, non-disabled --- # Design Study Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities. - Who are the stakeholders who you need to take into account for this study? - What is the right structure for the study? --- # Design Study (1 of 4) This is a usability study, so it should include the same tasks as tested for users without disabilities Metrics should be similar as well -- for example the [System Usability Scale](https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html) might be useful at the end --- # Design Study (2 of 4) This is also a study of accessibility, with disabled users. The study design needs to consider .left-column50[ - Choice of method - e.g. think aloud may be more difficult for a nonspeaking individual, or someone with fatigue - Check in on study measures and statistical design. Does user heterogeneity impact this? - Study venue (It may be difficult to duplicate participant hardware/software) ] .right-column50[ - Ensure that participants' hardware/software also addresses study needs (e.g. do participants have a working monitor?) - Study length (estimate before and after piloting) - Study length may increase for some disabled participnats - Study length may negatively impact participants with fatigue-related concerns ] --- # Design Study (3 of 4) Study metrics may also need revision. .left-column50[ Consider adding - Click errors - Success criteria & ratings - Accessibility errors ] .left-column50[ Consider changing - Any metrics that include abelist assumptions - Approach to time based measurements ] --- # Design Study (4 of 4) It is also important to consider ethical issues .left-column60[ - What responsibility do you have to participants in terms of providing skills and help in & out of study? - What responsibility do you have to give participants a participatory role in design? - How do you ensure the integrity of your record of their experience? - Are you compensating them appropriately and addressing costly transportation needs? ] .right-column40[ These changes may in turn impact - Expense - Scalability ] --- # Plan Study Tasks Supposed you are hired to perform usability testing of new food-delivery app with people with sensory disabilities. - Who are the stakeholders who you need to take into account for this study? - What is the right structure for the study? - What study tasks should this team prepare for? --- # Tasks List To perform standard usability testing on the app based on disabilities represented in sample 1. Assign Tasks 2. Train study team 3. Handle recruitment 4. Pilot with people with disabilities 5. Run the Study - Prepare study materials (e.g., print out protocols) - Hire access support staff - Participate in the study - Compensate participants 6. Analyze the data --- --- # 1. Assign Tasks - Consider: - access needs - effects of familiarity with the task - personal preferences - other institutional constraints - For this scenario: - Lua has significant experience running studies; Lua needs a study script in large print with a specific font - El wants to gain more experience running studies - Jay has significant experience with ASL and Deaf Space; Jay prefers to have interpreters --- # Task Assignment List 1. ~~Assign Tasks~~ 2. **Lua**: Train study team 3. **Jay**: Handle recruitment 4. **Jay**: Pilot with people with disabilities 5. **Lua**: Run the Study - **Lua**: Prepare study materials (e.g., print out protocols) - **Jay**: Hire access support staff - **Participants**: Participate in the study - **El**: Compensate participants 6. **All**: Analyze the data --- # 2. Train study team (1 of 2) As study lead, and a person with both disability and study related experience, Lua will train the study team. Her goal is to ensure that El and Jay are aware of best practices and considerations that might impact study design before they begin interacting with participants. --- # 2. Train study team (2 of 2) Some things they might discuss include - Ensuring that they address access needs and communication support from recruitment onward - Providing basic DEIA training as needed so that all those with participant contact know basics. - Don’t be overly sensitive (don’t overcompensate) - Don’t rely on useless cues (audio/visual) to convey encouragement - Monitor participant fatigue carefully - When to help and when to end a task if participants have trouble --- # 3. Handle Recruitment As a person already connected to the disability community, the team determins that Jay will be most likely to be trusted, and to identify opportunities to build trust throughout the recruitment process Lots of details here - Are your recruitment materials accessible - How will you find participants? - How much do you know about the group your are targeting and what to expect from/of them? - Will they see you as trustworthy? - Are you giving them enough agency in the study process? --- # The Importance of Gatekeepers Negotiating access often requires working with gatekeepers Participants may see researchers as “just another professional, conducting professional surveillance” --- # 4. Pilot study First, plan accommodations .left-column50[ Consider: - Communication - Materials - Time - Space ] -- .right-column50[ Prepare: - Hire interpreters (at least 2 weeks in advance) - Print out accessible study materials, consent form, etc - Set up the room ] --- # Reflect - Lua gets tired from reading overtime - Jay knows ASL and Deaf norms - Study length was problematic for one Blind participant who also had a chronic illness - Sometimes sightlines made it hard for Jay to see the interpreter --- # Update Study Design Based on Reflection - Institute a protocol for monitoring fatigue - Add a cutoff for tasks that take a long time - Rearrange the room to better support communication with interpreters - ... --- # Update Tasks List Based on Reflection 1. ~~Assign Tasks~~ 2. ~~Lua: Train study team~~ (done) 3. **Jay**: Handle recruitment (ongoing) 4. ~~Jay: Pilot with people with disabilities~~ (done) 5. **Jay & El**: Run the Study - **El**: Prepare study materials (e.g., print out protocols) - **Jay**: Hire access support staff - **Participants**: Participate in the study - **El**: Compensate participants 6. **All**: Analyze the data --- # 5. Run the Study At this point the team should be well prepared. However, it is still important to monitor both the team's and the participants' needs and adjust as accessibility concerns arise. --- # 6. Analyze the Data This is another area where access concerns may come up. - Data analysis tools may not be accessible - Team members may have different needs with respect to written versus audio transcripts - <q>Member Checking</q> --- # Try it You read about Alexa's use by people with disabilities in class. Form small groups - Identify potential tasks to test - Write down a sampling goal (who is included) - Write down a list of accessibility assessment goals (metrics) - Identify potential accessibility concerns Post on [Ed](https://edstem.org/us/courses/31170/discussion/2427390)
