name: inverse layout: true class: center, middle, inverse --- # Running a Quantitative Study Lauren Bricker CSE 340 Spring 2021 --- layout: false # Today's Agenda - Administrivia - Menus part 1-4 due Thursday 10pm - IMPORTANT change in the code posted to [Ed](https://edstem.org/us/courses/4985/discussion/445177). - Learning goals - Designing the Menus experiment - Practice data analysis - Go over parts 5 & 6 of Menus - Practice Quiz 7 [review](wk8review.html) (if time) --- # Code.org Problem Solving Process .left-column40[ ![:img Code.org problem solving process, 70%, width](https://images.code.org/4684a4cd531d3750d2a11051f6e0729f-image-1493396308445.png)
] .right-column60[ - **Define** - What problem are you trying to solve? - What resources, priorities, and constraints do you need to consider? - What does success look like? - **Prepare** - Brainstorm / research possible solutions - Compare pros and cons - Make a plan - **Try** - Put your plan into action - **Reflect** - How do your results compare to the goals you set while defining the problem? - What can you learn from this or do better next time? - What new problems have you discovered? ] --- # You are not the user .left-column40[ ![:img Code.org problem solving process, 70%, width](https://images.code.org/4684a4cd531d3750d2a11051f6e0729f-image-1493396308445.png)
![:img Code.org problem solving process with empathy,70%, width](https://code.org/curriculum/docs/csd/PSPE.png) ] .right-column60[ User testing is typically done at all stages of a software development cycle - **Define**: Understanding the user, task, and constraints through Task Analysis, Contextual Inquiry, Ethnographic studies - **Prepare**: Develop use case scenarios and prototype with storyboards then test the prototypes (Participatory Design) - **Try**: Continuous involvement by end users as you develop) - **Reflect**: User Studies done on the current version of the product. ] ??? Find and fix problems in a design (from 440) - Removes the expert blind spot - Obtain data to unify team around changes - Uncover unexpected behaviors Saves money in the long run --- # Types of User Testing .left-column40[ Was this A/B Testing? ![:img Facebook interface with pink like buttons, 90%, width](img/studies/pinklike.png)
![:img Facebook interface with blue like buttons, 130%, width](img/studies/pinklike-abtesting.png) ] .right-column60[ - Qualitative or quantitative usability testing - In-person - Remote (Conference-based, Semi-automatic, Controlled A/B testing) - Wizard-of-Oz testing - Diary Studies - Accessibility evaluation - Surveys - Usability-bug review - Frequently-asked-questions (FAQ) review ] .footnote[ Nielsen Norman Group [UX Research Cheat Sheet](https://www.nngroup.com/articles/ux-research-cheat-sheet/) ] ??? Not a scientific experiment but focus is on improving the design --- # Experiment Design
graph LR S(.) --> Hypothesis(Hypothesis) Hypothesis -- "Study Design" --> Method(Method) Method -- "Run Study" --> Data(Data) Data -- "Clean and Prep" --> Analysis(Analysis) Analysis --> Conclusions(Conclusions) classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em; classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF linkStyle 0 stroke-width:3px; linkStyle 1 stroke-width:3px; linkStyle 2 stroke-width:3px; linkStyle 3 stroke-width:3px; linkStyle 4 stroke-width:3px; class S invisible class Hypothesis,Conclusions start class Method,Data,Analysis normal
Think: What is the Hypothesis for the Menus assignment -- Pair: We'd chat for a minute in person but I'm not putting you in breakouts for this question... -- Share: Type your thoughts in the chat window --- # Method
graph LR S(.) --> Hypothesis(Hypothesis
Decreased seek
time and errors) Hypothesis -- "Study Design" --> Method(Method) Method -- "Run Study" --> Data(Data) Data -- "Clean and Prep" --> Analysis(Analysis) Analysis --> Conclusions(Conclusions) classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em; classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em; classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF linkStyle 0 stroke-width:3px; linkStyle 1 stroke-width:3px; linkStyle 2 stroke-width:3px; linkStyle 3 stroke-width:3px; linkStyle 4 stroke-width:3px; class S invisible class Hypothesis start class Conclusions startsmall class Method,Data,Analysis normal
- 3 tasks x 3 menu types = 9 *conditions* - Each condition will have a total of totalTrials = `ITEM_MAX` x `NUM_REPEATS` - In each *condition* we test `ITEM_MAX` different menu items - For each menu item, we repeat `NUM_REPEATS` times | | Normal | Pie | Custom | |--|--|--|--| | **Linear** | totalTrials | totalTrials | totalTrials | | **Relational** | totalTrials | totalTrials | totalTrials | | **Unclassified** | totalTrials | totalTrials | totalTrials | --- # Other Method considerations For Menus Part 5-6, an experimental *session* consists of 3 tasks x 3 menu types x `ITEM_MAX` items x `NUM_REPEATS` repetitions = 108 *trials* You have to run at least three participants through a complete session = 108 x 3 or 324 data points. -- In some experimental designs, participants only do some conditions - Called *between subjects design* Our participants do *all* trials - Our study is a *within subjects design* Order of presentation of conditions and items is randomized (why?) ??? Between subjects design: Person A compared to person B doing different tasks --- # Document all of this in your [report](/courses/cse340/21sp/assignments/menu-report) **Introduce study purpose** *Write two sentences describing the purpose of the experiment. This can be the same text you use in your [consent form](/courses/cse340/21sp/assignments/consent) under `Introduction and Purpose of study (Beneficience)`* --- # Document all of this in your [report](/courses/cse340/21sp/assignments/menu-report) **Introduce study method** **Menus** *Mention that there are three types of menus: Pie, Linear and Custom. Then include the following:* - *The design your custom menu by including your sketches and a clear explanation of how you expect a user to interact with your custom menu. Also include the description and calculations for your essential geometry.* - *Describe some of the design choices you made when you were conceiving your custom menu, and how you expected it would perform against Normal and Pie menus* - *Include the screenshots of your custom menu in both a selected and unselected state.* - *Discuss how your final product matches/doesn't match your original vision. * --- # Document all of this in your [report](/courses/cse340/21sp/assignments/menu-report) **Introduce study method** **Tasks** *Describe the 9 conditions of the study. Explain how many items were selected per menu, and how many times each item was repeated. Describe how many trials each participant completed.* **Setting** *What device was used? Was it an emulator? Did they use a mouse or a finger? Where did the experiment take place?* --- # Study Ethics
graph LR S(.) --> Hypothesis(Hypothesis:
Decreased seek
time and errors) Hypothesis -- "Study Design" --> Method(3 menus x
3 task conditions ) Method -- "Run Study" --> Data(Data) Data -- "Clean and Prep" --> Analysis(Analysis) Analysis --> Conclusions(Conclusions) classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em; classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em; classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em; classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF linkStyle 0 stroke-width:3px; linkStyle 1 stroke-width:3px; linkStyle 2 stroke-width:3px; linkStyle 3 stroke-width:3px; linkStyle 4 stroke-width:3px; class S invisible class Hypothesis start class Conclusions startsmall class Method normalbig class Data,Analysis normal
Ethical Principles for running participants. - Why are they needed? What examples have you heard about? --- # Study Ethics
graph LR S(.) --> Hypothesis(Hypothesis:
Decreased seek
time and errors) Hypothesis -- "Study Design" --> Method(3 menus x
3 task conditions ) Method -- "Run Study" --> Data(Data) Data -- "Clean and Prep" --> Analysis(Analysis) Analysis --> Conclusions(Conclusions) classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em; classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em; classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em; classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF linkStyle 0 stroke-width:3px; linkStyle 1 stroke-width:3px; linkStyle 2 stroke-width:3px; linkStyle 3 stroke-width:3px; linkStyle 4 stroke-width:3px; class S invisible class Hypothesis start class Conclusions startsmall class Method normalbig class Data,Analysis normal
Ethical Principles for running participants. - Federal Policy for the Protection of Human Subjects (known as the Common Rule ) published in 1991 - Driven by [Criminal/Racist/Harmful studies](https://www.nytimes.com/2017/05/22/science/social-science-research-institutional-review-boards-common-rule.html) - Nazi war crimes - Tuskegee Syphilis study - Epilepsy studies of institutionalized children - [16,000 people involuntarily included in radiation studies](https://www.nytimes.com/1995/08/20/us/count-of-subjects-in-radiation-experiments-is-raised-to-16000.html?module=inline) - [Milgram's study of electric shocking](https://www.simplypsychology.org/milgram.html) - [Stanford prison experiment](https://www.simplypsychology.org/zimbardo.html) ??? IRB = Institutional Review Board Protocol (and get it approved.) --- # Study Ethics
graph LR S(.) --> Hypothesis(Hypothesis:
Decreased seek
time and errors) Hypothesis -- "Study Design" --> Method(3 menus x
3 task conditions ) Method -- "Run Study" --> Data(Data) Data -- "Clean and Prep" --> Analysis(Analysis) Analysis --> Conclusions(Conclusions) classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em; classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em; classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em; classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF linkStyle 0 stroke-width:3px; linkStyle 1 stroke-width:3px; linkStyle 2 stroke-width:3px; linkStyle 3 stroke-width:3px; linkStyle 4 stroke-width:3px; class S invisible class Hypothesis start class Conclusions startsmall class Method normalbig class Data,Analysis normal
Basic ethics ([Belmont Report](https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/read-the-belmont-report/index.html), 1979) - Beneficence --> - Value of research higher than risks - Do no harm - Respect for Persons --> - Fully informed of intent and purpose - Informed consent - May opt out at any time, for any reason - Justice --> - Equitable, representative selection of participants --- # Consent Write your [consent](/courses/cse340/21sp/assignments/consent) form - Purpose of study (Beneficience) - Requirements for participation (Respect for Persons) - Study procedures (Respect for Persons) - Voluntariness (Respect for Persons) - Benefits to Society (Beneficience) - Contact (of IRB typically; Me in this case) ??? - Beneficence --> - Value of research higher than risks - Do no harm - Respect for Persons --> - Fully informed of intent and purpose - Informed consent - May opt out at any time, for any reason - Justice - equitable, representative selection of participants --- # Choosing and Consenting Participants For remote learning only, we are assigning you in "groups" to help you find testers. - If you can not safely test your app with someone who is co-present you will need to ask your [groupmates](https://docs.google.com/document/d/1SsMpSef4f5PIjnyyoYdcI93VoPdzWxRaMbe3XXAZuUs/edit?usp=sharing) - Consenting your participants will be very similar whether they are co-present or remote. - Set up a time when can speak to your participant (in real time) - phone or video call - For a co-present tester - print out two copies of the consent form for each participant -- one for them and one for you. - For remote testers - send a copy of the consent form to the participant prior to your phone call or video meeting. --- # Choosing and Consenting Participants During the consent meeting - Briefly explain what your user study is about. - Ensure participants understand their participation in the study is voluntary. - Do not *coerce* anyone into participating in your study. - Make sure they know they have a choice, and have read the consent form. - Participants must acknowledge their consent by "signing" via the Google form (linked at the bottom of the consent form template). - This form should send you an anonymized copy of their consent in email, which you can save and turn in with the rest of your report and reflection. --- # Choosing and Consenting Participants If you are unable to find 3 partipants from your friends, family, or pre-assigned group - Please reach out to a TA during Office Hours or to have them be a participant. - You may also reach out to other students in the class via Ed. --- # Document your participants in your [report](/courses/cse340/21sp/assignments/menu-report.html) **Introduce study method** **Participants:** *Describe your participants (without identifying them). How were they recruited? How many were there? Were they consented? You should also add some optional information such as: What was there average age? What genders were present? How experienced were they with android?* --- # Data Collection
graph LR S(.) --> Hypothesis(Hypothesis:
Decreased seek
time and errors) Hypothesis -- "Study Design" --> Method(3 menus x
3 task conditions ) Method -- "Run Study" --> Data(Data) Data -- "Clean and Prep" --> Analysis(Analysis) Analysis --> Conclusions(Conclusions) classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em; classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em; classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em; classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF linkStyle 0 stroke-width:3px; linkStyle 1 stroke-width:3px; linkStyle 2 stroke-width:3px; linkStyle 3 stroke-width:3px; linkStyle 4 stroke-width:3px; class S invisible class Hypothesis start class Conclusions startsmall class Method normalbig class Data,Analysis normal
If you have co-present users: 1. **Clear your data** file before you start the **first participant only** 2. Have participant read and sign the consent form 3. Emphasize key points verbally 4. Be consistent in how you present the study to all your users (it's helpful to write yourself a script) 5. Let them run through the session (all 108 trials) 5. Download result (use the `Device File Manager`) --- # Data Collection
graph LR S(.) --> Hypothesis(Hypothesis:
Decreased seek
time and errors) Hypothesis -- "Study Design" --> Method(3 menus x
3 task conditions ) Method -- "Run Study" --> Data(Data) Data -- "Clean and Prep" --> Analysis(Analysis) Analysis --> Conclusions(Conclusions) classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em; classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em; classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em; classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em; classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF linkStyle 0 stroke-width:3px; linkStyle 1 stroke-width:3px; linkStyle 2 stroke-width:3px; linkStyle 3 stroke-width:3px; linkStyle 4 stroke-width:3px; class S invisible class Hypothesis start class Conclusions startsmall class Method normalbig class Data,Analysis normal
If you have a remote user 1. Generate your APK 2. Consent your user via phone or video conference. 3. Send your APK to the participant so they can load it onto their own device or into their own emulator - You can't email your APK through UW's servers, so upload the APK to your Google Drive or One Drive and send your participant a link to download it for testing. 4. Have your participant use the hamburger menu to select `Clear Result CSV` before starting your study 5. Have your participant run one experiment session. 6. When the study is done, your participant will need to download their data and send it back for analysis. 7. Combine the three participants' data into one common .csv file. --- layout: true class: center, middle --- # Collecting data file (Demo) --- layout: false # Document what all of this in your [report](/courses/cse340/21sp/assignments/menu-report.html) **Introduce study method** **Data Collected** *What information was collected (time, errors, etc)* --- # Data Collection .left-column[ ![:img Picture of a dialogue box called Import file showing that you should replace current sheet and automatically detect separator type and convert text to numbers dates and formulas,100%, width](img/studies/import.png) ] .right-column[ Select the 'raw' sheet of your spreadsheet then load your file into the spreadsheet ![:img Picture of a spreadsheet with the tab titled 'Raw' selected, 100%, width](img/studies/raw.png) ] --- # Data Collection .left-column[ ![:img Image of bar chart comparing tasks to menu type showing that normal menus get progressively slower as items become nonlinear while pie menus are about the same, 100%, width](img/studies/chart.png) ] .right-column[ Now click on `Example Chart`. Here you can - Analyze and chart data: Simple Statistics - Min, Max, Mean (Sum/#), Median (Middle #), Mode (Most Common #) Demo Do this for speed *and* error. ] --- # Document what all of this in your [report](/courses/cse340/21sp/assignments/menu-report.html) **Speed Results** *Describe your thoughts about overall speed in different conditions. Use at least one chart to illustrate what you say. Describe the axes and explain why this graph was chosen. Here is an example chart generated using our data, when you paste your data into the spreadsheet you'll see that it updates to reflect your data. * **Error Results** *Describe what happened in terms of errors -- provide at least one chart showing what you learned about errors in different conditions. Describe the axes and explain why this graph was chosen. * --- # Other Ideas - How can you measure things *other* than what Fitts Law predicts: - How strenuous this was to the user - How easily they could recall what they did - How did they feel about the tasks they completed