Running a Quantitative Study

name: inverse
layout: true
class: center, middle, inverse
---
# Running a Quantitative Study

Lauren Bricker

CSE 340 Spring 2020
---
layout: false

# Today's goals

- Review our designs from Friday
- Discuss steps of running a study
- Practice onboarding participants
- Practice data analysis
- Go over parts 5 & 6 of Menus

---
# Case Study: UbiGreen - In Class activity

Take 2 minutes to go back into our [shared google doc](https://docs.google.com/document/d/1SqvaQyhFYs6AsOPoDcBkG4egCRRq3jEaW17HOi0hSkM/edit?usp=sharing)

Find a design by a different group than you were in. Use the google doc comment feature to leave
a short critique.

Phrases to get you started

- I like ...
- I wish ...
- What if...

.footnote[I like I wish critique method from [Stanford Design School](https://www.youtube.com/watch?v=QkWM2--3TQo)]

---
# Experiment Design

<div class="mermaid">
graph LR
S(.) --> Hypothesis(Hypothesis)
Hypothesis -- "Study Design" --> Method(Method)
Method -- "Run Study" --> Data(Data)
Data -- "Clean and Prep" --> Analysis(Analysis)
Analysis --> Conclusions(Conclusions)

linkStyle 0 stroke-width:3px;
linkStyle 1 stroke-width:3px;
linkStyle 2 stroke-width:3px;
linkStyle 3 stroke-width:3px;
linkStyle 4 stroke-width:3px;

class S invisible
class Hypothesis,Conclusions start
class Method,Data,Analysis normal
</div>

‍Think: What is the Hypothesis for the Menus assignment
--

‍Pair: We'd chat for a minute in person but I'm not putting you in breakouts for this question...
--

‍Share: Type your thoughts in the chat window

---
# Method
<div class="mermaid">
graph LR
S(.) --> Hypothesis(Hypothesis <br>Decreased seek <br>time and errors)
Hypothesis -- "Study Design" --> Method(Method)
Method -- "Run Study" --> Data(Data)
Data -- "Clean and Prep" --> Analysis(Analysis)
Analysis --> Conclusions(Conclusions)

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em;
classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em;
classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:3px;
linkStyle 1 stroke-width:3px;
linkStyle 2 stroke-width:3px;
linkStyle 3 stroke-width:3px;
linkStyle 4 stroke-width:3px;

class S invisible
class Hypothesis start
class Conclusions startsmall
class Method,Data,Analysis normal
</div>

- 3 tasks x 3 menu types = 9 *conditions*
- Each condition will have a total of totalTrials = `ITEM_MAX` x `NUM_REPEATS`
  - In each *condition* we test `ITEM_MAX` different menu items
  - For each menu item, we repeat `NUM_REPEATS` times

---
# Other Method considerations

For Menus Part 5-6, an experimental *session* consists of 3 tasks x 3 menu types x
`ITEM_MAX` items x `NUM_REPEATS` repetitions = 108 *trials*

You have to run at least three participants through a complete session = 108 x 3 or 324 data points.

--
In some experimental designs, participants only do some conditions
- Called *between subjects design*

Our participants do *all* trials
- Our study is a *within subjects design*

Order of presentation of conditions and items is randomized (why?)

???
Between subjects design: Person A compared to person B doing different tasks

---
# Document all of this in your [report](/courses/cse340/20sp/assignments/menu-report)

Introduce study purpose

`Write two sentences describing the purpose of the experiment. This
can be the same text you use in your consent form`

---
# Document all of this in your [report](/courses/cse340/20sp/assignments/menu-report)

Introduce study method - menus

`Mention that there are three types of menus, Pie, Linear and Custom.
Then describe your custom menu and include an screenshots of your custom menu in both a selected
and unselected state. Describe some of the design choices you made when you were conceiving your
custom menu and how your final product match (or didn’t) your original vision? Be sure to explain
clearly how your custom menu works and how a user interacts with it?`

Introduce study method - tasks

`Describe the 9 conditions of the study. Explain how many
items were selected per menu, and how many times each item was
repeated. Describe how many trials each participant completed.`

---
# Study Ethics
<div class="mermaid">
graph LR
S(.) --> Hypothesis(Hypothesis:<br>Decreased seek <br>time and errors)
Hypothesis -- "Study Design" --> Method(3 menus x <br> 3 task conditions )
Method -- "Run Study" --> Data(Data)
Data -- "Clean and Prep" --> Analysis(Analysis)
Analysis --> Conclusions(Conclusions)

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em;
classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em;
classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em;
classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:3px;
linkStyle 1 stroke-width:3px;
linkStyle 2 stroke-width:3px;
linkStyle 3 stroke-width:3px;
linkStyle 4 stroke-width:3px;

class S invisible
class Hypothesis start
class Conclusions startsmall
class Method normalbig
class Data,Analysis normal
</div>

Ethical Principles for running participants. Driven by
[Criminal/Racist/Harmful studies](https://www.nytimes.com/2017/05/22/science/social-science-research-institutional-review-boards-common-rule.html)
 - Nazi war crimes
 - Tuskegee Syphilis study
 - Epilepsy studies of institutionalized children
 - [16,000 people involuntarily included in radiation
   studies](https://www.nytimes.com/1995/08/20/us/count-of-subjects-in-radiation-experiments-is-raised-to-16000.html?module=inline)
 - [Milgram's study of electric shocking](https://www.simplypsychology.org/milgram.html)
 - [Stanford prison experiment](https://www.simplypsychology.org/zimbardo.html)

???
IRB = Institutional Review Board Protocol (and get it approved.)

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em;
classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em;
classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em;
classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:3px;
linkStyle 1 stroke-width:3px;
linkStyle 2 stroke-width:3px;
linkStyle 3 stroke-width:3px;
linkStyle 4 stroke-width:3px;

class S invisible
class Hypothesis start
class Conclusions startsmall
class Method normalbig
class Data,Analysis normal
</div>

Basic ethics ([Belmont Report](https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/read-the-belmont-report/index.html))
- Beneficence -->
 - Value of research higher than risks
 - Do no harm
- Respect for Persons -->
 - Fully informed of intent and purpose
 - Informed consent
 - May opt out at any time, for any reason
- Justice -->
 - Equitable, representative selection of participants

---
# Consent

Write your [consent](/courses/cse340/20sp/assignments/consent) form

- Purpose of study (Beneficience)
- Requirements for participation (Respect for Persons)
- Study procedures (Respect for Persons)
- Voluntariness (Respect for Persons)
- Benefits to Society (Beneficience)
- Contact (of IRB typically; Me in this case)

???
- Beneficence -->
 - Value of research higher than risks
 - Do no harm
- Respect for Persons -->
 - Fully informed of intent and purpose
 - Informed consent
 - May opt out at any time, for any reason
- Justice
 - equitable, representative selection of participants

---
# Choosing and Consenting Participants

For 20sp only, we are assigning you in "groups" to help you find testers.
- If you can not safely test your app with someone who is co-present you will need to ask your [groupmates](https://docs.google.com/document/d/11nsEWs3TubV5Zy0zOIGguqZJohkS34uizIrDv-bgnYs/edit)
- Consenting your participants will be very similar whether they are co-present or remote.
  - Set up a time when can speak to your participant (in real time) - phone or video call
  - For a co-present tester - print out two copies of the consent form for each participant -- one for them and one for you.
  - For remote testers - send a copy of the consent form to the participant prior to your phone call
or video meeting.

---
# Choosing and Consenting Participants

During the consent meeting
- Briefly explain what your user study is about,
- Ensure participants understand their participation in the study is voluntary.
  - Do not *coerce* anyone into participating in your study.
  - Make sure they know they have a choice, and have read the consent form.
- Participants must acknowledge their consent by "signing" via the Google form (linked
at the bottom of the consent form template).
  - This form should send you an anonymized copy of their
consent in email, which you can save and turn in with the rest of your report and reflection.

---
# Choosing and Consenting Participants

If you are unable to find 3 partipants from your friends, family, or pre-assigned group
- Please reach out to a TA during Office Hours or to have them be a partipipant.
- You may also reach out to other students in the class via Ed.

---
# Document your participants in your [report](/courses/cse340/20sp/assignments/menu-report)

Method - Participants

`Describe your participants (without identifying
them). How were they recruited? How many were there?  Were
they consented? You can also add
some optional information such as: What was there average age? What
genders were present? How experienced were they with android?`

---
# Data Collection

<div class="mermaid">
  graph LR
  S(.) --> Hypothesis(Hypothesis:<br>Decreased seek <br>time and errors)
  Hypothesis -- "Study Design" --> Method(3 menus x <br> 3 task conditions )
  Method -- "Run Study" --> Data(Data)
  Data -- "Clean and Prep" --> Analysis(Analysis)
  Analysis --> Conclusions(Conclusions)

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
  classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
  classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em;
  classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em;
  classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em;
  classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:3px;
  linkStyle 1 stroke-width:3px;
  linkStyle 2 stroke-width:3px;
  linkStyle 3 stroke-width:3px;
  linkStyle 4 stroke-width:3px;

class S invisible
  class Hypothesis start
  class Conclusions startsmall
  class Method normalbig
  class Data,Analysis normal
</div>

If you have co-present users:

1. **Clear your data** file before you start the **first participant
only**
2. Have participant read and sign the consent form
3. Emphasize key points verbally
4. Be Consistent in how you present the study
5. Download result (you can use a tool window called `Device File Manager`)

---
# Data Collection

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
  classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
  classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em;
  classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em;
  classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em;
  classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:3px;
  linkStyle 1 stroke-width:3px;
  linkStyle 2 stroke-width:3px;
  linkStyle 3 stroke-width:3px;
  linkStyle 4 stroke-width:3px;

class S invisible
  class Hypothesis start
  class Conclusions startsmall
  class Method normalbig
  class Data,Analysis normal
</div>

If you have a remote user

1. Generate your APK
2. Consent your user via phone or video conference.
3. Send your APK to the participant so they can load it onto their own device or into their own
 emulator
  - You can't email your APK through UW's servers, so upload the APK to your Google Drive or One Drive and send your participant
 a link to download it for testing.
4. Have your participant use the hamburger menu to select `Clear Result CSV` before starting
 your study
5. Have your participant run one experiment session.
6. When the study is done, your participant will need to download their data
and send it back for analysis.
7. Combine the three participants' data into one common .csv file.

---
layout: true
class: center, middle
---
# Collecting data file (Demo)
---
layout: false

# Document what all of this in your [report](/courses/cse340/20sp/assignments/menu-report)

Method - Setting
`What device was used? Was it an emulator? Where did the experiment take place?`

Method - Data Collected
`What information was collected (time, errors, etc)`

---
# Data Collection
.left-column[

![:img Picture of a dialogue box called Import file showing that you should replace current sheet and automatically detect separator type and convert text to numbers dates and formulas,100%](img/studies/import.png)
]

.right-column[

Select the 'raw' sheet of your spreadsheet then load your file into the spreadsheet

![:img Picture of a spreadsheet with the tab titled 'Raw' selected](img/studies/raw.png)

]
---
# Data Collection
.left-column[

![:img Image of bar chart comparing tasks to menu type showing that
normal menus get progressively slower as items become nonlinear while
pie menus are about the same, 100%](img/studies/chart.png)
]
.right-column[

Now click on `Example Chart`. Here you can

- Analyze and chart data: Simple Statistics
  - Min, Max, Mean (Sum/#), Median (Middle #), Mode (Most Common #)

Demo

Do this for speed *and* error.

]

---
# Document what all of this in your [report](/courses/cse340/20sp/assignments/menu-report)

Speed Results

`Describe your thoughts about overall speed in different
conditions. Use at least one chart to illustrate what you say. Here is
an example chart generated using our data, when you paste your data
into the spreadsheet you’ll see that it updates to reflect your data`

Error Results

`Describe what happened in terms of errors -- provide at least one chart showing
what you learned about errors in different conditions`