Running a Quantitative Study

name: inverse
layout: true
class: center, middle, inverse
---
# Running a Quantitative Study

Lauren Bricker

CSE 340 Winter 2020
---
layout:false

[//]: # (Outline Slide)
.title[Today's goals]
.body[
- Discuss steps of running a study
- Practice onboarding participants
- Practice data analysis
- Go over part 4-5 of Menus
]
---
# Experiment Design
.left-column[
<div class="mermaid" style="font-size:.5em">
graph TD
S(.) --> Hypothesis(Hypothesis)
Hypothesis -- "Study Design" --> Method(Method)
Method -- "Run Study" --> Data(Data)
Data -- "Clean and Prep" --> Analysis(Analysis)
Analysis --> Conclusions(Conclusions)

linkStyle 0 stroke-width:3px;
linkStyle 1 stroke-width:3px;
linkStyle 2 stroke-width:3px;
linkStyle 3 stroke-width:3px;
linkStyle 4 stroke-width:3px;

class S invisible
class Hypothesis,Conclusions start
class Method,Data,Analysis normal
</div>
]

.right-column[

What is the Hypothesis for the Menus assignment?
]
---
# Method
.left-column[
<div class="mermaid" style="font-size:.5em">
graph TD
S(.) --> Hypothesis(Hypothesis:<br>Decreased seek <br>time and errors)
Hypothesis -- "Study Design" --> Method(Method)
Method -- "Run Study" --> Data(Data)
Data -- "Clean and Prep" --> Analysis(Analysis)
Analysis --> Conclusions(Conclusions)

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em;
classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em;
classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:3px;
linkStyle 1 stroke-width:3px;
linkStyle 2 stroke-width:3px;
linkStyle 3 stroke-width:3px;
linkStyle 4 stroke-width:3px;

class S invisible
class Hypothesis start
class Conclusions startsmall
class Method,Data,Analysis normal
</div>
]

.right-column[

- 3 tasks x 3 menu types = 9 *conditions*
- Each condition will have a total of totalTrials = `ITEM_MAX` * `NUM_REPEATS`
  - In each *condition* we test `ITEM_MAX` different menu items
  - For each menu item, we repeat `NUM_REPEATS` times

---
.left-column[
## Other Method considerations
]

.right-column[
For Menus Part 4-5, an experimental *session* consists of 3 tasks x 3 menu types x
`ITEM_MAX` items x `NUM_REPEATS` repetitions = 108 *trials*

You have to run five participants through a complete session = 108 * 5
or 540 data points.

--
Participants do *all* trials (some designs participants only do some conditions)
- In some experiments participants do only some conditions, called *between subjects design*
- Our study is a *within subjects design*

Order of presentation of conditions and items is randomized (why?)
]

???
Between subjects design: Person A compared to person B doing different tasks

---
.left-column[
## Document all of this in your [report](/courses/cse340/20wi/assignments/menu-report)
]

.right-column[
Introduce study purpose

`Write two sentences describing the purpose of the experiment. This
can be the same text you use in your consent form`

Introduce study method - tasks

`Describe the 9 conditions of the study. Explain how many
items were selected per menu, and how many times each item was
repeated. Describe how many trials each participant completed. This
should be at most one paragraph`
]
---
# Study Ethics
.left-column[
<div class="mermaid" style="font-size:.5em">
graph TD
S(.) --> Hypothesis(Hypothesis:<br>Decreased seek <br>time and errors)
Hypothesis -- "Study Design" --> Method(3 menus x <br> 3 task conditions )
Method -- "Run Study" --> Data(Data)
Data -- "Clean and Prep" --> Analysis(Analysis)
Analysis --> Conclusions(Conclusions)

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em;
classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em;
classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em;
classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:3px;
linkStyle 1 stroke-width:3px;
linkStyle 2 stroke-width:3px;
linkStyle 3 stroke-width:3px;
linkStyle 4 stroke-width:3px;

class S invisible
class Hypothesis start
class Conclusions startsmall
class Method normalbig
class Data,Analysis normal
</div>

]

.right-column[
Ethical Principles for running participants. Driven by
[Criminal/Racist/Harmful studies](https://www.nytimes.com/2017/05/22/science/social-science-research-institutional-review-boards-common-rule.html)
 - Nazi war crimes
 - Tuskegee Syphilis study
 - Epilepsy studies of institutionalized children
 - [16,000 people involuntarily included in radiation
   studies](https://www.nytimes.com/1995/08/20/us/count-of-subjects-in-radiation-experiments-is-raised-to-16000.html?module=inline)
 - [Milgram's study of electric shocking](https://www.simplypsychology.org/milgram.html)
 - [Stanford prison experiment](https://www.simplypsychology.org/zimbardo.html)
]

???
IRB = Institutional Review Board Protocol (and get it approved.)

---
# Study Ethics
.left-column[
<div class="mermaid" style="font-size:.5em">
graph TD
S(.) --> Hypothesis(Hypothesis:<br>Decreased seek <br>time and errors)
Hypothesis -- "Study Design" --> Method(3 menus x <br> 3 task conditions )
Method -- "Run Study" --> Data(Data)
Data -- "Clean and Prep" --> Analysis(Analysis)
Analysis --> Conclusions(Conclusions)

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em;
classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em;
classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em;
classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:3px;
linkStyle 1 stroke-width:3px;
linkStyle 2 stroke-width:3px;
linkStyle 3 stroke-width:3px;
linkStyle 4 stroke-width:3px;

class S invisible
class Hypothesis start
class Conclusions startsmall
class Method normalbig
class Data,Analysis normal
</div>
]

.right-column[

Basic ethics ([Belmont Report](https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/read-the-belmont-report/index.html))
- Beneficence -->
 - Value of research higher than risks
 - Do no harm
- Respect for Persons -->
 - Fully informed of intent and purpose
 - Informed consent
 - May opt out at any time, for any reason
- Justice
 - Equitable, representative selection of participants
]

---
.left-column[
## Consent
]

.right-column[
Write your [consent](/courses/cse340/20wi/assignments/consent) form

- Purpose of study (Beneficience)
- Requirements for participation (Respect for Persons)
- Study procedures (Respect for Persons)
- Voluntariness (Respect for Persons)
- Benefits to Society (Beneficience)
- Contact (of IRB typically; Me in this case)
]

???
- Beneficence -->
 - Value of research higher than risks
 - Do no harm
- Respect for Persons -->
 - Fully informed of intent and purpose
 - Informed consent
 - May opt out at any time, for any reason
- Justice
 - equitable, representative selection of participants

---
.left-column[
## Document what all of this in your [report](/courses/cse340/20wi/assignments/menu-report)
]

.right-column[
Method

- Participants

`Describe your participants (without identifying
them). How were they recruited? How many were there?  Were
they consented? You can also add
some optional information such as: What was there average age? What
genders were present? How experienced were they with android?`
]
---
# Data Collection
.left-column[
<div class="mermaid" style="font-size:.5em">
graph TD
S(.) --> Hypothesis(Hypothesis:<br>Decreased seek <br>time and errors)
Hypothesis -- "Study Design" --> Method(3 menus x <br> 3 task conditions )
Method -- "Run Study" --> Data(Data)
Data -- "Clean and Prep" --> Analysis(Analysis)
Analysis --> Conclusions(Conclusions)

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:2.5em;
classDef normalbig fill:#e6f3ff,stroke:#333,stroke-width:2px,font-size:.7em,height:4em;
classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:5em;
classDef startsmall fill:#d1e0e0,stroke:#333,stroke-width:4px,font-size:.7em,height:2.5em;
classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:3px;
linkStyle 1 stroke-width:3px;
linkStyle 2 stroke-width:3px;
linkStyle 3 stroke-width:3px;
linkStyle 4 stroke-width:3px;

class S invisible
class Hypothesis start
class Conclusions startsmall
class Method normalbig
class Data,Analysis normal
</div>

]

.right-column[
1. **Clear your data** file before you start the **first participant
only**
2. Have participant read and sign the consent form
3. Emphasize key points verbally
4. Be Consistent in how you present the study
5. Download result
 - you can use a tool window called `Device File Manager`.
]

???
Make sure they know it is a **voluntary** study

---
layout: true
class: center, middle
---
# Collecting data file (Demo)
---
layout: false

.left-column[
## Document what all of this in your [report](/courses/cse340/20wi/assignments/menu-report)
]

.right-column[
Method - Setting
`What device was used? Was it an emulator? Where did the experiment take place?`

Method - Data Collected
`What information was collected (time, errors, etc)`
]

---
# Data Collection
.left-column[

![:img Picture of a dialogue box called Import file showing that you should replace current sheet and automatically detect separator type and convert text to numbers dates and formulas,100%](img/studies/import.png)
]

.right-column[

Select the 'raw' sheet of your spreadsheet then load your file into the spreadsheet

![:img Picture of a spreadsheet with the tab titled 'Raw' selected](img/studies/raw.png)

]
---
# Data Collection
.left-column[

![:img Image of bar chart comparing tasks to menu type showing that
normal menus get progressively slower as items become nonlinear while
pie menus are about the same, 100%](img/studies/chart.png)
]
.right-column[

Now click on `Example Chart`. Here you can

- Analyze and chart data: Simple Statistics
  - Min, Max, Mean (Sum/#), Median (Middle #), Mode (Most Common #)

Demo

Do this for speed *and* error.

]
---
.left-column[
## Document what all of this in your [report](/courses/cse340/20wi/assignments/menu-report)
]
.right-column[
Speed Results

`Describe your thoughts about overall speed in different
conditions. Use at least one chart to illustrate what you say. Here is
an example chart generated using our data, when you paste your data
into the spreadsheet you’ll see that it updates to reflect your data`

Error Results

`Describe what happened in terms of errors -- provide at least one chart showing
what you learned about errors in different conditions`
]
---
# Can we determine *causality*?

Implies *dependence* between speed/error, task, and menu type

Assumes you have measured the right variables!

---
# Dependence/independence

Events that are independent
- Flipping heads and then tails
- Day of week and whether a patient had a heart attack (probably?)

Events that are dependent
- Vice presidential candidate and presidential nominee
- Diagnostic test being positive and whether patient has a disease

---
# Correlation

![:img Example of correlations for heigh vs age (positively correlated) and
height vs birth month (uncorrelated),80%](img/studies/correlation.png)

---
# Correlation != Causation

![:img Correlation of number of people who drowned per year and films
nicolas cage appeared in, 80%](img/studies/cagelation.png)

---
# Correlation != Causation

![:img Two people talking. Says one: I used to think correlation
implied causation. Says the other: Then I took a statistics
course. Now I don't. Says the other: Sounds like the class
helped. Says the first: Well maybe, 50%](img/studies/correlation-cartoon.png)

---
# Charting helps you check your assumptions

![:img Image of bar chart comparing tasks to menu type showing that
normal menus get progressively slower as items become nonlinear while
pie menus are about the same, 50%](img/studies/chart.png)

---
# But having the right chart matters

![:img Histogram of data about speed of pie menu selection and linear
menu selection for each task type, 40%](img/studies/histogram.png)

---
# Normal Vs Pie

|Normal | Pie|
|--|--|
|![:img Histogram of data about speed of pie menu selection and linear menu selection for each task type,100%](img/studies/normal-only.png)|![:img Histogram of data about speed of pie menu selection and linear menu selection for each task type, 100%](img/studies/pie-only.png)|

---
# Comparing two groups to see if they are different

![:img two distributions, 50%](img/studies/samples1.png)

---
# Comparing two groups

![:img two distributions showing overlapping 95% confidence intervals,
50%](img/studies/samples2.png)