Basic Toolkit Abstractions

name: inverse
layout: true
class: center, middle, inverse
---
# Core Toolkit Abstractions

Jennifer Mankoff

CSE 340 Winter 2020
---
name: normal
layout: true
class:
---
layout: false

| Hall of Shame | Hall of Fame|
|---|---|
|![:img Picture of a small sandcastle that is falling apart, 100%](img/toolkits/sandcastle1.png) | ![:img Picture of a complex sandcastle with LEDs lighting it up, 85%](img/toolkits/sandcastle2.jpg)
---
![:img Simple home tools for sandcastle making such as a bucket; funnel; brush; and spackle knife, 50%](img/toolkits/sandcastletools.png)
---
# Announcements
- We have Piazza.
  - Participation in Piazza careers is not required
  - It's best to join through the Canvas course. I've already added all your netids. Only UW netids are FERPA protected
- Prep for section
 - Android Basics: Follow [this](https://developer.android.com/training/basics/firstapp) tutorial to create an Android Application.
 - Continue with the tutorial to also [run your app](https://developer.android.com/training/basics/firstapp/running-app) either on an emulator or on an Android Device.
 - Git Setup: [macOS](https://courses.cs.washington.edu/courses/cse154/19au/resources/assets/atomgit/macosx/), [win](https://courses.cs.washington.edu/courses/cse154/19au/resources/assets/atomgit/windows/) (ignore pieces about Atom as we use Android Studio)
Make sure you have git setup on your computer, follow above instructions
 - If you have never used git before read through [this](http://cse340-20wi.pages.cs.washington.edu/website/docs/git.html#1) and [this](https://courses.cs.washington.edu/courses/cse331/18au/handouts/tools/versioncontrol.html) (Android Studio ≈ Intellij)
 - Ensure that you can clone from CSE GitLab, (we recommend cloning via SSH)

???
Our learning goals for the first lab are Android project structure, GitGrade and first assignment Doodle.
Please follow the instructions below to be familiar with Android and have your Git ready before this Thursday section:

---
# Announcements
- We have name tags. Willing to be part of an experiment?
 - Discussion of 'willing' (consent)
 - How do we tell if it works? What outcome measures matter?
 - Plan of action

???

Two groups
You are randomly assigned
It is optional. I will try to learn names whether or not you have cards.
two weeks of data collection
Then decide process

---
[//]: # (Outline Slide)
# Today's goals

- __Toolkits & Toolkit Users__
- Introduce the interactor hierarchy
- How does an interactor draw?

---
# What does it take to run an app on a phone?

![:youtube An android screen with many icons on it; then an app opens and the user can be seen drawing a line. When they finish an undo button appears, 2TJPjyMQGkM]

???

- Give them 60 seconds to think about it and talk with their neighbor.
- What you want to get at is the application stack

---
# The Application Stack
.left-column[
<div class="mermaid">
  graph LR
  ap[Application Program]
  hlt[High Level Tools]
  t[Toolkit]
  w[Window System]
  o[OS]
  h[Hardware]

class ap,o,w yellow
class hlt,t green
class h darkblue
</div>
]
.right-column[
![:img Picture of a woman in front of an open laptop holding a phone with an ipad next to the laptop; a wristwatch on her arm; and a keyfob to the left of the laptop, 50%](img/toolkits/hardware.jpg)
]

???
Get the class to brainstorm about what hardware is visible on the screen

Point out that they all have stacks and the things we are learning are not android specific

---
.left-column[
![:img Picture of the control buttons at the bottom of several different phones; with icons and order changing from phone to phone, 100%](img/toolkits/phones.jpeg)
]
.right-column[
## Android Hardware

![:img Picture of the Hardware Abstraction Layer for android including audio; automotive; bluetooth; camera; drm; graphics; input; media; peripherals; sensors; storage; and tv componnets, 70%](img/toolkits/ape_fwk_hal.png)]

???
Get the class to brainstorm about what hardware is available on android vs their laptop and how that might influence user interfaces

---
# The Application Stack
.left-column[
<div class="mermaid">
  graph LR
  ap[Application Program]
  hlt[High Level Tools]
  t[Toolkit]
  w[Window System]
  o[OS]
  h[Hardware]

class ap,h,w yellow
class hlt,t green
class o darkblue
</div>
]

.right-column[
What services does it provide?

How does this impact what you might build in an application?
]

???
Class brainstorming about what OS does
1) manages shared resources (e.g. memory, processor power)
2) has the power to start and pause and stop applicatinos

---
# Impact of Android OS on apps
.center[
![:img Android Activity Lifecycle which shows when the operating system starts; pauses and kills android processes, 25%](img/toolkits/activity_lifecycle.png)
]

???
How does the operating system influence what
you can build? How does this differ on an android device from a desktop?

The Android Activity Lifecycle starts when an activity is launched (by the OS); which leads to callbacks to the application such as onCreate, onStart and onResume. Once the activity is running, it may be paused, stopped or destroyed. Things like priority and available memory may influence this process, as does the user opening the app.

---
# The Application Stack
.left-column[
<div class="mermaid">
  graph LR
  ap[Application Program]
  hlt[High Level Tools]
  t[Toolkit]
  w[Window System]
  o[OS]
  h[Hardware]

class ap,o,h yellow
class hlt,t green
class w darkblue
</div>
]
.right-column[

- Manages window size and visibility across many applications
- May trigger clipping and redraw
  - Lots of cases where this arises on a desktop (overlapping windows, minimization, etc)
  - Toolkit handles this for you, but can lead to callbacks; redraw events

Where is this on Android?
]

???
---
# The Application Stack
.left-column[
<div class="mermaid">
  graph LR
  ap[Application Program]
  hlt[High Level Tools]
  t[Toolkit]
  w[Window System]
  o[OS]
  h[Hardware]

class ap,w,h,o yellow
class hlt,t green
class t darkblue
</div>
]

.right-column[
Services a toolkit supports (in the order we will learn them)
- Graphics
- Animation
- Layout
- Accessibility
- Event Handling
- Interactive Components
- Undo
- Sensing (not covered)
- Data (not covered)
 - databases; apis
]

???
What are some things you expect every interface to have that I didn't mention here?
Example: Undo (we will learn about this)

Do all user interface toolkits have all of these?

---
# What is an Interface Toolkit

(Main subject of this course)
- Difference between toolkit and IDE (Integrated Development Environment)?
- Difference between toolkit and API (Application Programming Interface)?

Structure of a Toolkit
- Library of components
- Architecture

???
Connect what they learned in 143 (i.e that they used the SDK for Java which had a bunch of libraries that had pre-defined code). The IDE most people used was JGrasp.

---
# Example: Doodle assignment: Makes use of the Android Toolkit

Inspired by Google's Doodles, and will include drawing animation (to be discussed): [Example from last year's class](img/drawing/doodlevid.mp4)

![:youtube Animation showing images of food moving in a line down the page around a U finally forming a W, Sx8oiJGjaIM]

---
.left-column-half[
## Doodle Assignment

Uses of Android Toolkit  **Library** (green)

<div class="mermaid">
  graph TD
  Activity[Activity]
  Activity -->|...| Doodler[Doodler]
  Doodler --> Part1[Part1]
  Part1 --> Part1Activity[Part1Activity]
  Part1 --> Part2Activity[Part2Activity]

View[View]
  View --> ImageView[ImageView]
  View --> TextView[TextView]
  View --> LineView[LineView]

class Part1,Part1Activity,Part2Activity,LineView yellow
class Activity,View,ImageView,TextView green
class Doodler blue

</div>
]

.right-column-half[
##  Classes we are using in Doodle

- Doodler (which you don't edit) *extends* [Activity](https://developer.android.com/reference/android/app/Activity)
- Part1 *extends* Doodler. It implements *helper methods*.
- Part1Activity and Part2Activity both extend Part1 (and use the helper methods)
- LineView *extends* [View](https://developer.android.com/reference/android/view/View) (so does [ImageView](https://developer.android.com/reference/android/widget/ImageView.html) and [TextView](https://developer.android.com/reference/android/widget/TextView.html)

]

???
Green: Android toolkit
Blue: Classes we provide you don't have to modify
Yellow: Classes you implement

Remind students that they did do inheritance in 142/143 like with Critters.

---
.left-column-half[
# Doodle: What you will learn

- How the Android Activity Lifecycle works (*Architecture*)
- How to use Android Toolkit `View` objects to show things on screen (*Library*)
- How to create a custom `View` to draw lines on screen (*Library*)
- All about Android's `Canvas` object and how to use it to draw on screen (*Library*)
- How the Android Toolkit supports animation (*Architecture & Library*)
]

.right-column-half[
# Doodle: What you will do

- Modify Android Activities (`Part1` and `Part1Activity`)
- Implement methods to add images, text and lines
- Create the custom view `LineView`
- Override `onDraw` to draw on the screen in `LineView`
- Use `ObjectAnimator` to animate Text and other things
]
---
# Subtle influence of tools

![:img Hand-drawn sketch of an interface very rough around the edges, 50%](img/toolkits/sketch.png)

---
# Subtle influence of tools

![:img High Fidelity Prototype, 40%](img/toolkits/prototype.png)

---
# Subtle influence of tools

Whorfian Effects
- The way in which our tools influence how we think and thus what we do.

- Origins are linguistic (e.g. the way in which language about color affects what we perceive)

- Can even affect perception of human emotion!
http://www.radiolab.org/story/91730-new-words-new-world/

---
# The Application Stack
.left-column[
<div class="mermaid">
  graph LR
  ap[Application Program]
  hlt[High Level Tools]
  t[Toolkit]
  w[Window System]
  o[OS]
  h[Hardware]

class w,ap,o,h yellow
class t green
class hlt darkblue
</div>
]
.right-column[
![:img Screenshot of the android layout tool, 80%](img/toolkits/android-editor.gif)
]

---
# The Application Stack
.left-column[
<div class="mermaid">
  graph LR
  ap[Application Program]
  hlt[High Level Tools]
  t[Toolkit]
  w[Window System]
  o[OS]
  h[Hardware]

class w,o,h yellow
class t,hlt green
class ap darkblue
</div>
]
.right-column[
![:img Screenshot of an android ui for exploring a neighborhood showing a range of locations; activities to engage in in those locations; and things that match the selected activity in a sequence of screens, 80%](img/toolkits/android-ui.png)
]

???
Get class to discuss all the things that a UI like this does a little bit. What do you have to think about to build an application? How might a toolkit help with this?
---

# Summary: The Application Stack
.left-column[
<div class="mermaid">
  graph LR
  ap[Application Program]
  hlt[High Level Tools]
  t[Toolkit]
  w[Window System]
  o[OS]
  h[Hardware]

class ap,w,o,h yellow
class hlt,t green

</div>
]

.right-column[
- **Application Program** - An application designed for an end user to perform specific tasks.
- **High Level Tools** - Graphical interfaaces that that let you specify parts of your interface (such as layout). Subject to Worfian Effects
- **Toolkit** - A set of libraries and tools you use to develop applications.
- **Window System** - Manages window size and visibility across applications
- **OS** - The operating system that is running on the device, providing system services such as acceess to displays, input devices, file I/O
- **Hardware** - The device that is running software, such as an Android Phone

High level tools and the the Toolkit are generally packaged as part of a toolkit but really should be thought of as separate things. (You can program the toolkit without the tools. )
]

---

.left-column[
## Developer roles

<div class="mermaid">
  graph LR
  ip[Interface Programmer]
  w[Component Developer]
  l[Library Extender]
  a[Architecture Extender]
  t[Toolkit Builder]

class ip,t yellow
class w,l,a green
</div>
]
.right-column[
## Let's say you want to build an app

In this class, you will use the Android Toolkit

Who are we when we build the app?
]

---
.left-column[
## Developer roles
<div class="mermaid">
  graph LR
  ip[Interface Programmer]
  w[Component Developer]
  l[Library Extender]
  a[Architecture Extender]
  t[Toolkit Builder]

class t yellow
class w,l,a green
class ip darkblue
</div>
]
.right-column[
Combine library elements according to toolkit rules & following
constraints of architecture; common to all UI toolkits

In Android
- Construct app out of `Views`
- Assembly happens in an `Activity` in `onCreate()` using `addView(View)`

In Doodle
- You add views when you implement `addImage`, `addText` and `addLine` in `Part1.java`
- You will need to call `mainCanvas.addView([view])` (`mainCanvas` is defined in `Doodler`)
]
---
# How does the toolkit know what to draw?

![:img Picture of a very simple interface showing a ringing bell at
left and an x at right to close the window with the words Google
Calendar reminder Christian and Anind (Jen Mankoff) is starting at
12:30pm. Video call between them, 70%](img/toolkits/interface.png)

- Discuss
???
discuss with your neighbor based on the reading what does the
toolkit need to know to draw this
- what to draw; where to draw it
- what are good abstractions for this?
- Hierarchy normally reflects spatial containment relationships

- Deconstruct

- Let's list all the components... and arrange them in a tree

???

- deconstruct this interface into an interactor hierarchy

---
![:img Picture of a very simple interface showing a ringing bell at
left and an x at right to close the window with the words Google
Calendar reminder Christian and Anind (Jen Mankoff) is starting at
12:30pm. Video call between them, 60%](img/toolkits/interface.png)

<div class="mermaid">
graph LR
W(Window) --> L(LeftSide:Icon)
W --> M(Middle:Label)
W --> R(Right:Close Button)

class W start
class M,R,L blue

</div>

---
# More complex example

![:img Picture of a messaging interface with two messages in it, 20%](img/drawing/messaging.png)

---
.left-column[
![:img Picture of a messaging interface with two messages in it, 80%](img/drawing/messaging.png)]

.right-column[
<div class="mermaid">
graph LR
W(Window) --> T(Controls)
W --> M(Chats)
W --> Q(Text Entry)
T --> A[Button:Messages]
T --> B[Label:288-88]
T --> C[Button:Edit]
M --> G[Button:Call]
M --> H[Button:Add to Contacts]
M --> E[Chat1]
M --> F[Chat2...]
E --> I[Date]
E --> J[Chat Msg]
E --> K[Arrow]

class W start
class T,M,E,Q,A,B,C,G,H,E,F,I,J,K blue

</div>
]
---
# How is the tree used for drawing?

Interface Programmer:
- Build and interface == build an interactor hierarchy
- Change an interface == change the interactor hierarchy (or its  parameters)

---

.left-column-half[
## How does Android let you do this
Architecture invokes the work to construct the *interactor hierarchy*
by calling  `onCreate()`

In Doodle
- *Doodler* overrides `onCreate()` (from it's grandparent *Activity* class)
- *Doodler* provides a helper method for this (`doodle()`) which you
   override in `Part1Activity` and `Part2Activity`.
]
.right-column-half[
```java

abstract class Doodler extends AppCompatActivity {
    // ...
    @Override
    protected void onCreate(@Nullable Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        // ...
        doodle(...)
    }

abstract protected void doodle(FrameLayout mainCanvas);
```
]

---
# How is the tree used for drawing?

The interface programmer is not really *drawing on a canvas*, they are *adding views* to a *parent view* in `onCreate()` (or later if something changes)

This is *setup* for the interface being rendered: We are constructing
the interactor (also called component) hierarchy when we call these methods.

This happens in `Doodle` inside of `Part1.java` in `addImage()` when we say `mainCanvas.addView(imageView);`

You have to do it in `addText()` and `addLine()` too.

---
# How is the tree used for drawing?
Toolkit Architecture:
- Draw an interface == walk the tree and tell interactors to draw
- Update an interface == walk the tree, only redraw things that are marked as dirty
- Deliver input == walk the tree and deliver events (after midterm)

---
# How does an interactor draw?

Interactors (Views in Android):
- Receive draw (`onDraw()` in Android)
- Show feedback () (Draw onto `Canvas` in Android)
- Respond to input (Future assignment!)

You will create a custom interactor, a `LineView`, which has to implement `onDraw(Canvas canvas)`

You can use all the [Canvas](https://developer.android.com/reference/android/graphics/Canvas) methods (we'll talk more about these in the [Drawing](Drawing.html) lecture on Friday)

???
HTML is like this too

---
.left-column[
## Developer Roles
<div class="mermaid">
  graph LR
  ip[Interface Programmer]
  w[Component Developer]
  l[Library Extender]
  a[Architecture Extender]
  t[Toolkit Builder]

class ip,t yellow
class w,l,a green
class w darkblue
</div>
]
.right-column[
Creates new, re-usable interaction techniques (e.g. to support new forms of
input/direct manipulation).

Supported by many toolkits
]
---
# Some innovation: Interaction Technique Developer
.left-column-half[
- A method for carrying out a specific interactive task
- Example: enter a number in a range
 - Swiping to invoke an action
 - Gesture based text entry

![img: A picture of gesture based text entry on an android phone, 20%](img/toolkits/android-type.jpg)
]

.right-column-half[
![img: A picture of a finger left-swiping to invoke an action on a
list on an android phone, 10%](img/toolkits/android-swipe.png)

]
---
.left-column[
## Developer Roles
<div class="mermaid">
  graph LR
  ip[Interface Programmer]
  w[Component Developer]
  l[Library Extender]
  a[Architecture Extender]
  t[Toolkit Builder]

class ip,t yellow
class w,a green
class l darkblue
</div>
]
.right-column[
Similar to interaction technique developer, but may create new forms of layout, types of input, etc.

Supported by a few toolkits
]
---
# Toolkit Support for Styling

.left-column-half[
![:img A picture of the same dialogue box styled 4 different ways, 100%](img/toolkits/novellibrary.png)
]
.right-column-half[
subArctic allowed visually rich, dynamically resizable, images to be
provided using primarily conventional drawing tools (and with no
programming or programming-like activities at all).  Scott E. Hudson
and Kenichiro Tanaka. (UIST '00)
]
---
.left-column[
## Developer Roles
<div class="mermaid">
  graph LR
  ip[Interface Programmer]
  w[Component Developer]
  l[Library Extender]
  a[Architecture Extender]
  t[Toolkit Builder]

class ip,t yellow
class w,l,a green
class a darkblue
</div>
]
.right-column[
Modifies the flow of information within a toolkit to create entirely new effects (e.g. adding support for command objects)

Supported by very few toolkits
]
---
# Architecture Extender: Animation

.right-column50[
![:img A picture of a phone interface using animation to show a menu, 80%](img/toolkits/animation.gif)
]

Integrated into existing GUI toolkit

Primary abstraction: transition
- models movement over time
- through arbitrary space of values (e.g., color)
- screen space is most common

.footnote[Great post about [types of animation on mobile phones](https://yalantis.com/blog/-seven-types-of-animations-for-mobile-apps/), also
source of the image]

---
.left-column[
## Developer Roles

<div class="mermaid">
  graph LR
  ip[Interface Programmer]
  w[Component Developer]
  l[Library Extender]
  a[Architecture Extender]
  t[Toolkit Builder]

class ip,t yellow
class w,l,a green
class t darkblue
</div>
]
.right-column[
Creates entirely new toolkits that enable radical new forms of interaction

e.g. RapID https://make4all.org/portfolio/rapid/
]
---
# Toolkit Builder: Physical Interfaces
Creates entirely new toolkits that changes what we can do

![:youtube Physical Interface made out of RFID tags,4k15uXpp7-g]

---
# Toolkit for Overloading Existing Interfaces
Prefab supports pixel based enhancements: write interpretation logic that can be composed, reused, and shared to manage the multi-faceted nature of pixel-based interpretation; robustly annotate interface elements with metadata needed to enable runtime enhancements.

![:youtube Prefab Demo,lju6IIteg9Q]

---
#  Summary: Developer Roles
.left-column[
<div class="mermaid">
  graph LR
  ip[Interface Programmer]
  w[Component Developer]
  l[Library Extender]
  a[Architecture Extender]
  t[Toolkit Builder]

class ip,w,l,a,t yellow
</div>
]
.right-column[
- **Interface Programmer** - Uses high level tools and toolkit to create a user interface.
- **Component Developer** - Creates new, re-usable interaction techniques.
- **Library Extender** - Similar to Component Developer, but may create new forms of layout, types of input, etc.
- **Architecture Extender** - Modifies the flow of information within a toolkit to create entirely new effects
- **Toolkit Builder** - Creates entirely new toolkits that enable radical new forms of interaction.
]
---
#  Summary: In this class we look at General Principals

Particular toolkits tend to be mired in a particular platform

Along comes the next big thing  (e.g., iPhone) and you’ll need to
learn a new one

We frequently will use Android as an example of these concepts

If you learn the structures/concepts (and esp. rationale behind them)
that survives change better