Event Handling III--Essential Geometry and View Updates

name: inverse
layout: true
class: center, middle, inverse
---
# View Updates and Essential Geometry

Jennifer Mankoff

CSE 340 Winter 2021
---
layout:false

[//]: # (Outline Slide)
# Today's goals

- Quick follow up from Monday - Click Listeners
- Other Android callbacks
- Input dispatching process (toolkit architecture)
  - Picking (alternative: Focus)
  - Capture
  - Bubble
- Propositional Production Systems in implementing interactors
- Essential Geometry

---
# Review of Monday

- Events: logical input device abstraction
- Event driven code: very different from procedural programming
- Listeners are notified about events by the system and toolkit
- Case study: implementing View.OnClickListener
  - Implementing an Interface
  - Creating an anonymous inner class
  - Lambdas
---

template: inverse

# Other Android Callbacks

---
layout: false
# Already registered by Android  (1/3)

- `onCreate`
  - called when the activity is first created
  - we've seen this in our Doodle, Layout, and even EventExampleActivity apps:

```java
@Override
protected void onCreate(Bundle savedInstanceState) {
  // We want to do any view initialization work here
  super.onCreate(savedInstanceState);

// Load the XML representation of our layout into the view
  setContentView(R.layout.activity_main);

// Any other work we need to do like adding views or
  // registering click listeners!
}
```

---
# Already registered by Android  (2/3)

- `onStart`
  - Called when activity is about to be visible to the user
  - Always gets called after `onCreate` or `onRestart`
--
- `onResume`
  - Called when the activity will start interacting with a use
  - Always gets called after `onStart`
- ...

---
template: inverse

## Review: Input Dispatch Process

---
layout: false

# Review: Input __Dispatch__ Process

Input thread:
- When a user interacts,  __events__ are created
- Events go into a queue

---
# Review: Input __Dispatch__ Process

Input Thread

Dispatch thread:
- Front event comes off queue
- Event gets sent to a listener (callback) attached to a `View` or `Activity`.
- The `View` or `Activity` implements the callback

---
# Dispatch Process (Positional)

.column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1))
V0 --> V4((V4))
V1 --> V2((V2))
V1 --> V3((V3))
V4 --> V5((V5))
V4 --> V6((V6))

class V0,V1,V2,V5,V4,V3,V6 blue
</div>
]
.column[
![:img Example of what this tree might look like in graphical form with V0 at the back, 100%, width](img/pps-geom/picklist.svg)
]
---
# Step 1: Picking
.column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1))
V0 --> V4((V4))
V1 --> V2((V2))
V1 --> V3((V3))
V4 --> V5((V5))
V4 --> V6((V6))

class V0,V1,V2,V5,V4,V3,V6 blue
</div>
]
.column[
![:img Example of what this tree might look like in graphical form with V0 at the back, 100%, width](img/pps-geom/picklist.svg)
]
.column[
Android traverses the view hierarchy (starting at root)

"Picks" `View` objects that respond to an input event
]

---
# Step 1: Picking

.column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1))
V0 --> V4((V4))
V1 --> V2((V2))
V1 --> V3((V3))
V4 --> V5((V5))
V4 --> V6((V6))

class V0,V1,V2,V5,V4,V3,V6 blue
</div>

{`V6`, `V5`, `V4`, `V3`, `V2`, `V1`, `V0`}
]
.column[
![:img Example of what this tree might look like in graphical form with V0 at the back, 100%, width](img/pps-geom/picklist.svg)
]
.column[
Picking is a traversal of the tree
- Order matters!
- It uses reverse draw (z) order

]

---
# Step 1: Picking

.column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1))
V0 --> V4((V4))
V1 --> V2((V2))
V1 --> V3((V3))
V4 --> V5((V5))
V4 --> V6((V6))

class V0,V1,V2,V5,V4,V3,V6 blue
</div>

What are some reasons to skip views?
]

???
- Not inside the view
- View doesn't care about that type of input

---
# Step 1: Picking (and Filtering)

.column[
<div class="mermaid">
graph TD
V0((V0*)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V2,V5 blue
class V0,V1,V4,V3,V6 bluegreen
</div>

{`V6`, `V4`, `V3`, `V1`, `V0`}
]
.column[
![:img Example of what this tree might look like in graphical form with V0 at the back, 100%, width](img/pps-geom/picklist.svg)
]
.column[
- Toolkit *may* discard components that do not contain the event's position
- Some toolkits check with components for example by calling `pick()` on each component (not Android) 
]
.footnote[*: denotes the element responds to the event because of position/type]

---
# Step 2: Delivering

.column[

<div class="mermaid">
graph TD
V0((V0*)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V2,V5 blue
class V0,V1,V4,V3,V6 bluegreen

</div>

{`V6`, `V4`, `V3`, `V1`, `V0`}
]
.doublecolumn[
**Top-down**: Walk the pick list right to left (reverse z order). Sometimes called *Capture*
**Bottom-up**: Walk the pick list left to right (approximately corresponds to z order). Sometimes called *Bubble*

Some toolkits do both (with different listeners for each type event), some just bottom-up.

Most events can only be *consumed* by a single component (event delivery usually asks for a `boolean` return value to determine this, e.g. `onTouch()`)
]

---
# Step 2: Delivering

.column[

<div class="mermaid">
graph TD
V0((V0*)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V2,V5 blue
class V0,V1,V4,V3,V6 bluegreen

</div>

{`V6`, `V4`, `V3`, `V1`, `V0`}
]
.doublecolumn[
Most stuff in Android uses Bottom Up (bubble)
- Dispatch starts at the start of the picked `View` object list (`V3`)
- Walking down the list, dispatch asks: will you consume this event? (e.g. by calling `onTouch()`)
  - If `true`: the event is consumed and the event propagation __stops__
  - If `false`: Move to the next element in the `View` list
]
---
# Step 2: Delivering

.column[

<div class="mermaid">
graph TD
V0((V0*)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V2,V5 blue
class V0,V1,V4,V3,V6 bluegreen

</div>

{`V6`, `V4`, `V3`, `V1`, `V0`}
]
.column[
![:img Example of what this tree might look like in graphical form with V0 at the back, 100%, width](img/pps-geom/picklist.svg)
]
.column[
Scenario: 
- `V6` returns `false` because outside 
- `V3` returns `false` because it is a decoration
- `V1` returns `true` because it is a draggable container
**Dispatch halts and redraw starts**
]

???
Circular interactor! Like ColorPicker

---
# Positional Input - Recap

.column[

<div class="mermaid">
graph TD
V0((V0*)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V2,V5 blue
class V0,V1,V4,V3,V6 bluegreen

</div>

{`V6`, `V4`, `V3`, `V1`, `V0`}
]
.column[
![:img Example of what this tree might look like in graphical form with V0 at the back, 100%, width](img/pps-geom/picklist.svg)
]
.column[

|*Picking*       | V6| V4 | V3 | V1|V0|
|:---------------|:-------------------|
|*Capture Order* | 5 | 4  | 3  | 2 | 1|
|*Bubbling Order*| 1 | 2  | 3  | 4 | 5|

- Android uses bubbling
- Picking and Delivery are based on interactor hierarchy
- A view can refuse an event (return `false`)
- Dispatch halts when event is consumed
]
 
---
# Stuff to ponder

How would you handle focus input?
- Some event types are focus based (e.g. keyboard input)
- Just skip picking: Focus list is globally created, and we walk it the same way

--
Implementing Drag and Drop
- Focus? Or Positional?
--
- [Android's tutorial is positional](https://developer.android.com/training/gestures/scale#java)

--
Scrollbars
- Focus? Or Positional?
--
- both

---
# How does *Android* decide where to send events?

- **Capture** Android does have *some* support for this
   - `onInterceptTouchEvent()` in Android
- **Pick** to identify objects of interest (or just focus())
   - `buildTouchDispatchChildList()` in Android; happens only after capture!
   - not generally accessible but you can override it in a custom interactor
- **Bubble** Standard in Android
   - example: `onTouchEvent()` in Android

---
# What about callbacks?

Event delivery in view happens using callbacks
- e.g. `onTouchEvent()` is a callback

--
For applications,
- Usually Views invoke callback (when interaction is complete)
  - This is part of your *controller* code in MVC parlance
  - We often implement this by creating a custom listener in Android
  - Can also create a custom event but not required (we'll do this in `Undo`)

.footnote[fascinating to look at [implementation in
ViewGroup](https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/view/ViewGroup.java)
-- not very general!]

---

# Creating a new type of listener

Application callbacks can be implemented as custom listeners

```java
// Defines a new named inner interface for listening to an interesting event
public interface MyListener {
    void onInterestingEvent(); // can include a parameter
}

// variable to store the listener in once registered. Could also be a list of
// listeners that are all called if the callback is triggered.
protected MyView.MyListener mListener;

public void setMyListener(MyListener mListener) {
    this.mListener = mListener;  // could also be stored in a list.
}
...

// somewhere else in your code, when callback time happens
public void someMethod() {
  mListener.onInterestingEvent();
}

```
---

name: inverse
layout: true
class: center, middle, inverse
---
# Using Input to Create Interaction Techniques

## State Machines, PPS, Essential Geometry

---
layout:false

# Interaction Technique

A method for carrying out a specific interactive task

For example: For entering a number in a range you could use...

???
have the class try to think of examples
--
- (simulated) slider
- (simulated) knob
- type in a number (text edit box)

Each is a different interaction technique
---
# Example: Specify the end points for a line

Could just specify two endpoints – click, click
- not good: no affordance
  - no feedback (/ feedforward)

Better feedback is to use “rubber banding”
- stretch out the line as you drag
- at all times, shows where you would end up <br> if you “let go”
???
Importance of feedback vs application callback
---
# Implementing rubber banding

```
Accept the press for endpoint P1; P2 = P1;
Draw line P1-P2;
Repeat
  Erase line P1-P2;
  P2 = current_position(); Draw line P1-P2;
Until release event;
Act on line input;
```

???
Discuss!
Not event based
Not in the basic event/redraw loop
Potentially locks up the system
---
# Implementing rubber banding

Need to get around this loop <br> absolute min of 5 times / sec

– 10 times better

– more would be better but might block system.

???
aside -- why 5-10 times per second?

---
# Event driven code

Needs to respond to events as they arrive

Needs to maintain state between events

---
# Solution: State Machine

.left-column50[
- State Machines are used to respond to incoming events and allows for us to store state between events.
- Start state - indicated with incoming arrow
- End state - indicated with double-layered shape (means "reset to start")
- Transition States - indicated with single-layered shape
- Event Arrows - indicated with an arrow between states, represent different actions taken up states.
]
.right-column20[
<div class="mermaid">
stateDiagram-v2

[*] --> A: a
A --> B: a
B --> B: b
B --> C: b
C --> [*]: a

</div>
]

--
.right-column30[
Is the folowing "accepted" by the state machine at left?

- ab
- abbbbbb
- aaba
- abbbbba
]
---
# Solution: State Machine

State Machines are used to respond to incoming events and allows for us to store state between events.
.left-column60[
- Actions always take place on arrows (*transitions*)
- Start state - indicated in black
- End state - indicated with double-layered shape (means "reset to start")
- Transition States - indicated with single-layered shape
- Event Arrows - indicated with an arrow between states, represent different actions taken up states.
]
.right-column40[
<div class="mermaid">
stateDiagram-v2

[*] --> B: Event1/?Callback1()
B --> B: Event2/?Callback2()
B --> [*]: Event3/?Callback3()
</div>
]

---
# Propositional Production System (PPS)

- PPS is a state machine with extra conditions required to fire on the arrows.
  - ? predicate (Boolean expr) before event, adds extra conditions required to transition
  - action calls
- Typical notation

`event / pred ? action()`

- Example

`MouseUp/inView?Start_Line()`

- Finite State Machine (FSM) augmented with guards is Turing complete

---
# PPS Example: Rubber Banding

.left-column60[
Compare to previous implementation:

```txt
Accept the press for endpoint P1; P2 = P1;
Draw line P1-P2;
Repeat
  Erase line P1-P2;
  P2 = current_position(); Draw line P1-P2;
Until release event;
Act on line input;
```
]
.right-column40[

- Determine the Events (triggers)

- Determine the States

- Determine the Actions

- Determine the Queries
]
---
# PPS Example: Rubber Banding
.left-column60[
Compare to previous implementation:

```txt
Accept the press for endpoint P1; P2 = P1;
Draw line P1-P2;
Repeat
  Erase line P1-P2;
  P2 = current_position(); Draw line P1-P2;
Until release event;
Act on line input;
```
]

.right-column40[

<div class="mermaid">
stateDiagram-v2

[*] --> Drawing : Mouse Down/inView?Start_Line()
Drawing --> Drawing : Mouse_Move/inView?Update()
Drawing --> [*] : Mouse_Release/inView?Finish_Line()

</div>

]

---
# PPS Example: Rubber Banding
.left-column50[

Reading a state machine: translates input sequence into action!
- When you are in Start State, and a `Mouse_Down` event arrives, do
  the action `Start_line()` and go to Drawing State.
- Update  the line end point position every time the mouse moves.
- When it releases (`Mouse_Release` event), finish the line (at this
  stage a callback to the application might be appropriate)
]

.right-column50[

<div class="mermaid">
stateDiagram-v2

[*] --> Drawing : Mouse_Down/inView?Start_Line()
Drawing --> Drawing : Mouse_Move/inView?Update()
Drawing --> [*] : Mouse_Release/inView?Finish_Line()

</div>

]

???

How could we provide a better affordance?

Does it matter if we are using a mouse or a touch screen?

---
name: inverse
layout: true
class: center, middle, inverse
---
# Essential Geometry as the basis for state

---
layout:false

# Essential Geometry as the basis for state

.left-column-half[
![:img google doc with scrollbar, 80%, width](img/pps-geom/window.png)

]

.right-column-half[
- What is the essence (or nature) of this scrollbar? Where can you interact with it?
]

---

# Essential Geometry as the basis for state

.left-column-half[
![:img google doc with scrollbar, 80%, width](img/pps-geom/window.png)

]

.right-column-half[
- What is the essence (or nature) of this scrollbar? Where can you interact with it?
  - on the thumb
  - inside the scrollbar above the thumb
  - inside the scrollbar below the thumb
  - inside up arrow
  - inside the down arrow

- How do we implement a scroll bar like this?
]

---
# Scrollbar State machine with Essential Geometry

.left-column30[
<div class="mermaid">
stateDiagram-v2

[*] -->Scrolling: A
[*] --> [*]: B
[*] --> [*]: C
Scrolling --> Scrolling: D
Scrolling --> [*]: E
</div>

]
.right-column70[

```java
// A 
MouseDown / inThumb ? startScroll()

// B
MouseClick / InsideAboveThumb ?  scrollUp()

// C 
MouseClick / InsideBelowThumb ? scrollDown()

// D
MouseMove / updateThumbDocument()

// E
MouseUp / keepLocation()

```
]

---
# Let's try it for a button

.left-column50[
Essential geometry is:
- InsideButton
- OutsideButton

and methods for
- `indentButton()` (when button is pressed)
- `normalButton()` (when button is not pressed)
- `invokeAction()` (when the user releases in the button)
- `cancelAction()` (when the user releases outside the button)

]

---
# You should have something like this

.left-column50[
Essential geometry is:
- Inside
- Outside

]
.right-column50[
<div class="mermaid">
stateDiagram-v2

[*] --> INSIDE: DOWN / indentButton()
INSIDE --> OUTSIDE: MOVE / <BR> Outside ? normalButton()
OUTSIDE --> [*]: UP / cancelAction()
INSIDE --> [*]: UP / invokeAction()
OUTSIDE --> INSIDE: MOVE / <BR> Inside ? indentButton()
</div>

]

---
# How do we implement this?
- Implement in `onTouch()` using a switch statement
- Assume there is an
`essentialGeometry(MotionEvent event)` method. It returns a enum that tells you what
 part of the geometry you are in for that point.
- Assume implementations of all the methods
- Assume a field, `state` which is the current state of the state machine
- enums `EssentialGeometry` and `State` for comparing against

```java
@Override
public boolean onTouch(MotionEvent e) {
  EssentialGeometry geometry = essentialGeometry(event);
  switch (state) {
    case State.START:
      // arrow from start to INSIDE
      break;
    case State.INSIDE:
      // need both arrows
    case State.OUTSIDE: 
	  // need both arrows
    default:
      break; // ignore everything else
  }
  return false;
}
```

---
# How do we implement this?

```java
@Override
public boolean onTouch(MotionEvent e) {
  EssentialGeometry geometry = essentialGeometry(event);
  switch (state) {
    case State.START:
      if (e.getAction() == MotionEvent.ACTION_DOWN && geometry == Geometry.INSIDE) {
         indentButton();
         state = State.INSIDE;
         return true;
      }
      break;
   case State.INSIDE: 
	  ...
```
---
# How do we implement this?

```java
   case State.INSIDE:

case State.OUTSIDE: 
   
   
   
   
   
   
   ...
```
---
# How do we implement this?
```java
   case State.INSIDE: 
      if (e.getAction() == MotionEvent.ACTION_UP) {
         invokeAction();
         state = State.START;
         return true;
      } else if (e.getAction() == MotionEvent.ACTION_MOVE && geometry == Geometry.OUTSIDE) {
	     normalButton()
		 state = State.OUTSIDE
		 return true;
	  }
      break;
   case State.OUTSIDE: 
      if (e.getAction() == MotionEvent.ACTION_UP) {
         cancelAction();
         state = State.START;
         return true;
      } else if (e.getAction() == MotionEvent.ACTION_MOVE && geometry == Geometry.INSIDE) {
         indentButton()
		 state = State.INSIDE
		 return true;
	  }
      break;
   default: break;
```
---
# Enums

Enums are a group of named constants

- Often used for developing interactions for defining PPS States and Essential Geometry
- Will be used in ColorPicker and in Menu assignment for interaction *and* for experimental conditions
- Good code quality: limits choices of what a variable can be.

[Documentation](https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html)

---

.left-column[
## Harder Button

![:img FB Messenger Animation, 100%, width](img/pps-geom/messenger-bubble.gif)

]
.right-column[
- Determine the Events (triggers)

- Determine the States

- Determine the Queries (essential geometry, context)

- Determine the Actions
]

???
What constitutes an “event” varies

- may be just low level events, or
- higher level (synthesized) events
- e.g. region-enter, press-inside

What is missing? Query fields

--
.right-column[

<div class="mermaid">
stateDiagram-v2

[*] -->Active: Press/inside?highlight(),start_animation()
Active --> Active: AnimateStep/update()
Active --> Active: AnimateFinish/!small
Active --> [*]: Release,inside/small, unhighlight
Active --> [*]: Release,inside/!small,add_to_chat(),unhighlight()
</div>
]

---
# When to use PPSs

You're probably already using them, just not intentionally (and maybe
less well as a result)

PPSs are a good way to do control flow in event driven systems

Can do (formal or informal) analysis
- are all possible inputs (e.g. errors) handled from each state
- what are next legal inputs: can use to enable / disable

Can be automated based on higher level specification

---

# Summary

State machines are very good (for this job) but do have limits

State machines don't handle independent actions very well (state explosion)

Mostly useful for smaller things

- Great for individual components
- Not so great for whole dialogs

Path of least resistance is rigid sequencing
 Ask: is this good for what I am doing?

???