Event Handling II--Essential Geometry and View Updates

layout: false

.title[# Hall of Fame/Shame]
.body[
![:img Picture of the course website with the mouse clicking on an unresponsive hamburger menu,80%](img/viewupdate/noresponse.png)
]
---
name: inverse
layout: true
class: center, middle, inverse
---
# Essential Geometry and View Updates

Jennifer Mankoff
CSE 340 Winter 2020
---
layout:false

[//]: # (Outline Slide)
.title[Today's goals]
.body[

- Input dispatching process (toolkit architecture)
 - Picking (alternative: Focus)
 - Capture
 - Bubble

Introduce how interactors process them

Discuss use of Propositional Production Systems in implementing interactors

Understand Essential Geometry

Try creating a PPS

Understand View Updates 
]

---
template: inverse

## Input Dispatch Process

---
layout: false

# Input __Dispatch__ Process

Input thread:
- When a user interacts,  __events__ are created
- Events go into a queue

---
# Input __Dispatch__ Process

Input Thread

Dispatch thread:
- Front event comes off queue
- How does a toolkit decide where to send events?

---
# How do we decide where to send events?

---
# Input Process - Picking

.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1))
V0 --> V4((V4))
V1 --> V2((V2))
V1 --> V3((V3))
V4 --> V5((V5))
V4 --> V6((V6))

</div>
]
.right-column[
- In what order does dispatch "pick" the `View` objects?
]
--
.right-column[
- Hint: Post order traversal of the tree
]
--
.right-column[
- Picked Views = {`V2`, `V3`, `V1`, `V5`, `V6`, `V4`, `V0`}
- Order matters!
]
---
# Input Process - Picking

.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1))
V0 --> V4((V4))
V1 --> V2((V2))
V1 --> V3((V3))
V4 --> V5((V5))
V4 --> V6((V6))
</div>
]

.right-column[

But we can refine this! We can filter!
]

???
What are some reasons to skip views?
- Not inside the view
- View doesn't care about that type of input

--
.right-column[
What are some reasons to skip views?
- Not inside the view
- View doesn't care about that type of input
]

---
# Input Process - Picking only those interested

.left-column[

<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]

.footnote[*: denotes the element responds to the event because of position/type]

---
# Input Example - Capture

.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]
.right-column[
- Picked Views = {`V3`, `V1`, `V6`, `V4`}
- What happens next?
]
--
.right-column[
- Dispatch starts at the end of the picked `View` object list (`V4`)
- Walking down the list, dispatch asks: will you consume this event?
  - If `true`: the event is consumed and the event propagation __stops__
  - If `false`: Move to the next element in the `View` list
]
---
# Input Example - Capture

.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]
.right-column[
- Picked Views = {`V3`, `V1`, `V6`, `V4`}
- Dispatch starts at the end of the picked `View` object list (`V4`)
- What happens when we reach the last `View` in the list and no one has consumed the event?
]

---
# Input Example - Bubbling

.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]
.right-column[
- Picked Views = {`V3`, `V1`, `V6`, `V4`}
]

--
.right-column[
- Android backtracks through the list, asking if they want to consume the event
- Who gets asked first ?
]
---
# Input Example - Recap

.left-column[
<div class="mermaid">
graph TD
V0((V0)) --> V1((V1*))
V0 --> V4((V4*))
V1 --> V2((V2))
V1 --> V3((V3*))
V4 --> V5((V5))
V4 --> V6((V6*))

class V0,V2,V5 blue
class V1,V4,V3,V6 bluegreen
</div>
]
.right-column[
|State           |Ordering         |
|:---------------|:----------------|
|*Picking*       | V3| V1 | V6 | V4|
|*Capture Order* | 4 | 3  | 2  | 1 |
|*Bubbling Order*| 1 | 2  | 3  | 4 |
]

---
# How would you handle input in a circular component?
???
Consumption is based on the bounding box...
--

Delivery is based on bounding box

Return `false` if input is outside circle even if it's in the bounding box

---
.left-column-half[
## Event Dispatch

![:img Picture of interactor hierarchy connected to an interface and a
dotted line indicating application interface, 100%](img/viewupdate/callbacks.png)]
.right-column-half[
Dispatch Strategies
- Positional (Bottom-first and Top-down)
- Focus-based
]
---
.left-column-half[
## Event Dispatch

![:img Picture of interactor hierarchy connected to an interface and a
dotted line indicating application interface and a do_action() call
happening below the line in response to a button_pressed(), 100%](img/viewupdate/callbacks2.png)]
.right-column-half[
Callbacks handle *application* response to events
- Update Application Model

]

---
.left-column-half[
## Event Dispatch

![:img Picture of interactor hierarchy connected to an interface and a
dotted line indicating application interface with do_action() replaced
with an actionListener, 100%](img/viewupdate/callbacks3.png)]
.right-column-half[
Callbacks handle *application* response to events
- Update Application Model
- Best implemented using custom listeners

]

---
# What about focus input?

- Some event types are focus based (e.g. keyboard input)
- Just skip picking: Focus list is globally created, and we walk it the same way

---
# Implementing Drag and Drop

Focus? Or Positional?
???
[Android's tutorial is positional](https://developer.android.com/training/gestures/scale#java)

---
# How does *Android* decide where to send events?

- **Capture** (most things don’t) top to bottom deliver to target
   object (bottom)
   - example: `onInterceptTouchEvent()` in Android
- **Pick** to identify objects of interest (or just focus())
   - `buildTouchDispatchChildList()` in Android; happens only after capture!
- **Bubble** (bottom to top)
   - example: `onTouchEvent()` in Android
- Invoke callback (wait until complete)
  - we do this by creating a custom listener in Android

.footnote[fascinating to look at [implementation in
ViewGroup](https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/view/ViewGroup.java)
-- not very general!]
---
# Creating a new type of callback

```java
protected MyView.MyListener mListener;

public void setMyListener(MyListener mListener) {
        this.mListener = mListener;
    }

// Creates a new named inner interface for listening to color selection events
    public interface MyListener {
        void onInterestingEvent(); // can include a parameter
    }
	// somewhere else in your code, when callback time happens
    public void someMethod() {
		mListener.onInterestingEvent();
	}
```

---
name: inverse
layout: true
class: center, middle, inverse
---
# Using Input to Create Interaction Techniques
---
layout:false

# Interaction Technique

A method for carrying out a specific interactive task

Example: enter a number in a range

could use...

???
have the class try to think of examples
--
- (simulated) slider
- (simulated) knob
- type in a number (text edit box)

Each is a different interaction technique
---
# Example: Specify the end points for a line

Could just specify two endpoints – click, click
- not good: no affordance, <br>no feedback (/ feedforward)

Better feedback is to use “rubber banding”
- stretch out the line as you drag
- at all times, shows where you would end up <br> if you “let go”
???
Importance of feedback vs application callback
---
# Implementing rubber banding

```
Accept the press for endpoint P1; P2 = P1;
Draw line P1-P2;
Repeat
  Erase line P1-P2;
  P2 = current_position(); Draw line P1-P2;
Until release event;
Act on line input;
```

???
Discuss! 
Not event based
Not in the basic event/redraw loop
Potentially locks up the system
---
# Implementing rubber banding

Need to get around this loop <br> absolute min of 5 times / sec

– 10 times better

– more would be better

???
aside -- why 5-10 times per second?

---
# Event driven code

Needs to respond to events as they arrive

Needs to maintain state between events

---
# Solution: State Machine

.left-column-half[
Special circle start state <br> (arrow going into it)

Special circle for 'final state' <br> (really means 'reset to start')

Transitions represent actions (callbacks).
]
.rigth-column-half[

<div class="mermaid">
graph TD
S((.)) --> A((A))
A -- "Event/Callback()" --> B((B))
B -- "Event2/Callback2()" --> C[C]

linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;

class S invisible
class A start
class C finish
class B normal
</div>
]

???

---
# PPS Example: Rubber Banding

.left-column-half[
Compare to previous implementation:

```
Accept the press for endpoint P1; P2 = P1;
Draw line P1-P2;
Repeat
  Erase line P1-P2;
  P2 = current_position(); Draw line P1-P2;
Until release event;
Act on line input;
```
]
.right-column-half[

- Determine the Events (triggers)

- Determine the States

- Determine the Actions

- Determine the Queries
]
---
# PPS Example: Rubber Banding
.left-column-half[
Compare to previous implementation:

```
Accept the press for endpoint P1; P2 = P1;
Draw line P1-P2;
Repeat
  Erase line P1-P2;
  P2 = current_position(); Draw line P1-P2;
Until release event;
Act on line input;
```
]

.right-column-half[

<div class="mermaid">
graph TD
S((.)) --> A((Start))
A -- "Mouse Down:?inView/Start_Line()" --> B((Drawing))
B -- "Mouse_Move:?inView/Update()" --> B
B -- "Mouse_Release:?inView/Finish_Line()" --> C[Finished]

linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;

class S invisible
class A start
class C finish
class B normal
</div>

]

---
# PPS Example: Rubber Banding
.left-column-half[

Reading a state machine: translates input sequence into action!
- When you are in Start State, and a Mouse Down event arrives, do
  the action ```Start_line()``` and go to Drawing State. 
- Update  the line end point position every time the mouse moves. 
- When it releases (Mouse Release event), finish the line (at this
  stage a callback to the application might be appropriate)
]

.right-column-half[

linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;

class S invisible
class A start
class C finish
class B normal
</div>

]

???

How could we provide a better affordance?

Does it matter if we are using a mouse or a touch screen?

---
name: inverse
layout: true
class: center, middle, inverse
---
# Using Essential Geometry as the basis for state

---
layout:false

# Using Essential Geometry as the basis for state

What is the essence of this scrollbar?

![:img google doc with scrollbar, 30%](img/viewupdate/window.png)

How do we capture/implement that?

---
# Scrollbar State machine with Essential Geometry
.left-column-half[

Sometimes also use “guards” --> **Propositional Production System**

- predicate (Boolean expr) before event

- adds extra conditions required to fire

- typical notation: event : pred ? action

- e.g. MouseDown : InsideAboveThumb? Scrollup()

Note: FSM augmented with guards is Turing complete 
]
.right-column-half[
<div class="mermaid">
graph LR
S((.)) --> START((START))
START -- "MouseDown:InThumb?" --> SCROLLING((SCROLLING))
START -- "MouseClick:InsideAboveThumb?Scrollup()" --> DONE((DONE))
START -- "MouseClick:InsideBelowThumb?Scrolldown()" --> DONE((DONE))
SCROLLING -- "MouseMove:updateThumbDocument()" --> SCROLLING
SCROLLING -- "MouseUp:Keep()" --> DONE

linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;
linkStyle 4 stroke-width:4px;
linkStyle 5 stroke-width:4px;

class S invisible
class START start
class SCROLLING normal
class DONE finish
</div>

]
---
# Let's try it for a button

.left-column-half[
Essential geometry is:
- InsideButton
- OutsideButton

and methods for
- `indentButton()` (when button is pressed)
- `normalButton()` (when button is not pressed)
- `invokeAction()` (when the user releases in the button)
- `cancelAction()` (when the user releases outside the button)

]

---
# You should have something like this

.left-column-half[
Essential geometry is:
- InsideButton
- OutsideButton

]
.right-column-half[
<div class="mermaid">
graph TD
S((.)) --> START((START))
START -- "DOWN:Inside?indentButton()" --> PRESSED((PRESSED))
PRESSED -- "MOVE:Outside?normalButton()" --> PRESSED
PRESSED -- "UP:Outside?cancelAction()" --> DONE[DONE]
PRESSED -- "UP:Inside?invokeAction()" --> DONE
PRESSED -- "MOVE:Inside?indentButton()" --> PRESSED

linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;
linkStyle 4 stroke-width:4px;
linkStyle 5 stroke-width:4px;

class S invisible
class START start
class PRESSED normal
class DONE finish
</div>

]

---
# How do we implement this?
.left-column-half[
- Implement in `onTouch()` using a switch statement
- Assume there is an
`essentialGeometry(Point p)` method. It returns a struct that tells you what part of the geomery you are in for that point.
- Assume implementations of all the methods
- Assume a field, `state` which is the current state of the state machine
- enums `Geometry` and `State` for comparing against
]
.right-column-half[

```java
public boolean onTouch(MotionEvent e) {
 geometry = essentialGeometry(new Point(e.getX(), e.getY());
 switch(state) {
  case State.START: 
   if (geometry == Geometry.INSIDE && e.getAction() == MotionEvent.ACTION_DOWN) {
    indentButton();
    state = State.PRESSED;
   }
  case PRESSED 
    if (e.getAction() == MotionEvent.ACTION_MOVE) {
      if (geometry == Geometry.INSIDE) {
       indentButton()
  	  } else {
       normalButton()
      }
	} else if (e.getAction() == MotionEvent.ACTION_UP) {
	  state = State.START; // note we don't actually use the DONE state
	  if (geometry == Geometry.INSIDE) {
	   invokeAction()
	  } else {
	   cancelAction()
	  }
	}
	
    ...
```
]

---
# Enums

Group of named constants

- Used for PPS in colorPicker (PPS States; Essential Geometry)

- Used for PPS in Menu assignment *and* for experimental conditions

- Easy to inspect if you want to (we do this in ExperimentSession)

[Documentation](https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html)

---

.left-column[
## Harder Button

![:img FB Messenger Animation, 100%](img/viewupdate/messenger-bubble.gif)

]
.right-column[
- Determine the Events (triggers)

- Determine the States

- Determine the Queries (essential geometry, context)

- Determine the Actions
]

???
What constitutes an “event” varies

- may be just low level events, or
- higher level (synthesized) events
- e.g. region-enter, press-inside

What is missing? Query fields

---
.left-column[
## Facebook Button Solution
![:img FB Messenger Animation, 100%](img/viewupdate/messenger-bubble.gif)
]
--
.right-column[

Press:?inside => highlight(), start_animation(), small, active<br>
AnimateStep ==> update(), active<br>
AnimateFinish ==> !small, active<br>
Release:inside,small => unhighlight(), exit()<br>
Release:inside,!small => add_to_chat(), small, unhighlight(),
exit()<br>

__rest is unknowable from this animation__

<div class="mermaid">
  graph LR
  S((.)) --> A((Start))
  A -- "Press:?inside/highlight(), start_animation()" --> B((Active))
  B -- "AnimateStep,update()" --> B
  B -- "AnimateFinish,!small"--> B
  B -- "Release,inside:small, unhighlight" -->D(End)
  B -- "Release,inside:!small,add_to_chat(),unhighlight()" --> D

classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px;
classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px;
classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px;
classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF

linkStyle 0 stroke-width:4px;
linkStyle 1 stroke-width:4px;
linkStyle 2 stroke-width:4px;
linkStyle 3 stroke-width:4px;
linkStyle 4 stroke-width:4px;
linkStyle 5 stroke-width:4px;    
 
class S invisible
class A start
class D finish
class B normal
</div>

]

---
# When to use PPSs

You're probably already using them, just not intentionally (and maybe
less well as a result)

PPSs are a good way to do control flow in event driven systems

Can do (formal or informal) analysis
- are all possible inputs (e.g. errors) handled from each state
- what are next legal inputs: can use to enable / disable
Can be automated based on higher level specification

---

#Summary

State machines are very good (for this job) but do have limits

State machines don't handle independent actions very well (state explosion)

Mostly useful for smaller things

- Great for individual components
- Not so great for whole dialogs

Path of least resistance is rigid sequencing
 Ask: is this good for what I am doing?

???
xxx TODO decide whether to keep
xxx TODO decide how to end this deck and/or what other material needs
to be covered
---