name: inverse layout: true class: center, middle, inverse --- # View Updates and Essential Geometry Lauren Bricker CSE 340 Spring 2020 --- layout:false [//]: # (Outline Slide) # Today's goals - Quick follow up from Monday - Click Listeners - Other Android callbacks - Input dispatching process (toolkit architecture) - Picking (alternative: Focus) - Capture - Bubble - Propositional Production Systems in implementing interactors - Essential Geometry --- # Before anyone asks - Accessibility is due Thursday night. There is no reflection. - Examlet is on Friday (starting the last half of class). - To avoid distraction this will entirely be a "take home" examlet. You may leave "class" once we begin. (I will stay in class in case people need to come back in for clarifications). - Instructions will be on Ed. - You will be given a Google doc link to copy to your drive. You MUST have [UW's GSuite enabled](https://itconnect.uw.edu/connect/email/google-apps/getting-started/). - You will complete the quiz in your version of the Google doc - You will your solution as a PDF (File->Download->PDF) to your machine - You will then upload the quiz to Gradescope Quiz 5. - Any issues will be added to an erratta document (link will also be in the ed post.) - We will give you (some) more time to handle the technology. --- # Review of Monday - Events: logical input device abstraction - Event driven code: very different from procedural programming - Listeners are notified about events by the system and toolkit - Case study: implementing View.OnClickListener - Creating a separate class/file or an inner class - Creating an anonymous inner class - Implementing an Interface - Creating an anonymous class as a parameter - Lambdas Please open [Ed Sway](https://us.edstem.org/courses/381/sway/) --- template: inverse # Other Android Callbacks --- layout: false # Event listeners Android sets up by default (1/3) - `onCreate` - called when the activity is first created - we've seen this in our Doodle, Layout, and even EventExampleActivity apps: ```java @Override protected void onCreate(Bundle savedInstanceState) { // We want to do any view initialization work here super.onCreate(savedInstanceState); // Load the XML representation of our layout into the view setContentView(R.layout.activity_main); // Any other work we need to do like adding views or // registering click listeners! } ``` --- # Event listeners Android sets up by default (2/3) - `onStart` - Called when activity is about to be visible to the user - Always gets called after `onCreate` or `onRestart` -- - `onResume` - Called when the activity will start interacting with a use - Always gets called after `onStart` -- - `onPause` - Called when the system is about to start resuming another previous activity - __Commit changes__ to persistent data, __stop animations & intense operations__, __kill network__ requests - .red[Fast implementation required!] - the next activity will not resume until this finishes in your activity -- - `onStop`: Called when the activity is no longer visible to the user --- # Event listeners Android sets up by default (3/3) - `onRestart` - Called when the activity was previously stopped but is coming back to the user - Followed by a call to `onStart` -- - `onDestroy`: the final notice before your activity is .red[__destroyed__] -- .center.half-width-img[ ![Cookie Monster Eating](img/event-delivery/cookie-monster.gif) ] --- template: inverse ## Input Dispatch Process --- layout: false # Input __Dispatch__ Process Input thread: - When a user interacts, __events__ are created - Events go into a queue --- # Input __Dispatch__ Process Input Thread Dispatch thread: - Front event comes off queue - Event gets sent to a listener (callback) attached to a `View` or `Activity`. - The `View` or `Activity` implements the callback --- .left-column-half[ ## Event Dispatch (Summary) ![:img Picture of interactor hierarchy connected to an interface and a dotted line indicating application interface, 100%](img/pps-geom/callbacks.png)] .right-column-half[ Theoretical Dispatch Strategies - Positional - Bottom-up: The event sent to the "lowest", frontmost interactor in the tree that contains the mouse position. If it's not wanted there it goes up the tree. - Top-down: Event is sent to the topmost interactor that contains the mouse location, then passed down recursively to children. - Bubble out: Traversal starts top down, bounding rectangles are hints. Event is sent to bottom most item (drawn last), the event can bubble back (with knowledge of what was hit). - Focus-based: Windowing system determines which interactor gets the event. ] --- .left-column-half[ ## Event Dispatch ![:img Picture of interactor hierarchy connected to an interface and a dotted line indicating application interface and a do_action() call happening below the line in response to a button_pressed(), 100%](img/pps-geom/callbacks2.png)] .right-column-half[ Callbacks handle *application* response to events - Update Application Model ] --- .left-column-half[ ## Event Dispatch ![:img Picture of interactor hierarchy connected to an interface and a dotted line indicating application interface with do_action() replaced with an actionListener, 100%](img/pps-geom/callbacks3.png)] .right-column-half[ Callbacks handle *application* response to events - Update Application Model - Best implemented using custom listeners ] --- # Another type of event: `onTouchEvent` - `onTouchEvent` is a callback that can be implemented on a `View` to react to a user's touch - Often used when creating your own new component. ```java public class MyView extends View extends View.OnTouchListener { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); ... // register the touch event setOnTouchListener(this) } ... @Override public boolean onTouchEvent(MotionEvent event) { // Handle the touch event here } } ``` --- # Example: Overlapping windows .left-column40[
graph TD V0((V0)) --> V1((V1)) V0 --> V4((V4)) V1 --> V2((V2)) V1 --> V3((V3)) V4 --> V5((V5)) V4 --> V6((V6))
] .right-column50[ ![:img Example of what this tree might look like in graphical form with V0 at the back, 60%](img/pps-geom/picklist.svg) ] --- # How does _Android_ decide where to send events? .left-column40[
graph TD V0((V0)) --> V1((V1)) V0 --> V4((V4)) V1 --> V2((V2)) V1 --> V3((V3)) V4 --> V5((V5)) V4 --> V6((V6))
] .right-column50[ - Android traverses the view hierarchy (starting at root) - "Picks" `View` obects that respond to an input event - Loops through the list of `View` objects, checks if they will "capture" the event - If none capture the event, the event "bubbles" back up the `View` object list ] --- # Input Process - Picking .left-column40[
graph TD V0((V0)) --> V1((V1)) V0 --> V4((V4)) V1 --> V2((V2)) V1 --> V3((V3)) V4 --> V5((V5)) V4 --> V6((V6))
] .right-column50[ - In what order does dispatch "pick" the `View` objects? ] -- .right-column50[ - Hint: Post order traversal of the tree ] -- .right-column50[ - Picked Views = {`V2`, `V3`, `V1`, `V5`, `V6`, `V4`, `V0`} - Order matters! ] --- # Input Process - Picking .left-column40[
graph TD V0((V0)) --> V1((V1)) V0 --> V4((V4)) V1 --> V2((V2)) V1 --> V3((V3)) V4 --> V5((V5)) V4 --> V6((V6))
] .right-column50[ But we can refine this! We can filter! What are some reasons to skip views? ] -- .right-column50[ - Not inside the view - View doesn't care about that type of input ![:img Example of what this tree might look like in graphical form with V0 at the back, 40%](img/pps-geom/picklist.svg) ] --- # Input Process - Picking only those interested .left-column40[
graph TD V0((V0)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V0,V2,V5 blue class V1,V4,V3,V6 bluegreen
] .footnote[*: denotes the element responds to the event because of position/type] --- # Input Example - Capture .left-column40[
graph TD V0((V0)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V0,V2,V5 blue class V1,V4,V3,V6 bluegreen
] .right-column50[ Picked Views = {`V3`, `V1`, `V6`, `V4`} What happens next? ] -- .right-column50[ - Dispatch starts at the end of the picked `View` object list (`V4`) - Walking down the list, dispatch asks: will you consume this event? - If `true`: the event is consumed and the event propagation __stops__ - If `false`: Move to the next element in the `View` list ] --- # Input Example - Capture .left-column40[
graph TD V0((V0)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V0,V2,V5 blue class V1,V4,V3,V6 bluegreen
] .right-column50[ Picked Views = {`V3`, `V1`, `V6`, `V4`} - Dispatch starts at the end of the picked `View` object list (`V4`) - Android backtracks through the list, asking if they want to consume the event - Does this correspond to a top-down or bottom-up dispatch strategy? - What happens when we reach the last `View` in the list and no one has consumed the event? ] ??? It's top-down --- # Input Example - Bubbling .left-column40[
graph TD V0((V0)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V0,V2,V5 blue class V1,V4,V3,V6 bluegreen
] .right-column50[ Picked Views = {`V3`, `V1`, `V6`, `V4`} ] -- .right-column50[ - Android starts at the front of the list `V3`, asking if they want to consume the event - Does this correspond to a top-down or bottom-up dispatch strategy? Note: Bubbling in this case is not the bubble-out in the theoretical case. ] ??? bottom-up! --- # Input Example - Recap .left-column40[
graph TD V0((V0)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V0,V2,V5 blue class V1,V4,V3,V6 bluegreen
] .right-column50[ |State |Ordering | |:---------------|:----------------| |*Picking* | V3| V1 | V6 | V4| |*Capture Order* | 4 | 3 | 2 | 1 | |*Bubbling Order*| 1 | 2 | 3 | 4 | ] --- # Stuff to ponder: How would you handle input in a circular component? ??? Consumption is based on the bounding box... -- Delivery is based on bounding box Return `false` if input is outside circle even if it's in the bounding box --- # Stuff to ponder: How would you handle focus input? - Some event types are focus based (e.g. keyboard input) - Just skip picking: Focus list is globally created, and we walk it the same way --- # Stuff to ponder: Implementing Drag and Drop Focus? Or Positional? -- [Android's tutorial is positional](https://developer.android.com/training/gestures/scale#java) --- # How does *Android* decide where to send events? - **Capture** (most things don’t) top to bottom deliver to target object (bottom) - example: `onInterceptTouchEvent()` in Android - **Pick** to identify objects of interest (or just focus()) - `buildTouchDispatchChildList()` in Android; happens only after capture! - **Bubble** (bottom to top) - example: `onTouchEvent()` in Android - Invoke callback (wait until complete) - we do this by creating a custom listener in Android .footnote[fascinating to look at [implementation in ViewGroup](https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/view/ViewGroup.java) -- not very general!] --- # Note about event records - Where - Where is the location of the cursor (x,y position on the screen) - For focus based events, where is often the target - Value is anything you might need to know (like whether it's a 1 or 2 finger swipe; or which key was pressed) Why is this important? For *picking*: - When input is positional, *only* interactors whose bounding box includes (x,y) -- where -- are picked - When input is focus based, *only* interactors in the focus list are picked. Where isn't used in dispatch --- # Creating a new type of listener Application callbacks can be implemented as custom listeners ```java // Defines a new named inner interface for listening to an interesting event public interface MyListener { void onInterestingEvent(); // can include a parameter } // variable to store the listener in once registered. Could also be a list of // listeners that are all called if the callback is triggered. protected MyView.MyListener mListener; public void setMyListener(MyListener mListener) { this.mListener = mListener; // could also be stored in a list. } ... // somewhere else in your code, when callback time happens public void someMethod() { mListener.onInterestingEvent(); } ``` --- name: inverse layout: true class: center, middle, inverse --- # Using Input to Create Interaction Techniques ## State Machines, PPS, Essential Geometry --- layout:false # Interaction Technique A method for carrying out a specific interactive task For example: For entering a number in a range you could use... ??? have the class try to think of examples -- - (simulated) slider - (simulated) knob - type in a number (text edit box) Each is a different interaction technique --- # Example: Specify the end points for a line Could just specify two endpoints – click, click - not good: no affordance - no feedback (/ feedforward) Better feedback is to use “rubber banding” - stretch out the line as you drag - at all times, shows where you would end up
if you “let go” ??? Importance of feedback vs application callback --- # Implementing rubber banding ``` Accept the press for endpoint P1; P2 = P1; Draw line P1-P2; Repeat Erase line P1-P2; P2 = current_position(); Draw line P1-P2; Until release event; Act on line input; ``` ??? Discuss! Not event based Not in the basic event/redraw loop Potentially locks up the system --- # Implementing rubber banding Need to get around this loop
absolute min of 5 times / sec – 10 times better – more would be better but might block system. ??? aside -- why 5-10 times per second? --- # Event driven code Needs to respond to events as they arrive Needs to maintain state between events --- # Solution: State Machine - State Machines are used to respond to incoming events and allows for us to store state between events. - Start state - indicated with incoming arrow - End state - indicated with double-layered shape (means "reset to start") - Transition States - indicated with single-layered shape - Event Arrows - indicated with an arrow between states, represent different actions taken up states. .left-column40[
graph LR S((.)) --> A((A)) A -- "a" --> B((B)) B -- "b" --> B B -- "b" --> C[C] linkStyle 0 stroke-width:4px; linkStyle 1 stroke-width:4px; linkStyle 2 stroke-width:4px; linkStyle 3 stroke-width:4px; class S invisible class A start class C finish class B normal
] -- .right-column50[ Is the folowing "accepted" by the above state machine? - ab - abbbbbb - aab - abbbbba ] --- # Solution: State Machine - State Machines are used to respond to incoming events and allows for us to store state between events. - Start state - indicated with incoming arrow - End state - indicated with double-layered shape (means "reset to start") - Transition States - indicated with single-layered shape - Event Arrows - indicated with an arrow between states, represent different actions taken up states.
graph LR S((.)) --> A((A)) A -- "Event1/Callback1()" --> B((B)) B -- "Event2/Callback2()" --> B B -- "Event3/Callback3()" --> C[C] linkStyle 0 stroke-width:4px; linkStyle 1 stroke-width:4px; linkStyle 2 stroke-width:4px; linkStyle 3 stroke-width:4px; class S invisible class A start class C finish class B normal
--- # Propositional Production System (PPS) - State machine is just the start, stop and interim states with arrows with nothing on them. - PPS is a state machine with extra conditions required to fire on the arrows. - ? predicate (Boolean expr) before event, adds extra conditions required to transition - action calls - Typical notation event : pred ? action - Example MouseUp:inView?Start_Line() - Finite State Machine (FSM) augmented with guards is Turing complete --- # PPS Example: Rubber Banding .left-column-half[ Compare to previous implementation: ``` Accept the press for endpoint P1; P2 = P1; Draw line P1-P2; Repeat Erase line P1-P2; P2 = current_position(); Draw line P1-P2; Until release event; Act on line input; ``` ] .right-column-half[ - Determine the Events (triggers) - Determine the States - Determine the Actions - Determine the Queries ] --- # PPS Example: Rubber Banding .left-column40[ Compare to previous implementation: ``` Accept the press for endpoint P1; P2 = P1; Draw line P1-P2; Repeat Erase line P1-P2; P2 = current_position(); Draw line P1-P2; Until release event; Act on line input; ``` ] .right-column50[
graph TD S((.)) --> A((Start)) A -- "Mouse Down:inView?Start_Line()" --> B((Drawing)) B -- "Mouse_Move:inView?Update()" --> B B -- "Mouse_Release:inView?Finish_Line()" --> C[Finished] linkStyle 0 stroke-width:4px; linkStyle 1 stroke-width:4px; linkStyle 2 stroke-width:4px; linkStyle 3 stroke-width:4px; class S invisible class A start class C finish class B normal
] --- # PPS Example: Rubber Banding .left-column40[ Reading a state machine: translates input sequence into action! - When you are in Start State, and a Mouse Down event arrives, do the action `Start_line()` and go to Drawing State. - Update the line end point position every time the mouse moves. - When it releases (Mouse Release event), finish the line (at this stage a callback to the application might be appropriate) ] .right-column50[
graph TD S((.)) --> A((Start)) A -- "Mouse Down:?inView/Start_Line()" --> B((Drawing)) B -- "Mouse_Move:?inView/Update()" --> B B -- "Mouse_Release:?inView/Finish_Line()" --> C[Finished] linkStyle 0 stroke-width:4px; linkStyle 1 stroke-width:4px; linkStyle 2 stroke-width:4px; linkStyle 3 stroke-width:4px; class S invisible class A start class C finish class B normal
] ??? How could we provide a better affordance? Does it matter if we are using a mouse or a touch screen? --- name: inverse layout: true class: center, middle, inverse --- # Using Essential Geometry as the basis for state --- layout:false # Using Essential Geometry as the basis for state .left-column-half[ ![:img google doc with scrollbar, 80%](img/pps-geom/window.png) ] .right-column-half[ - What is the essence (or nature) of this scrollbar? Where can you interact with it? ] --- # Using Essential Geometry as the basis for state .left-column-half[ ![:img google doc with scrollbar, 80%](img/pps-geom/window.png) ] .right-column-half[ - What is the essence (or nature) of this scrollbar? Where can you interact with it? - on the thumb - inside the scrollbar above the thumb - inside the scrollbar below the thumb - inside up arrow - inside the down arrow ] -- .right-column-half[ - How do we implement a scroll bar like this? ] --- # Scrollbar State machine with Essential Geometry
graph LR S((.)) --> START((START)) START -- "MouseDown:InThumb?StartScroll()" --> SCROLLING((SCROLLING)) START -- "MouseClick:InsideAboveThumb?Scrollup()" --> DONE((DONE)) START -- "MouseClick:InsideBelowThumb?Scrolldown()" --> DONE((DONE)) SCROLLING -- "MouseMove:updateThumbDocument()" --> SCROLLING SCROLLING -- "MouseUp:KeepLocation()" --> DONE linkStyle 0 stroke-width:4px; linkStyle 1 stroke-width:4px; linkStyle 2 stroke-width:4px; linkStyle 3 stroke-width:4px; linkStyle 4 stroke-width:4px; linkStyle 5 stroke-width:4px; class S invisible class START start class SCROLLING normal class DONE finish
--- # Let's try it for a button .left-column-half[ Essential geometry is: - InsideButton - OutsideButton and methods for - `indentButton()` (when button is pressed) - `normalButton()` (when button is not pressed) - `invokeAction()` (when the user releases in the button) - `cancelAction()` (when the user releases outside the button) ] --- # You should have something like this .left-column40[ Essential geometry is: - Inside - Outside and methods for - `indentButton()` (when button is pressed) - `normalButton()` (when button is not pressed) - `invokeAction()` (when the user releases in the button) - `cancelAction()` (when the user releases outside the button) ] .right-column50[
graph TD S((.)) --> START((START)) START -- "DOWN:Inside?indentButton()" --> PRESSED((PRESSED)) PRESSED -- "MOVE:Outside?normalButton()" --> PRESSED PRESSED -- "UP:Outside?cancelAction()" --> END[END] PRESSED -- "UP:Inside?invokeAction()" --> END PRESSED -- "MOVE:Inside?indentButton()" --> PRESSED linkStyle 0 stroke-width:2px; linkStyle 1 stroke-width:2px; linkStyle 2 stroke-width:2px; linkStyle 3 stroke-width:2px; linkStyle 4 stroke-width:2px; linkStyle 5 stroke-width:2px; classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px; classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px; classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px; classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF; class S invisible class START start class PRESSED normal class END finish
] --- # How do we implement this? .left-column-half[ - Implement in `onTouch()` using a switch statement - Assume there is an `essentialGeometry(MotionEvent event)` method. It returns a enum that tells you what part of the geometry you are in for that point. - Assume implementations of all the methods - Assume a field, `state` which is the current state of the state machine - enums `EssentialGeometry` and `State` for comparing against ] .right-column-half[ ```java @Override public boolean onTouch(MotionEvent e) { EssentialGeometry geometry = essentialGeometry(event); switch (state) { case State.START: if (geometry == Geometry.INSIDE && e.getAction() == MotionEvent.ACTION_DOWN) { indentButton(); state = State.PRESSED; return true; } break; case PRESSED if (e.getAction() == MotionEvent.ACTION_MOVE) { if (geometry == Geometry.INSIDE) { indentButton(); } else { normalButton(); } return true; } else if (e.getAction() == MotionEvent.ACTION_UP) { state = State.START; // note we don't actually use the END state if (geometry == Geometry.INSIDE) { invokeAction(); } else { cancelAction(); } return true; } break; default: break; } return false; } ``` ] --- # Enums Enums are a group of named constants - Often used for developing interactions for defining PPS States and Essential Geometry - Will be used in ColorPicker and in Menu assignment for interaction *and* for experimental conditions - Good code quality: limits choices of what a variable can be. [Documentation](https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html) --- .left-column[ ## Harder Button ![:img FB Messenger Animation, 100%](img/pps-geom/messenger-bubble.gif) ] .right-column[ - Determine the Events (triggers) - Determine the States - Determine the Queries (essential geometry, context) - Determine the Actions ] ??? What constitutes an “event” varies - may be just low level events, or - higher level (synthesized) events - e.g. region-enter, press-inside What is missing? Query fields --- .left-column[ ## Facebook Button Solution ![:img FB Messenger Animation, 100%](img/pps-geom/messenger-bubble.gif) ] -- .right-column[ Press:inside? => highlight(), start_animation(), small, active
AnimateStep ==> update(), active
AnimateFinish ==> !small, active
Release:inside,small => unhighlight(), exit()
Release:inside,!small => add_to_chat(), small, unhighlight(), exit()
__rest is unknowable from this animation__
graph LR S((.)) --> A((Start)) A -- "Press:inside?highlight(),start_animation()" --> B((Active)) B -- "AnimateStep,update()" --> B B -- "AnimateFinish,!small"--> B B -- "Release,inside:small, unhighlight" -->D(End) B -- "Release,inside:!small,add_to_chat(),unhighlight()" --> D classDef finish outline-style:double,fill:#d1e0e0,stroke:#333,stroke-width:2px; classDef normal fill:#e6f3ff,stroke:#333,stroke-width:2px; classDef start fill:#d1e0e0,stroke:#333,stroke-width:4px; classDef invisible fill:#FFFFFF,stroke:#FFFFFF,color:#FFFFFF linkStyle 0 stroke-width:2px; linkStyle 1 stroke-width:2px; linkStyle 2 stroke-width:2px; linkStyle 3 stroke-width:2px; linkStyle 4 stroke-width:2px; linkStyle 5 stroke-width:2px; class S invisible class A start class D finish class B normal
] --- # When to use PPSs You're probably already using them, just not intentionally (and maybe less well as a result) PPSs are a good way to do control flow in event driven systems Can do (formal or informal) analysis - are all possible inputs (e.g. errors) handled from each state - what are next legal inputs: can use to enable / disable Can be automated based on higher level specification --- # Summary State machines are very good (for this job) but do have limits State machines don't handle independent actions very well (state explosion) Mostly useful for smaller things - Great for individual components - Not so great for whole dialogs Path of least resistance is rigid sequencing Ask: is this good for what I am doing? ???