+ - 0:00:00
Notes for current slide
Notes for next slide

The Whole Toolkit Part I

Input

Lauren Bricker

CSE 340 Spring 23

Slide 1 of 30

Goals

  • Get input from the user (part 1)
  • Produce output for the user (part 2)
  • How to store application specific data (part 3)
Slide 2 of 30

The Whole Toolkit Architecture

  • Input
    • (C/M?) Input models (events)
    • (C/M?) Event dispatch
    • (C/M?) handling (state machine)
    • (C/M?) Callbacks to application
  • Output
    • (V) Interactor Hierarchy design & use
    • (V)Drawing models (onDraw())
    • (V)Layout (onLayout() or XML)
    • (V) Damage and redraw process
  • End User
    • Input: The user does actions with the mouse, keyboard and other peripherals.
    • Output: Changes views on the screen, to audio, etc.
  • App Developer
    • Input: writes callbacks
    • Output: may call invalidate()
  • Component Developer
    • Input: needs to understand everything we talked about
    • Output: needs to understand everything we talked about
Slide 3 of 30

The Whole Toolkit Architecture

  • Input
    • Input models (events) more info
    • Event dispatch
    • Event handling (state machine)
    • Callbacks to application
  • Output
    • Interactor Hierarchy design & use
    • Drawing models (onDraw())
    • Layout (onLayout() or XML)
    • Damage and redraw process
  • Storage
    • Bundles
    • Shared Preferences
Slide 4 of 30

Events

An event is a representation of user input

  • Lots of types of devices (sound, finger, mouse, light)
  • We need device independence
    • To handle a variety of ways to get input (finger, mouse, etc)
    • We need a uniform and higher level abstraction for input (events)
  • Device differences
    • Handled implicitly by only creating events they can generate
  • Types of events
    • Discrete events (e.g. mouse or key click, "when" something happens)
    • Sampled devices
      • Handled as “incremental change” events
      • Each measurable change: a new event with new value
Slide 5 of 30

Contents of Event Record

What do we need to know about each UI event?

What: Event Type (mouse moved, key down, etc)

Where: Event Target (the input component)

When: Timestamp (when did event occur)

Value: Mouse coordinates; which key; etc.

Context: Modifiers (Ctrl, Shift, Alt, etc); Number of clicks; etc.

Slide 6 of 30

Discuss each with examples

Events

Which of the following does NOT generate an event?

A. Pressing the ALT key
B. Pressing the K key
C. Pressing your finger on the phone screen
D. Moving a mouse across the screen (on a computer)

Slide 7 of 30

Events

Which of the following does NOT generate an event?

A. Pressing the ALT key
B. Pressing the K key
C. Pressing your finger on the phone screen
D. Moving a mouse across the screen (on a computer)

ALT does not generate an event (it is a modifier) Everything else does

Slide 8 of 30

The Whole Toolkit Architecture

  • Input
    • Input models (events)
    • Event dispatch more info here and here
    • Event handling (state machine)
    • Callbacks to application
  • Output
    • Interactor Hierarchy design & use
    • Drawing models (onDraw())
    • Layout (onLayout() or XML)
    • Damage and redraw process
  • Storage
    • Bundles
    • Shared Preferences
Slide 9 of 30

Event Dispatch: Picking

0: Drawing Canvas
1: CircleA, Green
2: LineB, Orange
3: LineA, Black
4: PointA, Blue
5: PointB, Black
A picture of lines and circles

An interactor hierarchy and the corresponding interface are shown to the left. If the user clicks on the image at (4,4) what will be the order of views in the filtered pick list? Assume that the drawing canvas does not accept touch input.

Slide 10 of 30
  • 3: LineA (Black)
  • 1: CircleA (Green)

Nothing else is positionally associated

Event Dispatch: Picking

0: Drawing Canvas
1: CircleA, Green
2: LineB, Orange
3: LineA, Black
4: PointA, Blue
5: PointB, Black
A picture of lines and circles

An interactor hierarchy and the corresponding interface are shown to the left. If the user clicks on the image at (4,4) what will be the order of views in the filtered pick list? Assume that the drawing canvas does not accept touch input.

  • 3: LineA (Black)
  • 1: CircleA (Green)

Nothing else is positionally associated

Slide 11 of 30

Event Dispatch: Delivering Review

  • Top-down, Bottom-Up, Bubble-out, Focused-based: these are theoretical Event dispatch approaches.
    • Capture is rarely used by component developers
      • Capture is top-down: start on the in the biggest interactor that contains the event, then narrow in on which window actually will use the event
    • Bubbling is far more common
      • Bubbling is bottom-up - start with the window at the front (the last drawn, lowest in the interactor tree) - see if the event is consumed by that interactor. If not, go up the tree.
  • Android does it a bit differently.
    • Hacky version of picking and filtering
    • Most stuff is in bubble order, but some callbacks assume you consume the event (no option to return true vs false)
Slide 12 of 30

Event Dispatch: Bubble

A tree and picture of lines and circles

For the same image, what interactor will get the event first in Bubble?

Slide 13 of 30

Event Dispatch: Bubble

A tree and picture of lines and circles

For the same image, what interactor will get the event first in Bubble?

LineA, Black

Slide 14 of 30

Event Dispatch: Bubble

A tree and picture of lines and circles

For the same image, what interactor will get the event first in Bubble?

LineA, Black

Because it is the last thing (under the locator event) drawn (reflected in its position in the interactor hierarchy)

Slide 15 of 30

Focus versus Positional dispatch

In which of the following situations is focus dispatch used during event handling, and in which is positional dispatch used?

  • When a key is typed?
  • When a button is clicked?
  • When a user swipes on the screen?
  • When doing rubberbanding of a line (click, drag, release)
Slide 16 of 30

The Whole Toolkit Architecture

  • Input
    • Input models (events)
    • Event dispatch
    • Event handling (state machine) more info
    • Callbacks to application
  • Output
    • Interactor Hierarchy design & use
    • Drawing models (onDraw())
    • Layout (onLayout() or XML)
    • Damage and redraw process
  • Storage
    • Bundles
    • Shared Preferences
Slide 17 of 30

Essential Geometry represents places

google doc with scrollbar

  • Example: The essence (or nature) of this scrollbar/where can you interact with it is:
    • on the thumb
    • inside the scrollbar above the thumb
    • inside the scrollbar below the thumb
Slide 18 of 30

PPS uses Essential Geometry

Button

DOWN / indentButton()
MOVE /
Outside ? normalButton()
UP / cancelAction()
UP / invokeAction()
MOVE /
Inside ? indentButton()
INSIDE
OUTSIDE
Syntax error in graphmermaid version 8.8.2

Scrollbar

MouseDown / Inside
MouseMove / Inside
MouseClick /
AboveThumb?
Scrollup()
MouseClick /
BelowThumb?
Scrolldown()
MouseMove /
Outside
MouseMove /
UpdateThumbDocument()
MouseUp /
KeepLocation()
Scrolling
Ready
Syntax error in graphmermaid version 8.8.2

Then from the PPS you can generate the code for this interactor.

Slide 19 of 30

Like Button Example


FB Messenger Animation

  • Determine the Events (triggers)

  • Determine the States

  • Determine the Queries (essential geometry, context)

  • Determine the Actions

Slide 20 of 30

What constitutes an “event” varies

  • may be just low level events, or
  • higher level (synthesized) events
  • e.g. region-enter, press-inside

What is missing? Query fields

Like Button Example


FB Messenger Animation

  • Determine the Events (triggers)
    • MouseDown, MouseUp, MouseMove
  • Determine the States
    • PRESS/NOT_PRESSED
  • Determine the Queries (essential geometry, context)
    • INSIDE/OUTSIDE
  • Determine the Actions
    • highlight()/unhighlight(), startAnimation(), updateAnimation(), finishAnimation(), addToChat()
Slide 21 of 30

Like Button Example


FB Messenger Animation

MouseDown/INSIDE? highlight(),startAnimation()
updateAnimation()
finishAnimation(), !small
MouseUp/INSIDE/small? unhighlight()
MouseUp,INSIDE/!small,addToChat(),unhighlight()
PRESSED
Syntax error in graphmermaid version 8.8.2

Note : This does not include the states if the mouse moves off the button while still the user still has the mouse button pressed.

Slide 22 of 30

Reminders:

  • PPSs are good for helping you to design your interactions/interactors
  • PPSs are a good way to do control flow in event driven systems
  • You can use PPSs to do (formal or informal) analysis
    • are all possible inputs (e.g. errors) handled from each state
    • what are next legal inputs: can use to enable / disable
  • PPSs can be automated based on higher level specification
Slide 23 of 30

The Whole Toolkit Architecture

  • Input
    • Input models (events)
    • Event dispatch
    • Event handling (state machine)
    • Callbacks to application more info
  • Output
    • Interactor Hierarchy design & use
    • Drawing models (onDraw())
    • Layout (onLayout() or XML)
    • Damage and redraw process
  • Storage
    • Bundles
    • Shared Preferences
Slide 24 of 30

Callbacks in Android

  • At the time Android was created the toolkit developers have no idea how every app may want to respond to events
  • Listeners are an interface that acts as a callback method
    • Recall interfaces specify behavior (but no data) that can be inherited.
  • Must be registered with a particular View
    • The toolkit architecture (implemented in View) then delivers events that arrive at that View to those listeners
    • i.e. once the View is interacted with by the user
  • A View can have listen for different types of interactions
Slide 25 of 30

Standard Listener Interfaces

The toolkit has pre-defined interfaces so apps or components can respond to events such as clicks or touches.

// User tapped a view
public static interface View.OnClickListener { ... }
// User long pressed on a View
public static interface View.OnLongClickListener { ... }
// State of view has changed
// (e.g. user clicked outside a EditText input box)
public static interface View.OnFocusChangeListener { ... }
// user typed something
public static interface View.OnKeyListener { ... }
Slide 26 of 30

Different types of callbacks

visual description of callbacks going through an interactor hierarchy

Some callbacks are

  • registered with the system (onCreate, onResume, ...)
  • triggered by a timer.
  • triggered by our Measure (onMeasure), Layout (onLayout), Invalidate (onDraw)

And ...

  • created by the Component Developer! (i.e. onColorSelected)
Slide 27 of 30

Review: Model View Controller (MVC)

ViewControllerModelUser provides inputInputChange stateUpdate state of View(s)Triggers redrawUser sees responseloop[Every time user provides input]ViewControllerModel

Goal: separation of the view from the underlying model (data)

Slide 28 of 30

Custom Callback Example (ColorPicker)

ApplicationAbstractColorPickerViewUseraddColorChangeListener(this)Interacts with color pickeronColorSelected()ApplicationAbstractColorPickerViewUser
ColorChangeListenerAbstractColorPickerView#mColorChangeListeners+addColorChangeListener()+removeColorChangeListener()+invokeColorChangeListeners()ApplicationonColorSelected()Syntax error in graphmermaid version 8.8.2
Slide 29 of 30

End of Part I

Slide 30 of 30

Goals

  • Get input from the user (part 1)
  • Produce output for the user (part 2)
  • How to store application specific data (part 3)
Slide 2 of 30
Paused

Help

Keyboard shortcuts

, , Pg Up, k Go to previous slide
, , Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
s Start & Stop the presentation timer
t Reset the presentation timer
?, h Toggle this help
Esc Back to slideshow