name: inverse layout: true class: center, middle, inverse --- # Core Toolkit Abstractions Lauren Bricker CSE 340 Spring 2020 --- name: normal layout: true class: --- layout: false | Hall of Shame | Hall of Fame| |---|---| | |  --- | Hall of Shame(?) | Hall of Fame | |---|---| | |  ---  --- # Announcements - We have Ed ([use us.edstem.org](https://us.edstem.org/dashboard) to log in) - Reminder: set up your machine before lab tomorrow! - Android Basics: Follow [this](https://developer.android.com/training/basics/firstapp) tutorial to create an Android Application. - Continue with the tutorial to also [run your app](https://developer.android.com/training/basics/firstapp/running-app) either on an emulator or on an Android Device. - Git Setup: [macOS](https://courses.cs.washington.edu/courses/cse154/19au/resources/assets/atomgit/macosx/), [win](https://courses.cs.washington.edu/courses/cse154/19au/resources/assets/atomgit/windows/) (ignore pieces about Atom as we use Android Studio) Make sure you have git setup on your computer, follow above instructions - If you have never used git before read through [this](http://cse340-20wi.pages.cs.washington.edu/website/docs/git.html#1) and [this](https://courses.cs.washington.edu/courses/cse331/18au/handouts/tools/versioncontrol.html) (Android Studio ≈ Intellij) - Ensure that you can clone from CSE GitLab, (we recommend cloning via SSH) - There are now two new helpful pages - An [Academic Conduct](/courses/cse340/20sp/academic-conduct) page with Application Content and Collaboration Policies section. (Linked to from home page) - A [GitGrade](/courses/cse340/20sp/docs/gitgrade) tutorial page linked to from the [docs](/courses/cse340/20sp/docs) page. Note: there are still more tutorials we are building up. ??? Our learning goals for the first lab are Android project structure, GitGrade and first assignment Doodle. Please follow the instructions below to be familiar with Android and have your Git ready before this Thursday section: --- # Java refresher course The TAs will be hosting a Java refresher that covers most facets of Java we will be using in this class (and maybe some additional Java 8+ material that is nice to know). Which includes: - Inheritance - Generics - Anonymous Inner Classes - Lambdas (and "::" notation) If these are not topics you're comfortable with (some are not directly covered in any CS class), come join us! Time: 5:30-6:30pm Thursday Zoom link in Ed --- [//]: # (Outline Slide) # Today's goals - **Toolkits & Toolkit Users** - Introduce the interactor hierarchy - How does an interactor draw? --- # What does it take to run an app on a phone? ![:youtube An android screen with many icons on it; then an app opens and the user can be seen drawing a line. When they finish an undo button appears, 2TJPjyMQGkM] ??? - Give them 60 seconds to think about it and talk with their neighbor. - What you want to get at is the application stack --- # The Application Stack .left-column[ <div class="mermaid"> graph LR ap[Application Program] hlt[High Level Tools] t[Toolkit] w[Window System] o[OS] h[Hardware] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class ap,o,w yellow class hlt,t green class h darkblue </div> ] .right-column[  ] ??? Get the class to brainstorm about what hardware is visible on the screen Point out that they all have stacks and the things we are learning are not android specific --- .left-column[  ] .right-column[ ## Android Hardware ] ??? Get the class to brainstorm about what hardware is available on android vs their laptop and how that might influence user interfaces --- # The Application Stack .left-column[ <div class="mermaid"> graph LR ap[Application Program] hlt[High Level Tools] t[Toolkit] w[Window System] o[OS] h[Hardware] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class ap,h,w yellow class hlt,t green class o darkblue </div> ] .right-column[ What services does it provide? How does this impact what you might build in an application? ] ??? Class brainstorming about what OS does 1) manages shared resources (e.g. memory, processor power) 2) has the power to start and pause and stop applicatinos --- .left-column[ # Impact of Android OS on apps ] .right-column[  ] ??? How does the operating system influence what you can build? How does this differ on an android device from a desktop? The Android Activity Lifecycle starts when an activity is launched (by the OS); which leads to callbacks to the application such as onCreate, onStart and onResume. Once the activity is running, it may be paused, stopped or destroyed. Things like priority and available memory may influence this process, as does the user opening the app. --- # The Application Stack .left-column[ <div class="mermaid"> graph LR ap[Application Program] hlt[High Level Tools] t[Toolkit] w[Window System] o[OS] h[Hardware] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class ap,o,h yellow class hlt,t green class w darkblue </div> ] .right-column[ - Manages window size and visibility across many applications - May trigger clipping and redraw - Lots of cases where this arises on a desktop (overlapping windows, minimization, etc) - Toolkit handles this for you, but can lead to callbacks; redraw events Where is this on Android? ] ??? --- # The Application Stack .left-column[ <div class="mermaid"> graph LR ap[Application Program] hlt[High Level Tools] t[Toolkit] w[Window System] o[OS] h[Hardware] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class ap,w,h,o yellow class hlt,t green class t darkblue </div> ] .right-column[ Services a toolkit supports (in the order we will learn them) - Graphics - Animation - Layout - Accessibility - Event Handling - Interactive Components - Undo - Sensing (not covered) - Data (not covered) - databases; apis ] ??? What are some things you expect every interface to have that I didn't mention here? Example: Undo (we will learn about this) Do all user interface toolkits have all of these? --- # What is an Interface Toolkit (Main subject of this course) - Difference between toolkit and IDE (Integrated Development Environment)? - Difference between toolkit and API (Application Programming Interface)? Structure of a Toolkit - Library of components - Architecture ??? Connect what they learned in 143 (i.e that they used the SDK for Java which had a bunch of libraries that had pre-defined code). The IDE most people used was JGrasp. --- # Example: Doodle assignment: Makes use of the Android Toolkit Inspired by Google's Doodles, and will include drawing animation (to be discussed): <br> [Example from last year's class](img/drawing/doodlevid.mp4) ![:youtube Animation showing images of food moving in a line down the page around a U finally forming a W, Sx8oiJGjaIM] --- .left-column40[ ## Doodle Assignment Uses of Android Toolkit **Library** (green) <div class="mermaid"> graph TD Activity[Activity] Activity -->|...| Doodler[Doodler] Doodler --> Part1[Part1] Part1 --> Part1Activity[Part1Activity] Part1 --> Part2Activity[Part2Activity] View[View] View --> ImageView[ImageView] View --> TextView[TextView] View --> LineView[LineView] classDef yellow font-size:12pt,text-align:center classDef green font-size:12pt,text-align:center classDef blue font-size:12pt,text-align:center class Part1,Part1Activity,Part2Activity,LineView yellow class Activity,View,ImageView,TextView green class Doodler blue </div> ] .right-column30[ ## Classes we are using in Doodle - Doodler (which you don't edit) *extends* [Activity](https://developer.android.com/reference/android/app/Activity) - Part1 *extends* Doodler. It implements *helper methods*. - Part1Activity and Part2Activity both extend Part1 (and use the helper methods) - LineView *extends* [View](https://developer.android.com/reference/android/view/View) (so does [ImageView](https://developer.android.com/reference/android/widget/ImageView.html) and [TextView](https://developer.android.com/reference/android/widget/TextView.html) ] ??? Green: Android toolkit Blue: Classes we provide you don't have to modify Yellow: Classes you implement Remind students that they did do inheritance in 142/143 like with Critters. --- .left-column40[ # Doodle: What you will learn - How the Android Activity Lifecycle works (*Architecture*) - How to use Android Toolkit `View` objects to show things on screen (*Library*) - How to create a custom `View` to draw lines on screen (*Library*) - All about Android's `Canvas` object and how to use it to draw on screen (*Library*) - How the Android Toolkit supports animation (*Architecture & Library*) ] .right-column40[ # Doodle: What you will do - Modify Android Activities (`Part1` and `Part1Activity`) - Implement methods to add images, text and lines - Create the custom view `LineView` - Override `onDraw` to draw on the screen in `LineView` - Use `ObjectAnimator` to animate Text and other things - Create a custom doodle in `Part2Activity` ] --- # The Application Stack .left-column[ <div class="mermaid"> graph LR ap[Application Program] hlt[High Level Tools] t[Toolkit] w[Window System] o[OS] h[Hardware] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class w,ap,o,h yellow class t green class hlt darkblue </div> ] .right-column[  ] --- # Subtle influence of tools  --- # Subtle influence of tools  --- # Subtle influence of tools Whorfian Effects - The way in which our tools influence how we think and thus what we do. - Origins are linguistic (e.g. the way in which language about color affects what we perceive) - Can even affect perception of human emotion! http://www.radiolab.org/story/91730-new-words-new-world/ --- # The Application Stack .left-column[ <div class="mermaid"> graph LR ap[Application Program] hlt[High Level Tools] t[Toolkit] w[Window System] o[OS] h[Hardware] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class w,o,h yellow class t,hlt green class ap darkblue </div> ] .right-column[  ] ??? Get class to discuss all the things that a UI like this does a little bit. What do you have to think about to build an application? How might a toolkit help with this? --- # Summary: The Application Stack .left-column[ <div class="mermaid"> graph LR ap[Application Program] hlt[High Level Tools] t[Toolkit] w[Window System] o[OS] h[Hardware] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center class ap,w,o,h yellow class hlt,t green </div> ] .right-column[ - **Application Program** - An application designed for an end user to perform specific tasks. - **High Level Tools** - Graphical interfaces that that let you specify parts of your interface (such as layout). Subject to Worfian Effects - **Toolkit** - A set of libraries and tools you use to develop applications. - **Window System** - Manages window size and visibility across applications - **OS** - The operating system that is running on the device, providing system services such as acceess to displays, input devices, file I/O - **Hardware** - The device that is running software, such as an Android Phone High level tools and the the Toolkit are generally packaged as part of a toolkit but really should be thought of as separate things. (You can program the toolkit without the tools. ) ] --- .left-column[ ## Developer roles <div class="mermaid"> graph LR ip[Interface Programmer] w[Component Developer] l[Library Extender] a[Architecture Extender] t[Toolkit Builder] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class ip,t yellow class w,l,a green </div> ] .right-column[ ## Let's say you want to build an app In this class, you will use the Android Toolkit Who are we when we build the app? ] --- .left-column[ ## Developer roles <div class="mermaid"> graph LR ip[Interface Programmer] w[Component Developer] l[Library Extender] a[Architecture Extender] t[Toolkit Builder] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class t yellow class w,l,a green class ip darkblue </div> ] .right-column[ **Interface Programmers** combine library elements according to toolkit rules & following constraints of architecture; common to all UI toolkits In Android - Construct app out of `Views` - Assembly happens in an `Activity` in `onCreate()` using `addView(View)` In Doodle - You add views when you implement `addImage`, `addText` and `addLine` in `Part1.java` - You will need to call `doodleView.addView([view])` (`doodleView` is defined in `Doodler`) ] --- # How does the toolkit know what to draw?  - Discuss ??? discuss with your neighbor based on the reading what does the toolkit need to know to draw this - what to draw; where to draw it - what are good abstractions for this? - Hierarchy normally reflects spatial containment relationships -- - Deconstruct - Let's list all the components... and arrange them in a tree ??? - deconstruct this interface into an interactor hierarchy ---  <div class="mermaid"> graph LR W(Window) --> L(LeftSide:Icon) W --> M(Middle:Label) W --> R(Right:Close Button) classDef start font-size:14pt,text-align:center classDef blue font-size:14pt,text-align:center class W start class M,R,L blue </div> --- # More complex example  --- .left-column[ ## More complex example ] .right-column[ <div class="mermaid"> graph LR W(Window) --> T(Controls) W --> M(Chats) W --> Q(Text Entry) T --> A[Button:Messages] T --> B[Label:288-88] T --> C[Button:Edit] M --> G[Button:Call] M --> H[Button:Add to Contacts] M --> E[Chat1] M --> F[Chat2...] E --> I[Date] E --> J[Chat Msg] E --> K[Arrow] classDef start font-size:14pt,text-align:center classDef blue font-size:14pt,text-align:center class W start class T,M,E,Q,A,B,C,G,H,E,F,I,J,K blue </div> ] --- # How is the tree used for drawing? Interface Programmer: - Build and interface == build an interactor hierarchy - Change an interface == change the interactor hierarchy (or its parameters) --- .left-column-half[ ## How does Android let you do this Architecture invokes the work to construct the *interactor hierarchy* by calling `onCreate()` In Doodle - *Doodler* overrides `onCreate()` (from it's grandparent *Activity* class) - *Doodler* provides a helper method for this (`doodle()`) which you override in `Part1Activity` and `Part2Activity`. ] .right-column-half[ ```java abstract class Doodler extends AppCompatActivity { // ... @Override protected void onCreate(@Nullable Bundle savedInstanceState) { super.onCreate(savedInstanceState); // ... doodle(...) } abstract protected void doodle(FrameLayout doodleView); ``` ] --- # How is the tree used for drawing? The interface programmer is not really *drawing on a canvas*, they are *adding views* to a *parent view* in `onCreate()` (or later if something changes) This is *setup* for the interface being rendered: We are constructing the interactor (also called component) hierarchy when we call these methods. This happens in `Doodle` inside of `Part1.java` in `addImage()` when we say `doodleView.addView(imageView);` You have to do it in `addText()` and `addLine()` too. --- # How is the tree used for drawing? Toolkit Architecture: - Draw an interface == walk the tree and tell interactors to draw - Update an interface == walk the tree, only redraw things that are marked as dirty - Deliver input == walk the tree and deliver events (after midterm) --- # How does an interactor draw? Interactors (Views in Android): - Receive draw (`onDraw()` in Android) - Show feedback () (Draw onto `Canvas` in Android) - Respond to input (Future assignment!) You will create a custom interactor, a `LineView`, which has to implement `onDraw(Canvas canvas)` You can use all the [Canvas](https://developer.android.com/reference/android/graphics/Canvas) methods (we'll talk more about these in the [Drawing](Drawing.html) lecture on Friday) ??? HTML is like this too <!-- --- --> <!-- .left-column-half[ --> <!-- ## Most GUIs still use the same 7-10 interaction techniques --> <!-- - Basic GUI components invented 1970s --> <!-- - Windows, Icons, Menus, Pointers (“WIMP”) --> <!-- - “Perfected” by Macintosh in 1984 --> <!-- - Not much change since then (even with web) --> <!-- ] --> <!-- .right-column-half[ --> <!-- ## Aside: Significant Stagnation --> <!--  --> <!-- ] --> <!-- ??? --> <!-- - Work well, uniform --> <!-- - Good for usability --> <!-- - GUI is victim of its own success --> <!-- - Opportunities lost by not customizing interaction techniques to tasks --> <!-- - Hard for better techniques to get traction --> <!-- - Only very recently with mobile devices (touch based) have we seen lots of new techniques get a major foothold --> <!-- --- --> <!-- .left-column[ --> <!-- ## In class Exercise: Interface Developer, using Android Toolkit --> <!-- Quick Tour | tutorial --> <!-- for [building a ui](https://developer.android.com/training/basics/firstapp/building-ui)] --> <!-- .right-column[ --> <!--  --> <!-- ] --> --- .left-column[ ## Developer Roles <div class="mermaid"> graph LR ip[Interface Programmer] w[Component Developer] l[Library Extender] a[Architecture Extender] t[Toolkit Builder] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class ip,t yellow class w,l,a green class w darkblue </div> ] .right-column[ **Components** are small, reusable, well defined pieces of code that allow programmers to design and develop our UIs in a consistent way ([source](https://blog.bitsrc.io/building-a-consistent-ui-design-system-4481fb37470f)). - A UI component can create both functional and visual consistency in an application or set of applications. - Often components support new forms of input/direct manipulation. - Component development is supported by many toolkits **Component Developers** create these new, re-usable components. Example: In the Doodle assignment you will be developing the `LineView` interactor, a new component for drawing lines on the screen at certain locations. ] --- # Component Developer: Developing Novel Interaction Techniques .left-column-half[ - A method for carrying out a specific interactive task - Example: enter a number in a range - Swiping to invoke an action - Gesture based text entry  ] .right-column-half[  ] --- .left-column[ ## Developer Roles <div class="mermaid"> graph LR ip[Interface Programmer] w[Component Developer] l[Library Extender] a[Architecture Extender] t[Toolkit Builder] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class ip,t yellow class w,a green class l darkblue </div> ] .right-column[ **Library Extenders** are similar to component developers, but may create new forms of layout, types of input, etc. - Supported by a few toolkits - Examples: - The Java JDK can be extended by developers who create a set of components in a `.jar` file that is included with a software project (example [JFreeChart](http://jfree.org/jfreechart/)) - JavaScript has been extended by [many libraries](https://en.wikipedia.org/wiki/List_of_JavaScript_libraries), such as Bootstrap or jQuery. ] --- # Library Extender: Toolkit Support for Styling .left-column-half[  ] .right-column-half[ subArctic allowed visually rich, dynamically resizable, images to be provided using primarily conventional drawing tools (and with no programming or programming-like activities at all). Scott E. Hudson and Kenichiro Tanaka. (UIST '00) ] --- .left-column[ ## Developer Roles <div class="mermaid"> graph LR ip[Interface Programmer] w[Component Developer] l[Library Extender] a[Architecture Extender] t[Toolkit Builder] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class ip,t yellow class w,l,a green class a darkblue </div> ] .right-column[ **Architecture Extenders** modifies the flow of information within a toolkit to create entirely new effects (e.g. adding support for command objects) Supported by very few toolkits ] --- # Architecture Extender: Animation .right-column50[  ] Integrated into existing GUI toolkit Primary abstraction: transition - models movement over time - through arbitrary space of values (e.g., color) - screen space is most common .footnote[Great post about [types of animation on mobile phones](https://yalantis.com/blog/-seven-types-of-animations-for-mobile-apps/), also source of the image] --- .left-column[ ## Developer Roles <div class="mermaid"> graph LR ip[Interface Programmer] w[Component Developer] l[Library Extender] a[Architecture Extender] t[Toolkit Builder] classDef yellow font-size:14pt,text-align:center classDef green font-size:14pt,text-align:center classDef darkblue font-size:14pt,text-align:center class ip,t yellow class w,l,a green class t darkblue </div> ] .right-column[ **Toolkit builders** create entirely new toolkits that enable radical new forms of interaction e.g. RapID https://make4all.org/portfolio/rapid/ ] --- # Toolkit Builder: Physical Interfaces Creates entirely new toolkits that changes what we can do ![:youtube Physical Interface made out of RFID tags,4k15uXpp7-g] --- # Toolkit for Overloading Existing Interfaces Prefab supports pixel based enhancements: write interpretation logic that can be composed, reused, and shared to manage the multi-faceted nature of pixel-based interpretation; robustly annotate interface elements with metadata needed to enable runtime enhancements. ![:youtube Prefab Demo,lju6IIteg9Q] --- # Summary: Developer Roles .left-column[ <div class="mermaid"> graph LR ip[Interface Programmer] w[Component Developer] l[Library Extender] a[Architecture Extender] t[Toolkit Builder] classDef yellow font-size:14pt,text-align:center class ip,w,l,a,t yellow </div> ] .right-column[ - **Interface Programmer** - Uses high level tools and toolkit to create a user interface. - **Component Developer** - Creates new, re-usable interators or interaction techniques. - **Library Extender** - Similar to Component Developer, creates a system of components such as new forms of layout, types of input, etc. - **Architecture Extender** - Modifies the flow of information within a toolkit to create entirely new effects - **Toolkit Builder** - Creates entirely new toolkits that enable radical new forms of interaction. ] --- # Summary: In this class we look at General Principals Particular toolkits tend to be mired in a particular platform Along comes the next big thing (e.g., iPhone) and you’ll need to learn a new one We frequently will use Android as an example of these concepts If you learn the structures/concepts (and esp. rationale behind them) that survives change better
