Virtual Reality Systems

CSE 490V | Winter 2020

Overview

Modern virtual reality systems draw on the latest advances in optical fabrication, embedded computing, motion tracking, and real-time rendering. In this hands-on course, students will foster similar cross-disciplinary knowledge to build a head-mounted display. This overarching project spans hardware (optics, displays, electronics, and microcontrollers) and software (JavaScript, WebGL, and GLSL). Each assignment builds toward this larger goal. For example, in one assignment, students will learn to use an inertial measurement unit (IMU) to track the orientation of the headset. In another assignment, students will apply real-time computer graphics to correct lens distortions. Lectures will complement these engineering projects, diving into the history of AR/VR and relevant topics in computer graphics, signal processing, and human perception. Guest speakers will participate from leading AR/VR companies, including by hosting field trips.

Acknowledgments

This course is based on Stanford EE 267. We thank Gordon Wetzstein for sharing his materials and supporting the development of CSE 490V. We also thank Brian Curless, David Kessler, John Akers, Steve Seitz, Ira Kemelmacher-Shlizerman, and Adriana Schulz for their support.

Requirements

This course is designed for senior undergraduates and early MS/PhD students. No prior experience with hardware is required. Students are expected to have completed Linear Algebra (MATH 308) and Systems Programming (CSE 333). Familiarity with JavaScript, Vision (CSE 455), and Graphics (CSE 457) will be helpful, but not necessary. Registration is limited to 40 students.

Teaching Staff
Douglas Lanman
Affiliate Instructor, University of Washington, CSE
Director, Display Systems Research, Facebook Reality Labs

Douglas is the Director of Display Systems Research at Facebook Reality Labs, where he leads investigations into advanced display and imaging technologies for augmented and virtual reality. His prior research has focused on head-mounted displays, glasses-free 3D displays, light-field cameras, and active illumination for 3D reconstruction and interaction. He received a BS in Applied Physics with Honors from Caltech in 2002, and his MS and PhD in Electrical Engineering from Brown University in 2006 and 2010, respectively. He was a Senior Research Scientist at Nvidia from 2012 to 2014, a Postdoctoral Associate at the MIT Media Lab from 2010 to 2012, and an Assistant Research Staff Member at MIT Lincoln Laboratory from 2002 to 2005. His recent work has focused on developing the Half Dome prototype series, showcasing wide-field-of-view and compact varifocal HMDs with AI-driven rendering.

Ethan Gordon
PhD Student, University of Washington, CSE

Ethan is a second-year PhD student in the Personal Robotics Lab at the University of Washington. His current research interests are in robotics and AI with a focus on physics-based deformable object manipulation and the application of assistive feeding. He has previously done work in virtual reality and integrated photonics.

Kirit Narain
Undergraduate, University of Washington, CSE

Kirit is an undergraduate in Computer Science with a special interest in AR/VR. He is President of the Extended Reality Association and has previously completed research into building virtual reality headsets with the UW Reality Lab. His current projects include writing an XR data visualization tool and a control system for an electric racing motorcycle.

VR Headset Development Kit
1 / 16
2 / 16
Step 1: Use an ultrasonic knife to remove excess plastic from the enclosure.
3 / 16
Step 2a: Begin assembling the display by connecting the HDMI driver board.
4 / 16
Step 2b: Affix the LCD panel to the acrylic plate with double-sided tape.
5 / 16
Step 2c: Fold over the HDMI driver board without damaging the flex circuit.
6 / 16
Step 2d: Affix the HDMI driver board to the acrylic plate and attach the cables.
7 / 16
Step 2e: Test the display.
8 / 16
Step 3a: Place the display into the enclosure and align the center of the screen.
9 / 16
Step 3b: Tune the distortion correction so that lines appear straight.
10 / 16
Step 4a: Assemble the IMU components.
11 / 16
Step 4b: Place the IMU components into the enclosure.
12 / 16
Step 4c: Tune the IMU filtering.
13 / 16
Step 5a: Create a custom ARToolKit marker and attach it to the HMD.
14 / 16
Step 5b: Measure the marker offset, relative to the halfway point between the lenses.
15 / 16
Step 5c: Tune the positional tracking.
16 / 16
Step 5d: Put everything together to complete your own HMD.

Students will be provided a kit to build their own head-mounted display, including an LCD, an HDMI driver board, an inertial measurement unit (IMU), lenses, an enclosure, and all cabling. Kits must be returned at the end of the course. All software will be developed through the homework assignments. Component details are listed below.

Component Model Details
HMD Enclosure View-Master Deluxe VR Viewer Mattel
Display Panel Topfoison 5.5″ 1920×1080 LCD Topfoison
Display Mount Acrylic Sheet (133mm × 75mm × 2.5 mm) TAP Plastics
Microcontroller Teensy 3.2 PJRC
IMU InvenSense MPU-9250 HiLetgo
Breadboards Elegoo Mini Breadboard Kit Elegoo
Jumper Wires Edgelec 30cm Jumper Wires (Male to Male) Edgelec
HDMI Cable StarTech 6′ High Speed HDMI Cable StarTech
USB Cables Anker 6′ Micro USB Cable (2-Pack) Anker
Tape Scotch Permanent Double-Sided Tape Scotch
Velcro Strenco 2″ Adhesive Hook and Loop Tape Strenco
Final Projects

Eye Tracking for VR Gaming by Alex Zhang
Virtual Batting Cage by Dylan Hayre
VR Galaxy Tour by Natalia Abrosimova and Wenqing Lan
Foveated Ray Tracing by Frank Qin
Finger Tracking using Magnetometers by Alexander Mastrangelo and Paul Yoo
360° Vision using FOV Minification by Neil Sorens
Ear Hands: Spatial Audio for VR Gaming by Christie Zhao and Thomas Hsu
Stereoscopic Ray Tracing for VR by Michal Piszczek
AR Basketball Training by Eugene Jahn
Inverse Kinematics and Full-Body Tracking for VR by Terrell Strong
Exploring Wide-Field-of-View VR Headsets by Andrew Wei Daoyi Zhu
3D Drawing in VR by Daniel Lyu and Lily Zhao
Bird-like Flight in VR by Rory Soiffer and Everett Cheng
VR Volume Rendering by Nguyen Duc Duong, Xiao Liang, and Jeffery Tian
Crime Scene Investigation by Zhu Li and Weihan Ji
Exploring Two-Handed Interactions by Andrew Rudasics

 Description Materials
 Hardware
 Exploring Wide-Field-of-View VR Headsets Proposal and Report
 360° Vision using FOV Minification Proposal and Report
 Finger Tracking using Magnetometers Proposal and Report
 Body and Hand Tracking
 Inverse Kinematics and Full-Body Tracking for VR Proposal and Report
 Exploring Two-Handed InteractionsProposal and Report
 Bird-like Flight in VR Proposal and Report
 Eye Tracking
 Eye Tracking for VR Gaming Proposal and Website
 Accelerated Raytracing for VR Proposal and Slides
 Rendering
 Stereoscopic Ray Tracing for VR Proposal and Report
 Foveated Ray Tracing Proposal and Report
 VR Volume Rendering Proposal and Report
 Audio
 Ear Hands: Spatial Audio for VR Gaming Proposal and Report
 Training and Education
 AR Basketball Training Proposal and Website
 VR Batting Cage Proposal and Report
 VR Galaxy Tour Proposal and Report
 Applications
 Crime Scene Investigation Proposal and Report
 3D Drawing in VR Proposal and Website
 Sketching in AR Proposal and Report
 VR Dueling Proposal and Report
Schedule
Date Description Materials
Wednesday
January 8
Introduction to VR/AR Systems Slides
Sutherland [1968]
Friday
January 10
Head-Mounted Displays
Part I: Conventional Optical Architectures
Slides
Kore [2018]
Wednesday
January 15
Head-Mounted Displays
Part II: Emerging Optical Architectures
Slides
Friday
January 17
The Graphics Pipeline and OpenGL
Part I: Overview and Transformations
Slides and Notes
Marschner (Ch. 6 & 7)
Wednesday
January 22
The Graphics Pipeline and OpenGL
Part II: Lighting and Shading
Slides
Marschner (Ch. 10 & 11)
Friday
January 24
The Graphics Pipeline and OpenGL
Part III: OpenGL Shading Language (GLSL)

Kirit Narain
Slides
Wednesday
January 29
The Human Visual System
Marina Zannoli (Facebook)
Slides
LaValle (Ch. 5 & 6)
Friday
January 31
The Graphics Pipeline and OpenGL
Part IV: Stereo Rendering

Ethan Gordon
Slides
Wednesday
February 5
Inertial Measurement Units
Part I: Overview and Sensors
Slides and Notes
LaValle (Ch. 9.1 & 9.2)
Friday
February 7
Inertial Measurement Units
Part II: Filtering and Sensor Fusion
Slides
Wednesday
February 12
Positional Tracking
Part I: Overview and Sensors
Slides and Notes
Friday
February 14
Positional Tracking
Part II: Filtering and Calibration
Slides
Wednesday
February 19
Advanced Topics
Part I: Spatial Audio

Lakulish Antani (Valve)
Slides
LaValle (Ch. 11)
Friday
February 21
Advanced Topics
Part II: Engines and Emerging Technologies
Slides
Wednesday
February 26
Advanced Topics
Part III: VR Video Capture
Slides
Friday
February 28
Advanced Topics
Part IV: Direct-View Light Field Displays
Slides
Wednesday
March 4
Final Project Working Session
Friday
March 6
Industry Presentation: Microsoft HoloLens
Mark Bolas, Joel Kollin, and Ishan Chatterjee
Wednesday
March 11
Industry Presentation: Valve Corporation
Jeremy Selan, Alan Yates, Kerry Davis, and Jeep Barnett
Friday
March 13
Final Project Working Session
Friday
March 20
Final Project Presentations (Open to the Public)
Zoom video conference from 12:30pm to 4:00pm
Slides
Assignments

Students will complete six homeworks and a final project. Each homework is accompanied by a lab (a tutorial video). Labs must be completed before starting the homeworks. We encourage formatting written portions of homework solutions using the CSE 490V LaTeX template. Students must submit a one-page final project proposal and a final report. Final reports may take the form of a website or a conference manuscript.

Due Date Description Materials
Thursday
January 23
Homework 1
Transformations in WebGL
Lab 1 and Video
Assignment and Code
Solutions
Thursday
January 30
Homework 2
Lighting and Shading with GLSL
Lab 2 and Video
Assignment and Code
Solutions
Monday
February 10
Homework 3
Stereoscopic Rendering and Anaglyghs
Lab 3 and Video
Assignment and Code
Solutions
Friday
February 14
Homework 4
Build Your Own HMD
Lab 4 and Video
Assignment and Code
Solutions
Monday
February 17
Final Project Proposal Directions and Template
Example
Saturday
February 22
Homework 5
Orientation Tracking with IMUs
Lab 5 and Video
Assignment and Code
Solutions
Sunday
March 1
Homework 6
Pose Tracking
Lab 6
Assignment and Code
Solutions
Thursday
March 19
Final Project Report Template
Grading and Collaboration

The grading breakdown is as follows: homeworks (70%) and final project (30%).

Projects are due by midnight on the due date. Late assignments are marked down at a rate of 25% per day. If you fail to turn in an assignment on time it is worth 75% for the first 24 hours after the deadline, 50% for the next 24 hours, 25% for the next 24 hours, and then it is worth nothing after that. Exceptions will only be given with prior instructor approval.

While the headset development kits will be shared, students are expected to individually write their homework solutions. Students may collaborate to discuss concepts for the homeworks, but are expected to be able to explain their solutions for the purposes of grading by the instructor and TAs. Final project groups can be as large as three students, subject to instructor approval.

Textbooks and Resources

Lectures are supplemented by course notes, journal articles, and textbook chapters. The following textbooks will be used for CSE 490V, which are freely available to University of Washington students via the links below.

All software will be developed using JavaScript, WebGL, and GLSL. Students should review the following tutorials and online resources to prepare for the labs, homeworks, and final projects.

Office Hours and Contacts

We encourage students to post their questions to Piazza. The teaching staff can also be contacted directly at . The instructor and TAs will hold weekly office hours at the following times.