01. Introduction

Course Administration

Course objectives:

MR theory
MR technology (mainly VR + AR)
Designing and building MR experiences
- Learn about the theories and technologies used to create MR experiences

Topics:

Introduction to Mixed Reality
AR development tools, and designing effective AR experiences
VR development tools, and designing effective VR experiences
Tracking, calibration and registration for AR
- With Richard Green
Mixed reality displays
Interaction in VR
Interaction in AR
Collaboration in mixed reality
Creating multiple-sensory VR experiences
- Haptics etc.
Human Perception and Presence in mixed reality
Data Visualization in Mixed Reality
- Multi-dimensional data sets
Evaluating immersive experiences

Staff:

Stephan Lukosch:
- Course coordinator, main lecturer, currently acting director for HIT lab
Adrian Clark
Rory Clifford
Richard Green
Rob Lindeman (HIT lab director, on sabbatical)
Tham Piumsomboon
- Lecturer in product design
Yuanjie Wu

Labs:

HIT lab 2nd floor (John Britten building)
- Limited lab assignments: focus on project work
- 9 workstations (27 students, 3 people per group)
  - Admin rights
  - Must sign HIT lab equipment-use policy
- TAs will provide technical support of setting up development environment (Unity 3D)
  - Extremely full-featured: focus on the features that are important
- Stephan will give general feedback on research project

Assessment:

Research project:
- 30% of course grades
- Max. 3 students
- Commented, documented source code
- Demonstration/video of project
Research paper:
- 30% of course grades
- 6-page conference-style paper (< 4000 words)
No contribution sheet for groups: assume equal effort
Exam: 2 hour open-book exam

Research project:

Teams of 3
Hybrid tabletop game: physical game elements augmented with virtual information (e.g. 3D objects, animations)
Requirements:
- Visualize several digital game elements anchored in the real world
- Support some form of interaction with the digital game elements (touch interaction, interaction between multiple targets, based on distance between device and marker)
- Players can see and interact with the digital game elements via their smartphone
  - Not enough HMDs for everyone
  - Unity runs on macOS, Linux, Windows
  - Vuphoria runs on Android and iOS, but iOS deployment requires a Mac
  - Unity integrates with Plastic SCM: free for <= 3 people
  - Can borrow a webcam if required
  - Can try use HMDs, but probably difficult

Mixed Reality

A continuum:

Real environment
AR: augmented reality
- Add digital information to the real world
AV: augmented virtuality
- In a virtual world, apart from a few things
  - e.g. VR car simulator, but user can see the real steering wheel
VR: virtual environment
- Everything is virtual

In terms of interactions:

Reality: ubiquitous computers which you interact with alongside the real world
AR: augmented using input from both the user and the environment
VR: completely cut off from the real world: only interaction with the computer

Virtual Reality

VR:

Replicates an environment, real or imagined
Simulates a user’s physical presence and environment to allow for user interaction

Defining characteristics of VR:

Environment simulation
Presence
Interaction

AIP Cube (Zeltzer, 1992)

Three axes:

Autonomy:
- User can react to events and stimuli
- Head tracking, body input
  - User can change their viewpoint
Interaction
- User can interact with objects environment
- User input devices, HCI
Presence
- User feels immersed through sensory input and output channels

VR is at extreme end of all three axes of the AIP cube.

Very hyped in 1980s/1990s:

Lagging technology
Lack of understanding, usability
No ‘killer app’
Except some specific scenarios
- Surgical simulation
- Military training
- Phobia therapy

Keys to success:

High fidelity/realism: graphics, audio, haptics, behaviors
Low latency: tracking, collision detection, rendering, networking
Ease of use: for programmers and users
Compelling content
Responsive expressiveness (natural behaviors)

Current state of senses:

Visual: good
- Hard to match eye’s FoV though
Aural: good spatialized audio
Olfactory (Smell): too many types of receptors; very hard
Haptics: application-specific and cumbersome
Gustatory (taste): base tases are known, but very hard

Simulator sickness:

General discomfort
Fatigue
Headache
Eye strain
Difficulty focusing
Salivation increasing
Sweating
Nausea
Difficulty concentration
‘Fullness of the head’
Blurred vision
Dizziness with eyes open
Dizziness with eyes closed
Vertigo
Stomach awareness
Burping

Factors negatively influencing VR:

Latency
Miss-calibration of tracking
Low-tracking accuracy
Low-tracking precision
Limited FoV
Low refresh rate
Low resolution
Flicker/stutter
Real-world stimuli
Lack of depth cues
Device weight
Heat
Fogging of screens

Delay/latency is one of the main contributing factors to simulator sickness. The system must complete several tasks in series, which can lead to noticeable high latency:

Tracking delay
Application delay
Rendering delay
Display delay

VR Output

Sound:

Display techniques:
- Multi-speaker output
- Headphones
- Bone conduction
Spatialization vs localization
- Spatialization: processing of sound signals to make them seem to emanate from a specific point in space
- Localization: our ability to identify the source position of a sound

Smell:

Two main problems
- Scent generation
  - The nose has tens of thousands of receptor types
- Delivery
  - How to deliver the scent to the user (and hopefully only to them) and remove it quickly

Touch:

Haptic feedback comes in many difference senses:
- Force/pressure
- Slipperiness
- Vibration
- Wind
- Temperature
- Pain
- Proprioception
- Balance (?)
Most density populated area: fingertips, lips, tongue
Two-point discrimination: how far away do two points need to be in order to sense them as two separate touches rather than one single touch?
- 2-3 mm in finger
- 6 mm on cheek
- 39 mm in back
Cyberglove:
- ~100K
- Tracks hand motion
- Contains motors to block finger movement: creates impression of actually grabbing something
Force-feedback arms:
- Stylus attached to robot arm
- Can be used for sculpting: resistance varies with the material

VR Interaction

Interaction with VR:

Keyboard/mouse not very attractive:
- Cannot see them
- Don’t want to be anchored to a desk: want to move around
- No good 3D mappings

Basic VR interaction tasks:

Object selection and manipulation
- Problems:
  - Ambiguity
  - Judging distance
- Selection approaches
  - Direct/enhanced grabbing
  - Ray-casting techniques
  - Image-plane techniques
- Manipulation approaches:
  - Direct position/orientation control
  - Worlds in miniature (God mode)
  - Skewers
  - Surrogates
Navigation
- Wayfinding: how do I know where I am, how do I get there?
- People get lost/disoriented easily: need maps
- Limited physical space; possibly infinite virtual space
  - Not a 1:1 mapping between their physical and virtual position, making it easy to get disoriented
- Different types of travel
  - Walking/running
  - Turning
  - Side-stepping
  - Back-stepping
  - Crawling
  - Quick start/stop
  - Driving
  - Flying
  - Teleporting
- Need to do other things while traveling
- Impossible spaces
  - Change blindness redirection: change the geometry of the space behind them
    - Suma, E. A.; Lipps, Z.; Finkelstein, S. L.; Krum, D. M. & Bolas, M. T., Impossible Spaces: Maximizing Natural Walking in Virtual Environments with Self-Overlapping Architecture, IEEE Trans. Vis. Comput. Graph., 2012, 18, 555-564
  - Humans trust their visual sense more than their memory
  - Changing rotation angle?
System control:
- Changing settings
- Manipulating widgets:
  - Lighting effects
  - Object representation
  - Data filtering
- Approaches:
  - Floating windows
  - Hand-held windows
  - Gestures
  - Menus on fingers
Symbolic input: typing/inputting text/numbers
Avatar control
- Body sensors to accurately map avatar to real user?
- Or approximate with head and hand position?

The “optimal” interface depends on:

The capabilities of the user
- Dexterity
- Level of expertise
The nature of the task being performed
- Granularity
- Complexity
The constraints of the environment
- Stationary, moving, noisy, etc.

Augmented Reality

Azuma (1997):

Fundamental article on AR
Defined AR as:
- Combining real and virtual images
- Interactive, real-time
- Registered in 3D: positioned at a real position in the world

AR feedback loop:

User:
- Observes AR display
- Controls the viewpoint
- Interacts the content
System:
- Tracks the user’s viewpoint
- Registers the pose in the real world with the virtual environment
- Presents situated visualization

Requirements:

Display: must combine real and virtual images
Interactive in real-time
Registered in 3d: viewpoint tracking

History:

1968: Sutherland HMD system
- System would hang from the ceiling
1970-80s: UC air force SuperCockpit program
1990s: Boeing wire harness assembly
Now:
- Magic books: virtual content shown over pages
- Magic mirror: ‘mirror’ overlays X-ray image over color image
- Remote support

Display types:

Head-attached
- Head-mounted display/projector
- Two types:
  - Occluded/video: essentially a VR headset with a camera feed streamed through it
    - e.g. Varjo XR-1:
      - Low-latency (~20 ms) cameras
      - 1080p resolution per eye
      - Tethered
      - 87 degree FoV
  - Optical see-through: transparent display that overlaying content
    - No/lower distortion
    - Safer: use will always be able to see real world
      - No latency for real content
    - Images will be a bit transparent as well
    - More connected with the real world
    - e.g. Hololens, Magic Leap
      - Hololens has ~30 degree FoV: very limited
Body-attached
- Hand-held display/projector (smartphones)
Spatial
- Spatially-aligned projector/monitor
  - Project images onto a real object
  - e.g. pool table
  - Everyone can see the content: not as awkward

Tracking:

Continually locating the user’s viewpoint when moving
Position (XYZ) and orientation (RPY)

Registration:

Positioning virtual objects in relation to the real world
Anchoring a virtual object to a real object when a view is fixed

Tracking requirements:

Augmented reality information display
- World-stabilized: hardest, must track position + rotation
- Body stabilized: fixed distance from your body: must track rotation
- Head stabilized: easiest

Tracking technologies:

Active:
- Mechanical, magnetic, ultrasonic
- GPS, Wi-Fi, cellular
Passive
- Inertial sensors (IMU: compass, accelerometer, gyro)
- Computer vision:
  - Marker-based tracking
    - ARToolkit
    - Research project will use marker-based tracking for reliability
  - Natural feature tracking
    - Vuforia texture tracking
      - Can handle partially-occluded markers
Hybrid tracking
- Combined sensors
- e.g. MonoSLAM

Evolution of AR interfaces (more expressive/intuitive going down):

Browsing:
- Simple input:
  - Very limited modification of virtual content
  - e.g. placing furniture in room: can control position and rotation
- Viewpoint control
- Handheld AR displays
- Information registered to real-world context:
  - e.g. AR map UI
3D AR:
- 3D UI
- Often use HMDs, 6DoF head-tracking
- Dedicated controllers (6DoF)
- 3D interaction: manipulation, selection, etc.
Tangible UI
- Augmented surfaces
- Object interaction
- Familiar controllers
- Indirect interaction
- Based on Tangible Bits vision (Ishii and Ullmer, 1997):
  - Give physical form to digital information
  - Make bits directly manipulable and perceptible
  - Seamless coupling between physical objects and virtual data
Tangible AR
- Tangible input principles applied to AR
- AR overlay
- Direct interaction
- Physical controllers for moving virtual content
- Support for spatial 3D interaction techniques
- Time and space multiplexed interaction
- Multi-hand interactions possible
Natural AR
- Interacting with AR content in the same way as real world objects
- Natural user input: body motion, gesture, gaze, speech
- e.g. overhead depth sensing camera
  - Create real-time hand model, point-cloud
  - Overlay graphics (spider)
  - Gesture interaction
  - Demo: spider on desk: occluded by hand, can crawl over hand