05. Mixed Reality Displays

Rob Lindeman: Professor & Director of HIT Lab.

Displays

Definitions

Virtual Reality

Rob first defined VR as:

Fooling the senses into believing they are experiencing something they are not actually experiencing

Lindeman, 1999 (PhD)

Today, he has a new definition:

Fooling the brain into believing it is experiencing something it is not actually experiencing

Mixed Reality

Mixing (not overlaying) of real-world (RW) and computer-generated (CG) stimuli.

This requires matching attributes such as:

• Visual: Lighting, shadows, occlusion, level of fidelity
• Aural: Sound occlusion, reflection
• Other senses?

Milgram’s Reality-Virtuality continuum: different displays influence the quality of the experience.

General Display Types

NB: humans are animals and as such, were evolutionary pressures have guided the development of ours senses. Displays that leverage the different strengths and weaknesses are more likely to be effective.

Senses:

• Visual
  • Very good visuals: high framerate, good lighting simulation
• Auditory
  • Very good spatialized audio
• Haptic
  • Application-specific, cumbersome
  • Catch-all for many different senses:
    • Force/pressure
    • Slipperiness
    • Vibration
    • Wind
    • Temperature
    • Pain
    • Proprioception
  • Sensitivity varies greatly
  • Haptics is bidirectional:
    • Tight coupling between sensing and acting on the environment
    • e.g. picking up a cup: use haptics to tread the line between slipping and crushing the cup
    • Tactile/force devices:
      • Pin arrays for the fingers: individually actuated pins
      • Force-feedback ‘arms’
      • ‘Pager’ motors
      • Particle brakes: stopping motion
      • Passive haptics
      • Most successful haptics are very application-specific (e.g. surgical devices)
    • Virtual contact
      • What should we do when contact has been made with a virtual object?
        • Should the virtual hand continue to mirror the pose of physical hand, or be blocked by the wall?
      • The output of collision detection is the input to virtual contact
      • Cues for understanding the nature of contact with objects is typically over-simplified (e.g. sound)
    • Vibrotactile displays:
      • Use of vibration motors as a display
      • US Navy TSAS project: communicate which direction is ‘down’ to pilots during maneuvers
      • Haptic vest: communicate collision direction, strength to users
      • Wind feedback: head tracking + fans
• Olfactory
  • Very hard - too many types of receptors
  • Almost all human-perceivable colors can be produced from just three sub-pixel types
  • Nose has ~15,000 types of receptors
• Gustatory
  • Know the base tastes, but no way of producing or delivering them
  • Meta cookie: AR display, air pumps with different smells, (tasteless?) cookie with marker burned into it

Display anchoring:

• World-fixed
• View-fixed
• Body-worn
• Hand-held

Visual display types:

• World-fixed displays
  • Fishtank/desktop VR
  • Projection AR
• Body-worn displays:
  • Opaque HMDs (VR)
  • Transparent HMDs (AR)
• Hand-held displays:
  • Tablet/phone VR/AR
  • Boom-mounted screens (not too common today)

Mixing Reality

Visual

NB: we don’t need to simulate reality, just need to make it good enough to make the brain believe it is physically correct.

Direct:

                                    Human
Real-world ----> Environment ----> sensory ----> Nerves ----> Brain
 signal                           subsystem

                  Display?         Retina        Optic    Direct cranial
                                                 nerve     stimulation

Captured/mediated

Real-world ----> Environment ----> Capture device ----> Post-processing ----> Captured signal

Audio

Real-world ----> Environment ----> Outer ear ----> Middle ear ---> Inner ear ----> Nerves ----> Brain

• Typical AR/VR systems use speakers (environment) or headphones (outer ear)
• Mixing could also be performed in the inner middle ear using bone conduction

Mic-through AR:

• Microphone glued to earbuds
• PC mixes audio for virtual user

Hear-through AR:

• Acoustic-hear-through AR: multiple speakers placed around the room
• Bone-conduction: ears are not covered so can continue to hear
• Mixing at the sensory subsystem
• Own voice: combination of sound reaching ears through air, plus vibration through cocela

Visual Mixing

Projection:

• Project virtual content on top of the physical world
• Examples:
  • Microsoft IllumiRoom (2013):
    • Use projector to ‘extend’ TV content
    • Can also distort and re-project room texture

Optical-see-through AR:

• HMD with transparent display
• e.g. Microsoft Hololens, Magic Leap

Optical-see-through Projective AR:

• Projection onto retro-reflective surfaces: only visible to the user wearing the projector

Video-see-through AR:

• Camera on headset: camera feed mixed with virtual content and displayed in headset display
• Benefit: easy to remove things from reality: hard/impossible in optical-see-through systems
• e.g. Varjo XR-1

Visual Cues

Do we need stereo, which is one of the major things added by VR compared to traditional displays?

Monoscopic cues:

• Overlap (interposition)
• Sizing/shadows
• Size
• Linear perspective
• Texture gradient
• Height
• Atmospheric effects
• Brightness

Stereoscopic cues:

• Parallax between two images
• Only good for within a few meters of the cameras

Motion depth:

• Changing relative position of head and objects
• User (e.g. head movement) and/or object movement
  • Proprioception can disambiguate between these two cases

Physiological cues:

• The eye changes during viewing
• Accommodation: muscular changes of the eye
• Convergence: movements to bring images to the same location on both retinas

Masking/Occlusion

Making a physical object block a virtual one.

• CAVE (CAVE Automatic Virtual Environment)
  • Projection of VR content onto room surface
  • Need to create mask to prevent projection on physical objects
• HMD: not necessary; mixing being done virtually
• Fishtank VR: display edge/bezels can break effect

Real-world Problems with Immersion

• Feeling sick after using VR for a prolonged period
• Popcorn problem: can’t interact with physical objects (or eat) without taking the headset on and off
• Communication: very difficult to talk with someone using a VR headset

Dynamic immersion:

• Open VR headsets which allow the user see the real world from the user’s peripheral vision
• Linderman: replaced part of Google cardboard’s frame with LCD panels (no backlight) that could be turned on and off
• Later version added eye tracker + tiny LCDs with eyes on the outside

Visuals & Sound

Non-intrusive senses: touch etc. requires something on or in your body.

Final Thoughts

• Real world stimuli: high fidelity/low control
• CG stimuli: lower fidelity but complete control
• Far, far future: a 3D printer and robot that quickly creates objects and puts them in the environment
• Later mixing point = more ‘personal’ stimuli (closer to the brain)
• Multi-sensory approaches compensates for weaknesses in one sense with another sense