09. Creating Multiple-Sensory VR Experiences

Part 1: Yuanjie Wu

Yuanjie Wu, post-doc researcher at HIT Lab.

(currently in Auckland).

Senses

Creating a realistic experience must provide a multi-sensory experience and create a sense of presence.

A VR system can be modeled as a loop of:

• Input: data coming into the system from the user
• Application: physics simulation, user interaction
• Rendering: transform of data in computer-friendly format into human-friendly format - visual, aural, haptic, olfactory, gustatory
• Output: feedback perceived by the user

What is ‘input’ and ‘output’ depends on the point of view: the system or the human.

Subjective reality: the way an individual experiences and perceives the external world in their own mind.

Brains consciously and sub-consciously find patterns. The sub-conscious can be thought of as a filter that only allows information that does not conform to the patterns to pass through.

Perceptual illusions provide insight into some of the shortcuts the brain makes:

• Jastrow and Ponzo railroad illusion: brain can misinterpret size
• Moon illusion: moon appears larger when on the horizon (compared to high in the sky) as there are foreground items that can be used as a frame of reference.
• Ouchi illusion: rectangles appear to move

Mental models: NLP (neuro-linguistic programming)

• External stimuli (senses) pass through
• Filters, which delete, distort and generalize the information
  • Based on meta programs, values, beliefs, attitudes, memories, decisions
• Which consciously and unconsciously impacts the person’s
• Internal state:
  • Mental model
  • Emotional state
  • Physiology

VR research problems:

• Avatars
• Tracking
• Cybersickness
• Locomotion
• Navigation
• Perception/cognition
• Social dynamicsoSafety
• Ethics
• Sensory delivery:
  • Tactile (e.g. force feedback, temperature, pressure)
  • Olfactory/gustatory
• Evaluation metrics
• Interaction/manipulation
• Latency/FOV
• Fatigue

Multi-sensory VR systems

• Sub-systems:
  • Stimulation of the senses
    • Requires specific hardware, software and protocols
• Data processing
  • Pre-processing:
    • Filtering
    • Serialization
  • Transmission
• Integration: combining all data into one rendering system
  • Data fusion
  • Application

Subject wearing HMD in a cage:

• Enough space to walk around a little
• External cameras track position
• Fans mounted on cage used to direct wind
  • Aroma diffusers using multiple scent bottles
• Speakers attacked to floor used for vibration
  • e.g. simulating off-road driving

Avatar system:

• Control system
  • Full body tracking with multiple Kinect cameras
    • Needed to estimate orientation - Kinects could not determine if they were looking at the person’s front or back
  • Leap motion attached to the headset for natural hand tracking
    • Limited tracking range: users had to put their hands directly in front of them
    • Fix: stick 5 Leap motion sensors onto the headset
  • HTC Vive lighthouse used for HMD positioning?

Realism:

• Appearance realism
• Behavior realism
  • Verbal behavior
  • Non-verbal behvaior
    • Body movement, facial expressions etc.

Part 2: Rory Clifford

Dr. Rory Clifford, post-doc research fellow at HIT Lab.

Focus on training simulations, cultural preservation.

What creates a profound VR experience?

• Emotion
• Sound
• Movement
  • Makes users feel present and localized within the space

In the first 30 seconds, you must:

• Grab the person’s attention
  • e.g. flashing light to grab the user’s attention
• Provide affordances to navigate the environment
  • They may be going the wrong way
  • Although both diagetic and non-diagetic affordances can be used, diagetic cues keep the user more immersed
• Provide a natural and intuitive method of interaction

Sound:

• Induces mood
• Deepens the presence
• Adds believability
• 3D spatial sound especially deepens immersion
  • Can also help with UI problems like navigation and discovery
• Don’t over do it

Movement:

• Movement types:
  • Teleportation
    • 360 video:
      • Quick and easy way to produce VR content
      • Can only teleport to pre-defined positions
  • Gaze-based
  • Physical controls
    • e.g. replica of steering wheel

Smell:

• Olfactory sensory system
• Direct connection to brain through crainal nerves: most other sensory input passes through hypothalamus - an additional step of processing
• Must limit amount of smell to prevent simulator sickness
• Can trigger memories
  • Theory: help users remember VR training when in actual scenarios

Vibro-tactile feedback:

• e.g. jolts, earthquakes, engine vibration
• Low-frequency audio passing through subwoofers or audio transducers
• Can be external (e.g. floor or other hard surface) or fitted (e.g. vest)
  • Vests: portable, but users are aware that the vest is there, reducing immersion

Haptics:

• Independent of the sound channel
• Assists with spatial awareness and helping anchor the user in virtual space
• More control over the vibration (supported in game engines)

Fire Emergency NZ (FENZ):

• Arial firefighting training
  • Can only train once a year before fire training
  • Can’t exactly start fires for training
  • Expensive: requires several aircraft
• Projector-based windows
• Headsets with multiple simulated audio channels mimicking real headsets
• Vibro-tactile feedback in chairs

Modeling the real world:

• Photogrammetry:
  • Low accuracy but provides good textures
  • Requires cleanup to reduce number of polygons
• LiDAR:
  • High-accuracy, high-polygon count
  • Camera used for texturing, but not great - should be combined with photogrammetry