10. Human Perception and Presence in MR

Rob Lindeman, Director HITLab NZ.

In popular media:

• UI about complementing the character: their personality, proficiency in technology, basic scene state (e.g. red blaring lights for bad information coming).
• Impressions matter
• Flow is a good concept to study
• Popular media can give us good ideas

Terms:

• Presence: sense of ‘being there’
• Immersion: being surrounded
• Flow: heightened state of awareness/action
• Situation awareness: clear understanding of surroundings
• Natural interaction: interaction that recedes into the background
  • Low cognitive load

‘Being there’

What does it mean to ‘be here’?

• Experience of going through some process to get to a place (e.g. walking through the door)

What does it mean to be together?

• Eye contact with others, talking, shaking hands

How can we re-create these using technology?

In a real environment, we can use:

• Hand-held mobile device
  • Phones/tablets
• In-vehicle system
  • Navigation/traffic
• Augmented reality
  • There++: augmenting reality

For a remote physical environment:

• Phone
• Video conference
  • Eye contact difficult: looking at the camera (for eye contact) means you can’t see others
• Teleoperated robots
  • Allows movement and possibly even manipulating the environment
• Drones

In virtual environments:

• Video games: FPS, MMOs
  • Can be present even without VR
  • Multiplayer games mimic physical co-presence
• Immersive learning environments
  • e.g. immersive chemistry
• Surgical simulations
  • Allows more precision and manipulators
  • Allows training on simulated data

In described environments:

• Movies
• Books
  • As long as you have the essence, the brain is able to fill in the blanks through their imagination
  • However, everyone imagines a different scene: this can lead to disappointment when a book is adapted into a movie

Game Design

What makes a good game?

A great game is a series of interesting and meaningful choices made by the player in pursuit of a clear and compelling goal

• Sid Meier

‘Natural Funativity’:

• Survival-skill training
• Needs to have the player develop a set of skills with increasing levels of difficulty
• Putting them to the test: missions, quests, levels etc.
• Prize at the end (or in the middle)
  • e.g. unlocking items, badges, leaderboards

Game structure:

• Movies:
  • (typically) have a linear structure
  • Are fixed - controlled by the writer/director/cinematographer
• In comparison, games must provide ‘interesting and meaningful choices’
  • User must be in control
  • Not fun to die due to circumstances outside your control
• Choices must make sense in the context of the story

Flow

Mihály Csíkszentmihályi, Flow: The Psychology of Optimal Experience (1990):

• Hightened sense of perception
• Highly focused on the primary task
• In the ‘sweet spot’ between frustration and boredom Occurs in athletes, writers, video gamers, programmers For game design:
• The ‘sweet spot’ for difficulty is relatively large
• Game difficulty must match the player skill (and increase over time)
  • But if it matches exactly, this itself will cause the player to get boredom. Hence, difficulty should oscillate slightly

Convexity of game play:

• Provide a choke point: all paths, regardless of what the player chose, should lead to a single result
  • e.g. bosses at the end of game stages, story progression
  • The number of choices available after every iteration should increase
    • e.g. unlocked items, skills, regions to explore
• This addresses the narrative paradox: writers can create a complete story while providing players with (the perception of) choice

Flow and convexity can be combined:

• The choke point should be at the higher end of difficulty
• After the choke point, choices can be provided to the user and the difficulty slightly decreased (relative to the player’s skill)

flOw

• By Jenova Chen (Thatgamecompany)
• Adaptive difficulty: game tries to determine player skill and adaptively change the difficulty level to match

Characterizing Flow:

• A challenge activity that requires skills
• The merging of action and awareness
  • Tight coupling between actions and responses
• Clear goals
• Direct feedback
• Concentration on the task at hand
• Sense of control
• Loss of self-consciousness
• Transformation of time

Immersion

Immersion:

• To completely surround/envelope it
  • e.g. swimming, intensive language course
• Affects all the senses
  • Sound can be as important as the visuals
  • Also need to consider touch and smell
• How can we immerse MR users?

Haptic ChairIO (Feng et al., 2016):

• Chair that looks like a joystick
• And acts like a joystick: it leans, and tilt sensors can be used as input
• HITLab added vibration floor, pan-tilt fan units:
  • Combined with VR headset for audio/video
  • Footstep vibrations and fans (wind from the ‘motion’) provide movement cues
  • Non-fatiguing: sitting down, hands free to do other work
  • Clear mapping of seat movement to camera movement

Natural interaction:

• Recedes into the background:
  • Low cognitive load for interaction techniques
  • Stimuli/feedback can be easily digested
  • Low cumber
• Multi-sensory feedback
• Multi-modal user input
  • e.g. ‘put that over there’: combines pointing (gesture) and voice
• Hybrid ways of executing commands
  • Interactions should evolve with the user
  • Provide scaffolding to novices
  • Provide fast and efficient interactions for experts

Personal experiences:

• We all filter our senses
• Variations in eyesight, hearing etc.
• Different childhood experiences
• Different moods

Presence

Types of presence:

• Presence: sense of ‘being there’
  • How virtual characters react to you
  • The depth of the interactions with the environment
    • Can you turn on the tap? Open a cupboard? Pick up a cat?
    • Every interaction has a cost, both in terms of development and performance
  • The invisibility/naturalness of the interface
  • The lack of distractions (e.g. cables)
• Co-presence: ‘being there together’
  • Multiple people can be in the same shared space without feeling ‘together’
• Tele-presence: ‘being over there’
  • Remotely present in a partially physical space
• Tele-co-presence: ‘being over there together’?

Measuring presence:

• How can be measure if someone feels ‘present’ in a game or other virtual environment?
• How can we measure the depth of presence?
• Methods:
  • Questionnaires
    • Slater Usoh Steed
    • Witmer & Singer
    • Questions must be written carefully and validated
      • Ensure they are unambiguous
    • Measurement is done after the fact
  • Behaviors
    • Watch the user and see how they react
      • If you throw something at them, do they duck?
      • If they get hit, do they scream?
      • Will they refuse to walk off a ledge?
    • Hard to measure the depth of presence (but easy to see it)
    • Issue: you may need to invent/incorporate events
  • Physiological measures
    • Possible metrics:
      • Heart rate
      • Sweat (galvanic skin response or skin conductance)
      • Breathing rate/regularity
    • Hard to fake
    • Issues:
      • Some measures take time to settle
      • May need to calibrate to a baseline
      • Need to wear sensors

The Real World

The real world is great:

• Fast update rate
• Multi-modal rendering
• Really good physics
• Nearly infinite fidelity
• Can handle massive numbers of objects and players
  • Realistic crowd behavior
• Minimal lag

Hence, it is useful to use existing things from the real world: this makes AR easier than VR in terms of fidelity.

But beyond perceptual, there is:

• Anticipation
• Expectations
• Previous experiences

We can tap into experiences already anchored in the mind of the user: provide the essence and let the brain fill in the details, or plant new experiences: seeds that can grow and become scaffolding for future experiences.

To do this:

• Prime the user to expect what you are about to show
  • A VR experience starts long before the physical experience:
    • Advertising
    • Word of mouth
    • To plant the seed, tell give them some specific information: this reduces variability between users.
      • e.g. while you wait in line at a Disney park, you are shown videos, newspaper clips describing the backstory etc. which immerse you and reduce perceived wait time
• Remove all distractions
  • Non-interactable objects (e.g. cupboards that you can’t open)
  • Lack of interaction precision
  • Fatigue
  • Bumping into cables
  • Wearing a lot of gear

The myth of technical immersion:

• Technology is not necessary to achieve immersion
  • Books are very low-tech but can still transport us to fantastic places
• Our ‘high-fidelity’ technology is still relatively low-fidelity:
  • Leverage the mind to fill in the blank
  • e.g. in Alien, you don’t see the alien until the end
  • e.g. reading a ghost story at night with a window open: the environment and story are matched
• Tasks should be:
  • Easy to learn
  • Easy to carry out
  • Not fatiguing
  • Require appropriate precision
    • e.g. movement/velocity control: need both very fine and large movements
  • Support appropriate expressiveness

Impossible spaces:

• Have a non one-to-one mapping for rotation to redirect walking and effectively increase the size of the virtual space
• Change blindness for redirected walking: modify/reconfigure the virtual space when they are looking the other way
  • Redirection also works with reaching, touching