12. Evaluating Immersive Experiences

Can simply ask the player for their opinion, but these statements are qualitative.

Through validated instruments that use questionnaires, you can get quantitative data (e.g. on situational awareness, workload).

There are many methods to achieve this:

• Usability (System Usability Scale (SUS))
• Game Experience Questionnaire (GEQ)
• Situational Awareness (Overview and SART)
• NASA Task Load Index (TLX)
• Simulation Workload measure (SIM-TLX)
• Immersive Tendencies Questionnaire (ITQ)
• iGroup Presence Questionnaire (IPQ)
• User Experience Questionnaire (UEQ)
• Game Engagement Questionnaire (GEQ)
• Revised Game Engagement Model (R-GEM)
• Revised Personal Involvement Inventory (PII)
• Flow Short Scale

Engagement

What is engagement?

• Some disagreement between academics in what it is and how you quantify it
• Benyon et al. 2005:
  • Must be accessible, usable and acceptable
  • Should provide experiences that pull people in to create experiences that are:
    • memorable,
    • satisfying,
    • enjoyable,
    • rewarding
• IJsselsteijn et al. 2008:
  • Sensory and imaginative immersion
  • Tension
  • Competence that is asked of the user
  • Flow
  • Negative/positive effect on the user
  • Challenge

Elements of Flow (Csikszentmihalyi):

• Be feasible for the user to complete the task
• Allow the user to concentrate on the task
• Have clearly defined goals
• Provide feedback on the user’s actions
• Feel involved in the situation
• Give the user control over the situation and goals
• Allow for a loss of self-conciousness: stop being aware of themselves
• Transformation of time: forget about time passing by
• Autotelic experience: activities should be intrinsically rewarding

O’Brien & Toms:

• Point of engagement: user decides to use the system based on factors such as:
  • Aesthetics
  • Novelty
  • Interest
  • Personal motivations
  • Specific/experimental goals
• Engagement:
  • Aesthetics and sensory appeal
  • Attention
  • Awareness
  • Control
  • Interactivity
  • Novelty
  • Challenge
  • Feedback
  • Interest
  • Positive
• Disengagement attributes which prevent users from re-engaging with the system:
  • Usability
  • Challenge
  • Positive affect
  • Negative affect
  • Perceived time
  • Interruptions

Situational Awareness

AR promises the ability to provide additional information to the environment you are in.

Many jobs require high situational awareness to make effective and timely decisions.

Situational awareness (Endsley, 1995):

• Level 1: the perception of elements in the environment
• Level 2: the comprehension of their meaning
• Level 3: the projection of their status in the near future
• People make decisions based on their situational awareness, and their actions change the state of the environment, creating a feedback loop
• Situational awareness, decision-making and the performance of their actions can be influenced by:
  • System capability
  • Interface design
  • Stress/workload
  • Complexity
  • Automation
  • As well as more individual factors:
    • Abilities, experience, trailing
    • Their ability to process information
• Situational awareness and decision-making can be affected by:
  • The user’s goals/objectives
  • Their preconceptions and expectations

Assessing situational awareness:

• Self-rating:
  • Non-intrusive: ask questions post-trial
  • Subjective
  • Situation Awareness Rating Technique (SART):
    • Most well-known self rating system
    • 10 dimensions on a Likert scale from 1 to 7
    • Applicable when:
      • The task is dynamic, collaborative and changeable (e.g. long tasks that can’t be frozen)
      • Task outcome is not known (e.g. real world task)
      • U-(D-S) score:
        • U: summed understanding
          • Information quantity/quality
          • Familiarity with the situation
        • D: summed attentional demand
          • Instability, variability and complexity of the situation
        • S: summed attentional supply
          • Arousal (alert/ready for activity or low alertness level)
          • Spare mental capacity
          • Concentration of attention
          • Division of attention
• Freeze probe:
  • Task randomly frozen: questions asked about the current or recent state of the system
  • May negatively affect performance
• Real-time probe:
  • Experts ask questions asked during the experiment
  • No task freeze
  • Response time indicator of situational awareness
• Observer rating:
  • Experts observe participants while they do the task and rate their situational awareness

Metrics

NASA Task Load Index (TLX):

• Subjective workload assessment tool designed for human-machine systems
• Users rate workload on five dimensions:
  • Mental demands
  • Physical demands
  • Temporal demands (how hurried/rushed was the pacing?)
  • Performance: success in achieving the task
  • Effort/frustration:
    • How much effort did they put into the task
    • How insecure/discourage/irritated/stressed/annoyed were they
• Overall workload score takes weighted average, with weights being defined by the experimenters based on their expert judgment

Simulation workload measure (SIM-TLX):

• Based on NASA-TLX
• Released 2020
• Considers degree of immersion, perceptual difficulties, novel methods of controlling the environment
• In addition to mental, physical, temporal demands, and frustration, asks:
  • Task complexity: how complex was the task
  • Situational stress: how stressed were they while perfuming the task
  • Distraction: how distracting was the task environment
  • Perceptual strain: how uncomfortable/irritating were the visual/auditory aspects
  • Task control: how difficult was control/navigation

System Usability Scale (SUS):

• 10 questions on a Likert scale from 1 to 5:
  1. I think that I would like to use this system frequently
  2. I found the system unnecessarily complex
  3. I thought the system was easy to use
  4. I think that I would need the support of a technical person to be able to use this system
  5. I found the various functions in this system were well integrated
  6. I thought there was too much inconsistency in this system
  7. I would imagine that most people would learn to use this system very quickly
  8. I found the system very cumbersome to use
  9. I felt very confident using the system
  10. I needed to learn a lot of things before I could get going with this system
• Items 1, 3, 5, 7, 9: take sum of val - 1
• Items 2, 4, 6, 8, 10: take sum of 5 - val
• Multiply sums by 2.5
• ‘Good’ usability: average SUS value > 68

Game Experience Questionnaire (GEQ (another also has the same acronym)):

• Modular structure with core, social presence, and post-game modules
• Likert scale from 0-4; each question assesses one of seven components:
  • Competence
  • Immersion
  • Flow
  • Tension
  • Challenge
  • Positive/negative
• Slightly controversial: heavily used and can provide some insights, but was never validated
• If GEQ score is low while the usability score is high, it likely means the game is bad
  • Bad usability will usually lead to a bad game experience

Igroup Presence Questionnaire (IPQ):

• How much an individual believes they are really in the virtual environment (VE)
• Constructed with ~500 participants
• Three subscales:
  • Spatial presence: sense of being physically present in the VE
  • Involvement: attention devoted to the VE and the involvement experience
    • Awareness of real world surroundings (e.g. sound, room temperature, other people etc.)
  • Experienced realism: subjective experience of realism
• 14 questions, including a general question that does not belong to any of the subscales
• Answered on a Likert scale from 0 to 6 (-3 to 3)
• Answers for each sub-scale summed together (with a few questions inverted)
  • Results in a 3D scale

Immersive Tendency Questionnaire (ITQ):

• Measuring the tendency of individuals to be involved/immersed: how much of the immersion comes from the experience you created versus the participant’s tendencies?
  • Participants group may be biased - people taking part in VR studies likely to have more experience with VR compared to the general population
• 7 point scale per item
• Three subscales:
  • INVOL: tendency to become involved in activities:
    • Difficulties with people getting your attention/being aware of surroundings when watching tv/movie/reading book
    • Identifying closely with characters
    • Becoming scared/apprehensive/fearful after watching TV show/movie
  • FOCUS: tendency to maintain focus on current activities:
    • How physically fit/mentally alert they feel currently
    • How well they can block out external distractions
    • Losing track of time
  • GAMES: tendency to play games
    • Feeling like they are inside the game rather than controlling it through a controller
    • How often do they play video games

User Experience Questionnaire (UEQ):

• Measure UX of interactive projects
• 7-step Likert scale from -3 to 3
• Fully validated (in multiple languages)
• 6 scales, 26 items:
  • Attractiveness: overall impression of the product
  • Perspicuity: how quickly/easily they can learn to use the product
  • Efficiency: how fast/efficient the interaction (and feedback) is; amount of perceived ‘unnecessarily’ effort
  • Dependability: how in-control they feel; can the user predict system behavior and feel ‘safe’ while using the product
  • Stimulation: how exciting/fun is the product
  • Novelty: how innovative/creative is the product

Game Engagement Questionnaire (GEQ):

• Uses engagement as an indicator of game involvement
• Attempts to quantify absorption, flow, presence and immersion
• Questions on a no/maybe/yes scale with each question having a unique mapping to a numeric scale

Revised Game Engagement Model (R-GEM):

• Evaluates subjective gameplay experience
• Extends the GEQ
• At some point users shift from low-level to high-level engagement
• Low level:
  • Immersion: felling of being enveloped by the game’s stimuli/experiences
  • Involvement: motivation to play
• High level:
  • Presence: feeling of being physically located within the game
  • Flow: optimal experience of intrinsically-motivated enjoyment
• Questionnaire based on SUS, ITQ, PQ, Flow Short Scale (FSS), Personal Involvement Inventory (PII), Technology Acceptance Model (TAM)

Case Studies

AR game to assess upper extremity motor dysfunctions:

• Existing validated tools to assess motor performance for e.g. stroke, Alzheimer, Parkinson’s patients
• Instead of moving physical items, use AR and motion capture to assess health
• Goals:
  • Evaluate usability/game experience
  • Compare characteristics of movements in AR versus real world
• Used NASA-TLX, SUS, GEQ and Kinect motion capture to collect data
• Results:
  • More engaging than standardized tests
  • Motion capture was not accurate enough (at least not for initial assessment)
    • Technology is probably good enough today

Human augmentation for distributed situational awareness:

• Collaboration with Dutch police and fire department
• Virtual co-location of local and remote experts
  • Person at crime scene wears HMD (and backpack laptop), remote expert can annotate crime scene
  • Remote expert also has audio connection
• Results:
  • AR increased workload and situational awareness
  • Remote colleague appreciated: acted as advisor
  • Local user wanted avatar for the remote colleague for more presence, not just voice

Aerial wildfire firefighting training:

• Air attack supervisor (AAS):
  • Coordinates fire crews fighting wildfires
  • Communicates hazards and gives advice
  • Stressful and dangerous
• Transitional AAS training expensive, rarely done
• Conditions:
  • Cylindrical projection display with 270 degree field of regard
    • AAS and pilot can see and interact with each other
  • HMD with 360 degree field of regard (but limited by headset’s FoV)
    • AAS can see an avatar of the pilot
• Methods:
  • SART for situational awareness
    • Non-significant difference
  • NASA TLX for workload
    • HMD had slightly lower workload
  • IPQ for presence
    • HMD had slightly higher presence

Superhuman Sports

Designing MR games that motivate and engage users in physical activity.

You can:

• Augment the senses
  • Extra-sensory perception:
    • X-ray vision
    • ‘Spider sense’
    • Clairvoyance
  • Sensory augmentation:
    • Map an ‘invisible play world’ onto existing senses (substitution)
  • Change properties of one sensory modality into stimuli for another
• Augment the body
  • Laser tag: vest with PGM force feedback with constricted movements the wearer’s movements the more they got shot
  • Mechanical tail: affected balance as the player moved around
  • MetaArmS: remap feet to mechanical arms
• Augment the playing field:
  • Adding virtual elements:
    • New physics
    • New equipment
    • New opponents
  • Train in a safe environment:
    • Climbing treadmill:
      • Circular wall which rotates as you move up, allowing users to climb endlessly
      • Users always ~50 cm above the ground, but there is a button which causes a ‘platform’ to appear
      • Physical excursion adds immersion to the experience?
• Technology is still not quite ready, so interactions should be kept simple