03. Developing Virtual Reality Experiences

Dr Tham Piumsomboon, School of Product Design.

Current VR Development Tools

2016: rise of consumer HMDs. Oculus, HTC Vive.

XR Fragmentation: different vendors all had their own proprietary APIs (e.g. Steam VR, Hololens, Oculus, HTC Vive, Magic Leap).

Kronos group (which created OpenGL, Vulkan, etc.) developed the OpenXR standard: cross-platform API supported by many hardware vendors.

Toolkits:

• VRTK
  • Open source, Microsoft
• MRTK
• XRTK
  • Fork of MRTK developed by Unity
• [Oculus Interaction SDK]
  • May be depreciated

Game Engines:

• HITLab using Unity engine
• A few others available: Unreal, CRYENGINE, GameMaker Studio, Amazon Lumberyard
• Social platforms (e.g. Breakroom)

Developing VR Experiences

Immersion:

• Feel like you are physically and mentally in the virtual world
• How to simulate a large world in a limited physical space?
• In non-VR games:
  • Suspension of disbelief: enough realism in the experience that you can ignore the issues

Models of immersion:

• Three types
  • Sensory immersion:
    • Disassociation with the real world
  • Challenge-based immersion:
    • Control:
      • Ask what made great games from the past (e.g. Mario) great?
      • Input mechanisms (e.g. game controllers, hand gestures)
    • Challenge:
      • Challenges must be achievable: too difficult -> frustrated; too easy -> bored
      • Hence, the difficulty must be balanced to make the experience rewarding
    • Cognitive involvement:
      • What you get out of the experience:
        • Fun
        • Social aspects
        • Learning/training
        • Work
  • Imaginative immersion:
    • Emotional involvement

Unity:

• Unity Settings -> XR Plug-in Management: use OpenXR (e.g. to support Windows MR headset)
  • Windows MR app needs to be running in the background
• Add interaction profile
• Package manager -> XR Interaction toolkit (com.unity.xr.interaction.toolkit)
• Add XR Origin GameObject: virtual camera maps to headset position and orientation
  • Add input action component: XR default input action
  • Also adds controller game objects

Analysis -> Profiler

• Visualize frame rate of application and components (e.g. rendering, scripting, physics, display sync) that make up the processing time
• High frame rate important to prevent motion sickness
• Project Settings -> Quality to increase/reduce quality and decrease/increase frame rate

Game engines components:

• Core functionality:
  • Rendering engine
  • Physics engine
    • And collision detection
• Sound, scripting, animation, networking, streaming, memory management etc.

Game loop:

• Read HID (human input device) state
• Update scene state
  • Physics engine
  • User input
  • Multiplayer networking
  • Collisions
  • Animations
  • NPC state
  • Audio
• Render the scene

The subsystems will often update at different rates (e.g. NPC behavior may update at ~1 FPS, physics engine at 120 FPS, renderer at 60 FPS).

Camera placements and control mappings:

• Relatively easy to convert 2.5D to 3D
• FPS games: camera mapping can be done easily by adding XR Origin
  • Controllers need to be mapped to existing controls
  • XROrigin:
    • Copy tracking origin, put it in player
    • Put controllers under player
    • Controllers: Add default SOMETHING
    • Player: add locomotion, input system
      • XR origin set
      • Add reference to XRI default input actions
    • Player: add continuous move provider, left hand XRI LeftHand LocomotionMove
    • Player: add snap turn provider, left hand XRI SnapMove
• XR Grab Intractable: add to a game object to allow a user to grab it from afar
• XR Socket Interactor: allows interactables to be ‘docked’ to the game object
  • Box collider: disable Is trigger TODO
• Scripts: using UnityEngine.InputSystem, create InputActionReference property (or private property with [SerializeField] annotation)

ProBuilder package: allows you to create new primitive shapes.

VR Interaction Design

Even if the graphics are good, you need to be able to interact with the environment.

Seven principles of fundamental design (Norman):

• Discoverable: users can discover what actions are possible
• Feedback: full and continuous feedback about the results of your actions and the current state of the world
• Conceptual model: design informs the user of the system’s conceptual model, making it seem intuitive
• Affordances:
  • The perceived affordance should match the actual affordance
  • Clues as to how something works:
• Signifiers: something to indicate the existence of affordances (e.g. arrows, sound)
• Mappings: relationship between the controls and actions is understood
• Constraints: physical, logical, semantic, cultural constraints which guide actions

VR: mappings are from games, not reality (e.g. using scissors: clicK a button to use, not picking it up with your fingers). Chairs: cannot sit on VR chairs in real life

VR affordances:

• Use visual cues to show possible affordances
• Perceived affordances should match actual affordances
• Good cognitive model: map object behavior to expected behavior
  • May vary by culture
• Controllers have different controls:
  • Some have joysticks, some have touchpads, some have buttons
  • Trigger buttons
• Examples:
  • Buttons that can be pushed
  • Objects that can be picked up
  • Doors that can be opened and walked through
  • Mutual human actuation

User groups:

• Age:
  • Children require different interface designs
  • Older people have different needs
• Prior experiences with HMDs
• Different physical characteristics: left/right-handed, height, arm reach
• Perceptual/cognitive/motor abilities
  • e.g. color perception
  • Cognitive/motor disabilities

Whole user needs:

• Social: don’t make them look stupid
• Cultural: follow local cultural norms
• Physical: can they physically use the interface
• Cognitive: can they understand the interface
• Emotional: make the user feel good and in control

Summary:

• High-fidelity graphics in VR is possible if we can afford it computationally
  • But not sufficient for immersion
• SCI immersive model: engage user through sensory, challenge-based, and imaginative immersion
• General design principles can be applied to VR design
• Plenty off opportunities of richer VR interaction beyond what general design principles could govern