06. Interaction in VR

Rob Lindeman, Director of HITLab NZ.

User interaction:

• How can we do things in VR environments?
• What controllers/inputs do they support?
• As a user, what kinds of thins would I even want to do?
  • Is the task fatiguing? Will it make me feel sick

The state of VR:

• 1980s/1990s:
  • Much hype
  • No inroads into everyday life:
    • Lagging technology
      • But the technology we have today seems close
    • Lack of understanding of usability issues
    • Lack of ‘killer apps’ (games?)
  • Some use in specific scenarios:
    • Surgical simulation
    • Military training
    • Phobia therapy
    • Oil/gas visualization
    • Automotive design
• 2000:
  • Growth of video games led to:
    • Immense increase in hardware power
    • Reduction in hardware costs
    • Immense growth in number of users/gamers
    • Many ‘new’ interface devices
  • Better understanding of 3D-UI
  • Gaming emerged as the killer app (?) (at least for now)
• 2010s:
  • Lots of new hardware:
    • WiiMote, WiiMotion Plus
    • Kinect
    • Powerful smartphones (economies of scale)
    • ~12 VR/AR headsets announced
    • Lots of controllers
  • 2015:
    • A lot of VR headsets
    • Very few AR headsets
    • Many phone-integrated headsets (e.g. Google Cardboard, Samsung Gear VR)
    • Some locomotion controllers (e.g. treadmills), but no consumer hardware
    • A lot of companies go out of business: no use case for the products

Motivation for studying VR interaction:

• Mouse + Keyboard great for general desktop UI tasks:
  • Text entry, selection, drag/drop, scrolling, rubber banding etc.
  • Fixed computing environment
  • 2D windows, 2D mouse
• But how do we design effective techniques for 3D?
  • With a 2D device?
  • With multiple n-D devices?
  • New devices?
  • 2D interface widgets?
  • With a new language; new interaction techniques that can support the new environment
• Gaming:
  • Tight coupling between action and reaction
  • Requires precision
• VR gives real first-person experiences, not just views
  • HMDs:
    • Look behind by turning your head
  • Selecting/manipulating objects:
    • Reach out with your hand and grab it
  • Travel:
    • Walk
    • Except that you are still in a confined physical space with objects and cats in the way
  • Doing things that have no physical analog is more problematic
    • How do you change font size?

Existing input methods:

• Joystick, trackballs, trackpoints, trackpads, tablets, gaming controllers
• General vs purpose-built controllers:
  • General purpose:
    • Single device used for many thing
    • Mouse, joystick, gamepad, game controllers, Vive/Touch controllers, WiiMote etc.
    • Okay for many tasks, but not optimal
  • Special purpose:
    • Typically used for a specific task (e.g. driving, playing guitar)
    • Very effective for a given task
• Current devices:
  • PlayStation
    • Vibration motors with different weights in each wing to allow varying vibration strength
    • PS5: shoulder/trigger buttons have motors for force feedback (variable resistance)
  • Xbox controllers
  • WiiMote
  • Leap Motion hand tracking
  • Kinect body tracking
  • Hand-held devices:
    • Smartphones
    • Tablets
    • Nintendo DS
    • Sony PSP

Classification Schemes

Relative vs absolute movement:

• Mice return a delta: relative motion
• Touch screens, pen tablets return absolute position

Integrated vs separable degrees of freedom:

• e.g. Etch-a-sketch has separate X and Y controls
• Motions that are easy with one are hard with the other

Analog vs digital:

• Continuous vs digital input: prefer former

Isometric vs isotonic:

• Isometric: infinite resistance (no motion), but force sensing
  • e.g. ThinkPad TrackPoint
• Isotonic: zero resistance
• In reality, devices exist on a continuum of elasticity
  • Mice are mostly isotonic, but they do have mass and hence inertia
  • Some controls (e.g. joysticks) are self-centering

Rate control vs position control:

• Mice
  • Usually position control
  • Scrolling:
    • Scroll wheel: position control
    • Windows middle click and drag: rate-controlled scrolling
• Trackballs: usually position control
• Joysticks: usually position (cross-hair) or velocity (e.g. aircraft)
• Rate control eliminates need for clutching/ratcheting
• Isotonic-rate and isometric-position control usually poor

Special-purpose vs general-purpose:

• Game controllers must support many types of games
  • Few ‘standard’ mappings: each game can do things differently (for the most part)
• Some special-purpose devices:
  • Mostly based on things that already exist in the real world
  • Examples:
    • Guitar controllers
    • Steering wheels
    • RPG keyboard/joystick
    • Drum kits, dance pads, bongos etc.

Direct vs indirect:

• Direct:
  • Click and drag with mouse/stylus/finger
  • Touch screen gestures: swipe, two-finger rotate
  • Problems:
    • Works well for things that have a physical analogue, but not for those without
    • May have low precision
    • Selection/de-selection may be messy
• Indirect
  • Use some widget to indirectly change something

3D Input Devices:

• SpaceBall/SpaceMouse
  • Isometric device which senses force
• CyberGlove II
  • From the 1990s
  • Strain sensors used to sense hand movement
• PHANTOM Omni Haptic Device
  • Stylus attached to robot arm
  • Force feedback allows user to feel surface
  • Limited working volume, only one point of contact, very limited use case
• HMD with 3/6-DOF trackers (e.g. Oculus Quest)

3D Spatial Input Devices:

• Microsoft Kinect
• Leap Motion
• Oculus Touch
• HTC Vive
  • Used touch surfaces which could be displayed by a little, rather than a joystick
    • Very different experience
• Valve Index
  • Strapped to your hand: don’t need to actively hold the controller
  • Had finger tracking

Motion-Capture/Tracking Systems:

• Probably won’t be that common in the future: people will just use inside-out tracking built into headsets
• Used in movies/TV and games
  • Capture actual motion and re-use
• Can be done interactively or offline
• Can capture three or more DoF
  • Positions orientation, limbs
• No good general purpose approaches to high-fidelity, full-body tracking without markers
• Attempts:
  • Magnetic tracking
    • Transmitter creates a magnetic field
    • Wired receivers stuck to clothes
    • Receivers tracked (relative to the transmitter) using changes in magnetic field
    • Pros:
      • Fairly lightweight
      • Six DoF
    • Cons:
      • Very noise near ferrous metal (e.g. rebar)
      • Limited range
  • Ultrasonic tracking
    • Speakers emitting ultrasonic sound, captured by receivers containing microphones
    • Used to compute distance
    • Receivers had to point in the right direction (hemisphere)
    • High resolution and accuracy
    • Requires ‘line-of-sight’
  • Inertial trackers
    • Accelerometers, gyroscopes
    • Pros:
      • Lightweight
      • Wireless
    • Cons:
      • Error accumulates
      • Only moderate accuracy
    • e.g. Wii MotionPlus
  • Optical tracking
    • Multiple fixed makers
    • Known camera parameters
    • Inside-out tracking: camera attached to user, lab-mounted fixed landmarks
    • Outside-in tracking: cameras attached to lab, landmarks attached to user
    • Active vs passive:
      • Active: markers are lights
      • Passive: reflective markers with external light
    • PlayStation MOVE:
      • Stick with illuminated ball of known size and color
      • Camera tracker + internal tracker used for tracking
    • Leap Motion:
      • Three IR LEDs for illumination
      • Stereo cameras used for depth
  • Hybrid tracking
    • Compensate negative characteristics of one approach with another

Other Input Devices:

• Speech input
• Gestures (e.g. pointing at an object)
• Device actions (e.g. buttons, joysticks)
• Head/gaze, eye blinks
  • ‘Put that, there’: hybrid speech + gesture

Special Purpose Input Devices:

• Some applications are more ‘real’ with a device that matches the real action
• Examples:
  • Light gun
  • Flight simulator motion platform
  • Snowboard/surfboard
  • Pod racer
  • Motorcycle
• Sensors are very cheap today: you may be able to simply attach some sensors to a passive object

Interaction in VR

Mapping Devices to Actions:

• For each (user, task, environment)
  • For the four basic VR tasks
    • For each device DOF
      • Choose a mapping to an action

VR interaction:

• Must take advantage of people’s real-world experience
• And for those without real-world analogues, allow users to express their intent
• Without making people tired
• Without making people sick
• While making it easy to learn and use

Main interaction tasks (Bowman et al.):

• Object selection/manipulation:
  • How does the user select the object they wish to manipulate?
  • How do they actually manipulate it?
• Navigation:
  • Wayfinding (mental): where am I now, and how do I get to where I am going?
  • Locomotion (motor): how do I travel there?
• System control:
  • Changing system parameters
  • Manipulating widgets
    • Lighting effects
    • Object representation
    • Data filtering
  • Approaches:
    • Floating windows
    • Hand-held windows
    • Gestures
    • Menus on fingers
• Symbolic input:
  • Text/number input
• Avatar control (Lindeman):
  • How do you control you?
• And throwing things 😆

Objects:

• Issues:
  • Ambiguity when there are multiple objects the user could be pointing to
  • Distance
  • Selecting multiple objects
  • Releasing objects
• Selection approaches:
  • Direct/enhanced grabbing (latter: items further away than arms reach)
  • Ray-casting
  • Image-plane
• Manipulation approaches:
  • World in miniature (WIM): miniature world representing the world you are in
    • Can you pick yourself up?
  • Skewers
  • Surrogates
• Modifying objects:
  • Choose among object properties
  • Natural mappings of actions to changes
  • Arbitrary mappings

Object selection in the real world:

• Touching/grabbing
• Pointing
  • Finger: direct
  • Pointer: extended
  • Mouse: indirect
• Voice: ask someone
• Context
• Eye gaze

Selection-task decomposition:

• Indicate:
  • Denote which open we intend to select
  • On desktop: move mouse
  • In VR:
    • Avatar hand-movement
    • Device movement
    • Virtual ‘beam’ TODO raytracing
• Confirm:
  • On desktop: mouse click
  • In VR:
    • Click
    • Dwell (timeout)
    • Verbal cue

Reaching objects:

• Indicating at a distance
  • Go-go: greater than 1:1 mapping when arm more than a certain distance away
  • Two-handed pointing
  • World in miniature
  • Flashlight
  • Voodoo dolls
• Image plane technique: user pinches object, determine XY location on the image plane and then use determine front-most object at that point

Manipulation:

• Typical tasks:
  • Repositioning
  • Rotation
  • Property modification
• Approaches
  • WIM
  • 3D widgets:
    • Virtual sphere for rotation
    • Jack for scaling
  • Non-isomorphic translation/rotation
  • Skewers
  • 2D widgets

Design Guidelines:

• Use existing techniques unless you really need to
• Match the interaction technique with the device
  • Use task analysis (e.g. does the task need high precision?)
• Use techniques that can help reduce clutching
  • When dragging with mice: if you reach the end of the mouse pad, you need to grip and pick up the mouse, then move it to the opposite end of the mouse pad
• Non-isomorphic techniques are more useful and intuitive
• Use pointing techniques for selection; virtual hand techniques for manipulation
• Use grasp-sensitive object selection
• Constrain degrees of freedom when possible
  • Fewer mistakes, less annoyance
• There is no single best interaction technique: just test, test, test

Research papers:

• Object Impersonation (Wang, IEEE VR 2015)
  • Open headset: user can simultaneously use tablet
  • User becomes the object:
    • e.g. moving a light: user’s head becomes the light, can turn head to position beam
    • e.g. paving a road: the path you move in becomes the road
• Navigation: wayfinding
  • Easy to get lost in VR
  • Traditional tools:
    • Maps (North vs forwards up)
    • Landmarks
    • Spoken dierction
  • Non-traditional
    • Callouts
    • Zooming
• Navigation: travel
  • Limited physical space, possibly infinite virtual space
  • Different travel types:
    • Walking, running, turning, strafing, back stepping, crawling, quick start/stop, driving, flying, tale_porting
    • Also lying down, kneeling, ducking, jumping
  • Travel isn’t the goal: usually doing other things while travelling
• Initial Exploration of a Multi-Sensory Design Space:
  • Spinning chair: user leans forwards/backwards to move
  • Fans opposing direction to travel to simulate feeling of movement
• Finger Walking (Yan et al., 2016)
  • Short distance: finger tapping on a touchpad to ‘walk’
  • Long distance: two fingers (like feet on a hoverboard), force determines speed and angle between fingers determines angle
• TriggerWalking (Sarupuri et al., 2016):
  • Tap controller trigger to walk: controller orientation controls moment direction
• System Control using Hybrid Virtual Environments (Wang, 3DUI 2013):
  • Open headset
  • Tablet taped to arm used for selecting objects
  • Drop items in the scene in VR
• Avatar Control
  • Paul Yost
  • IMU controllers attached to arms, feet for full body tracking

The ‘optimal’ interface depends on:

• The capability of the user:
  • Dexterity
  • Expected level of expertise
• The task:
  • Task complexity
  • Granularity: how precise does the input need to be? -The environment:
  • Stationary, moving, noisy environments