Dava Visualization in Mixed Reality

Master in Human Interface Technology (MHIT)

HITLab NZ:

• Founded in 2002
• Research focuses: VR, AR, applied immersive gaming
• Philosophy:

  We put people before technology, start with the person, look at all the tasks they are trying to perform, TODO

MHIT:

• Application, development and evaluation of HIT
• Learn:
  • Interface design principles
  • Describe/evaluate interface hardware/technology
  • Research/development skills
• Engage with industry
• 3 months of course work:
  • HITD602 design & evaluation
    • Relationship between aesthetics, function, UX
    • Evaluation of design/experience
  • HITD603 prototyping and projects
    • Requirement analysis, engaging with clients/problem owners
• 9 month thesis project
  • Develop prototype
  • Run user study
  • Write thesis
• Requirements:
  • BEHons
  • Min. B+ grade
• Scholarships available: more or less certain that you could get fees-only scholarship
  • One student getting stipend from industry
• 22% of MHIT students remain in academia (enrolled in PhD program)

Data Visualization in Mixed Reality

Immersive analytics (Immersive Analytics, Springer, 2018):

• Coping with the ever-increasing amount and complexity of data around us that surpasses our ability to understand/utilize in decision-making:
  • Business analysis
  • Science
  • Policy making
  • General public (e.g. personalized health data)
• Removing barriers between people, their data and tools used for analysis
• Support data understanding and decision-making everywhere by everyone
• Allows both individual and collaborative works
• Engagement helps support data understanding and decision maknig
• Builds upon:
  • Data visualization
  • Visual analytics
  • VR/AR
  • Computer graphics
  • HCI

Very dependent on availability of immersive technologies:

• HMDs for AR/VR
• Large wall-mounted, hand-held or wearable displays
• ML to interpret user gestures/utterance

Immersive analytics allows engagement:

• With wider audience through tools/technologies that more fully engage the senses
• With a new generation whose primary input device is not the mouse/keyboard
• In situations are desktop computing is impossible
• In groups where all participants are equally empowered

Opportunties:

• Situated analytics
  • User-controlled data analytic linked with objects in the physical world
    • Energy consumption
    • Construction progress
    • Supermarket (e.g. nutritional value of foods, comparison)
    • Instruments in a lab
• Embodied data exploration
  • Touch/gesture/voice/TUI for more intuitive/engaging data exploration
  • Computer becomes invisible to the user
• Collaboration: colocated or remote; synchronous or asynchronous
• Spatial immersion: 3D (or 2.5D) rather than 2D visualization
• Multi-sensory presentation
  • Beyond visual/audio (e.g. haptics)
  • Augmented cognition
• Engagement in data-informed decision-making
  • Involve the general public/other stakeholders
  • Allows immersive interactive narrative visualizations (e.g. climate change, carbon footprint)

Possible Values of 3D for Data Visualizations

Additional visual channel (3rd spatial dimension) for data visualization:

• Prone to occlusion, depth disparity, foreshortening
• Studies demonstrate some benefits to this channel

Immersive display technologies have advanced considerably: higher resolution, lower latency, wider range of interaction technologies

Immersive workspaces:

• Use the space around you as a workspace
• Place data visualizations where you want, anchored to the physical space (or relative to your position)
• Beyond task effectiveness:
  • Focus not on accuracy/speed
  • Does spatial immersion support deeper collaboration, greater engagement, or a more memorable experience?

Depth Cues and Display Technology

• Linear perspective:
  • Consequence of the projective properties of the eye as a sensor:
  • Occlusion: objects closer in space prevent us from seeing objects behind it
  • Foreshortening
    • Relative size: two objects of the same size at different distances from the observers project differently
    • Relative density: spatial patterns of objects/visual features appear denser as the distance to the pattern increases
    • Height in visual fields:
    • Objects are bound to rest on the ground
    • Bottom of objects can be used as a reference
• Aerial perspective: changes in color properties of objects at large distances
• Motion perspective: moving object/observers provide information about 3D structure
• Binocular disparity/stereopsis: small differences in the images received by the left/right eye
• Accommodation (depth of field):
  • Effects of dynamic physiological changes in the shape of each eye
  • Amount of blur of the background and other objects provides information about their relative distance
  • Dependent on the lightness of the scene
• Depth cues:
  • Shadows
    • Cue for judging the height of an object above the plane
    • Useful for floating objects
  • Convergence
    • Reflex of the visual system: change in rotation of the eyes that takes place to align the object/region of interest in the center of the eyes’ fovea
    • Eye orientation/angle (and differences between the two eyes) can be used to infer short distances
• Controlled point of view
  • Ability to manipulate the point of view in a virtual space (without physically moving)
  • User knows positional changes, expects visual changes
  • Relies on touch/proprioception
  • Complementary to visual cues
  • e.g. moving joystick to move your avatar/camera
• Subjective motion
  • Actual physical motion in the space of the observer
  • Information through the vestibular system (balance, movement detection)
  • Complementary to visual cues
• Object manipuation
  • Change position of objects with respect to the observer
  • Trigger motion perspective, changes in other cues
  • Does not trigger vestibular signals; uses touch (somatic), motor, priprioception

Limitations of depth perception:

• 30% of population may experience binocular deficiency
• Binocular acuity decreases with age
• Line-of-sight ambiguity: rays can only intersect once (occlusion)
• Text legibility
  • Low resolution of HMDs
  • Foreshortening, 3D orientation

Comparing 2D with 3D Representations - Potential Benefits of Immersive Visualization

Cone Trees:

• Indented lists/tree structures in 3D, where nodes are arranged in a cone that you can rotate
• Linear perspective provides a focus+context view of the tree
• 3D cues of perspective, lighting, shadows help with understanding
• More effective use of display space
• Interactive animation reduces cognitive load
• Study results:
  • Poor representation for hierarchical data: occlusion, slow tree rotation
  • May help in improving understanding of the underlying structure

Data mountains:

• Arrange documents on a virtual 3D desktop
• More objectives on the desktop
  • Linear perspective provides focus + context view
• Natural metaphor for grouping
• Leverages 3D spatial memory
• Study results:
  • 2D data mountains outperformed 3D, although participants thought otherwise
  • 2.5D data (2D + linear perspective) outperform 2D
  • i.e. 3D < 2D < 2.5D

Aviation:

• Show position and predicted flight path in 3D
• Study results:
  • Better for lateral/altitude flight path tracking
  • Worse for accurate measurement of airspeed
  • ATC found it worse for everything other than collision avoidance

3D shapes/landscapes:

• 3D better for:
• Understanding the overall shape
• Approximate navigation and relative positioning
• 2D better for precise manipulation

Network visualization:

• 3D better for judging if there is a path between highlighted nodes
• Motion cues beneficial for:
  • Path following in 3D mazes
  • Viewing graphs in AR
• Egocentric spherical layout of 3D graph with HMD outperforms 2D for:
  • Finding common neighbors
  • Finding paths
  • Recalling node location

Multivariate data visualization:

• 3D scatter plots better for:
  • Distance comparisons
  • Outlier detection
  • Cluster identification and shape identification
  • Answering integrative questions

Spatial and spatio-temporal data visualization:

• 2D vs 3D representations in VR:
  • Exocentric: globe in front of view
  • Egocentric: standing inside globe
  • Flat map
  • Curved map around the user
  • Exocentric globe more accurate for distance comparison and estimation
  • More time required for task completion compared to maps

Overall:

• Clusters/other structures may be clearer in 3D
• Sufficient depth cues required for the viewer to see clusters
• 3D may benefit path following
• Binocular ‘pop-out’ may be beneficial for highlighting elements
• Using the 3rd dimension to show time is a successful idiom

Summary:

• 3D not generally better than 2D
• 3D may show overall structures in multi-dimensional spaces better
• 2D preferable for precise manipulation or accurate data value measurement
• Choice of technology and depth cues can make a significant difference to the effectiveness:
  • Binocular presentation, head-tracking increased spatial judgment accuracy
  • Binocular 3D beneficial for depth-related tasks: spatial understanding and manipulation

Data Visualization in AR - Situated Analytics

• Data visualizations integrated into the physical environment
• Needs to take into account the existence of the physical world
• Examples:
  • Supermarket (e.g. viewing detailed product information, price comparison)
  • Attendees at a conference (e.g. displaying name, affiliation)
  • Machinery in a lab (e.g. showing progress)
  • Objects at a building site

Conceptual model:

• The raw data and the visualization pipeline exist in a logical world
• Raw data is turned into a visual form fit for human consumption
• Data is brought into the physical world through a physical presentation
• A physical referent (real-world items) may be present

Physically vs perceptually-situated visualizations:

• Physical distance separating a physical presentation and its physical referent may not necessarily match the perceived distance (e.g. visualizing microchip vs mountain)
• Spatial situatedness needs to be refined:
  • Physically situated in space: if its physical presentation is physically close to the data’s physical referent
  • Perceptually situated in space: if its physical/virtual presentation appears to be close to the data’s physical referent (e.g. mountain and its data visualization)

Embedded vs non-embedded visualizations:

• Embedded visualizations are deeply integrated within their physical environment
  • Different virtual sub-elements align with their related physical sub-elements

Interaction:

• By altering its pipeline (e.g. filtering data)
• By altering the physical presentation (e.g. moving around, re-arranging elements)
• Using insights to take immediate action