07. Interaction in AR

Stephan Lukosch

AR Interface Foundations:

• AR requirements:
  • Combining real/virtual images: display technologies
  • Interactive in real-time: input & interactive technologies
  • Registered in 3D: viewpoint tracking technologies
• AR feedback loop:
  • User input, camera movement
  • Pose tracking
  • Registration of virtual content
  • Situated visualization
• Augmentation Placement:
  • Relative to:
    • Head
    • Body
    • Hand
    • Environment: tables, walls, mid-air
• Displays:
  • Head-mounted (glasses)
  • Hand-held projector
  • Hand-help display (smartphones)

Designing AR system = interface design which satisfies the user and allows real-time interaction

Interacting with AR content:

• Augmented reality content is spatially registered: how do you interact with it?
• By touch:
  • Hololens clicker: one button controller
• By raycasting:
  • Cast a ray passing through eye and controller
• By hand tracking:
  • Hand is recognized and mapped to a hand model
  • Gestures (e.g. pinch) allow interaction
• Body tracking
  • Skeleton tracking provides whole-body input
  • Requires some sort of tracking system (e.g. external cameras)

Evolution of AR interfaces

Expressiveness and intuitiveness has increased over time:

• Browsing:
  • 2D elements registered to real-world content
  • For visualizing; limit interaction with the content
  • Mostly hand-held devices
• 3D AR:
  • Allows manipulation of 3D objects anchored in the real world
  • Dedicated controllers, head-mounted displays, 6DOF tracking
  • One of the most important interaction classes within AR
  • No tactile feedback: just visual
• Tangible UI:
  • Rekimoto, Saitoh, 1999
    • Virtual objects projected onto a surface
    • Physical objects used as controls for virtual objects
    • Supports collaboration
  • Ishii and Ullmer, 1997
    • Tangible bits
  • [Augmented Groove, 2000]:
    • Mapping physical actions to MIDI
  • Limitations:
    • Difficult to change object properties
    • Limited display: projected onto a surface or screen
    • Separation between object and display
  • Advantages:
    • Natural: user’s hands can be used to interact with both real and virtual objects
    • No need for special purpose input device
    • User intuitively knows how to use the interface
• Tangible AR
  • Tangible interfaces have a tangible gap: interaction and presentation are on 2D surfaces. However, there is no interaction gap: same input devices can be used for physical and virtual objects. Tangible AR tries to close both gaps:
    • Physical controllers for moving virtual content
    • Support spatial 3D interaction
    • Support multi-handed interaction
    • Time and space multiplex interaction
      • Space-multiplexed:
        • Many devices with one function
        • More intuitive, quicker to use
        • e.g. (physical) toolbox
        • Poupyrev et. al., 2003
          • Different functionality assigned to markers
          • Opaque functionality: couldn’t tell what the marker would do by looking at it
      • Time multiplexed
        • One device with many functions
        • Space efficient
        • e.g. mouse
        • VOMAR:
          • Catalog book; tap a paddle against a page/section to choose the functionality of the paddle
• Natural AR:
  • Use of natural user input: freehand gestures, body motion, gaze, speech
  • Multimodal input: not all input methods are appropriate in all situations
  • HITLabNZ spider demo:
    • Overhead camera with depth captures real-time hand model
    • Can get spider to crawl over your hand
    • Presence: how believable the virtual content is to the usre
  • Hololens 2:
    • Continuous 3D hand tracking
      • Hololens sometimes overlays blue hand over user’s hand in order to reassure them that the tracking is working without distracting the user by overlaying the virtual hand for the whole time: less is better
    • Gesture-driven interface
    • Speech input:
      • Commands applied to the object you are currently looking at (gaze tracking)
      • Good for quantitative input (numbers, text)
        • Precise input difficult in AR

Designing AR Systems

Basic design guidelines:

• Provide a good conceptual model and metaphor
  • True for any kind of user interface
• Make things visible:
  • If an object has a function, then the user interface should show it
  • Even if a function is obvious, the user may not realize that the system supports this
• Map interface controls to the customer’s model
  • Not that of the system implementation
• Provide feedback: WYSIWYG
• Interface components:
  • Physical objects
  • Interaction metaphor
  • Virtual objects

Affordances

Objects are purposely built: they include affordances and make them obvious.

Affordances: an attribute of an object that allows people to know how to use it

Physical affordances:

• Chairs are to sit
• Handles are to twist and pull
• Scissors are to cut
• Surface Dial: we expect circular objects to be spun

Interfaces:

• Virtual objects do not have ‘real’ affordances
• They are better conceptualized as ‘perceived’ affordances
  • Based on people’s prior experiences
  • Common/repeated metaphors become ingrained in users

Augmented reality:

• Physical: tangible controllers and objects
• Virtual: virtual graphics and audio

Case Studies

Navigating a spatial interface:

• Menu displayed over hand; other hand used as pointer
• Menu attached to marker held in one hand; other hand used as pointer
• Interaction with cylinder: rotate to select
• One hand interaction: gesture to select?
• Place marker on surface: another marker used for selection
  • Menu fixed at one location: stable

Workspace awareness in collaborative AR:

• Local player solving puzzle
• Remote instructor gave advice in AR
  • Knew solution, gave advice either visually or aurally
  • Audio made users more aware of the instructor’s actions, but was also more distracting
  • Visual allowed players to follow instructions better than audio
• HMD had very limited FoV: local player may not notice when instructor marked a piece

Depth perception in AR:

• Object selection with HMD
• Tried reducing brightness of background and blurring: neither worked out
• Lighting plays an important role in depth perception

3D AR lens:

• Magnifying glass with physical handle and a marker where the lens would be
• When in AR, acted like a real magnifying glass

Magic book:

• 3D model shown in book using marker/texture tracking

Interaction Design

The process of:

• Discovering requirements, designing to fulfil requirements, produce prototypes and evaluates them
  • Often requirements will be conflicting: you must make trade-offs that will best suit your future users
  • Focus on users and their goals
  • Trade-offs to balance conflicting requirements
• Approaches:
  • User-centered design: user knows the best and guides the designer; the designer translates user needs and goals
  • Activity-centered design: focus on user behavior around tasks: their behavior determines the goals
  • System design: system is in the focus and sets the goal
  • Genius design/rapid expert design: design based on the experience and creativity of the designer

Solving the right problem:

• Engineers and business people are trained to solve problems
• Designers are trained to discover problems
• We should rather have no solution than a brilliant solution to a non-existent problem
• Designers should:
  • Never start by trying to solve the problem
  • Start by trying to understand what the real issues are
  • Diverging upon a solution
  • Studying people, their needs and their goals

Double diamond of design:

• Four phases of design:
  • Diamond 1:
    • Discovery:
      • Understand the problem. Never assume what the problem is
      • Talk to the users
    • Define: use insights from discovery phase to describe and define the problem
  • Diamond 2:
    • Develop:
      • Explore alternatives solutions to the problem
      • Seek inspiration from elsewhere
    • Deliver:
      • Test out the solutions; give them to the user and gather feedback
      • Reject under-performing solutions; improve promising ones
  • Loop through each diamond as many times as required, and return to the start if required
• Principles:
  • Put people first: understand the people using the service; their needs, strengths, aspirations
  • Communicate visually and inclusively: help people gain a shared understanding of the problem and ideas
  • Collaborate and co-create: work with others
  • Iterate: spot errors early

Involving users:

• Expectation management:
  • Must give them realistic expectations: no surprises, no disappointments
  • Timely training
  • Communication, but no hype
• Ownership:
  • Make users active stakeholders
  • More likely to forgive/accept problems
  • Can make a big difference in the acceptance and success of the problem

Interaction design:

• Discover requirements
• Design alternatives
  • Which may lead to requirements being refined
• Prototype alternatives
• Evaluate the product and its user experience throughout
  • If it sucks, this means that the requirements were probably incorrect

Practical issues:

• Who are the users?
• What are the users’ needs?
• How do you generative alternative designs?
• How do you choose among alternatives?
• Who are the stakeholders?
  • They can influence the success/failure of the project, so involve them and keep them happy

What are the users’ needs?

• Users don’t know what is possible
• Instead:
  • Explore the problem space
  • Investigate user activities: see what can be improved
  • Try out potential ideas

Alternative generation:

• Humans tend to stick with what works
• Considering alternatives: the design space, helps identify better designs
• Where do alternative designs come from?
  • Flair and creativity: research and synthesis
  • Cross-fertilization of ideas from different perspectives
  • From users
  • Product evolution based on changed use
  • Inspiration from similar and different products/domains
• Balance constraints and trade-offs
• Morphological charts:
  • List the functions: what does the product need to do?
    • e.g. for a beverage container, it must contain the beverage, provide access to the contents, and display product information
  • Then for each function, list its means:
    • Methods of addressing the functions/user needs
    • e.g. for the beverage container, access to the contents could be done through a pull tab, straw, or a cap
  • Pick one means for each function
    • Not every combination will be practical or possible

Choosing between alternatives:

• Interaction design focuses on externally-visible and measurable behavior
• Technical feasibility
• Evaluation with users or peers
  • Use prototypes, not static documentation: behavior is key
• A/B testing:
  • Defining appropriate metrics is non-trivial
• Quality thresholds:
  • Different stakeholder groups have different quality threshold
  • Use usability, user experience goals to defined criteria

Prototyping:

• Allows the designer and their users to explore interactions and capture key interactions
• Focuses on use experience
• Communicates design ideas
• Learn through doing
• Avoids premature commitment

Typical development:

• Sketching
  • Helps to express, develop and communicate design ideas
• Storyboards
• UI mockups
• Interaction flows
• Video prototypes
• Interaction prototypes
• Final native application

Low fidelity prototypes:

• Low development cost allows evaluation of multiple design concepts
• Limits the feedback you can get: error checking, navigational and flow limitations

High fidelity prototypes:

• Fully interactive
• Has look and feel of the final product
• Clearly defined navigational scheme
• Much higher development cost
• Sunk cost bias: more reluctant to make changes given the time/effort