02. Developing Augmented Reality Experiences

Adrian Clark, senior lecturer, School of Product Design.

Introduction to Unity

Unity: ‘real time development platform’. Not just for games.

Unity is so big, no one knows the full extent of what it can do.

Resources:

• Unity Learn
  • AR stuff now under the XR section
• Community
• User manual
  • Pretty decent docs
• Asset Store
  • Also third-party stores like Turbo Squid
  • Prefer fbx files
  • To download assets purchased from the store: Window -> Package manager -> My assets

Unity:

• Use 2020.3 LTS release
• Use 3D template

Editor:

• Many windows
  • Scene
    • See and position all GameObjects in the scene
    • Top left, switch between translation, rotation and scale tools
    • Add Rigidbody to a GameObject to add physics
  • Game
    • What the camera sees
    • Any changes made during play mode are not saved
  • Project
    • Assets folder: contents update automatically when FS changes
  • Hierarchy
    • All game objects in the scene in a hierarchy
    • Parent nodes affect child nodes
  • Inspector
    • Modification of GameObject properties
    • Components: behaviors or extensions to the game objects
      • Game objects are essentially containers for components
      • UI objects have their own event systems: cannot attach click listeners etc. to 3D objects
    • Examples:
      • Renderer
      • MeshFilter
      • Camera
      • Light
        • Directional by default:
          • Infinitely far away: rays are parallel
          • Node position does not matter: only rotation
        • Spotlight: cone of influence
        • Point: sphere of influence
      • Collider
  • Console
    • Warnings, errors

Shader rendering mode:

• Opaque: alpha ignored
• Cutout: binary transparency; on or off. Use if the object has holes
• Fade: change transparency of all aspects of the material based on alpha
• Transparent: TODO

Scripting:

// Behavior script is another component that attaches to a GameObject

// https://docs.unity3d.com/Manual/ExecutionOrder.html

// A massive number of lifecycle callbacks
public class NodeBehaviorScript: MonoBehaviour {
  // Called before first frame update
  void Start() {
    Debug.log("Instantiated");
  }

  // Called every frame
  void Update() {
    if (Input.GetKey(KeyCode.UpArrow)) {
      // transform: transform of the object the script is attached to
      // localPosition: position relative to parent
      transform.localPosition += new Vector3(0, 0, 0.1f);
    }
  }

  void onMouseDown() {
    // Set up a collider on the game object
    // e.g. box collider: invisible box (hopefully) around the object
  }

  void onCollisionEnter(Collision collision) {
    // Use collider that is larger than the object: when two objects
    // come close together you can add custom behavior (e.g. 'picking up'
    // the object in AR
  }

}

Unity Remote:

• App installed on your phone
• Project Settings -> Editor -> Unity Remote
• Set game resolution to match device screen
• Game view is streamed to your phone screen
  • But not the camera
• Touch events etc. on phone are sent the host computer

AR

Many different SDKs available. Some deciding factors:

• Price (free, paid per scan/month/app/licence period)
• Supported hardware platforms (iOS, Android, desktop, HMDs, web)
• Tracking (Fiducial, 2D natural feature, SLAM, 3D object, face, GPS/IMU)
• Performance

Unity also has AR foundation: a common interface to platform-specific AR frameworks. No way of running it in the editor, which makes development very frustrating (although there are some rumblings of a Unity Remote-like app which does on-device processing).

This course will use Vuforia:

• Initially developed by Qualcomm (and optimized for their chips), bought in 2015 by PTC and slowly becoming monetized
• Works on black and white
• Available as a UnityPackage
• Each target is its own separate game object
  • Add game objects as children of the target in to anchor them to the target
  • Except for ground/mid-air planes: ‘finder’ objects to find the plane, and ‘stage’ objects containing content
• Target types:
  • 2D Image
    • Single image: import image into Unity, drag image into texture field
      • These have Image Target Preview component which displays a preview of the image
    • Can create databases in the cloud, then download
    • Can have cloud image target which does processing online:
      • Useful for databases with large (hundreds) numbers of images
      • Requires paid license
  • Cylinder
  • Multi
    • Box: six images, one for each face
    • Add occlusion object: create 6 planes with depth mask material which is rendered before the game objects
    • Add target representation: renders the box on top of (where Vuforia thinks) the actual box (is)
  • 3D models:
    • CAD models (model targets)
      • Being deprecated
    • Scanned 3D objects (object targets)
      • Supposedly getting better
    • Scanned 3D environments (area targets)
      • Doesn’t work that well
    • Ground planes (ground or mid-air)
  • VuMarks
    • Vuforia’s custom fiducial markers
    • Can have multiple VuMarks with similar visual content but with different data
    • Created as SVG files: Illustrator template available
• Ground plane targets
  • Anchoring virtual content to ‘ground planes’ - horizontal planes in the environment
  • Uses SLAM: requires IMUs etc., so not supported on all devices
    • Can emulate in editor with a PDF print-out of a texture
    • Project -> Packages -> Vuforia Engine AR -> Vuforia -> Database -> ForPrint -> Emulator -> Emulator Ground Plane pdf file
  • Requires ‘track device pose’ to be enabled in Vuforia engine configuration
  • Ground plane finder:
    • Places a reticle on ground planes (think crosshair)
    • Interactive hit test: on tap, instantiates a prefab on the ground plane
    • Can also use automatic hit tests
  • Create ground plane stage:
    • Link to the ground plane finder (content positioning behavior, anchor stage)
      • Can enable duplication to have multiple copies of the ground stage content
    • Size is in real-world units (1m x 1m)
    • Can save the ground plane as a prefab (works somewhat)
• Mid-air positioner:
  • Fixed distance from the ground
  • Position is tracked relative to the ground plane
  • No automatic hit-tracking
  • Ground plane stage: change anchor behavior to MID_AIR
• Adding custom targets:
  • Requires license
    • In AR camera, go to Vuforia configuration
      • Enter license
      • Can also change settings such as scale

Vuforia AR camera:

• GameObject -> Vuforia Engine -> Camera
• Replaces default camera
• Vuforia engine configuration
  • Can also just edit configuration as a text file
    • Asset/Resources/VuforiaConfiguration.asset
    • Global settings: apply to all scenes
  • World center mode:
    • First target: first target detected is the world origin
    • Device: camera always at origin
      • Need to turn off track device pose
    • Origin impacts things such as physics simulations

Targets:

• Default observer event handler:
  • Responsible for turning on/off AR content when target is visible
    • Can run custom scripts or change properties of GameObjects when a target is found or lost
  • Can choose definition of Visible for each target:
    • Tracked: visible to the camera
    • Extended Tracked: the area immediately surrounding the target is visible
    • Limited: vague idea of position using IMU or something

UI:

• Can position element on screen anchored to a corner, center etc.
• Elements not scaled for different pixel densities: may appear tiny on phones
  • Game view, set display resolution to some high portrait resolution to preview
• UI elements automatically have a ‘Canvas Scaler’ component
  • Change UI scale mode from ‘constant pixel size’ to ‘scale with screen size’
  • Set a reference resolution and an axis which it scales along
  • Can also use ‘constant physical size’
• Button click events:
  • Can call any public method from any (instance of a) script assigned to a GameObject

Prefabs:

• Saved chunk of a scene (like a snapshot)
• Drag the game object into the project window to create a new prefab
  • Saves components, children etc.
• Creating elements from script:

  • GameObject -> new empty object

  • Add component -> new script

    public class SomeCreator: MonoBehaviour {
      // In the inspector, script component, can assign any GameObject (including prefabs) to the property
      public GameObject SomeGameObject;
      // Works with a lot more types to (e.g. int, Rect)
      void Start() {
        // Create new copy of the `SomeGameObject` object
        GameObject someNewCopy = GameObject.Instantiate(SomeGameObject);
        // Can optionally pass in position, rotation (as a quaternion)
        
        // Primitives can also be created programatically
        GameObject cube = GameObject.CreatePrimitive(PrimitiveType.Cube);
    
        // Can set the material properties for the cube's default MeshRenderer
        // GameObjects start off sharing the same default material
        // Setting the color creates a new instance of the material.
        cube.GetComponent<Renderer>().material.color = Random.ColorHSV();
        // `Color` components have a `[0, 1]` floating point range,
        // whereas `Color32` has a [0, 255] integer range
    
        // used `sharedMaterial` to modify the material instead, (potentially)
        // affecting multiple objects. Don't call if it uses the default material
        cube.GetComponent<Renderer>().sharedMaterial
    
        // For complex models (or prefabs) composed of multiple GameObjects,
        // we need to access the children components.
        // Find the first child object in the tree with the given name
        baseObject.transform.Find("ChildName");
      }
    }
    

Building for Mobile:

• File -> Build Settings
  • Set to iOS/Android
  • Click switch platform (and wait a while)
  • iOS:
    • Builds: recommend new folder for each project
    • Set signing team
• Project settings -> player
  • Can set company, product name, icons etc.
  • Device orientation: typically just portrait
  • Other settings:
    • Can set build number
    • Configuration:
      • Camera usage description: camera privacy description text
      • Scripting backend: IL.2CPP (intermediate language which compiles C# rather than interpreting it)
    • For Vuforia:
      • iOS:
        • Target iOS version >= 11
        • Architecture ARM64
        • Product name cannot be ‘vuforia’ (there is a library called ‘vuforia’ which it gets mixed up with)
      • Android:
        • ARCore: if available, Vuforia will use it
          • Minimum API level 24, remove Vulkan (check if this is still the case)
• Android logs: adb logcat -s "Unity:*" (s for silence all but logs that match the given string)

Adrian:

• Co-founded AR company
• AR overused and often misunderstood
• Before developing AR applications, ask if there is benefit to doing it in AR:
  • Could it be done in VR?
  • As a desktop/mobile app?
  • As a webpage?
• AR useful for visualizing spatial data, especially if it has an intrinsic link to the real world-environment
  • If it fails the latter, it should be done in VR instead (or even just on a flat screen)
  • Data should have at least 3 spatial dimensions
    • You move around in 3 dimensions and hence, the data should have at least that many dimensions
• Awkward interactions:
  • Holding your phone up with one hand while tapping the screen
    • Requires powerful phones and drains battery
  • Trying to tap targets in mid air while wearing a heavy HMD
• Hard to find a balance for visualization realism:
  • Shouldn’t perfectly match the environment: people need to know they can interact with it
  • Shouldn’t be so out-of-place to be jarring
  • AR visualization considerations:
    • Real/virtual object occlusion
      • Never pixel perfect
    • Lighting which matches the real world (brightness, color, reflection)
      • Can’t get high-quality reflections: don’t know what’s behind the camera
      • Shadowing to give the perception of distance
      • Clutter/contrast between real/virtual objects
• Interactions
  • Still in its infancy
  • With touch:
    • How do we ensure people can accurately touch?
      • Fat fingers, and phone is held in one arm outstretched
    • How do we choose where they touch in 3D space?
  • With gestures:
    • What gestures are intuitive? How does it vary by culture
    • How do we combat fatigue?
    • How do we stop it from looking embarrassing?
    • How do we deal with a lack of haptic response?
      • Is my finger past the target? In front of it? On it?
      • Without good occlusion, haptics is important
  • Forget about the WIMP (windows, icons, menus and pointers) metaphor
    • Why bring an intrinstically 2D metaphor into an intrinstically 3D experience?
    • Think about new interactions and visual affordances
      • Tangible user interfaces
        • e.g. magic lenses: https://www.youtube.com/watch?v=PKegByAZ0kM
      • AR should be seamless: we should forget it even exists