Getting started with AR development in Unity
To get started with AR development, Unity recommends using AR Foundation to create your application for Unity’s supported handheld AR and wearable AR devices.
AR Foundation allows you to work with augmented reality platforms in a multi-platform way within Unity. This package presents an interface for Unity developers to use, but doesn’t implement any AR features on its own.
To use AR Foundation on a target device, you also need to download and install separate packages for each of the target platforms officially supported by Unity:
- ARCore XR Plug-in on Android
- ARKit XR Plug-in on iOS
- Magic Leap XR Plug-in on Magic Leap
- Windows XR Plug-in on HoloLens
For instructions on how to configure your Project using the XR Plug-in Management system, see the Configuring your unity Project for XR page.
AR Foundation supports the following features:
| Feature | Description |
|---|---|
| Device tracking | Track the device’s position and orientation in physical space. |
| Raycast | Commonly used to determine where virtual content will appear, where a ray (defined by an origin and direction) intersects with a real-world feature detected and/or tracked by the AR device. Unity has built-in functions that allow you to use raycasting in your AR app. |
| Plane detection | Detect the size and location of horizontal and vertical surfaces (e.g. coffee table, walls). These surfaces are called “planes”. |
| Reference points | Track the positions of planes and feature points over time. |
| Point cloud detection | Detect visually distinct features in the captured camera image and use these points to understand where the device is relative to the world around it. |
| Gestures | Recognize gestures as input events based on human hands. |
| Face tracking | Access face landmarks, a mesh representation of detected faces, and blend shape information, which can feed into a facial animation rig. The Face Manager configures devices for face tracking and creates GameObjects for each detected face. |
| 2D image tracking | Detect specific 2D images in the environment. The Tracked Image Manager automatically creates GameObjects that represent all recognized images. You can change an AR application based on the presence of specific images. |
| 3D object tracking | Import digital representations of real-world objects into your Unity application and detect them in the environment. The Tracked Object Manager creates GameObjects for each detected physical object to enable applications to change based on the presence of specific real-world objects. |
| Environment probes | Detect lighting and color information in specific areas of the environment, which helps enable 3D content to blend seamlessly with the surroundings. The Environment Probe Manager uses this information to automatically create cubemaps in Unity. |
| Meshing | Generate triangle meshes that correspond to the physical space, expanding the ability to interact with representations of the physical environment and/or visually overlay the details on it. |
| 2D and 3D body tracking | Provides 2D (screen-space) or 3D (world-space) representations of humans recognized in the camera frame. For 2D detection, humans are represented by a hierarchy of seventeen joints with screen-space coordinates. For 3D detection, humans are represented by a hierarchy of ninety-three joints with world-space transforms. |
| Human segmentation | The Human Body Subsystem provides apps with human stencil and depth segmentation images. The stencil segmentation image identifies, for each pixel, whether the pixel contains a person. The depth segmentation image consists of an estimated distance from the device for each pixel that correlates to a recognized human. Using these segmentation images together allows for rendered 3D content to be realistically occluded by real-world humans. |
| Occlusion | Apply distance to objects in the physical world to rendered 3D content, which achieves a realistic blending of physical and virtual objects. |
| Participant tracking | Track the position and orientation of other devices in a shared AR session. |
AR platform support
AR Foundation doesn’t implement any AR features on its own. Instead, it defines a multi-platform API that allows developers to work with functionality common to multiple platforms.
AR Foundation supports the following features across different platforms:
For more information on how to download and use AR Foundation, see the AR Foundation package documentation.