Recording and Playback

Vuforia Engine allows developers to record and play back AR sessions. Camera frames and device tracking data is recorded in order to play back the AR session at a later time. This enables developers to rapidly iterate on their app and test AR experiences very efficiently. In particular, this is helpful for building AR experiences that use Model Targets or Area Targets, where the physical target is not always available for testing. Recorded sequences can then be played back either on a device or using Unity Editor Play Mode.

Overview and Recommended Workflow

Sessions can be recorded in any application using Vuforia Engine. See Vuforia Play Mode for more details on how to replay recordings in Unity.
Vuforia Engine also offers a SessionRecorder API that can be used directly in a native or Unity app. See the last section of this article for more information about this API.

While it’s possible to record a session without using any Vuforia targets, not having any visual feedback for target tracking removes any visual confirmation that Vuforia has detected the target.
We recommend the following development process for incorporating recordings in your workflow.

It is worth mentioning that it is important to work out the scenario you wish to capture before recording. A clear strategy on what to capture; whether it is user centric, sequential steps, or targets under environmental constraints, will reward you with an ideal recording that can help in the development process and result in a capable AR application.

NOTE: The following restrictions apply:

  • Recording on device is currently only supported in landscape orientation.
  • Currently devices with cameras that capture frames in the following pixel formats are supported:
    • NV21
    • NV12
    • Luminance-only (Grayscale)
  • Recording and playback of sessions is currently not supported for Ground Plane experiences.
  • Recording on Magic Leap is currently not supported

Session Recorder in Unity

In Unity, there is a readily available Session Recorder GameObject that can be added to any existing application. It will display a customizable UI for recording and will call the underlying API for you.

NOTE: For advanced use cases, recording can also be started and stopped using the scripting API. See the section further below for more details. 

  1. First, set up a Vuforia enabled scene with the Session Recorder GameObject, ARCamera GameObject, and a Vuforia Target. These are easily added by right-clicking in the Hierarchy panel and selecting Vuforia Engine.

NOTE: if the Session Recorder GameObject is not present, please ensure that you are using the latest Vuforia Engine package for Unity (9.5 and later). 

Session Recorder configuration

From the Session Recorder Behaviour component, it is possible to tweak the default UI and behaviour. If not needed, you can hide the Clean button or disable sensor data recording in the Advanced Settings. However, recording sensor data is strongly advised.

The Overlay Canvas contains the UI for mobile devices, while the Holographic Canvas is used on HoloLens. On HoloLens, the voice commands “SessionRecorder Start” “SessionRecorder Stop” and “SessionRecorder Clean” control to the Session Recorder functionality.

The Session Recorder Behaviour is fully customizable and can interact with any other MonoBehaviour in the scene. For inspiration, see the Model Target scene from the Vuforia Core Samples for Unity.

NOTE: For iOS builds, the UIFileSharingEnabled plist entry is automatically enabled when the Session Recorder GameObject is added to your scene. You can also configure it manually in Vuforia Configuration.  

  1. Set the build platform and Player Settings according to your connected device and press build and run.
  2. When the app is launched, it may be required to allow permission to use and access the camera. 
  3. Once the camera feed is present, press the record button and record your scene and Vuforia Target. 
    Please note the following limitations: 
    • Recording a session is only possible from a device, not from the Unity Editor with Vuforia’s Webcam Play Mode 
      (however, a recorded session can be played back in Play Mode, which is described below in Using Recordings)
    • Recording a session is currently only possible in landscape left mode.
  4. Pressing the Stop button will conclude the recording.
  5. Between recordings, you can choose to remove existing recorded sessions by pressing the Clean button to free up space on the device.

NOTE: The Model Target scene in the Vuforia Engine Core Unity Samples includes an example of a customized recording API.

Transferring Recordings

The recorded session will be stored in the app’s private storage of your device and there are different procedures to retrieve the session that depend on the device platform.

Retrieving a recording is easiest on iOS, therefore it is recommended to use an iOS device for recording, if possible.

  1. On Android devices, the recorded data can be accessed using the adb command or using a tool like Android File Transfer or Windows Explorer. The recordings are found within the shared files directory of the app on your device (this is what the getExternalFilesDir Android API method returns). The exact path may vary from device to device. One known variant is /storage/emulated/0/Android/data, so the whole path to the recordings might look like 

/storage/emulated/0/Android/data/[PACKAGE NAME]/files/Recordings

The section Querying the Location of a Recording  in the Session Recorder API Overview article explains how to obtain the path to each recording. The [PACKAGE NAME] will have a form like com.mycompany.MyApps.MyCoolApp (specified within the AndroidManifest.xml of the app using the package property). 

To copy a recording to the current working directory on your computer, you can use the following adb command : 

adb pull /storage/emulated/0/Android/data/[PACKAGE NAME]/files/Recordings/VuforiaSession-[DATE]-[TIME] ./

The recording can alternatively be copied using Android Studio, Android File Transfer on Mac, or Windows File Explorer on Windows. 

In Android Studio, select View -> Tool Windows -> Device File Explorer and browse to the files and right-click them and select Save As… It may be necessary to right-click and select Synchronize first, if no files appear in the Data folder.

For Windows, you can use the File Explorer and browse the MTP device from This PC. Connect your device with a USB and press the Allow on the pop-up message on the device as it connects. The recordings can now be transferred.

For macOS, you can use the Android File Transfer software from Google to browse to the Data folder and retrieve the recordings.

  1. On iOS devices, the recorded data can be accessed using iTunes, macOS’s Finder or a tool like iExplorer. iExplorer will list the app under the Apps menu on the bar on the left hand-side. Clicking on the app’s name reveals a number of directories on the main window belonging to the app. The recordings are to be found within the Documents/Recordings sub-directory and can simply be copied using the export functionality of iExplorer. For this, right-click on the directory you’d like to copy and select the Export to Folder-> option. It can also be dragged-and-dropped into a directory that is open in Finder.
  1. On UWP devices, the recorded data can be accessed using the operating system’s file browser. The recordings are to be found in the recording app’s private storage space. The path to the recordings directory has the form: 

C:\Users\[USER NAME]\AppData\Local\Packages\[PACKAGE NAME]_[PACKAGE FAMILY]\LocalState\Recordings.

The [PACKAGE NAME] will have a form like My.Apps.MyCoolApp and [PACKAGE FAMILY] will be a cryptic string containing numbers and letters.  

  1. On the HoloLens, the Device portal API of Microsoft (see Microsoft's device portal api reference article) can be used for downloading the recorded datasets on to a computer. This can be done in two steps using HTTP GET calls:
  1. Get a list of all files in the Recordings directory,
  2. Get all the files in the list the first call returns.

Please see below for the details of the HTTP calls.

For step 1:

  • URL: http://[HOLO LENS IP]:10080/api/filesystem/apps/files
  • Parameters:
    • knownfolderid = LocalAppData
    • packagefullname = [PACKAGE NAME]_[PACKAGE FAMILY]
    • path = \\LocalState\Recordings (note the double backslashes, which might need to be escaped in a string)

For step 2:

  • URL: http://[HOLO LENS IP]:10080/api/filesystem/apps/file (note the singular file, in contrast to files above)
  • Parameters:
    • knownfolderid = LocalAppData
    • packagefullname = [PACKAGE NAME]_[PACKAGE FAMILY]
    • path = [PATH TO FILE FROM RETRIEVED LIST]
    • filename = [FILENAME FROM RETRIEVED LIST]

Please note that you will need to include your HoloLens credentials (username and password) as part of this HTTP call. For the [PACKAGE NAME] and [PACKAGE FAMILY], please see the explanation in the UWP section above.
 

Using Recordings

Using Recordings in Unity Editor Play Mode

Using a recording in Unity Editor Play Mode allows you to rapidly prototype your AR experience without building to a mobile device, and, without having to be close to the physical target that you use for testing.

Follow the steps below to use a recording in Play Mode:

  1. Once the data is transferred to your workstation, the Vuforia Configuration can be set to Play Mode type – RECORDING.
  2. Browse to the directory of the saved recording data and select the recording session you wish to use.
  3. The recorded session will be played back once Play Mode is started in the editor.

See Vuforia Play Mode in Unity for more information.

Using Recordings in a mobile application

In cases where you want to test your application on the actual mobile target device, but still do not want to test on the physical object / location (e.g. because it is not available), you can play back a recording using the File Driver sample in a native or Unity application.

See the File Driver Sample for more information.
 

Creating Recordings using the SessionRecorder API

Using the Unity scripting API

Instead of using the Session Recorder GameObject, the recording functionality can also be used by directly calling into the SessionRecorder scripting API. This allows you to create custom interfaces and i.e. instructions of what to scan and how long.

Start by calling the SessionRecorder.Instance and initiate a recording using the SessionRecorder.Start call. During recording you can check the status with GetRecordingStatus and enlist conditions depending on the returned status. Conclude the recording with SessionRecorder.Stop and call the method GeRecordingPath to identify the location of the stored recording.

For more information, please refer to the Session Recorder API Overview.

Creating Recordings in a native application

To use the Session Recorder in native, similarly start with accessing the singleton of the SessionRecorder with SessionRecorder::getInstance. Use start and stop to control the recording session and output the recording data consisting of camera images and an .xml file. These files are stored in a directory named VuforiaSession followed by time and date stamps.

The camera images include data on device pose and similar information that helps you in testing and troubleshooting on your device and scenario. The types of data that can be recorded can be identified with getSupportedSources, where the Camera source includes the device pose.

Once a recording is concluded the stored location can be queried with getRecordingPath. To retrieve the recordings, please see the above section and note the different approaches for different operating systems.

NOTE: If you plan to record on an iOS device, the UIFileSharingEnabled plist entry needs to be set in Xcode.

For additional SessionRecorder methods, please see Session Recorder API Overview.
 

Learn More

Session Recorder API Overview

How to Use the File Driver Sample

Vuforia Play Mode in Unity

Can this page be better?
Share your feedback on our community forum