Using the Vuforia Fusion Platform Handle

Introduction

In Vuforia Engine 8.1, a new experimental API has been added that provides native [Objective-C (iOS), C++ (Android NDK)] developer access to ARKit and ARCore specific functionality. This, when used with care, can be used to combine Vuforia functionality with functionality that currently only exists in ARKit and ARCore. For instance, an application can access plane boundaries or additional lighting information.

Detailed information on ARKit can be found here, while that on ARCore can be found here.

This page gives you a feature overview of using the plafrom handle and provides several examples on usage:

Using the Platform Handle

Platform Access via getFusionProviderPlatformInfo()

The API to use is getFusionProviderPlatformInfo(), which returns a FusionProviderPlatformInfo struct that contains pointers to the session and frame of ARKit or ARCore. The values contained within the struct are platform-specific and need to be used with platform-specific code. The code block below provides an overview of the API. More details can be found in the Vuforia Reference Documentation.

/// The types of platform Fusion provider information available
enum FUSION_PROVIDER_PLATFORM_TYPE
{
  /// Unknown provider
  FUSION_PROVIDER_PLATFORM_NOT_AVAILABLE = 0,

  /// The provider is ARKit
  FUSION_PROVIDER_PLATFORM_ARKIT,

  /// The provider is ARCore
  FUSION_PROVIDER_PLATFORM_ARCORE
};

/// struct returned from getFusionProviderPlatformInfo()
struct FusionProviderPlatformInfo
{
  /// The type of provider
  FUSION_PROVIDER_PLATFORM_TYPE mType;

  /// Pointer to the Provider Session
  void *mSession;

  /// Pointer to the Provider Frame
  void *mFrame;
};

/// Returns information about the underlying Platform AR currently in use
FusionProviderPlatformInfo VUFORIA_API getFusionProviderPlatformInfo();

Notes on the API Usage - Caveats

Pointer Ownership

Vuforia owns the pointers contained in this struct; developers must exercise caution while using them. 

  • Do not release or destroy the session or the frame.
  • Do not pause the session.
  • Do not reconfigure the session.

Note: Doing any of the above will cause Vuforia's handling of the information from the provider to fail in undefined ways.

Camera Ownership

Vuforia owns the camera throughout this process. In order to change any camera properties such as the video mode or autofocus mode, you need to use existing Vuforia camera control APIs.

Pointer Validity

  • A valid value for the session will be available after the camera has been started and will remain valid until the camera is stopped.
  • A valid value for the frame will be available after the camera has been started and will only be valid for the duration of one frame.
  • The caller needs to request the frame information for every new frame.

Platform-Specific

  • No ARKit delegate callbacks are available on iOS.
    • All information supplied through the callbacks is available via the ARSession.
  • No Java-binding support on Android.
    • Developers should use JNI to access this functionality from Java applications.
  • Some parameters of the returned struct also have a platform-specific lifecycle.

Expected Usage

It is expected that these calls will be most useful during the rendering phase of the application, and largely in a read-only mode from within an application's render loop.

Vuforia Tracker Behavior

The behavior of Vuforia Engine has been modified to provide developers with more flexibility when making use of the getFusionProviderPlatformInfo() API call.

 

Plane Detection

(getFusionProviderPlatformInfo() not called)

Plane Detection

(getFusionProviderPlatformInfo() called)

Vuforia HitTest Results 

Tracker Reset

(getFusionProviderPlatformInfo() not called)

Tracker Reset

(getFusionProviderPlatformInfo() called)

ARCore

 Horizontal and vertical plane detection enabled, this is for performance reasons. Horizontal and vertical plane detection always enabled. Horizontal plane results only. Vuforia hitTest behavior remains consistent with previous releases independent of the mode. Future releases will make use of the HITTEST_HINT described here. When reset is called on a tracker, all anchors that have been created will be removed, as Vuforia Engine is assumed to own all anchors. When reset is called on a tracker, only anchors that have been created by Vuforia Engine will be removed.

ARKit

 Horizontal plane detection enabled, as needed. Horizontal and vertical plane detection always enabled. Horizontal plane results only. Vuforia hitTest behavior remains consistent with previous releases independent of the mode. Future releases will make use of the HITTEST_HINT described here. When reset is called on a tracker, all anchors that have been created will be removed, as Vuforia Engine is assumed to own all anchors. When reset is called on a tracker, only anchors that have been created by Vuforia Engine will be removed.

 

Converting to a Platform-Specific Pointer

ARKit

For complete details of the ARKit API, developers should refer to Apple documentation. As Vuforia owns the pointers, developers must use the (__bridge ARSession*) cast in Objective-C to turn the void* into an ARSession or ARFrame. This tells Objective-C's reference counting that the pointer should not be handled by the application.

e

#import <ARKit/ARKit.h>
 
// Code omitted ...
 
Vuforia::FusionProviderInfo info = Vuforia::getFusionProviderInfo();
 
ARSession* arSession = (__bridge ARSession*)info.mSession;
 
ARFrame* arSession = (__bridge ARFrame*)info.mFrame;
/*
 * Alternatively for ARKit the frame can also be obtained directly from the
 * ARSession, using arSession.currentFrame;
 */
 
// Code omitted ...

ARCore

For complete details of the ARCore API, developers should refer to Google documentation. As ARCore is a C-based API, it is sufficient to use a static_cast<> to turn the pointer from a void* into an ArSession or ArFrame. As this is an NDK pointer, this functionality cannot be used directly from within a Java Android application. If developers wish to use these APIs within a Java application, they will have to make the calls in a JNI section of the application.

  #include <arcore_c_api.h>
   
  // Code omitted ...
   
  Vuforia::FusionProviderInfo info = Vuforia::getFusionProviderInfo();
   
  ArSession *session = static_cast<ArSession>(info.mSession);
   
  /*
   * In ARCore developers must get the ArFrame using this call. Updating
   * the ArSession to get the ArFrame will cause the Vuforia library to
   * enter an undefined state.
   */
  ArFrame *frame = static_cast<ArFrame *>(info.mFrame);
   
  // Code omitted ...

 

Example 1 - Printing Tracking Information to the Console

This first example describes how to print the tracking state to the console using the platform. In further examples, we will build on this example while looking at rendering plane boundaries.

Adding ARKit into the Vuforia Samples Application

Prerequisites

Steps

To add ARKit into an existing Vuforia Xcode project (in this case, the Vuforia Samples Application) as part of the GroundPlane feature, follow these steps:

  1. Add #import <ARKit/ARKit.h to GroundPlaneEAGLView.mm so that the top of the file looks like this, see line 6. Line 9 was added as we use std::string in the sample code. For this sample, we also add a new method testFusionProviderPlatformInfo, line 18, which we will implement later in the documentation.
    #import <QuartzCore/QuartzCore.h>
    #import <OpenGLES/ES2/gl.h>
    #import <OpenGLES/ES2/glext.h>
    #import <sys/time.h>
    #import <GLKit/GLKit.h>
    #import <ARKit/ARKit.h>
 
    // std::string is used in the sample code
    #import <string>
 
    // ... Code omitted for brevity
 
    @interface GroundPlaneEAGLView ()
    - (void)initShaders;
    - (void)createFramebuffer;
    - (void)deleteFramebuffer;
    - (void)setFramebuffer;
    - (void)testFusionProviderPlatformInfo;
    - (BOOL)presentFramebuffer;
  1. In the "Build Phases" tab of the Vuforia Samples Xcode project, add ARKit.framework into the "Link Binary With Libraries" section, so that it appears like this:
  1. Edit GroundPlaneEAGLView.mm and add a call to the testFusionProviderPlatformInfo into renderFrameWithState.
- (void) renderFrameWithState:(const Vuforia::State&) state
                projectMatrix:(Vuforia::Matrix44F&) projectionMatrix
{
    [self testFusionProviderPlatformInfo];
  1. Now, add this code into the class which implements testFusionProviderPlatformInfo.
- (void) testFusionProviderPlatformInfo
{
    Vuforia::FusionProviderPlatformInfo info = Vuforia::getFusionProviderPlatformInfo();
     
    if (info.mType != Vuforia::FUSION_PROVIDER_PLATFORM_TYPE::FUSION_PROVIDER_PLATFORM_ARKIT)
    {
        NSLog(@"Fusion provider platform type does not match the expected type");
        return;
    }
     
    if ((info.mSession == nullptr) ||
        (info.mFrame == nullptr))
    {
        NSLog(@"Fusion provider platform pointers are not set");
        return;
    }
     
    ARSession* arSession = (__bridge ARSession*)info.mSession;
     
    // In ARKit the same ARFrame* can also be extracted from the session
    // as shown below, and should be the preferred method.
    ARFrame* arFrame = (__bridge ARFrame*)info.mFrame;
     
    if (arSession != nil)
    {
        ARFrame* arSessionFrame = arSession.currentFrame;
        if (arFrame != arSessionFrame)
        {
            // This test would not normally be needed, it is here for
            // example purposes only
            NSLog(@"Error arFrame does not match the one on the ARSession");
            return;
        }
         
        // NSLog(@"ARSession is %@", arSession);
        if ((arSessionFrame != nil) &&
            (arSessionFrame.camera != nil))
        {
            std::string trackingStateStr;
            switch(arSessionFrame.camera.trackingState)
            {
                case ARTrackingStateNotAvailable:
                    trackingStateStr = "Available";
                    break;
                     
                case ARTrackingStateLimited:
                    trackingStateStr = "Limited";
                    break;
                     
                case ARTrackingStateNormal:
                    trackingStateStr = "Normal";
                    break;
            }
            NSLog(@"the current tracking state is: %s", trackingStateStr.c_str());
        }
    }
}
  1. When running the application and launching the ground plane feature, you will now see messages such as this one:
2019-02-18 17:20:58.360473+0000 Vuforia[4670:1150633] the current tracking state is: Limited
2019-02-18 17:20:58.377147+0000 Vuforia[4670:1150633] the current tracking state is: Limited
2019-02-18 17:20:58.393734+0000 Vuforia[4670:1150633] the current tracking state is: Limited
2019-02-18 17:20:58.477162+0000 Vuforia[4670:1150633] the current tracking state is: Normal

 

Adding ARCore into the ImageTargets Native Sample Application

Prerequisites

Steps

In this example, we will modify the ImageTargetsNative sample as it is a native NDK application. Adding ARCore to an existing application requires use of the ARCore NDK API bundle. This can be downloaded automatically as part of the gradle build.

  1. Update the build.gradle file to include support for ARCore; more details can be found in the Google ARCore documentation.
  1. Add these lines at the top of the file.
apply plugin: 'com.android.application'
 
/*
The arcore aar library contains the native shared libraries.  These are
extracted before building to a temporary directory.
 */
def arcore_libpath = "${buildDir}/arcore-native"
 
// Create a configuration to mark which aars to extract .so files from
configurations { natives }
  1. Add line 5 below into the dependencies section of the build.gradle file, where 1.6.0 is the version of ARCore you wish to use, and pass the location of ARCore to the NDK build used by Android.mk (line 14 below).
dependencies {
    implementation files("$VUFORIA_SDK_DIR/build/java/Vuforia/Vuforia.jar")
 
    // ARCore library
    natives "com.google.ar:core:1.6.0"
    implementation 'com.google.ar:core:1.6.0'
}
 
// .. code omitted
// Add the ARCORE_LIBPATH arguments onto the ndkBuild
        externalNativeBuild {
            ndkBuild {
                arguments ndk_args.split()
                arguments += "ARCORE_LIBPATH=${arcore_libpath}/jni".toString()
            }
        }
  1. At the end of the build.gradle file, add these lines to download the ARCore native code.
// Extracts the shared libraries from aars in the natives configuration.
// This is done so that NDK builds can access these libraries.
task extractNativeLibraries() {
    // Always extract, this insures the native libs are updated if the version changes.
    outputs.upToDateWhen { false }
    doFirst {
        configurations.natives.files.each { f ->
            copy {
                from zipTree(f)
                into arcore_libpath
                include "jni/**/*"
            }
        }
    }
}
 
tasks.whenTaskAdded {
    task-> if (task.name.contains("external") && !task.name.contains("Clean")) {
        task.dependsOn(extractNativeLibraries)
    }
}
  1. Edit app/src/main/Android.mk with this patch.
  --- a/QCAR/samples/android/ImageTargetsNative/app/src/main/jni/Android.mk
+++ b/QCAR/samples/android/ImageTargetsNative/app/src/main/jni/Android.mk
@@ -21,11 +21,24 @@ LOCAL_PATH := $(call my-dir)
  
 include $(CLEAR_VARS)
 VUFORIA_SDK_DIR := ../../../../../../../../../Vuforia_install/Android/Public/Vuforia-sdk
+
 LOCAL_MODULE := Vuforia-prebuilt
-LOCAL_SRC_FILES = $(VUFORIA_SDK_DIR)/build/lib/$(TARGET_ARCH_ABI)/libVuforia.so
+LOCAL_SRC_FILES := $(VUFORIA_SDK_DIR)/build/lib/$(TARGET_ARCH_ABI)/libVuforia.so
 LOCAL_EXPORT_C_INCLUDES := $(LOCAL_PATH)/$(VUFORIA_SDK_DIR)/build/include
 include $(PREBUILT_SHARED_LIBRARY)
  
+include $(CLEAR_VARS)
+LOCAL_MODULE := arcore-prebuilt
+LOCAL_SRC_FILES := $(ARCORE_LIBPATH)/$(TARGET_ARCH_ABI)/libarcore_sdk_c.so
+
+# The ARCore header file has been included in the Local Path. If the header file
+# is elsewhere add the include path here and uncomment these lines
+#ARCORE_HEADER_PATH := $(ARCORE_HEADER_PATH)/include
+#LOCAL_EXPORT_C_INCLUDES := $(ARCORE_HEADER_PATH)
+include $(PREBUILT_SHARED_LIBRARY)
+
+
+
 #-----------------------------------------------------------------------------
  
 # The CLEAR_VARS variable is provided by the build system and points to a
@@ -55,7 +68,7 @@ OPENGLES_DEF  := -DUSE_OPENGL_ES_2_0
 # The flag "-Wno-write-strings" removes warnings about deprecated conversion
 #   from string constant to 'char*'.
  
-LOCAL_CFLAGS := -Wno-write-strings $(OPENGLES_DEF) -std=c++11
+LOCAL_CFLAGS := -Wno-write-strings $(OPENGLES_DEF) -std=c++14
  
 # The list of additional linker flags to be used when building your
 # module. Use the "-l" prefix in front of the name of libraries you want to
@@ -69,7 +82,7 @@ LOCAL_LDLIBS := \
 # in the generated file. Here we reference the prebuilt library defined earlier
 # in this makefile.
  
-LOCAL_SHARED_LIBRARIES := Vuforia-prebuilt
+LOCAL_SHARED_LIBRARIES := Vuforia-prebuilt arcore-prebuilt
  
 # The LOCAL_SRC_FILES variables must contain a list of C/C++ source files
 # that will be built and assembled into a module. Note that you should not

3. Copy the arcore_c_api.h file from the Android SDK into the app/src/main/jni directory or include it in a path that is defined in the app/src/main/Android.mk file.

4. Add an include of arcore_c_api.h to SampleAppRenderer.cpp.

#include "SampleAppRenderer.h"
#include "SampleUtils.h"
 
#include "arcore_c_api.h"
  1. Add a call to testFusionProviderPlatformInfo into renderFrame() in SampleAppRenderer.cpp.
void SampleAppRenderer::renderFrame()
{
  pthread_mutex_lock(& renderingPrimitivesMutex);

  testFusionProviderPlatformInfo();

6. Now, add a new method to SampleAppRenderer.h.

// Updates rendering primitives on start and configuration changes
void updateRenderingPrimitives();

private:
void testFusionProviderPlatformInfo();
  1. Create the implementation in SampleAppRenderer.cpp. This sample describes how to create a configuration to see if autofocus has been enabled and, similar to the ARKit sample, prints the tracking state.
  void SampleAppRenderer::testFusionProviderPlatformInfo()
{
    Vuforia::FusionProviderPlatformInfo info = Vuforia::getFusionProviderPlatformInfo();
    if (info.mType != Vuforia::FUSION_PROVIDER_PLATFORM_ARCORE)
    {
        LOG("Fusion provider platform type does not match the expected type");
        return;
    }
 
    if ((info.mSession == nullptr) || (info.mFrame == nullptr))
    {
        LOG("Fusion provider platform pointers are not set");
        return;
    }
 
    ArSession *arSession = (ArSession *)info.mSession;
    ArFrame *arFrame = (ArFrame *)info.mFrame;
 
    ArConfig *config;
    ArConfig_create((const ArSession *) arSession, &config);
    ArSession_getConfig(arSession, config);
 
    ArFocusMode focusMode;
    ArConfig_getFocusMode((const ArSession *) arSession, config, &focusMode);
    LOG("focusMode is %d", focusMode);
    ArConfig_destroy(config);
 
    ArCamera *arCamera;
    ArFrame_acquireCamera(arSession, arFrame, &arCamera);
    ArTrackingState trackingState;
    ArCamera_getTrackingState(arSession, arCamera, &trackingState);
    switch (trackingState)
    {
    case AR_TRACKING_STATE_STOPPED:
        LOG("trackingState is stopped");
        break;
    case AR_TRACKING_STATE_TRACKING:
        LOG("trackingState is tracking");
        break;
    case AR_TRACKING_STATE_PAUSED:
        LOG("trackingState is paused");
        break;
    }
}

8. Now, build and run the application you should see logging, as per the example below. Printing out the focus mode per frame is shown only for demonstration; it is recommended that developers omit it after proving that it works.

02-19 18:03:29.987 31262-31421/com.vuforia.engine.ImageTargets I/Vuforia: trackingState is tracking
02-19 18:03:30.037 31262-31421/com.vuforia.engine.ImageTargets I/Vuforia: focusMode is 1
 
02-19 18:03:30.608 31262-31421/com.vuforia.engine.ImageTargets I/Vuforia: trackingState is paused

 

Example 2 - Creating and Destroying Anchors

Vuforia Engine already has an API to create, manage, and destroy anchors; hence, there shouldn't be a need to do this outside of Vuforia Engine. However, if this is something the developer needs to do, it can be done using the existing ARKit or ARCore APIs. This may be the case if the developer is dealing with cloud anchors, for example.

ARKit

The ARKit API for anchors is available here. All anchors are available from the ARFrame using the currentFrame. All anchors in ARKit have a unique UUID assigned to them; if a developer creates an anchor outside of Vuforia Engine, the UUID should allow the developer to track changes in the pose or other anchor properties by searching by UUID on the list of anchors.

Below is an example of creating and removing an anchor using ARKit.

std::string anchorUUID;
 
Vuforia::FusionProviderPlatformInfo info = Vuforia::getFusionProviderPlatformInfo();
ARSession* arSession = (__bridge ARSession*)info.mSession;
ARFrame* arFrame = (__bridge ARFrame*)info.mFrame;
 
if (arSession != nil && arFrame != nil)
{
    // Create a transform with a translation of 0.2 meters in front of the camera
    matrix_float4x4 translation = matrix_identity_float4x4;
    translation.columns[3].z = -0.2;
    matrix_float4x4 transform = matrix_multiply(arFrame.camera.transform, translation);
 
    // Add a new anchor to the session
    ARAnchor *anchor = [[ARAnchor alloc] initWithTransform:transform];
    [arSession addAnchor:anchor];
    anchorUUID = anchor.identifier.UUIDString.UTF8String;
 
    // Remove the anchor
    [arSession removeAnchor:anchor];
}

The ARAnchor pointer has a unique ID, which helps identify the anchor when looping around the anchors present on the currentFrame. That anchor can then be used to remove the anchor. The pointer address of the anchor can change; it is therefore recommended to use the UUID as a means of identifying the anchor in future.

  NSArray<aranchor*>* arKitAnchors = arSession.currentFrame.anchors;
  bool foundMyAnchor {false};
  ARAnchor* myAnchor {};
  for (ARAnchor* arAnchor in arKitAnchors)
  {
      if (arAnchor.identifier.UUIDString.UTF8String ==  anchorUUID)
      {
          foundMyAnchor = true;
          myAnchor = arAnchor;
      }
  }
   
  // At this point if the anchor transform has been updated the application
  // would update any augmenations using that anchor rather than delete it
  // as in this example.
   
  if (foundMyAnchor)
  {
      // Delete it
      [arSession removeAnchor:myAnchor];
  }

ARCore

The ARCore API for anchors is available here. Once tracking has been established, it is possible to create anchors, for instance, using the camera pose. The following code will create an anchor in front of the camera:

ArPose* arCameraPose = nullptr;
ArPose_create(arSession, nullptr, &arCameraPose);
 
ArCamera* arCamera;
ArFrame_acquireCamera(arSession, arFrame, &arCamera);
 
ArTrackingState trackingState;   
ArCamera_getTrackingState(arSession, arCamera, &trackingState);
 
if (trackingState == AR_TRACKING_STATE_TRACKING)
{
    ArCamera_getPose(arSession, arCamera, arCameraPose);
 
   // Get the raw pose and transform it with a translation of 0.2 meters in front of the camera
   float cameraPoseRaw[7] = {0.f};
   ArPose_getPoseRaw(arSession, arCameraPose, cameraPoseRaw);
   cameraPoseRaw[6] = {-0.2f};
 
   ArPose* arAnchorPose = nullptr;
   ArPose_create(arSession, cameraPoseRaw, &arAnchorPose);
 
   ArAnchor* arAnchor = nullptr;
   if (ArSession_acquireNewAnchor(arSession, arCameraPose, &arAnchor) != ArStatus::AR_SUCCESS)
   {
       LOG("ArSession_acquireNewAnchor error");
   }
 
   ArPose_destroy(arCameraPose);
   ArPose_destroy(arAnchorPose);
}
 
ArCamera_release(arCamera);

 

Example 3 - Visualizing Plane Boundaries

ARKit

ARKit 1.0 only defines a bounding box for the ARPlaneAnchor class. ARKit 1.5 (iOS 11.3 and above) defines a crude 3D mesh that can be used to visualize the plane in the class.

 

ARCore

This example details how to create a mash-up of the Google hello_ar_c sample with the ImageTargetsNative application showing the rendering of planes and point clouds.

Steps to Mash Up the Two Applications

In this example, the shortest possible route has been taken to combine the two applications.

Adding Files

  • Add arcore_c_ap.h to the ImageTargetsNative/app/src/main/jni directory.
  • JNI files that need to be extracted from the hello_ar_c sample application, the files were added to a new directory ImageTargetsNative/app/src/main/jni/hello_ar_c.
    • glm.h
    • plane_renderer.cc
    • plane_renderer.h
    • point_cloud_renderer.cc
    • point_cloud_renderer.h
    • util.cc
    • util.hpp
  • Java files that need to be extracted from the hello_ar_c application and added to ImageTargetsNative/app/src/mainjava/com/vuforia/engine/ImageTargets/.
    • JniInterface.java
  • Asset files that  need to be extracted from the hello_ar_c application and added to the ImageTargetsNative/app/src/main/assets directory.
    • models/
      • trigrid.png
    • shaders/
      • plane.frag
      • plane.vert
      • point_cloud.frag
      • point_cloud.vert

Modifications Required

In addition to the changes in Example 1 to support ARCore, the following changes need to be made.

  • Android.mk 
    • Needs to include the new JNI files to make sure that they are compiled.
    • The hello_ar_c app uses GLM, the required header files for glm.h are included in the Android NDK; the include path for header files needs to be added.
      • LOCAL_C_INCLUDES := $(ANDROID_NDK)/sources/third_party/vulkan/src/libs/glm
    • Add -landroid to the application LOCAL_LDLIBS.
  • Application.mk
    • Needs to add support for using C++ STL support. 
      APP_CPPFLAGS := -fno-rtti -std=c++14
APP_STL := c++_static
  • The hello_ar_c sample makes use of the Android Asset manager and changes need to be made to the ImageTargets code and the hello_ar_c to accommodate this.
    • JniInterface.java needs to be changed to remove most of the code, except that used for loading resources; it needs to be initialized with the asset manager from ImageTargets.
    • plane_renderer.cc has to have the asset manager passed into it; this is done by modifying the initRendering call from Java to the native code.
      • AAssetManager is created from Java in the ImageTargets initRendering C++ code, passed to the SampleRenderer initialization, which in turns passes into the plane_renderer.cc code.
      • JNIEnv* is also needed to initialize the plane_renderer and point_cloud_renderer, and their initialization calls are modified.
    • In order to use the Asset manager from the C++ side, we need to modify the LoadPngAssetFromManager call to find the Java code in a different Java package com/vuforia/engine/ImageTargets/JniInterface.
  • SampleAppRenderer.cpp
    • Add member variables for the PlaneRenderer and PointCloudRenderer.
    • As new planes are added, add the code to change the color used to render the planes.
    • Initialize the PlaneRenderer and PointCloudRenderer classes.
    • Add code to get the planes from ARCore and then pass them to the hello_ar_c code to render them.
    • Add code to get the point cloud from ARCore and then pass them to the hello_ar_c code to render them.
    • Pass in the Vuforia projection matrix into the plane point cloud rendering code.
      • Using the ARCore projection matrix places the planes in the wrong location.
    • The rendering of the planes and point cloud must be done after the video background has been rendered.
  • Code to call plane drawing:
void SampleAppRenderer::drawPlanes(const ArSession *arSession, const ArFrame *arFrame,
                                   Vuforia::Matrix44F &
                                   vProjectionMatrix)
{
  ArTrackableList *planeList = nullptr;
  ArTrackableList_create(arSession, & planeList);
  if (planeList == nullptr)
  {
    return;
  }

  ArTrackableType plane_tracked_type = AR_TRACKABLE_PLANE;
  ArSession_getAllTrackables(arSession, plane_tracked_type, planeList);

  int32_t plane_list_size = 0;
  ArTrackableList_getSize(arSession, planeList, & plane_list_size);
  mPlaneCount = plane_list_size;

  ArCamera *arCamera;
  ArFrame_acquireCamera(arSession, arFrame, & arCamera);

  glm::mat4 viewMatrix;
  ArCamera_getViewMatrix(arSession, arCamera, glm::value_ptr(viewMatrix));

  // Note using the ARCore projection matrix, e.g.
  // glm::mat4 projectionMatrix;
  // ArCamera_getProjectionMatrix(arSession, arCamera,
  //                             /*near=*/0.1f, /*far=*/100.f,
  //                             glm::value_ptr(projectionMatrix));
  // is not possible due to the setup of the ArSession to make it work with
  // Vuforia.
  // To get the correct result we must use the Vuforia projection matrix which
  // incorporates the display rotation in its creation.
  glm::mat4 projectionMatrix = glm::make_mat4(vProjectionMatrix.data);
  ArCamera_release(arCamera);

  for (int i = 0; i & lt; plane_list_size; ++i)
  {
    ArTrackable *arTrackable{};
    ArTrackableList_acquireItem(arSession, planeList, i, & arTrackable);
    ArPlane *arPlane = ArAsPlane(arTrackable);
    ArTrackingState outTrackingState;
    ArTrackable_getTrackingState(arSession, arTrackable,
                                 &
                                 outTrackingState);

    ArPlane *subsume_plane;
    ArPlane_acquireSubsumedBy(arSession, arPlane, & subsume_plane);
    if (subsume_plane != nullptr)
    {
      ArTrackable_release(ArAsTrackable(subsume_plane));
      continue;
    }

    if (ArTrackingState::AR_TRACKING_STATE_TRACKING != outTrackingState)
    {
      continue;
    }

    ArTrackingState plane_tracking_state;
    ArTrackable_getTrackingState(arSession, ArAsTrackable(arPlane),
                                 &
                                 plane_tracking_state);
    if (plane_tracking_state == AR_TRACKING_STATE_TRACKING)
    {
      const auto iter = mPlaneColorMap.find(arPlane);
      glm::vec3 color;
      if (iter != mPlaneColorMap.end())
      {
        color = iter - >
        second;

        // If this is an already observed trackable release it so it doesn't
        // leave an additional reference dangling.
        ArTrackable_release(arTrackable);
      }
      else
      {
        // The first plane is always white.
        if (!mFirstPlaneFound)
        {
          mFirstPlaneFound = true;
          color = {255, 255, 255}; // White
        }
        else
        {
          color = GetRandomPlaneColor();
        }
        mPlaneColorMap.insert({arPlane, color});
      }

      // Draw the plane make sure it is drawn correctly and blended
      // with the video background
      glEnable(GL_BLEND);
      glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
      mPlaneRenderer.Draw(projectionMatrix, viewMatrix, *arSession, *arPlane,
                          color);
      glDisable(GL_BLEND);
    }
  }

  ArTrackableList_destroy(planeList);
}
  • Code to call the drawing of the point cloud:
void SampleAppRenderer::drawPointCloud(const ArSession *arSession, const ArFrame *arFrame,
                                       Vuforia::Matrix44F &
                                       vProjectionMatrix)
{
  ArPointCloud *pointCloud{};
  ArStatus status = ArFrame_acquirePointCloud(arSession, arFrame, & pointCloud);
  if (status != AR_SUCCESS)
  {
    LOG(" Unable to retrieve a point cloud status is
        : % d "
        , status);
  }

  ArCamera *arCamera;
  ArFrame_acquireCamera(arSession, arFrame, & arCamera);
  glm::mat4 viewMatrix;
  ArCamera_getViewMatrix(arSession, arCamera, glm::value_ptr(viewMatrix));

  // Note using the ARCore projection matrix, e.g.
  // glm::mat4 projectionMatrix;
  // ArCamera_getProjectionMatrix(arSession, arCamera,
  //   /*near=*/0.1f, /*far=*/100.f,
  //   glm::value_ptr(projectionMatrix));
  // would require to change the session in order to work.
  // We should be setting the display geometry every time the
  // display rotation changes. We use Vuforia projection matrix instead.
  glm::mat4 projectionMatrix = glm::make_mat4(vProjectionMatrix.data);

  ArCamera_release(arCamera);

  mPointCloudRenderer.Draw(projectionMatrix * viewMatrix, arSession, pointCloud);
  ArPointCloud_release(pointCloud);
}

Coordinate Systems

In Vuforia Engine, the world has its own origin independent of the camera and, often, of any target. The world coordinate system is a right-handed coordinate system (similar to OpenGL) with the Y-axis being "Up". The camera and targets all report their poses relative to the world-origin using the world's coordinate system. ARCore follows an analogous choice: poses always describe the transformation from object's local coordinate space to the world coodinate space. The world coordinate space can change potentially every frame.

When drawing planes, use the Vuforia projection matrix as it is recommended not to change the current AR session. The Vuforia projection matrix takes into account the display orientation. To achieve a similar result using ARCore, the projection matrix would need to set the display geometry and rotation on the ARSession. Changing the session could impact the behavior of Vuforia Engine.

Additional details about coordinate systems can be found here: Vuforia, ARKit, ARCore.

 

Example 4 - Classifying Planes Using ARKit

In ARKit on iOS 12 (iPhone XS, iPhone XS Max, and iPhone XR), the phone will classify the type of planes found; more documentation here. Developers can access this functionality using the session and frame pointers. The sample code below displays how to print out various classifications.

Vuforia::FusionProviderPlatformInfo info = Vuforia::getFusionProviderPlatformInfo();

if (info.mType != Vuforia::FUSION_PROVIDER_PLATFORM_TYPE::FUSION_PROVIDER_PLATFORM_ARKIT)
{
  NSLog(@ " Fusion provider platform type does not match the expected type ");
  return;
}

if ((info.mSession == nullptr) ||
    (info.mFrame == nullptr))
{
  NSLog(@ " Fusion provider platform pointers are not set ");
  return;
}

ARSession *arSession = (__bridge ARSession *)info.mSession;

// In ARKit the same ARFrame* can also be extracted from the session
// as shown below.
ARFrame *arFrame = (__bridge ARFrame *)info.mFrame;

if (arSession != nil)
{
  ARFrame *arSessionFrame = arSession.currentFrame;
  if (@available(iOS 12.0, *))
  {
    // Look for planes and determine their type
    if (ARPlaneAnchor.classificationSupported)
    {
      NSArray <
      ARAnchor *>
      *arKitAnchors = arSessionFrame.anchors;
      NSLog(@ " ARPlane
            : planes found = % lu "
            , (unsigned long)[arKitAnchors count]);
      for (ARAnchor *arKitAnchor in arKitAnchors)
      {
        // We are only interested in PlaneAnchors
        if ([arKitAnchor class] != [ARPlaneAnchor class])
        {
          continue;
        }

        // Get the classifaction status
        ARPlaneAnchor *planeAnchor = (ARPlaneAnchor *)arKitAnchor;
        NSLog(@ " ARPlane Classifying plane with uuid
              : % s "
              , planeAnchor.identifier.UUIDString.UTF8String);
        auto classificationStatus = planeAnchor.classificationStatus;
        switch (classificationStatus)
        {
        case ARPlaneClassificationStatusNotAvailable:
          NSLog(@ " ARPlaneClassificationStatusNotAvailable ");
          break;
        case ARPlaneClassificationStatusUndetermined:
          NSLog(@ " ARPlaneClassificationStatusUndetermined ");
          break;
        case ARPlaneClassificationStatusUnknown:
          NSLog(@ " ARPlaneClassificationStatusUnknown ");
          break;
        case ARPlaneClassificationStatusKnown:
          // NSLog(@"ARPlaneClassificationStatusKnown");
          break;
        }

        if (classificationStatus == ARPlaneClassificationStatusKnown)
        {
          switch (planeAnchor.classification)
          {
          case ARPlaneClassificationNone:
            NSLog(@ " ARPlaneClassificationNone ");
            break;
          case ARPlaneClassificationWall:
            NSLog(@ " ARPlaneClassificationWall ");
            break;
          case ARPlaneClassificationFloor:
            NSLog(@ " ARPlaneClassificationFloor ");
            break;
          case ARPlaneClassificationCeiling:
            NSLog(@ " ARPlaneClassificationCeiling ");
            break;
          case ARPlaneClassificationTable:
            NSLog(@ " ARPlaneClassificationTable ");
            break;
          case ARPlaneClassificationSeat:
            NSLog(@ " ARPlaneClassificationSeat ");
            break;
          }
        }
      }
    }
  }
}

Example 5 - Accessing Color Correction Information Using ARCore

NOTE: With Vuforia Engine 8.3 color correction informarmation can be accessed directly using Vuforia, via the Illumination class. The use is described in Illumination Values section on the Ground Plane sample user guide page.

Related Topics

Can this page be better?
Share your feedback on our community forum