3. Function Introduction

3.1 UnityXR

Based on UnityXR Plug-in Framework (https://docs.unity3d.com/Manual/XR.html (opens new window))， Qiyu UnityXR SDK supports the subsystems and APIs as below:

Subsystems	References
XRDisplaySubsystem	https://docs.unity3d.com/Manual/xrsdk-display.html (opens new window)
XRInputSubsystem	https://docs.unity3d.com/Manual/xrsdk-input.html (opens new window)
XRSettings	https://docs.unity3d.com/ScriptReference/XR.XRSettings.html (opens new window)
XRNodeState	https://docs.unity3d.com/ScriptReference/XR.XRNodeState.html (opens new window)

3.2 XRRig

This guide introduces the Unity XR Rig Prefab and its features.

3.2.1 Tracking Origin Mode

Sets the tracking origin type.

• Device: tracks the position and orientation relative to the device’s position.

• Floor: tracks the position and orientation relative to the floor, based on the user’s standing height as specified in Device.

Note: For use of XR Rig in multi-scenes APP, refer to 6.1 XRRig (opens new window) Notes

3.3 QiyuManager

This guide introduces some common features that Qiyu provides for developers. please drag the QiyuManager prefab into the scene, which located in Packages/QIYU XR Plugin/Assets/QiyuManager.prefab directory.

3.3.1 Foveation Level

The higher the Foveation Level, the higher the FPS, but the more the pixel loss, which will affect the clarity of the edge area, please choose the appropriate Level according to the rendering needs. Refer to 3.6.1 Foveation Rendering (opens new window) for more details.

3.3.2 Eye Resolution Scale Factor

A value of 0.7(recommended) will use the default eye texture resolution specified by the XR device. The lower scale value, the lower resolution eye textures, which can improve performance at the expense of a less sharp image. The higher scale value, the higher resolution eye textures, resulting in a potentially sharper image at a cost to performance and increased memory usage. Please not exceed 1.0.

3.3.3 Switch to 3Dof Mode

• HMD 3Dof：Select "MainCamera" under XRRig, and set "Rotation Only" in "Tracking Type".
• Controller 3Dof：Call the API Utils.SetTrackingPosition(false).

Tips: The effect of 3Dof is unnatural, because there are not neck model and arm model in the "Track Pose Driver" from Unity XR. We suggest the developers use QIYU 3Dof APIs, which includes the neck and arm models, to simulate the body movement vividly. Please refer the 4.4 QIYU 3Dof Mode (opens new window).

3.4 Input

This guide introduces the input interaction capabilities supported by the SDK.

3.4.1 UnityXR Input

UnityEngine.XR.InputDevices provide Cross-platform XR controller input. Please refer to the XR Interaction Toolkit package (opens new window) to integrate the function.

Please note: to improve the user experience of controller, we change the deviceRotation (set the 35 degrees as default) from v1.4.0. Please refer to the following figures. If you have already changed the angle for the controllers, please modify your angle back 35 degree after you upgrade the SDK.

3.4.2 QIYU Controller

If you want to use the QIYU Controller models in the scene, please drag The controller model to specify Model Prefab location of XRcontroller.

Notes:

• If you using QIYU Dream to develop your application, please using the controller models at Controller_Dream file in Model Prefab.
• If the QIYU controller Model needs to be used in your application which is also released on QIYU APP Store, the application should filling the right controller model in the Model Prefab.
• If the QIYU controller picture needs to be used in your application, click here (opens new window) to download.

Demo：

QiyuXRController scripts can instead the default XRController to switch the models automatically (Programming for this function by the device types) . The prefabs of controller models should be dragged to the model prefab as the Fig3.4.2.3 shows.

3.4.3 VR Input Keyboard

SDK provides a VR input keyboard, which helps developers to place virtual keyboards in their applications.

1. Drag the VrinputField Prefab into the UI.
2. Click the component to pop up the virtual keyboard.
3. Please refer to the "SGKeyboard.Unity" sample scene.

4. You can adjust the sensitivity parameters if needed. In order to improve the keyboard operation experience, it is recommended that the gamepad button threshold should be 0.5 in Device-based mode.

5. Set the left and right hand input. If you only need to use one, you can set the other value to NULL.

6. (Required)"Active Input Handling" in "ProjectSettings" must be set to "Both".

3.5 Audio

This guide introduces VR audio experiences in Unity.

3.5.1 Unity AudioSource

• Add "AudioSource" component, and set Spatial Audio.

• You can configure third-party audio plug-ins in ProjectSettings. You can download "ResonanceAudio" and import it into the project.

3.5.2 Resonance Audio

Resonance Audio is a multi-platform spatial Audio SDK that provides high-fidelity, powerful spatial Audio technology. It is a great help for real-life VR gaming and video experiences. The download link is https://resonance-audio.github.io/resonance-audio/develop/unity/getting-started.

3.5.3 Third Party Audio Plugin

QIYU devices support multiple third party audio plugins, such as FMOD, Wwise, Photon voice and so on. however, some of default settings are not compatible with QIYI OS, which will cause the recorded video no sound. Please refer to 6.8 General Audio Plugin (opens new window) to modify the settings.

3.6 Advanced Rendering

This guide introduces advanced rendering features that can help improve APP performance.

3.6.1 Foveation Rendering

Foveated rendering can optimize the ability to render scenes by providing high resolution for the center of the field of view, and greatly reducing peripheral views the details of the scene. This reduces the GPU load as a result of the reduction in pixel shading requirements. FR can dramatically increase performance, improving the image show in the headset.

"Foveated Rendering" option as shown in the image below. Choose a Foveation Level other than “None” and then foveated rendering is enabled.

Developers can also set the fixation rendering level and parameters through the APIs provided in 5.1 FR APIs.

Known Issues

• FFR is invalid when Graphics HDR enabled.

3.6.2 Multi View (Single Pass)

With Single Pass Stereo rendering (https://docs.unity3d.com/Manual/SinglePassStereoRendering.html), objects are rendered once to the left eye buffer, then duplicated to the right buffer automatically with appropriate modifications for vertex position and view-dependent variables such as reflection. It primarily reduces CPU usage, and the GPU performance is unchanged largely. If your application is CPU-bound or draw call bound, we strongly recommend using Single Pass Stereo rendering to improve performance. You can select "MultiView" in Project Settings->XR Plug-in Management ->QIYU. MultiView takes effect globally.

3.6.3 Vulkan

Vulkan is a modern graphics API, in many ways the successor to OpenGL ES. It has numerous benefits, both from an API design perspective, as well as possible performance improvements. Any performance benefits that you might derive from Vulkan will only be of rendering CPU performance, not of GPU performance. You can enable "Vulkan" as follow:

1. Open Unity Editor -> Build Settings -> Player Settings -> Settings for Android.
2. Open Other Settings，and disable Auto Graphic API. Add Vulkan in Graphic APIs List.
3. Place Vulkan on top of OpenGLES.

Known Issues

• In the complex scene, the frame rate of Vulkan is lower than OpenGLES, we will fix this issue in the future version.
• If enable the Vulkan and Multiview at the same time, please using the Unity 2020.3.14 or above version, otherwise, the UI will be abnormal.
• If enable the HDR, Vulkan and Multiview at the same time, please using the Unity2021.2.8f1or above version, otherwise, the UI will be abnormal.
• In URP project, if enable the Vulkan and Multiview at the same time, the screen will be black after switching scenes multiple times.
• The Vulkan API does NOT support Foveation Rendering right now.

3.7 QIYU Compositor Layers

Compositor layers let you render certain forms of content such as text, video, or textures with higher quality than traditional eye-buffer rendering can achieve. Because this will increase the system load, using more than 5 overlay or underlay is NOT recommended.

3.7.1 Methods

1. In XR Camera, adding QiyuOverlayManager.

2. In scene, create Empty GameObject and add QiyuOverlay.

3.7.2 Settings

• Overlay Type: Rendering types which include Overlay or Underlay.

  • Overlay: Will cover all the models in the scene.
  • Underlay: The compositor must punch a hole in the eye buffer with an special mask so that underlays are visible.

• Layer ID: the order of rendering, Smaller values render first.

  Note: The scene should follow the rule that nearby objects block distant objects to set Layer ID, so as to ensure the correct depth-of-field blocking relationship. Otherwise, there may be uncomfortable focusing.

• Overlay Shape

  • Quad: Quad Mesh,layers are a flat texture with four vertices. Most commonly used as a panel to display text or information in a scene.
  • Cylinder: Cylinder Mesh, layers are a single texture that is wrapped around the camera in a cylinder. Most commonly used for curved UI interfaces.
  • Square: Square Mesh, layers are a single texture that is wrapped into a sphere and projected to surround the user’s view. Most commonly used for 360/180 video playback.
  • Customize: Customized type, the current Mesh can be used to render.

• Mesh Face: define rendering the front or back of compositor layer.

• Is External Surface: to identify that the layer will be used to pass through textures or video from an Android Surface. Use External Surface Width and External Surface Height to define the size of the output, and the SetAndroidOESTextureID(int oesID) should be used to set the texture ID. Please refer to the Fig 3.7.2.2.

• Override Color Scale: Override any global color settings for the compositor layers.
  • Color Scale: Set color scale input.
  • Color Offset: Set color offset input.

3.7.3 Notes

1. Enable the HDR, the Underlay will be disabled.
2. Is External Surface is not support by Vulkan.

3.7.4 Sample Scene

Referring to 5.2.4 for an example.

3.8 Passthrough

Passthrough provides the possibility to make mixed reality content. By taking the real-time environment of the real world captured by the camera on the VR headset as the background, it presents the effect of combining virtual and real after superimposing and merging the virtual content. Please refer to the Passthrough support by different types of QIYU devices:

Device	Passthrough
QIYU MIX	Supported (binocular color passthrough)
QIYU Dream Pro	Supported (mono passthrough)
QIYU Dream	Not Supported
QIYU 3	Not Supported

3.8.1 How to Use

Click "QiyuManager" in the scene, and check "Passthrough As Background" in the property panel. When loading the scene, Passthrough will be automatically enabled as the background and it will be automatically destroyed when the scene exits.

Note：

• If you need to dynamically enable or disable Passthrough during runtime, you can call the Seethrough API, refer to 4.6 Passthrough (opens new window)。
• The switch of Passthrough is an asynchronous process with certain delay. You can register a callback function to control the display and hide of scene objects through “QiyuXRCore.RegisterSeeThroughCallback”. For detailed usage, refer to SeethroughTest sample scene.
• If your app adds MR mode or MR scenes based on existing features and needs to be released in the app stores of all types of Qiyu devices, please differentiate device types/capabilities. For example, on devices that do not support color passthrough, do not display the MR mode button or hide the entrance to the MR scene, refer to 4.6 Passthrough (opens new window) calling example.
• If your app is a brand-new MR app, to ensure users' experience, we recommend launching it to the devices that support color passthrough. Thus, you don't need to differentiate device types.
• If customize passthrough (Passthrough Window, Surface Projected Passthrough, Styling Passthrough, etc) is needed, please contact with vrsupport@qiyi.com for OpenXR SDK.

3.8.2 Samples

Qiyu SDK provides some MR example scenes for developers to refer to and experience. After importing QIYU UnityXR SDK and then importing QiyuMRSamples.unitypackage, you can see the following scenes. After building and running the APP on Qiyu VR device, you can browse the MR effect:

• MRSolarSystem
• MRBalloon
• MRFarmland

Steps of importing samples:

1. Click the link to download QiyuMRSamples.unitypackage (opens new window)

2. After downloading, import the project: import Qiyu SDK first, then import samples through Menu-Assets-Import Package-Custom Package.

   

3. Set XR Rig-Tracking Origin Mode to Floor.

   

4. Follow the building steps provided in Getting Started (opens new window) to compile and run the build in the Qiyu VR device.

At the same time, we provide a complete Sample Project: MRDinosaur, Click the link to download MRDinosaur project. (opens new window)

3.8.3 Known Issues

1. Passthrough does not support enabling HDR.
2. Passthrough does not support Vulkan.
3. For Dream Pro device, before developing MR applications, you need to manually install two services and reboot the device.Click the link to download the installation package (opens new window)
4. Performance description: due to the increase of rendering layers, after enabling Passthrough background, CPU occupancy will increase by 12%, GPU occupancy will increase by 9%, and memory occupancy will increase by about 100M. Please optimize the complexity of the scene models as much as possible to reduce GPU occupancy and ensure a reasonable frame rate at runtime. Performance data of the application can be viewed, analyzed and optimized through the performance tools provided by Qiyu. Refer to: Performance and Optimization. (opens new window)
5. Due to the particularity of the Passthrough rendering layer for Alpha channel processing, if semi-transparent materials in the scene are not displayed normally, change the shader used in the project to ensure the correct display of Alpha channel.

If the ColorMask in the shader is RGB, please change it to RGBA. For example:

Please change to:

If you want the shader to display the texture's alpha at the end, please make sure the alpha of finalRGBA is correct. For example:

Please change to:

3.9 Menu features

This guide introduces some functions provided in Menu-QIYU which can simplify some developer 's work.

3.9.1 Modify Player Setting

One-click to modify project configuration to the recommended configuration. Refer to Step 8 of Quick Start for details.

3.9.2 Create/Delete Manifest

If you need to add additional content (such as permissions) to the Manifest file, click "Menu-QIYU-Tools-Create Manifest" to generate the Manifest file. Refer to Step 9 (opens new window) of Quick Start for details.

4.0 Quick Preview

4.0.1 QIYU Quick Scene Preview

To improve the efficiency, Scene Quick Preview can help developer to check the effects on the device, build with AssetBundle. If the codes are changed during preview, the application should be reinstalled.

Function Description

• Scene Select: Select the scene need to be built.
• Clean Cache Bundles: Delete the QiyuAssetBundles which is same level with Assets.
• Debug Mode: Enable the Debug.
• Build Scene: Build a scene as AssetBundles. If the apk has been already installed on VR device, just copying the AssetBundles into device. If the apk has not been installed, the apk will be installed into device and run. If the apk is running, the application will be reloaded the new resources.
• Build Scene And Restart: Copy the AssetBundles into device and restart the application.
• Start or restart APP: Start or restart the application.
• Uninstall APP: Uninstall the application.

4.0.2 PC Direct Preview

To improve the porting efficiency and prevent to build the package many time during the developing, PC direct Preview will provide a function running the scenes in Unity Editor, and you can also see the running views on the headset.

• PC configuration:

  • OS: Windows10 and above.

  • Graphics: NVIDIA GeForce GTX 1060 / AMD 5700.

  • CPU: Intel Core i7-6700 / AMD FX 8320 and above.

  • Memory: 8G and above.

• Method:

  1. Use USB cable to connect PC and headset.

2. install the QiyuPreviewAgent.apk to headset, please download at https://static-d.iqiyi.com/lequ/20220620/QiyuPreviewAgent_v1.0.0.rar (opens new window).
3. Pressing the "Play" in Unity Editor, and run the scene.
4. Select QIYU-Direct Previews in QIYU, and open the setting dashboard, you can please refer to the intrduction as below.
5. Pressing the "Connect Qiyu Device" button, connect QIYU device, and the Preview Agent application will be started automatically.
6. The synchronous display of the scene will be shown both in PC and Headset.

• Description for Settings

  1. Connect Type: USB connection is only accepted at the moment.
  2. IP Port: Data forwarding port of computer and all-in-one machine. If it is not necessary, this configuration item does not need to be modified.
  3. Render Frame: The frequency of the computer sending data to the all-in-one machine controls the display frame rate of the all-in-one machine, so the rendering frame rate of the computer is not consistent with the display frame rate of the all-in-one machine.
  4. Render Resolution: Rendering resolution, monocular size of eyebuffer used for engine rendering and all-in-one display. The higher the rendering resolution, the clearer the display effect and the lower the frame rate. Please balance the weight of frame rate and sharpness and choose the appropriate rendering resolution.

• Known Issue:

  1. Some graphics cards ( AMD 5600XT and lower models, Nvidia 1060 and lower models), and some types of CPU (Not higher than intel i7-6700 or same level of AMD CPU) may cause latency.
  2. Using Unity2019 or Unity2020, when turn to ActionBasedController scene, the screen will be stuck, and both headset and controller cannot be tracked.