Skip to content

WIP: Hololens 2 support using OpenXr Remoting

Lucas Gandel requested to merge LucasGandelKitware/vtk:OpenXR-DX-Remoting into master

Based on OpenXR WIP branch added in !7907 (closed).

Goal

Add OpenXR Remoting and OpenGL-DirectX interop to the original branch to allow rendering in the Hololens 2.

This work is based on Microsoft's Holographic remoting samples. It provides the "player" client application that should be deployed in the Hololens. This branch implements the server side, similar to the "remote_openXR" sample.

Your GPU must support :

How to try

VTK
  • Build and Install the OpenXR SDK with CMake option DYNAMIC_LOADER=ON.
  • Build this branch with CMake option VTK_MODULE_ENABLE_VTK_RenderingOpenXR and set the OpenXR_LIBRARY and OpenXR_INCLUDE_DIRS CMake variable.
    ! Manual steps required when building (see TODOs to fix):
  • Error: Cannot open "openxr_msft_holographic_remoting.h".. -> Install the following nuget packages in Visual Studio (Right click on RenderingOpenXR project -> Manage Nuget Packages) : Microsoft.Holographic.Remoting.OpenXr / OpenXR.Headers / OpenXR.Loader
  • vtkOpenXRManager::EnableRemotingXR() uses filesystem (cpp17) to find the RemotingXR.json file -> Force cpp17 for the RenderingOpenXR project (Properties -> C / C++ -> Language -> Standard)
  • The 4 .json actions file (in VTK/Rendering/OpenXR) must be placed next to the test executable/solution
Player
  • Build the sample player application from Microsoft's Holographic remoting samples. ! The commit message indicates the required version of the Microsoft.Holographic.Remoting.OpenXr package.
  • Deploy it to the Hololens. Send the IP address of the Hololens as first argument.
Test application
  • Construct a small VTK scene as usual but using the OpenXR versions of the rendering classes:
  vtkNew<vtkOpenXRRenderer> renderer;
  renderer->SetBackground(0.0, 0.0, 0.0);
  vtkNew<vtkOpenXRRenderWindow> renderWindow;
  vtkNew<vtkOpenXRRenderWindowInteractor> iren;
  vtkNew<vtkOpenXRCamera> cam;
  renderWindow->AddRenderer(renderer);
  iren->SetRenderWindow(renderWindow);
  renderer->SetActiveCamera(cam);

  renderWindow->SetHost(argv[1]);
  • Set the IP address of the Hololens using renderWindow->SetHost("XX.XX.XX.XX")

IMPORTANT: If nothing appears in the Hololens and you have both an Intel and NVidia GPU, disabling one should work.

TODOs

This WIP and the master branch have diverged too much. Best is to start from the current master branch.

1. C++
  • 1.1. Add Holographic Remoting connection logic (refactor vtkOpenXRManager::Initialize)

  • 1.2. Integrate vtkWin32OpenGLD3D11RenderWindow (clean/rename/use FrameBufferFlipYOn())

  • 1.3. Abstraction: Split OpenXR-OpenGL (master) and OpenXR-D3D-Remoting (this branch) implementations:

    - Replace HelperWindow with vtkWin32OpenGLD3D11RenderWindow in vtkOpenXRRenderWindow
    - OpenXR uses DirectX backend -> Replace vtkOpenXRManager OpenGL related enums/structs...
  • 1.4. Don't use filesystem to construct the RemotingXR.json file path. (Use vtksys??)

2. CMake
  • 2.1. Add CMake variable to build with Holographic Remoting
  • 2.2. Clean Rendering/OpenGL2/CMakeLists.txt to find DirectX
  • 2.3. Handle finding of the 3 required Nuget packages using CMake. (Nuget package could be downloaded locally and their dir specified in CMake)
Edited by Lucas Gandel

Merge request reports