VTK merge requestshttps://gitlab.kitware.com/LucasGandelKitware/vtk/-/merge_requests2021-06-24T04:57:33-04:00https://gitlab.kitware.com/LucasGandelKitware/vtk/-/merge_requests/1WIP: Hololens 2 support using OpenXr Remoting2021-06-24T04:57:33-04:00Lucas GandelWIP: Hololens 2 support using OpenXr RemotingBased on OpenXR WIP branch added in [!7907](https://gitlab.kitware.com/vtk/vtk/-/merge_requests/7907).
### Goal
Add OpenXR Remoting and OpenGL-DirectX interop to the original branch to allow rendering in the Hololens 2.
This work is ba...Based on OpenXR WIP branch added in [!7907](https://gitlab.kitware.com/vtk/vtk/-/merge_requests/7907).
### Goal
Add OpenXR Remoting and OpenGL-DirectX interop to the original branch to allow rendering in the Hololens 2.
This work is based on Microsoft's [Holographic remoting samples](https://github.com/microsoft/MixedReality-HolographicRemoting-Samples).
It provides the "player" client application that should be deployed in the Hololens. This branch implements the server side, similar to the "remote_openXR" sample.
Your GPU must support :
- H.265(HEVC) codec. ([support matrix](https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new))
- [NV_DX_Interop](https://www.khronos.org/registry/OpenGL/extensions/NV/WGL_NV_DX_interop.txt) extension.
### How to try
##### VTK
- Build and Install the [OpenXR SDK](https://github.com/KhronosGroup/OpenXR-SDK).
- Build this branch with CMake option VTK_MODULE_ENABLE_VTK_RenderingOpenXR and set the OpenXR_LIBRARY and OpenXR_INCLUDE_DIRS CMake variable.
###### ! Manual steps required when building (see TODOs to fix):
- Error: Cannot open "openxr_msft_holographic_remoting.h"..
-> Install the following nuget packages in Visual Studio (Right click on RenderingOpenXR project -> Manage Nuget Packages) : Microsoft.Holographic.Remoting.OpenXr / OpenXR.Headers / OpenXR.Loader
- vtkOpenXRManager::EnableRemotingXR() uses filesystem (cpp17) to find the RemotingXR.json file
-> Force cpp17 for the RenderingOpenXR project (Properties -> C / C++ -> Language -> Standard)
- The 4 .json actions file (in VTK/Rendering/OpenXR) must be placed next to the test executable/solution
##### Player
- Build the sample player application from Microsoft's [Holographic remoting samples](https://github.com/microsoft/MixedReality-HolographicRemoting-Samples). _! The commit message indicates the required version of the Microsoft.Holographic.Remoting.OpenXr package._
- Deploy it to the Hololens. Send the IP address of the Hololens as first argument.
##### Test application
- Construct a small VTK scene as usual but using the OpenXR versions of the rendering classes:
```
vtkNew<vtkOpenXRRenderer> renderer;
renderer->SetBackground(0.0, 0.0, 0.0);
vtkNew<vtkOpenXRRenderWindow> renderWindow;
vtkNew<vtkOpenXRRenderWindowInteractor> iren;
vtkNew<vtkOpenXRCamera> cam;
renderWindow->AddRenderer(renderer);
iren->SetRenderWindow(renderWindow);
renderer->SetActiveCamera(cam);
renderWindow->SetHost(argv[1]);
```
- Set the IP address of the Hololens using ``` renderWindow->SetHost("XX.XX.XX.XX") ```
IMPORTANT: If nothing appears in the Hololens and you have both an Intel and NVidia GPU, disabling one should work.
### TODOs
The last commit overrides the original WIP to show what pieces are required to add support for the Hololens 2. The following tasks must be addressed:
##### 1. Architecture / C++
- [ ] 1.1. Clean OpenXR initial branch and refactor to use the same initialization and rendering approach as this branch.
- [ ] 1.2. Abstraction: Split OpenXR-OpenGL (initial branch) and OpenXR-DX-Remoting (this branch) implementations.
- [ ] 1.3. Don't use filesystem to construct the RemotingXR.json file path. (Use vtksys??)
##### 2. CMake
- [ ] 2.1. Add CMake variable to build with Holographic Remoting
- [ ] 2.2. Clean Rendering/OpenGL2/CMakeLists.txt to find DirectX
- [ ] 2.3. Clean Rendering/OpenXR/CmakeLists.txt to handle .json action files correctly (or fix path in vtkOpenXRRenderWindowInteractor::LoadActions())
- [ ] 2.4. Handle finding of the 3 required Nuget packages using CMake. (Nuget package could be downloaded locally and their dir specified in CMake)
##### 3. User experience
- [ ] 3.1. Investigate common interactions for both implementations
- [ ] 3.2. GUI: vtkQWidgetWidget ?
- [ ] 3.3. Hologram stability
##### 4. Rendering
- [ ] 4.1. Hardware picking (requires texture flip ?)
- [ ] 4.2. Blit into OpenXR textures: OpenGL: unneeded temporary framebuffers? Remoting: Improve texture flip approach currently using Frame()Lucas GandelLucas Gandel