# Adapt the ParaView OpenVR plugin code to compile with the new VR module - [x] Adapt the OpenVR plugin to make it work with the VR module: in https://gitlab.kitware.com/paraview/paraview/-/merge_requests/5166 # Improve the generic VR module "VR" - Create VRInteractorStyle. NB: Could be almost fully generic, the only problem is in `SetInteractor` because actions names are different in OpenVR/OpenXR (OpenXR do not accept actions with slashes inside its name eg. _"/actions/vtk/in/StartMovement"_ must be _"startmovement"_) : * Remove occurences of _vr::TrackedDevicePose_t_: - [x] Create and use `vtkVRRenderWindow::GetTrackedDevicePose()` that return a vtkMatrix4x4 (in absolute tracking coordinates) from a vtkEventDataDevice - [x] Create and use the new generic function `VRRenderWindowInteractor::ConvertPoseToWorldCoordinates()` that convert a vtkMatrix4x4 into world coordinates * Remove occurences of _vr::TrackedDeviceIndex_t_: - [x] Create the `VRRenderWindow::GetTrackedDeviceModel()` function that returns a _vtkVRModel_ from an _EventDataDevice_ (a lot of changes in VRRenderWindow here, think about just storing model with their associated EventDataDevice) - [x] Get the VRModel from the RenderWindow by using the new `VRRenderWindow::GetTrackedDeviceModel` * [x] Create `OpenVRInteractorStyle` with the overidden method => SetInteractor to add actions (_"/actions/vtk/in/StartMovement"_ for OpenVR, _"startmovement"_ for OpenXR) (**OR** change OpenVR action names to a string without slashes to be generic) - Create VRRenderWindowInteractor : * [x] Create a generic `ConvertPoseToWorldCoordinates` (not using vr::TrackedDevicePose_t but a vtkMatrix4x4 directly, like in GenericVRModel::Render) to be used in the `VRInteractorStyle` * [x] Make DoOneEvent virtual pure (or empty) ? * [x] Create all shared functions : PhysicalToWorldMatrix functions, StartEventLoop, GetPointerDevice, Initialize (when relevant), etc. * [x] Create `OpenVRRenderWindowInteractor` to define `DoOneEvent` and `Initialize` * [x] Implement _InternalCreateTimer_ and _InternalDestroyTimer_, as noted in the [comment](https://gitlab.kitware.com/vtk/vtk/-/merge_requests/8504/diffs#note_1048653), then have the subclasses use the generic implementation.: https://gitlab.kitware.com/vtk/vtk/-/issues/19102 - VRRenderWindow: * [x] Store matrices of tracked devices in a vector member variable (to factorize `TrackedDevicePoses` in the subclass) * [x] Store models in a vector associated with their EventDataDevice (to factorize `TrackedDeviceModels`) * [x] What is the use of the member variable `VTKRenderModels`? * [x] Create the generic `ConvertPoseToMatrices` that take a vtkMatrix4x4 pose instead of a vr::TrackedDevicePose * [x] Remove useless functions such as GetTrackedDeviceModel, GetTrackedDevicePose, ConvertOpenVRPoseToMatrices .. * [x] Implement `UpdateHMDMatrixPose()` in OpenXR - Camera: * [x] Compute the actual view angle from the headset projection matrix - Actions: * [x] Find the _controller_type_ name for the valve index to update vtk_openvr_actions.json > The controller type for valve index is named "knuckles" (see https://docs.unity3d.com/2017.3/Documentation/Manual/OpenVRControllers.html) * [x] Create a vtk_openvr_knuckles.json file for bindings with this controller. See https://github.com/1runeberg/SunRock/blob/master/Config/SteamVRBindings/knuckles.json for an example * [x] Create a generic binding .json to fallback for all other controllers: https://gitlab.kitware.com/vtk/vtk/-/issues/19103 * [x] Implement a generic way of handling actions (action paths, action updates/processing, etc.): https://gitlab.kitware.com/vtk/vtk/-/issues/19103 - Code cleaning * Update documentation : - [x] _vtkVRCamera_ - [x] _vtkVRControlsHelper_ : https://gitlab.kitware.com/vtk/vtk/-/issues/19104 - [x] _vtkVRModel_ and its virtual pure function to load the model : https://gitlab.kitware.com/vtk/vtk/-/issues/19104 - [x] _vtkVRRenderWindow_ - [x] _vtkVRInteractorStyle_ - [x] _vtkVRRenderWindowInteractor_ * [x] Make sure includes in VR / OpenVR classes are minimal * [x] Move CameraPose from OverlayInternal to either VRCamera or a new small class. It doesn't depend on overlays so it should not be in that file. And it ended up being used publicly all over the place so Internal doesn't make sense. * [x] Make sure vtk.module dependances are minimal * [x] Review of !8470 (adding OpenXR module) revealed several issues that likely need to be fixed in OpenVR module as well (renderer `MakeCamera` method needs `VTK_NEW_INSTANCE` attribute, etc...) * [x] Rework the enum classes `vtkEventData...` in `vtkEventData.h` to be put in a namespace - CMake: * [x] Add a factory mechanism to be able to compile both OpenVR and OpenXR modules and choose the backend at runtime * [x] Add a VTK_ENABLE_OPENVR_COLLABORATION variable (in OpenVR module) that will enable VTK_ENABLE_VR_COLLABORATION (in the VR module) NB: - controller models and ray drawing can't be tested with this method as OpenVR do not send the controller model through the Mixed Reality Portal - To make the Mixed Reality Portal work with OpenGL, the OpenXR runtime must be set to SteamVR in the Developper tab of SteamVR settings window. # Create the "OpenXR" module and merge it in VTK #### Basic Integration [Use this working branch that adds the support for OpenXR](https://gitlab.kitware.com/vtk/vtk/-/merge_requests/7907) - [x] _vtkOpenXRManager_ tasks * [x] Copy the _vtkOpenXRManager_ (communication class between VTK and OpenXR), _vtkOpenXRUtilities.h_ and _XrExtensions.h_ * [x] In _SelectExtensions_ examine whether we should query additional runtime capabilities and store whether they're enabled in _OptionalExtensions_, (use _EnableExtensionIfSupported()_ with _XR_KHR_COMPOSITION_LAYER_DEPTH_EXTENSION_NAME_, _XR_MSFT_UNBOUNDED_REFERENCE_SPACE_EXTENSION_NAME_, _XR_MSFT_SPATIAL_ANCHOR_EXTENSION_NAME_, _XR_EXT_HAND_TRACKING_EXTENSION_NAME_). * [x] Decide if we want (or need) to provide api to get _ActionSet_ instances by name, in which case, we need to store them in a map rather than vector (see _CreateActionSet()_). UPDATE: Implementation détail of #19116 * [x] In _UpdateActionData_, decide if we need to store the action paths currently kept in _this->SubactionPaths_ in the interactor (Paul's comment in the code didn't make it clear if it should be stored there in addition or instead). Note that we currently publicly expose api on the _vtkOpenXRManager_ to get _SubActionPaths_, _Instance_, and _Session_, so that the _vtkOpenXRRenderWindowInteractor_ can use them to interact with the OpenXR runtime. It may be better to move that code from the interactor into the manager, which may be what Paul's comment was about. UPDATE: Implementation détail of #19116 and #19103 * [x] Figure out if we need flag to invalidate/validate pose at each frame, or just at begin frame (see the _GetViewPose_ method). * [x] Android/handheld support will require making the form factor an option, rather than hardcoding to _XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY_ as we do currently. - [x] Make the _vtkOpenXRCamera_ inherit from _vtkVRCamera_ and define function `GetTrackingToDCMatrix()` - [x] Make the _vtkOpenXRRenderer_ inherit from _vtkVRRenderer_ and override `MakeCamera()` to return an OpenXRCamera - [x] Make _vtkOpenXRRenderWindow_ inherit from _vtkVRRenderWindow_ and: * [x] override `MakeInteractor()` to return a _vtkOpenXRRenderWindowInteractor_ * [x] override `CreateFramebuffers()` (look at `vtkOpenXRRenderWindow::CreateFramebuffers` + render mechanism in `vtkOpenXRRenderWindow::RenderOneEye()`) * [x] override `GetSizeFromAPI()` to get the size of the HMD from `vtkOpenXRManager::GetRecommandedImageRectSize()` * [x] override `GetPoseMatrixWorldFromDevice()` * [x] Initialization / Render functions * [x] Create function to convert OpenXRPoseToMatrices * [x] Once !8504 is merged, remove methods that can be defined only in _vtkVRRenderWindow_, such as `GetTrackedDeviceModel`, `GetPoseMatrixWorldFromDevice`, etc. - [x] Make _vtkOpenXRInteractorStyle_ inherit from _vtkVRInteractorStyle_ (once !8504 is merged), or else use only the generic _vtkVRInteractorStyle_ depending on the actions names and the `SetInteractor()` problem. * [x] change interaction entry point methods to use just `override` instead of `virtual`. * [x] update PrintSelf to print whatever is left specific to this class after re-parenting - [x] Make _vtkOpenXRRenderWindowInteractor_ inherit from _vtkVRRenderWindowInteractor_ (once !8504 is merged): * [x] Initialization * [x] Override `AddAction()` functions * [x] Override `DoOneEvent()` by using `ProcessXrEvents()`, `PollXrActions()`, and all `Handle***Actions()` #### Improve the OpenXR module - [x] RenderWindow PhysicalScale seems bugged (ray is not drawn where it should be if physicalScale / Translation is set) - [x] Reduce creation and destruction of objects in the render loop, specifically local instances of vtkMatrix4x4 and vtkTransform. - [x] MenuRepresentation does not always follow the camera view (sometimes, when opening the menu it opens behind the head) - [x] MenuRepresentation does not capture the start position of the controller to swipe in the option. It seems that the starting point is always horizontal * [x] Create the Overlay : extension `XR_EXTX_overlay` (provisional), or with a composition layer directly ? To investiguate #19113 - [x] Create _vtkOpenXRModel_ * [x] use the extension `XR_MSFT_CONTROLLER_MODEL` (To add an extension, see the file `XrExtensions.h`) to get the model in _gltf_ format: https://gitlab.kitware.com/vtk/vtk/-/issues/19105 * [x] Create the `OpenXRControlsHelper::InitControlPosition()`: iterate over the node of the gltf format to get the position of each component #18332 * [x] Create `vtkOpenXRRenderWindow::RenderModels()` * [x] Be sure to check active before rendering each model #19117 - [x] Generify the handling of actions in _vtkOpenXRRenderWindowInteractor_ * [x] create _vtkOpenXRActionData_ classes (see very WIP classes https://gitlab.kitware.com/paul.lafoix/vtk/-/blob/OpenXRWithActionData/Rendering/OpenXR/vtkOpenXRActionData.h). This will be useful to remove all `HandleBooleanActions()`, `HandleFlaotActions()`, etc... functions in _vtkOpenXRRenderWindowInteractor_ * [x] move loading of actions to generic VR module (currently in _vtkOpenXRRenderWindowInteractor::Initialize()_ method) * [x] also support loading of action sets from json (currently handled in _vtkOpenXRRenderWindowInteractor::LoadActions_). Storing action sets in the json file will allow to support more than one action set as well. #19116 * [x] Also, the current _LoadActions_ implementation hardcodes the controller type to "_vive_controller_". When this section of refactoring is addressed, we should also support other controller types (e.g. "_khr_simple_", "_valve_index_controller_", etc). - [x] [18332](https://gitlab.kitware.com/vtk/vtk/-/issues/18332) OpenXR: Implement controls helper - [x] Since _vtkOpenXRUtilities.h_ contains only static methods, it does not need to be a class: https://gitlab.kitware.com/vtk/vtk/-/issues/19106 - [x] Once finished will all of the above items, make a pass to remove/turnoff debugging: * [x] _vtkOpenXRManager::Initialize()_ - remove _this->DebugOn()_ - [x] Fix volume rendering. Camera matrices look wrong when using vtkOpenGLGPURaycastMapper. #### CI * [X] Add openxr in CI (both VTK and ParaView) * [X] Testing is the same as OpenVR, duplicate tests * [X] Install OpenVR on Linux and enable smoke tests * [x] Install [monado](https://monado.dev/) in linux CI and start it in the CI and enable rendering test: tested, not that simple * [x] Move TestAvatar in the "VR" module: https://gitlab.kitware.com/vtk/vtk/-/issues/19107 * [x] Find a way to test interactions in CI: https://gitlab.kitware.com/vtk/vtk/-/issues/19108 * [x] TestPicking * [x] TestMenu * [x] TestControlsHelper # Improve the ParaView XRInterface plugin - [x] Rename the old "VRPlugin" to "CAVEInteraction" and rename "OpenVRPlugin" to "XRInterface" - [x] Remove occurences of OpenVR when possible (missing _vtkVROverlay_, _vtkVRCameraPose_ though) - [x] adapt actions.json files to be used with OpenXR and OpenVR - [x] CMake: Do not depend on _VTKRenderingOpenVR_ but on _VTKRenderingVR_ - [x] General cleanup - [x] Add a Developper oriented documentation in a .md - [ ] Cleanup the XRInterfaceRepresentations code - [x] Remove mineview related code / integrate mineview feature in ParaView (Locations/Skyboxes/View): https://gitlab.kitware.com/paraview/paraview/-/issues/22316 - [x] Cleanup/integrate cleanly collaboration code: https://gitlab.kitware.com/paraview/paraview/-/issues/22316 - [x] Avoid using setText in the UI but rely on cleaner UX design - [ ] Handle backward compatibility to actual deprecation mechanism in loadState - [x] Cleanup in-XR UI to be more usable - [x] Cleanup Imago support: https://gitlab.kitware.com/paraview/paraview/-/issues/22317 - [x] Fix and improve Reset All Positions feature - [x] Fix Crop Planes features - [x] Fix grounded movement in OpenXR - [x] Make base station visible in OpenXR: not needed - [x] Use nice controller model in OpenXR - [x] Fix warnings with widgets in OpenXR - [x] Add OpenXR to the ParaView-superbuild and enable it in CI - [x] Make OpenXR the default: https://gitlab.kitware.com/paraview/paraview/-/issues/22318 - [x] Deprecate OpenVR support: https://gitlab.kitware.com/paraview/paraview/-/issues/22318 # Add support for HoloLens using the OpenXR module [Use this WIP branch that adds support for HoloLens using OpenXR Remoting](https://gitlab.kitware.com/LucasGandelKitware/vtk/-/tree/prepare-holographic-remoting) This branch allows to stream VTK inside an HoloLens using : * [OpenXR Remoting extension](https://docs.microsoft.com/en-us/windows/mixed-reality/develop/platform-capabilities-and-apis/holographic-remoting-create-remote-openxr) * [OpenGL/DirectX interop](https://www.khronos.org/registry/OpenGL/extensions/NV/WGL_NV_DX_interop.txt) TODO: * [x] Split OpenGL/D3D implementation in vtkOpenXRManager: Strategy design pattern (set singleton interface at runtime) * [x] Split OpenGL/D3D implementation in vtkOpenXRRenderWindow: Introduce vtkOpenXRRemotingRenderWindow subclass to override behavior. * [x] CMake dependency: [Microsoft.Holographic.Remoting.OpenXr nuget package](https://docs.microsoft.com/en-us/windows/mixed-reality/develop/platform-capabilities-and-apis/holographic-remoting-create-remote-openxr#get-the-holographic-remoting-nuget-package) : Implement [FindOpenXRRemoting](https://gitlab.kitware.com/alexy.pellegrini/vtk/-/commit/ee7c166cbd8ddf477bb31c885eff0d1e52ee1ce4) vs add support for Nuget packages references in CMake ([already supported](https://gitlab.kitware.com/cmake/cmake/-/merge_requests/3389) for C# project). * [x] Introduce CMake variable to enable remoting and conditionally include the required changes based on this variable. e.g. add #ifdef VTK_OPENXR_REMOTING in OpenXr.h to prevent remoting/d3d headers to be always included. * [x] CMake dependency: D3D libraries: d3d11.lib, dxgi.lib (available on all machine? no need to find library?) * [x] Interaction: Handle (missing) models and actions * [x] First frame not being rendered by the runtime (Warning macro "Not renderer"). Try to skip. * [x] Clean + document (+ rename?) `vtkWin32OpenGLD3D11RenderWindow` * [x] Depth support: Blit Hololens depth texture (D3D) into the render window framebuffer depth attachment #19114 * [x] Hologram stability (XrSpace, infinite near/far plane) -> Depth reprojection #19115 * [x] Add support for hand tracking mesh using [XR_MSFT_hand_tracking_mesh](https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#XR_MSFT_hand_tracking_mesh) #19105 * [x] Add `vtkWin32OpenGLD3D11RenderWindow` test (blit VTK into D3D window) * [x] Handle multisampling in vtkWin32OpenGLD3D11RenderWindow