VR / OpenVR / OpenXR improvements
Adapt the ParaView OpenVR plugin code to compile with the new VR module
-
Adapt the OpenVR plugin to make it work with the VR module: in paraview/paraview!5166 (merged)
Improve the generic VR module "VR"
- Create VRInteractorStyle. NB: Could be almost fully generic, the only problem is in
SetInteractor
because actions names are different in OpenVR/OpenXR (OpenXR do not accept actions with slashes inside its name eg. "/actions/vtk/in/StartMovement" must be "startmovement") :- Remove occurences of vr::TrackedDevicePose_t:
-
Create and use vtkVRRenderWindow::GetTrackedDevicePose()
that return a vtkMatrix4x4 (in absolute tracking coordinates) from a vtkEventDataDevice -
Create and use the new generic function VRRenderWindowInteractor::ConvertPoseToWorldCoordinates()
that convert a vtkMatrix4x4 into world coordinates
-
- Remove occurences of vr::TrackedDeviceIndex_t:
-
Create the VRRenderWindow::GetTrackedDeviceModel()
function that returns a vtkVRModel from an EventDataDevice (a lot of changes in VRRenderWindow here, think about just storing model with their associated EventDataDevice) -
Get the VRModel from the RenderWindow by using the new VRRenderWindow::GetTrackedDeviceModel
-
-
Create OpenVRInteractorStyle
with the overidden method => SetInteractor to add actions ("/actions/vtk/in/StartMovement" for OpenVR, "startmovement" for OpenXR) (OR change OpenVR action names to a string without slashes to be generic)
- Remove occurences of vr::TrackedDevicePose_t:
- Create VRRenderWindowInteractor :
-
Create a generic ConvertPoseToWorldCoordinates
(not using vr::TrackedDevicePose_t but a vtkMatrix4x4 directly, like in GenericVRModel::Render) to be used in theVRInteractorStyle
-
Make DoOneEvent virtual pure (or empty) ? -
Create all shared functions : PhysicalToWorldMatrix functions, StartEventLoop, GetPointerDevice, Initialize (when relevant), etc. -
Create OpenVRRenderWindowInteractor
to defineDoOneEvent
andInitialize
-
Implement InternalCreateTimer and InternalDestroyTimer, as noted in the comment, then have the subclasses use the generic implementation.: #19102
-
- VRRenderWindow:
-
Store matrices of tracked devices in a vector member variable (to factorize TrackedDevicePoses
in the subclass) -
Store models in a vector associated with their EventDataDevice (to factorize TrackedDeviceModels
) -
What is the use of the member variable VTKRenderModels
? -
Create the generic ConvertPoseToMatrices
that take a vtkMatrix4x4 pose instead of a vr::TrackedDevicePose -
Remove useless functions such as GetTrackedDeviceModel, GetTrackedDevicePose, ConvertOpenVRPoseToMatrices .. -
Implement UpdateHMDMatrixPose()
in OpenXR
-
- Camera:
-
Compute the actual view angle from the headset projection matrix
-
- Actions:
-
Find the controller_type name for the valve index to update vtk_openvr_actions.json
The controller type for valve index is named "knuckles" (see https://docs.unity3d.com/2017.3/Documentation/Manual/OpenVRControllers.html)
-
Create a vtk_openvr_knuckles.json file for bindings with this controller. See https://github.com/1runeberg/SunRock/blob/master/Config/SteamVRBindings/knuckles.json for an example -
Create a generic binding .json to fallback for all other controllers: #19103 -
Implement a generic way of handling actions (action paths, action updates/processing, etc.): #19103
-
- Code cleaning
- Update documentation :
-
Make sure includes in VR / OpenVR classes are minimal -
Move CameraPose from OverlayInternal to either VRCamera or a new small class. It doesn't depend on overlays so it should not be in that file. And it ended up being used publicly all over the place so Internal doesn't make sense. -
Make sure vtk.module dependances are minimal -
Review of !8470 (merged) (adding OpenXR module) revealed several issues that likely need to be fixed in OpenVR module as well (renderer MakeCamera
method needsVTK_NEW_INSTANCE
attribute, etc...) -
Rework the enum classes vtkEventData...
invtkEventData.h
to be put in a namespace
- CMake:
-
Add a factory mechanism to be able to compile both OpenVR and OpenXR modules and choose the backend at runtime -
Add a VTK_ENABLE_OPENVR_COLLABORATION variable (in OpenVR module) that will enable VTK_ENABLE_VR_COLLABORATION (in the VR module)
-
NB:
- controller models and ray drawing can't be tested with this method as OpenVR do not send the controller model through the Mixed Reality Portal
- To make the Mixed Reality Portal work with OpenGL, the OpenXR runtime must be set to SteamVR in the Developper tab of SteamVR settings window.
Create the "OpenXR" module and merge it in VTK
Basic Integration
Use this working branch that adds the support for OpenXR
-
vtkOpenXRManager tasks -
Copy the vtkOpenXRManager (communication class between VTK and OpenXR), vtkOpenXRUtilities.h and XrExtensions.h -
In SelectExtensions examine whether we should query additional runtime capabilities and store whether they're enabled in OptionalExtensions, (use EnableExtensionIfSupported() with XR_KHR_COMPOSITION_LAYER_DEPTH_EXTENSION_NAME, XR_MSFT_UNBOUNDED_REFERENCE_SPACE_EXTENSION_NAME, XR_MSFT_SPATIAL_ANCHOR_EXTENSION_NAME, XR_EXT_HAND_TRACKING_EXTENSION_NAME). -
Decide if we want (or need) to provide api to get ActionSet instances by name, in which case, we need to store them in a map rather than vector (see CreateActionSet()). UPDATE: Implementation détail of #19116 -
In UpdateActionData, decide if we need to store the action paths currently kept in this->SubactionPaths in the interactor (Paul's comment in the code didn't make it clear if it should be stored there in addition or instead). Note that we currently publicly expose api on the vtkOpenXRManager to get SubActionPaths, Instance, and Session, so that the vtkOpenXRRenderWindowInteractor can use them to interact with the OpenXR runtime. It may be better to move that code from the interactor into the manager, which may be what Paul's comment was about. UPDATE: Implementation détail of #19116 and #19103 -
Figure out if we need flag to invalidate/validate pose at each frame, or just at begin frame (see the GetViewPose method). -
Android/handheld support will require making the form factor an option, rather than hardcoding to XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY as we do currently.
-
-
Make the vtkOpenXRCamera inherit from vtkVRCamera and define function GetTrackingToDCMatrix()
-
Make the vtkOpenXRRenderer inherit from vtkVRRenderer and override MakeCamera()
to return an OpenXRCamera -
Make vtkOpenXRRenderWindow inherit from vtkVRRenderWindow and: -
override MakeInteractor()
to return a vtkOpenXRRenderWindowInteractor -
override CreateFramebuffers()
(look atvtkOpenXRRenderWindow::CreateFramebuffers
+ render mechanism invtkOpenXRRenderWindow::RenderOneEye()
) -
override GetSizeFromAPI()
to get the size of the HMD fromvtkOpenXRManager::GetRecommandedImageRectSize()
-
override GetPoseMatrixWorldFromDevice()
-
Initialization / Render functions -
Create function to convert OpenXRPoseToMatrices -
Once !8504 (merged) is merged, remove methods that can be defined only in vtkVRRenderWindow, such as GetTrackedDeviceModel
,GetPoseMatrixWorldFromDevice
, etc.
-
-
Make vtkOpenXRInteractorStyle inherit from vtkVRInteractorStyle (once !8504 (merged) is merged), or else use only the generic vtkVRInteractorStyle depending on the actions names and the SetInteractor()
problem.-
change interaction entry point methods to use just override
instead ofvirtual
. -
update PrintSelf to print whatever is left specific to this class after re-parenting
-
-
Make vtkOpenXRRenderWindowInteractor inherit from vtkVRRenderWindowInteractor (once !8504 (merged) is merged): -
Initialization -
Override AddAction()
functions -
Override DoOneEvent()
by usingProcessXrEvents()
,PollXrActions()
, and allHandle***Actions()
-
Improve the OpenXR module
-
RenderWindow PhysicalScale seems bugged (ray is not drawn where it should be if physicalScale / Translation is set) -
Reduce creation and destruction of objects in the render loop, specifically local instances of vtkMatrix4x4 and vtkTransform. -
MenuRepresentation does not always follow the camera view (sometimes, when opening the menu it opens behind the head) -
MenuRepresentation does not capture the start position of the controller to swipe in the option. It seems that the starting point is always horizontal -
Create the Overlay : extension XR_EXTX_overlay
(provisional), or with a composition layer directly ? To investiguate #19113
-
-
Create vtkOpenXRModel -
use the extension XR_MSFT_CONTROLLER_MODEL
(To add an extension, see the fileXrExtensions.h
) to get the model in gltf format: #19105 -
Create the OpenXRControlsHelper::InitControlPosition()
: iterate over the node of the gltf format to get the position of each component #18332 -
Create vtkOpenXRRenderWindow::RenderModels()
-
Be sure to check active before rendering each model #19117
-
-
Generify the handling of actions in vtkOpenXRRenderWindowInteractor -
create vtkOpenXRActionData classes (see very WIP classes https://gitlab.kitware.com/paul.lafoix/vtk/-/blob/OpenXRWithActionData/Rendering/OpenXR/vtkOpenXRActionData.h). This will be useful to remove all HandleBooleanActions()
,HandleFlaotActions()
, etc... functions in vtkOpenXRRenderWindowInteractor -
move loading of actions to generic VR module (currently in vtkOpenXRRenderWindowInteractor::Initialize() method) -
also support loading of action sets from json (currently handled in vtkOpenXRRenderWindowInteractor::LoadActions). Storing action sets in the json file will allow to support more than one action set as well. #19116 -
Also, the current LoadActions implementation hardcodes the controller type to "vive_controller". When this section of refactoring is addressed, we should also support other controller types (e.g. "khr_simple", "valve_index_controller", etc).
-
-
18332 OpenXR: Implement controls helper -
Since vtkOpenXRUtilities.h contains only static methods, it does not need to be a class: #19106 -
Once finished will all of the above items, make a pass to remove/turnoff debugging: -
vtkOpenXRManager::Initialize() - remove this->DebugOn()
-
-
Fix volume rendering. Camera matrices look wrong when using vtkOpenGLGPURaycastMapper.
CI
-
Add openxr in CI (both VTK and ParaView) -
Testing is the same as OpenVR, duplicate tests -
Install OpenVR on Linux and enable smoke tests -
Install monado in linux CI and start it in the CI and enable rendering test: tested, not that simple -
Move TestAvatar in the "VR" module: #19107 -
Find a way to test interactions in CI: #19108 -
TestPicking -
TestMenu -
TestControlsHelper
Improve the ParaView XRInterface plugin
-
Rename the old "VRPlugin" to "CAVEInteraction" and rename "OpenVRPlugin" to "XRInterface" -
Remove occurences of OpenVR when possible (missing vtkVROverlay, vtkVRCameraPose though) -
adapt actions.json files to be used with OpenXR and OpenVR -
CMake: Do not depend on VTKRenderingOpenVR but on VTKRenderingVR -
General cleanup -
Add a Developper oriented documentation in a .md -
Cleanup the XRInterfaceRepresentations code -
Remove mineview related code / integrate mineview feature in ParaView (Locations/Skyboxes/View): paraview/paraview#22316 -
Cleanup/integrate cleanly collaboration code: paraview/paraview#22316 -
Avoid using setText in the UI but rely on cleaner UX design -
Handle backward compatibility to actual deprecation mechanism in loadState -
Cleanup in-XR UI to be more usable -
Cleanup Imago support: paraview/paraview#22317 -
Fix and improve Reset All Positions feature -
Fix Crop Planes features -
Fix grounded movement in OpenXR -
Make base station visible in OpenXR: not needed -
Use nice controller model in OpenXR -
Fix warnings with widgets in OpenXR -
Add OpenXR to the ParaView-superbuild and enable it in CI -
Make OpenXR the default: paraview/paraview#22318 -
Deprecate OpenVR support: paraview/paraview#22318
Add support for HoloLens using the OpenXR module
Use this WIP branch that adds support for HoloLens using OpenXR Remoting
This branch allows to stream VTK inside an HoloLens using :
TODO:
-
Split OpenGL/D3D implementation in vtkOpenXRManager: Strategy design pattern (set singleton interface at runtime) -
Split OpenGL/D3D implementation in vtkOpenXRRenderWindow: Introduce vtkOpenXRRemotingRenderWindow subclass to override behavior. -
CMake dependency: Microsoft.Holographic.Remoting.OpenXr nuget package : Implement FindOpenXRRemoting vs add support for Nuget packages references in CMake (already supported for C# project). -
Introduce CMake variable to enable remoting and conditionally include the required changes based on this variable. e.g. add #ifdef VTK_OPENXR_REMOTING in OpenXr.h to prevent remoting/d3d headers to be always included. -
CMake dependency: D3D libraries: d3d11.lib, dxgi.lib (available on all machine? no need to find library?) -
Interaction: Handle (missing) models and actions -
First frame not being rendered by the runtime (Warning macro "Not renderer"). Try to skip. -
Clean + document (+ rename?) vtkWin32OpenGLD3D11RenderWindow
-
Depth support: Blit Hololens depth texture (D3D) into the render window framebuffer depth attachment #19114 -
Hologram stability (XrSpace, infinite near/far plane) -> Depth reprojection #19115 -
Add support for hand tracking mesh using XR_MSFT_hand_tracking_mesh #19105 -
Add vtkWin32OpenGLD3D11RenderWindow
test (blit VTK into D3D window) -
Handle multisampling in vtkWin32OpenGLD3D11RenderWindow
Edited by Lucas Gandel