VR / OpenVR / OpenXR improvements
Adapt the ParaView OpenVR plugin code to compile with the new VR module
- Adapt the OpenVR plugin to make it work with the VR module: in paraview/paraview!5166 (merged)
Improve the generic VR module "VR"
- Create VRInteractorStyle. NB: Could be almost fully generic, the only problem is in
SetInteractorbecause actions names are different in OpenVR/OpenXR (OpenXR do not accept actions with slashes inside its name eg. "/actions/vtk/in/StartMovement" must be "startmovement") :
- Remove occurences of vr::TrackedDevicePose_t:
Create and use
vtkVRRenderWindow::GetTrackedDeviceMatrix()that return a float matrix (in absolute tracking coordinate) from a vtkEventDataDevice
Create and use the new generic function
VRRenderWindowInteractor::ConvertPoseToWorldCoordinates()that convert a float matrix into world coordinates
- Create and use
- Remove occurences of vr::TrackedDeviceIndex_t:
VRRenderWindow::GetTrackedDeviceModel()function that returns a vtkVRModel from an EventDataDevice (a lot of changes in VRRenderWindow here, think about just storing model with their associated EventDataDevice)
Get the VRModel from the RenderWindow by using the new
- Create the
OpenVRInteractorStylewith only one overidden method => SetInteractor to add actions ("/actions/vtk/in/StartMovement" for OpenVR, "startmovement" for OpenXR) OR change OpenVR action names to a string without slashes to be generic
- Remove occurences of vr::TrackedDevicePose_t:
- Create VRRenderWindowInteractor :
Create a generic
ConvertPoseToWorldCoordinates(not using vr::TrackedDevicePose_t but the submatrix in float  directly, like in GenericVRModel::Render) to be used in the
- Make DoOneEvent virtual pure (or empty) ?
- Create all shared functions : PhysicalToWorldMatrix functions, StartEventLoop, GetPointerDevice, Initialize (? can be factorized maybe) ..
- Implement InternalCreateTimer and InternalDestroyTimer, as noted in the comment, then have the subclasses use the generic implementation.
- Create a generic
Store matrices of tracked device in a vector member variable (to factorize
TrackedDevicePosein the subclass)
Store models in a vector associated with their EventDataDevice (to factorize
What is the utility of the member variabe
Create the generic
ConvertPoseToMatricesthat take a float pose instead of a vr::TrackedDevicePose
- Remove useless functions about GetTrackedDeviceModel, GetTrackedDevicePose, ConvertOpenVRPoseToMatrices ..
- Store matrices of tracked device in a vector member variable (to factorize
- Find the controller_type name for the valve index to update vtk_openvr_actions.json
The controller type for valve index is named "knuckles" (see https://docs.unity3d.com/2017.3/Documentation/Manual/OpenVRControllers.html)
- Create a vtk_openvr_knuckles.json file for bindings with this controller. See https://github.com/1runeberg/SunRock/blob/master/Config/SteamVRBindings/knuckles.json for an example
- Create a generic binding .json to fallback for all other controllers
- Implement a generic way of handling actions (action paths, action updates/processing, etc.)
- Code cleaning
- Update documentation :
- vtkVRModel and its virtual pure function to load the model
- Make sure includes in VR / OpenVR classes are minimal
- Make sure vtk.module dependances are minimal
Review of !8470 (adding OpenXR module) revealed several issues that likely need to be fixed in OpenVR module as well (renderer
- Update documentation :
- Add a factory mechanism to be able to compile both OpenVR and OpenXR modules and choose the backend at runtime
- Add a VTK_ENABLE_OPENVR_COLLABORATION variable (in OpenVR module) that will enable VTK_ENABLE_VR_COLLABORATION (in the VR module)
- Create a local test working with ctest using the Windows Mixed Reality Portal to emulate a headset :
- If local test is working, setup a Windows buildbot to work with the gitlab CI:
- Move TestAvatar in the "VR" module
- Create other tests:
NB: controller models and ray drawing can't be tested with this method as OpenVR do not send the controller model through the Mixed Reality Portal
Create the "OpenXR" module and merge it in VTK
- Copy the vtkOpenXRManager (communication class between VTK and OpenXR), vtkOpenXRUtilities.h and XrExtensions.h
- In PrepareRendering, when setting nearZ and farZ on this->RenderResources->DepthInfoViews[eye], we may be able to choose near / far better (e.g. use reversed-Z (near > far) for more uniform Z resolution).
- In SelectExtensions examine whether we should query additional runtime capabilities and store whether they're enabled in OptionalExtensions, (use EnableExtensionIfSupported() with XR_KHR_COMPOSITION_LAYER_DEPTH_EXTENSION_NAME, XR_MSFT_UNBOUNDED_REFERENCE_SPACE_EXTENSION_NAME, XR_MSFT_SPATIAL_ANCHOR_EXTENSION_NAME, XR_EXT_HAND_TRACKING_EXTENSION_NAME).
- Decide if we want (or need) to provide api to get ActionSet instances by name, in which case, we need to store them in a map rather than vector (see CreateActionSet()).
- In UpdateActionData, decide if we need to store the action paths currently kept in this->SubactionPaths in the interactor (Paul's comment in the code didn't make it clear if it should be stored there in addition or instead). Note that we currently publicly expose api on the vtkOpenXRManager to get SubActionPaths, Instance, and Session, so that the vtkOpenXRRenderWindowInteractor can use them to interact with the OpenXR runtime. It may be better to move that code from the interactor into the manager, which may be what Paul's comment was about.
- Figure out if we need flag to invalidate/validate pose at each frame, or just at begin frame (see the GetViewPose method).
- Android/handheld support will require making the form factor an option, rather than hardcoding to XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY as we do currently.
Make the vtkOpenXRCamera inherit from vtkVRCamera and define function
Make the vtkOpenXRRenderer inherit from vtkVRRenderer and override
MakeCamera()to return an OpenXRCamera
Make vtkOpenXRRenderWindow inherit from vtkVRRenderWindow and:
MakeInteractor()to return a vtkOpenXRRenderWindowInteractor
vtkOpenXRRenderWindow::CreateFramebuffers+ render mechanism in
GetSizeFromAPI()to get the size of the HMD from
- Initialization / Render functions
- Create function to convert OpenXRPoseToMatrices
- Instead of using vtkVRRay, create vtkOpenXRModel with gltf representation of the controller using the XR_MSFT_controller_model extension
Make vtkOpenXRInteractorStyle inherit from vtkVRInteractorStyle (once !8504 is merged), or else use only the generic vtkVRInteractorStyle depending on the actions names and the
change interaction entry point methods to use just
- update PrintSelf to print whatever is left specific to this class after re-parenting
- change interaction entry point methods to use just
Make vtkOpenXRRenderWindowInteractor inherit from vtkVRRenderWindowInteractor (once !8504 is merged):
PollXrActions(), and all
Improve the OpenXR module
- RenderWindow PhysicalScale seems bugged (ray is not drawn where it should be if physicalScale / Translation is set)
- MenuRepresentation does not always follow the camera view (sometimes, when opening the menu it opens behind the head)
MenuRepresentation does not capture the start position of the controller to swipe in the option. It seems that the starting point is always horizontal
Create the Overlay : extension
XR_EXTX_overlay(provisional), or with a composition layer directly ? To investiguate
- Create vtkOpenXRModel :
use the extension
XR_MSFT_CONTROLLER_MODEL(To add an extension, see the file
XrExtensions.h) to get the model in gltf format
OpenXRControlsHelper::InitControlPosition(): iterate over the node of the gltf format to get the position of each component
- use the extension
- Create the Overlay : extension
Generify the handling of actions in vtkOpenXRRenderWindowInteractor
create vtkOpenXRActionData classes (see very WIP classes https://gitlab.kitware.com/paul.lafoix/vtk/-/blob/OpenXRWithActionData/Rendering/OpenXR/vtkOpenXRActionData.h). This will be useful to remove all
HandleFlaotActions(), etc... functions in vtkOpenXRRenderWindowInteractor
- move loading of actions to generic VR module (currently in vtkOpenXRRenderWindowInteractor::Initialize() method)
- also support loading of action sets from json (currently handled in vtkOpenXRRenderWindowInteractor::LoadActions). Storing action sets in the json file will allow to support more than one action set as well.
- Also, the current LoadActions implementation hardcodes the controller type to "vive_controller". When this section of refactoring is addressed, we should also support other controller types (e.g. "khr_simple", "valve_index_controller", etc).
- create vtkOpenXRActionData classes (see very WIP classes https://gitlab.kitware.com/paul.lafoix/vtk/-/blob/OpenXRWithActionData/Rendering/OpenXR/vtkOpenXRActionData.h). This will be useful to remove all
- 18332 OpenXR: Implement controls helper
- Since vtkOpenXRUtilities.h contains only static methods, it does not need to be a class.
- Build process : must compile the OpenXR SDK
- Testing is the same as OpenVR, so we can maybe create tests in the generic "VR" module and execute them with both OpenVR / OpenXR (factory mechanism)
Improve the ParaView VR plugin
- Rename the old "VRPlugin" to "CAVEPlugin" and rename "OpenVRPlugin" to "VRPlugin"
- Remove occurences of OpenVR when possible (missing vtkVROverlay, vtkVRCameraPose though)
- adapt actions.json files to be used with OpenXR and OpenVR
Question: Why do we duplicate actions.json and bindings.json files with the OpenVR plugin and the Rendering/OpenVR module ?
- CMake: Do not depend on VTKRenderingOpenVR but on VTKRenderingVR
Add support for HoloLens using the OpenXR module
This branch allows to stream VTK inside an HoloLens using :
We need to:
- Install the nuget packages
Modify vtkOpenXRRenderWindow and vtkOpenXRManager to use the new
vtkWin32OpenGLD3D11RenderWindowand to be able to support both OpenGL and DirectX with minimal changes
- Hologram stability
- Fix CMakeList.txt to find DirectX and the Holographic.Remoting package
- Resolve using flipped quad vs inverting projection (lights issue)