Update remote rendering infrastructure
The remote rendering classes have been significantly refactored and documented. Here's a detail listing of changes.
View proxy can set the codec type, toggle hardware-acceleration, and adjust a few other encoding parameters at runtime either from the Qt Properties Panel or from python with the PropertyManager
.
New properties introduced in the RenderViewProxy XML definition:
- domain=enum, CodecType:
{-1: Lossless, 0: VP9, 1: AV1, 2: H264, 3: H265}
- domain=bool, UseHardwareAcceleration: via FFmpeg, this still performs the GPU->CPU pixel transfer because FFmpeg needs it.
- domain=enum, Quality:
{1: Low, 50: Medium, 100: High}
This is set up simply so we have something on the client to adjust server-side encode quality. The server-side object accepts any other integer (from 1-10) as well. - domain=bool, ForceKeyFrame: forces the encoder to treat all frames as key-frames.
- domain=bool, DecompressPackets: decode the compressed images from the render server and display them in the client view.
- domain=bool, StreamOutput: stream view's output in a well-recognized container format. the output chunks are instances of
vtkCompressedVideoPacket
- domain=bool, UseAsynchronousEncoding: when true, encoding happens asynchronously in a separate thread.
You may get the view output stream with view.GetViewOutputObservable()
. This observable emits instances of vtkCompressedVideoPacket
for each frame.
Some interesting properties were added to vtkCompressedVideoPacket for flexibility. Since this class sits in async/vtk, we can modify it for our needs. For lossless images:
package.GetMimeType() == ‘image/bmp’
-
package.GetData()
== vtkUnsignedCharArray that has bmp data. -
package.GetWidth()
,package.GetHeight()
Otherwise, with vp9:
package.GetMimeType() == "video/webm; codecs=vp09.00.10.08"
-
package.GetData()
== vtkUnsignedCharArray that has the webm chunk. -
package.GetWidth()
,package.GetHeight()
as usual. package.GetIsKeyFrame()
For the remaining codecs that we cannot save into a container due to lack of code/legal clarity,
- `package.GetMimeType() == "application/octet-stream"
-
package.GetData()
== vtkUnsignedCharArray that has a raw bitstream (.h264/.h265/.av1) -
package.GetWidth()
,package.GetHeight()
as usual. package.GetIsKeyFrame()
Summary of major changes:
- use OpenGL video frames on the render server for capture and on the client for rendering the image to the screen.
- use the new VTK::Streaming module.
- cleanup vtkSMViewProxy.
- separate data members into
vtkSMViewProxyInternals
. - set default codec type to -1 i.e, lossless.
- support output streams with mimetype='image/bmp' and mimetype='video/webm; codecs="vp09.00.10.08"'
- update vtk submodule for VTK::Streaming
There are a couple of things to watch out for:
- I deliberately slowed down the AV1 encoder to experiment with frame throttling on the desktop until we have a web client up and running.
- with VP9+VAAPI hardware acceleration through FFmpeg - lags exactly by 1 frame.
- h.264 and h.265 require the dimensions to be divisible by 2.