Skip to content

Update remote rendering infrastructure

The remote rendering classes have been significantly refactored and documented. Here's a detail listing of changes.

View proxy can set the codec type, toggle hardware-acceleration, and adjust a few other encoding parameters at runtime either from the Qt Properties Panel or from python with the PropertyManager.

New properties introduced in the RenderViewProxy XML definition:

  1. domain=enum, CodecType: {-1: Lossless, 0: VP9, 1: AV1, 2: H264, 3: H265}
  2. domain=bool, UseHardwareAcceleration: via FFmpeg, this still performs the GPU->CPU pixel transfer because FFmpeg needs it.
  3. domain=enum, Quality: {1: Low, 50: Medium, 100: High} This is set up simply so we have something on the client to adjust server-side encode quality. The server-side object accepts any other integer (from 1-10) as well.
  4. domain=bool, ForceKeyFrame: forces the encoder to treat all frames as key-frames.
  5. domain=bool, DecompressPackets: decode the compressed images from the render server and display them in the client view.
  6. domain=bool, StreamOutput: stream view's output in a well-recognized container format. the output chunks are instances of vtkCompressedVideoPacket
  7. domain=bool, UseAsynchronousEncoding: when true, encoding happens asynchronously in a separate thread.

You may get the view output stream with view.GetViewOutputObservable(). This observable emits instances of vtkCompressedVideoPacket for each frame.

Some interesting properties were added to vtkCompressedVideoPacket for flexibility. Since this class sits in async/vtk, we can modify it for our needs. For lossless images:

  • package.GetMimeType() == ‘image/bmp’
  • package.GetData() == vtkUnsignedCharArray that has bmp data.
  • package.GetWidth(), package.GetHeight()

Otherwise, with vp9:

  • package.GetMimeType() == "video/webm; codecs=vp09.00.10.08"
  • package.GetData() == vtkUnsignedCharArray that has the webm chunk.
  • package.GetWidth(), package.GetHeight() as usual.
  • package.GetIsKeyFrame()

For the remaining codecs that we cannot save into a container due to lack of code/legal clarity,

  • `package.GetMimeType() == "application/octet-stream"
  • package.GetData() == vtkUnsignedCharArray that has a raw bitstream (.h264/.h265/.av1)
  • package.GetWidth(), package.GetHeight() as usual.
  • package.GetIsKeyFrame()

Summary of major changes:

  • use OpenGL video frames on the render server for capture and on the client for rendering the image to the screen.
  • use the new VTK::Streaming module.
  • cleanup vtkSMViewProxy.
  • separate data members into vtkSMViewProxyInternals.
  • set default codec type to -1 i.e, lossless.
  • support output streams with mimetype='image/bmp' and mimetype='video/webm; codecs="vp09.00.10.08"'
  • update vtk submodule for VTK::Streaming

There are a couple of things to watch out for:

  1. I deliberately slowed down the AV1 encoder to experiment with frame throttling on the desktop until we have a web client up and running.
  2. with VP9+VAAPI hardware acceleration through FFmpeg - lags exactly by 1 frame.
  3. h.264 and h.265 require the dimensions to be divisible by 2.

Screencast_from_09-19-2022_08_23_44_AM

Edited by Jaswant Panchumarti (Kitware)

Merge request reports