Skip to content

views: get lossless streaming to work with new infrastructure

Changes to RenderViewProxy

The encoder configuration properties are now exposed on the new StillVideoEncoder and InteractiveVideoEncoder proxy property of the render view proxy.

With respect to async/paraview!145, here's a change set of properties on the RenderViewProxy

  1. domain=enum, CodecType: {-1: Lossless, 0: VP9, 1: AV1, 2: H264, 3: H265}
  2. domain=bool, UseHardwareAcceleration: via FFmpeg, this still performs the GPU->CPU pixel transfer because FFmpeg needs it.
  3. domain=enum, Quality: {1: Low, 50: Medium, 100: High} This is set up simply so we have something on the client to adjust server-side encode quality. The server-side object accepts any other integer (from 1-10) as well.
  4. domain=bool, ForceKeyFrame: forces the encoder to treat all frames as key-frames.
  5. domain=bool, UseAsynchronousEncoding: when true, encoding happens asynchronously in a separate thread.
  6. domain=bool, Display: decode the compressed images from the render server and display them in the client view.
  7. domain=bool, StreamOutput: stream view's output in a well-recognized container format. the output chunks are instances of vtkCompressedVideoPacket
  8. [new] domain=proxy_list, StillVideoEncoder: encoder used for still renders. {JPEGVideoEncoder, FFmpegSoftwareEncoder, FFmpegHardwareEncoder, NVIDIA Encoder}
  9. [new] domain=proxy_list, InteractiveVideoEncoder: encoder used for still renders. {JPEGVideoEncoder, FFmpegSoftwareEncoder, FFmpegHardwareEncoder, NVIDIA Encoder}
  10. [new] domain=bool, LosslessMode: Use lossless png. StillVideoEncoder and InteractiveVideoEncoder are hidden.

Addressed problems:

  • Fix the black background issue by using rgb24 instead of rgba32 for lossless and jpeg.
  • Verify viewport dimensions actually match up to the frame dimensions before the frame is overlaid. Otherwise, we end up corrupting display memory which can lead to driver crashes.
  • Fix payload 0 height

Performance enhancements and other goodies

  • [new] Consolidate view performance metrics and stats into a nice json string.
  • [new] Just like the view output was streamed into a view output observable, the json-stringified stats are also streamed in another. You may get it by listening to GetViewStatsObservable()
  • [new] Add InteractiveRender on the view proxy.
  • vtkPVView() will use interactive encoder during interactive rendering and still encoder otherwise.
  • [new] All streaming-related classes are now in a dedicated repository hosted in the async group https://gitlab.kitware.com/async/vtkstreaming
  • [new] The Nvidia encoder is always compiled in and has zero NVIDIA/CUDA compile time dependency. The CUDA driver API is lazy-loaded if the driver is installed. It seems to be the most performant encoder right now since it does not fetch the pixels from GPU memory to CPU memory.
  • [new] vtkOpenGLVideoFrame learned how to capture vtkRenderWindow pixels in IYUV or NV12. Conversion happens in the fragment shader. As a result, we save CPU cycles on the worker thread. It can directly get to encoding instead of converting rgba -> ycbcr and then encoding.
  • [new] VTKStreaming works with 8-byte aligned image dimensions always. It removed some weird bugs with the software encoders and decoders. As a result, you will see black padding bars on the right and top of the image.
  • Improved vtkOpenGLVideoFrame render IYUV or NV12 into a vtkRenderWindow. Conversion happens in the fragment shader. As a consequence, vtkSMViewProxy can directly render IYUV/NV12 instead of blocking on the client for converting ycbcr -> rgba.

TODO

  • Ponder on whether we need to containerize vp9/h.264/h.265 bitstreams into webm/mp4. It appears as though WebRTC supports native bitstreams with some limitations. I removed the webm muxing for now because it was not correct before. Time stamps were a mess.

The management of properties for the StillVideoEncoder and InteractiveVideoEncoder needs to be handled from the property manager interface. It needs more work on vtkPTSPropertyManager and/or vtkPTSPipelineBuilder to enable proxy creation (and not just source/view creation) from python like ParaView/master.

Edited by Jaswant Panchumarti (Kitware)

Merge request reports