views: get lossless streaming to work with new infrastructure
Changes to RenderViewProxy
The encoder configuration properties are now exposed on the new StillVideoEncoder
and InteractiveVideoEncoder
proxy property of the render view proxy.
With respect to async/paraview!145, here's a change set of properties on the RenderViewProxy
domain=enum, CodecType:{-1: Lossless, 0: VP9, 1: AV1, 2: H264, 3: H265}
domain=bool, UseHardwareAcceleration: via FFmpeg, this still performs the GPU->CPU pixel transfer because FFmpeg needs it.domain=enum, Quality:{1: Low, 50: Medium, 100: High}
This is set up simply so we have something on the client to adjust server-side encode quality. The server-side object accepts any other integer (from 1-10) as well.domain=bool, ForceKeyFrame: forces the encoder to treat all frames as key-frames.domain=bool, UseAsynchronousEncoding: when true, encoding happens asynchronously in a separate thread.- domain=bool, Display: decode the compressed images from the render server and display them in the client view.
- domain=bool, StreamOutput: stream view's output in a well-recognized container format. the output chunks are instances of
vtkCompressedVideoPacket
- [new] domain=proxy_list, StillVideoEncoder: encoder used for still renders. {
JPEGVideoEncoder
,FFmpegSoftwareEncoder
,FFmpegHardwareEncoder
,NVIDIA Encoder
} - [new] domain=proxy_list, InteractiveVideoEncoder: encoder used for still renders. {
JPEGVideoEncoder
,FFmpegSoftwareEncoder
,FFmpegHardwareEncoder
,NVIDIA Encoder
} - [new] domain=bool, LosslessMode: Use lossless png.
StillVideoEncoder
andInteractiveVideoEncoder
are hidden.
Addressed problems:
- Fix the black background issue by using
rgb24
instead ofrgba32
for lossless and jpeg. - Verify viewport dimensions actually match up to the frame dimensions before the frame is overlaid. Otherwise, we end up corrupting display memory which can lead to driver crashes.
- Fix payload
0
height
Performance enhancements and other goodies
- [new] Consolidate view performance metrics and stats into a nice json string.
- [new] Just like the view output was streamed into a view output observable, the json-stringified stats are also streamed in another. You may get it by listening to
GetViewStatsObservable()
- [new] Add
InteractiveRender
on the view proxy. -
vtkPVView()
will use interactive encoder during interactive rendering and still encoder otherwise. - [new] All streaming-related classes are now in a dedicated repository hosted in the async group https://gitlab.kitware.com/async/vtkstreaming
- [new] The Nvidia encoder is always compiled in and has zero NVIDIA/CUDA compile time dependency. The CUDA driver API is lazy-loaded if the driver is installed. It seems to be the most performant encoder right now since it does not fetch the pixels from GPU memory to CPU memory.
- [new]
vtkOpenGLVideoFrame
learned how to capturevtkRenderWindow
pixels in IYUV or NV12. Conversion happens in the fragment shader. As a result, we save CPU cycles on the worker thread. It can directly get to encoding instead of converting rgba -> ycbcr and then encoding. - [new]
VTKStreaming
works with 8-byte aligned image dimensions always. It removed some weird bugs with the software encoders and decoders. As a result, you will see black padding bars on the right and top of the image. - Improved
vtkOpenGLVideoFrame
render IYUV or NV12 into avtkRenderWindow
. Conversion happens in the fragment shader. As a consequence,vtkSMViewProxy
can directly render IYUV/NV12 instead of blocking on the client for converting ycbcr -> rgba.
TODO
- Ponder on whether we need to containerize vp9/h.264/h.265 bitstreams into webm/mp4. It appears as though WebRTC supports native bitstreams with some limitations. I removed the webm muxing for now because it was not correct before. Time stamps were a mess.
The management of properties for the StillVideoEncoder
and InteractiveVideoEncoder
needs to be handled from the property manager interface. It needs more work on vtkPTSPropertyManager
and/or vtkPTSPipelineBuilder
to enable proxy creation (and not just source/view creation) from python like ParaView/master
.
Edited by Jaswant Panchumarti (Kitware)