Issues with Depth Peeling when used in Qt Quick
Our software depends on depth peeling for transparency. In older version of VTK, the software worked just fine. After updating VTK to the newer version (8.1.1) with OpenGL2 back-end, trying to display transparent objects appears to break both VTK and Qt Quick because nothing is rendered correctly, not even other Qt Quick components. The issue doesn't disappear when we disable depth peeling during program execution. The issue also happens in an implementation written from scratch and this program when depth peeling is added to it: https://github.com/nicanor-romero/QtVtk. When trying to fix the problem we established that there might be a problem with framebuffer object binding or even OpenGL context when trying to use depth peeling. The program uses
vtkExternalOpenGLRenderWindow for rendering the scene. The testing was performed on Windows 10 but I was told it also doesn't work on Mac OS. Since VTK doesn't have Qt Quick components, we use a custom implementation based on this example:
The included program has a scene with two transparent objects intersecting with each other, a cone and a sphere. Depth peeling is off for the first 50 frames, then it is turned on for the next 50 frames and then the program it turns it off. In default configuration, the first 50 frames are rendered correctly, but then nothing renders at all, the window is all white, even after depth peeling is turned off at frame 101. To render next frames just keep scaling the window by grabbing an edge. The program will write frame counter in the terminal and whether depth peeling is on or off for the convenience.
There are four boolean flags in customitem.h which can be used to control the program. Setting
m_useInternalFBO to true will enable rendering to an FBO created in OpenGL. Turning it on doesn't change much. The scene is rendered to an FBO and then blitted to the main framebuffer but when the depth peeling is turned on, the screen turns white as before. Changing
m_useBlitFramebuffer to false will disable blitting but the screen will still turn white. When
m_useResetOpenGL is true,
resetOpenGLState() is called on the window right before rendering with VTK. Enabling this somehow fixes the problem of white screen but the scene is completely missing. If you also set
m_useSetFramebuffer to true, the framebuffer will be set right after the reset and the white screen problem somehow comes back.
m_useResetOpenGL is true while other flags are false, you can sometimes see a strange pattern emerging on the screen, as if one of the peels was partially rendered. An example of this is in one of the attached files. Maybe you can make something out of this.
Correct transparency is an vital feature in our software. Default alpha blending doesn't give correct results and depth sorting is too slow to use due to complexity of our geometry. If you could find and solve the issue or at least help us find a way to solve this, we would be grateful. We spent a lot of time trying to solve this but nothing worked. I attach a zip file containing the example program, an image of the scene rendered by the program with depth peeling off and an image with the result of a strange render.