About Parallel Rendering

One of ParaView's strengths is its ability to offload the often demanding rendering task. By offload, we mean that ParaView allows you to connect to a remote machine, ideally one that is closer to the data and to high-end rendering hardware, do the rendering on that machine and still interact with the data from a convenient location, such as the desktop or laptop in your office.

Abstracting away the location where rendering takes place opens up many possibilities. First, it opens up the possibility to parallelize the job of rendering to make it possible to render huge data sets at interactive rates. Rendering is done in the parallel Render Server component, which may be part of or separate from the parallel Data Server component. In the next section we describe how parallel rendering works and explain the controls you have over it. Second, huge datasets often require high resolution displays to view the intricate details within while maintaining a high level view to maintain context. In the following section we explain how ParaView can be used to drive tile display walls. Lastly, with the number of displays free to vary, it becomes possible to use ParaView to drive multi-display Virtual Reality systems. That is described in the final section of this chapter.

Parallel Rendering

X Forwarding - Not a good idea

Parallel Rendering implies that many processors have some context to render their pixels into. Even though X11 forwarding might be available, you should not run the client remotely and forward its X calls. ParaView will be far more efficient if you let ParaView directly handle the data transfer between local and remote machines. When doing hardware accelerated rendering in GPUs this implies having X11 or Windows display contexts (either offscreen of onscreen). Otherwise this implies using offscreen Mesa (OSMesa) linked in to ParaView to do the rendering entirely offscreen into software buffers.

Onscreen GPU Accelerated rendering via X11 Connections

One of the most common problems people have with setting up the ParaView server is allowing the server processes to open windows on the graphics card on each process's node. When ParaView needs to do parallel rendering, each process will create a window that it will use to render. This window is necessary because you need the X window before you can create an OpenGL context on the graphics hardware.

There is a way around this. If you are using the Mesa as your OpenGL implementation, then you can also use the supplemental OSMesa library to create an OpenGL context without an X window. However, Mesa is strictly a CPU rendering library so, use the OSMesa solution if and only if your server hardware does not have rendering hardware. If your cluster does not have graphics hardware, then compile ParaView with OSMesa support and use the --use-offscreen-rendering flag when launching the server.

Assuming that your cluster does have graphics hardware, you will need to establish the following three things.

1. Have xdm run on each cluster node at startup. Although xdm is almost always run at startup on workstation installations, it is not as commonplace to be run on cluster nodes. Talk to your system administrators for help in setting this up.

2. Disable all security on the X server. That is, allow any process to open a window on the x server without having to log in. Again, talk to your system administrators for help.

3. Use the -display flag for pvserver to make sure that each process is connecting to the display localhost:0 (or just :0).

To enable the last condition, you would run something like

mpirun -np 4 ./pvserver -display localhost:0

An easy way to test your setup is to use the glxgears program. Unlike pvserver, it will quickly tell you (or, rather, fail to start) if it cannot connect to the local X server.

mpirun -np 4 /usr/X11R6/bin/glxgears -display localhost:0

Offscreen software rendering via OSMesa

When running ParaView in a parallel mode, it may be helpful for the remote rendering processes to do their rendering in offscreen buffers. For example, other windows may be displayed on the node(s) where you are rendering; if these windows cover part of the rendering window, depending on the platform and graphics capabilities they might even be captured as part of the display results from that node. A similar situation could occur if more than one rendering process is assigned to a single machine, and the processes share a display. Also, in some cases the remote rendering nodes are not directly connected to a display and otherwise if your cluster does not have graphics hardware, then compile ParaView with OSMesa support and use the --use-offscreen-rendering flag when launching the server.

The first step to compiling OSMesa support is to make sure that you are compiling with the Mesa 3D Graphics Library. It is difficult to tell an installation of Mesa from any other OpenGL implementation (although the existence of an osmesa.h header and a libOSMesa library is a good clue). If you are not sure, you can always download your own copy from http://mesa3d.org. We recommend using either Mesa version 7.6.1 or 7.9.1.

There are three different ways to use Mesa as ParaView's openGL library:

- You can use it purely as a substitute for GPU enabled onscreen rendering. To do this, set the CMake variable OPENGL_INCLUDE_DIR to point to the Mesa include directory (the one containing the GL subdirectory), and set the OPENGL_gl_LIBRARY and OPENGL_glu_LIBRARY to the libGL and libGLU library files, respectively.

Variable | Value | Description
PARAVIEW_BUILD_QT_GUI | ON |
VTK_USE_COCOA | ON | Mac Only. X11 is not supported.
VTK_OPENGL_HAS_OSMESA | OFF | Disable off screen rendering.
OPENGL_INCLUDE_DIR | <mesa include dir> | Set this to the include directory in MESA.
OPENGL_gl_LIBRARY | libGL | Set this to the libGL.a or libGL.so file in MESA.
OPENGL_glu_LIBRARY | libGLU | Set this to the libGLU.a or libGLU.so file in MESA.

- You can use it as a supplement to onscreen rendering. This mode requires that you have a display (X11 is running). In addition to specifying the GL library (which may be a GPU implementation of the Mesa one above), you must tell ParaView where Mesa's OSMesa library is. Do that by turning the VTK_OPENGL_HAS_OSMESA variable to ON. After you configure again you will see a new CMake variable called OSMESA_LIBRARY. Set this to the libOSMesa library file.

  Variable | Value | Description
  PARAVIEW_BUILD_QT_GUI | ON |
  VTK_USE_COCOA | ON | Mac Only. X11 is not supported.
  VTK_OPENGL_HAS_OSMESA | ON | Turn this to ON to enable software rendering.
  OSMESA_INCLUDE_DIR | <mesa include dir> | Set this to the include directory for MESA.
  OPENGL_INCLUDE_DIR | <mesa include dir> | Set this to the include directory for MESA.
  OPENGL_gl_LIBRARY | libGL | Set this to the libGL.a or libGL.so file.
  OPENGL_glu_LIBRARY | libGLU | Set this to the libGLU.a or libGLU.so file.
  OSMESA_LIBRARY | libOSMesa | Set this to the libOSMesa.a or libOSMesa.so file.

- You can use it for pure offscreen rendering, which is necessary when there is no display. To do this, make sure that the OPENGL_gl_LIBRARY variable is empty and that VTK_USE_X is off. Specify the location of OSMesa and OPENGL_glu_LIBRARY as above and turn on the VTK_USE_OFFSCREEN variable.

Variable | Value | Description
PARAVIEW_BUILD_QT_GUI | OFF | When using offscreen rendering there is no gui
VTK_USE_COCOA | OFF | Mac only.
VTK_OPENGL_HAS_OSMESA | ON | Turn this to ON to enable Off Screen MESA.
OSMESA_INCLUDE_DIR | <mesa include dir> | Set this to the include directory for MESA.
OPENGL_INCLUDE_DIR | <mesa include dir> | Set this to the include directory for MESA.
OPENGL_gl_LIBRARY | <empty> | Set this to empty.
OPENGL_glu_LIBRARY | libGLU | Set this to the libGLU.a or libGLU.so file.
OSMESA_LIBRARY | libOSMesa | Set this to the libOSMesa.a or libOSMesa.so file.

Once again, once you build with OSMesa support, it will not take effect unless you launch the server with the --use-offscreen-rendering flag or alternatively, set the PV_OFFSCREEN environment variable on the server to 1.

Compositing

Given that you are connected to a server which is capable of rendering, you have a choice of whether to do the rendering remotely or to do it locally. ParaView’s server performs all data-processing tasks. This includes generation of a polygonal representation of the full data set and of decimated LOD models. Once the data is generated on the server, it is sometimes better to do the rendering remotely and ship pixels to the client for display, and sometimes better to instead shift geometry to the client and have the client render it locally.

In many cases, the polygonal representation of the data set is much smaller than the original data set. (In an extreme case, a simple outline may be used to represent a very large structured mesh.) In these cases, it may be better to transmit the polygonal representation from the server to the client, and then let the client render it. The client can render the data repeatedly, when the viewpoint is changed for instance, without causing additional network traffic. Only when the data changes will network traffic occur. If the client workstation has high-performance rendering hardware, it can sometimes render even large data sets interactively in this way.

The second option is to have each node of the server render its geometry and send the resulting images to the client for display. There is a penalty per rendered frame for compositing images and sending the image across the network. However, ParaView’s image compositing and delivery is very fast, and there are many options to ensure interactive rendering in this mode. Therefore, although small models may be collected and rendered on the client interactively, ParaView’s distributed rendering can render models of all sizes interactively.

ParaView automatically chooses a rendering strategy to achieve the best rendering performance. You can control the rendering strategy explicitly, forcing rendering to occur entirely on the server or entirely on the client for example, by choosing Settings… from the Edit menu of ParaView. Double click on Render View from the window on the left-hand side of the Settings dialog, and then click on Server. The rendering strategy parameters shown in Figure 10 will now be visible. Here we explain in detail the most important of these controls. For an explanation of all controls, see the Appendix.

Figure 10. Parallel rendering parameters

Remote Render Threshold: This slider determines how large the data set must be in order for parallel rendering with image compositing and delivery to be used (as opposed to collecting the geometry to the client). The value of this slider is measured in megabytes. Only when the entire data set consumes more memory than this value will compositing of images occur. If the check box beside the Remote Render Threshold slider is unmarked, then compositing will not happen; the geometry will always be collected. This is only a reasonable option when you can be sure the data set you are using is very small. In general, it is safer to move the slider to the right than to uncheck the box.

ParaView uses IceT to perform image compositing. IceT is a parallel rendering library that takes multiple images rendered from different portions of the geometry and combines them into a single image. IceT employs several image-compositing algorithms, all of which are designed to work well on distributed memory machines. Examples of two such image-compositing algorithms are depicted in Figure 11 and Figure 12. IceT will automatically choose a compositing algorithm based on the current workload and available computing resources.

Figure 11. Tree compositing on four processes.

Figure 12. Binary swap on four processes.

Interactive Subsample Rate: The time it takes to composite and deliver images is directly proportional to the size of the images. The overhead of parallel rendering can be reduced by simply reducing the size of the images. ParaView has the ability to subsample images before they are composited and inflate them after they have been composited. The Interactive Subsample Rate slider specifies the amount by which images are subsampled. This is measured in pixels, and the subsampling is the same in both the horizontal and vertical directions. Thus a subsample rate of 2 will result in an image that is ¼ the size of the original image. The image is scaled to full size before it is displayed on the user interface, so the higher the subsample rate, the more obviously pixilated the image will be during interaction as demonstrated in Figure 13. When the user is not interacting with the data, no subsampling will be used. If you want subsampling to always be off, unmark the check box beside the Interactive Subsample Rate slider.

[[File:ParaView_UsersGuide_NoSubsampling.jpg | link=]] | [[File:ParaView_UsersGuide_TwoPixelSubsampling.jpg | link=]] | [[File:ParaView_UsersGuide_EightPixelSubsampling.jpg | link=]]
No Subsampling | Subsample Rate: 2 pixels | Subsample Rate: 8 pixels |

Figure 13. The effect of subsampling on image quality

Squirt Compression: When ParaView is run in client/server mode, ParaView uses image compression to optimize the image transfer. The compression uses an encoding algorithm optimized for images called SQUIRT (developed at Sandia National Laboratories).

SQUIRT uses simple run-length encoding for its compression. A run-length image encoder will find sequences of pixels that are all the same color and encode them as a single run length (the count of pixels repeated) and the color value. ParaView represents colors as 24-bit values, but SQUIRT will optionally apply a bit mask to the colors before comparing them. Although information is lost when this mask is applied, the sizes of the run lengths are increased, and the compression gets better. The bit masks used by SQUIRT are carefully chosen to match the color sensitivity of the human visual system. A 19-bit mask employed by SQUIRT greatly improves compression with little or no noticeable image artifacts. Reducing the number of bits further can improve compression even more, but it can lead to more noticeable color-banding artifacts.

The Squirt Compression slider determines the bit mask used during interactive rendering (i.e., rendering that occurs while the user is changing the camera position or otherwise interacting with the data). During still rendering (when the user is not interacting with the data), lossless compression is always used. The check box to the left of the Squirt Compression slider toggles whether the SQUIRT compression algorithm is used at all.

Tile Display Walls

Tiled Display

ParaView’s parallel architecture makes it possible to visualize massive amounts of data interactively. When the data is of sufficient resolution that parallel processing is necessary for interactive display, it is often the case that high-resolution images are needed to inspect the data in adequate detail. If you have a 2D grid of display devices, you can run ParaView in tiled display mode to take advantage of it.

To put ParaView in tiled display mode, give pvserver (or pvrenderserver) the X and Y dimensions of the 2D grid with the --tile-dimensions-x (or -tdx) and --tile-dimensions-y (or tdy) arguments. The X and Y dimensions default to 0, which disables tiled display mode. If you set only one of them to a positive value on the command line, the other will be set to 1. In tiled display mode, there must be at least as many server (in client / server mode) or render server (in client / data server / render server mode) nodes as tiles. The example below will create a 3 x 2 tiled display.

pvserver -tdx=3 -tdy=2

Unless you have a high end display wall, it is likely that each monitor's bezel creates a gap between the images shown in your tiled display. You can compensate for the bezel width by considering them to be like mullions on a window. To do that, specify the size of the gap (in pixels) through the --tile-mullion-x (or -tmx) and --tile-mullion-y (or -tmy) command line arguments.

The IceT library, which ParaView uses for its image compositing, has custom compositing algorithms that work on tiled displays. Although compositing images for large tiled displays is a compute-intensive process, IceT reduces the overall amount of work by employing custom compositing strategies and removing empty tiles from the computation as demonstrated in Figure 14. If the number of nodes is greater than the number of tiles, then the image compositing work will be divided amongst all the processes in the render server. In general, rendering to a tiled display will perform significantly better if there are many more nodes in the cluster than tiles in the display it drives. It also greatly helps if the geometry to be rendered is spatially distributed. Spatially distributed data is broken into contiguous pieces that are contained in small regions of space and are therefore rendered to smaller areas of the screen. IceT takes advantage of this property to reduce the amount of work required to composite the final images. ParaView includes the D3 filter that redistributes data amongst the processors to ensure a spatially distributed geometry and thus improves tiled rendering performance.

Figure 14. Compositing images for 8 processes on a 4-tile display.

Unlike other parallel rendering modes, composited images are not delivered to the client. Instead, image compositing is reserved for generating images on the tiled display, and the desktop renders its own images from a lower resolution version of the geometry to display in the UI. In tiled display mode, ParaView automatically decimates the geometry and sends it to the client to make this happen. However, when the data is very large, even a decimated version of the geometry can overwhelm the client. In this case, ParaView will replace the geometry on the client with a bounding box.

You have several controls at run time over the tiled rendering algorithm that you can tune to maintain interactivity while visualizing very large data on very high resolution tiled displays. These are located on the Tile Display Parameters section of the Render View / Server page of the application settings dialog. These controls are described in the Application Settings section of the Appendix.

CAVE Displays

Introduction

- ParaView has basic support for visualization in CAVE like environments. These environments consists of multiple displays which are configured based on some coordinates in the room. Typically the base coordiante of the room is the reference coordinate of the head-tracker.

- ParaView can operate in a Client-Server fashion. The CAVE modules leverages this by assigning different process on the server to manage different displays. The client acts as a central point of control, sending tracking values to each of the processes on the server.

- ParaView has a VRPN (Virtual Reality Private Network) client plugin which reads tracked information from any VRPN server and relays the value to each process managing the display.

- Displays are configured using the PVX configuration XML file.

Prerequisites

- VRPN needs to be installed on the client side.

- MPI needs to be installed on the server side.

- Make sure to enable BUILD_SHARED_LIB, PARAVIEW_USE_MPI and PARAVIEW_BUILD_PLUGIN_VRPNPlugin when building for CAVE support.

- Consequently VRPN_INCLUDE_DIR and VRPN_LIBRARY should also be set.

Structure of config file

<?xml version="1.0" ?>
<pvx>
<Process Type="server|dataserver|renderserver">
<!--
The only supported Type values are "server", "dataserver" or "renderserver".
This controls which executable this configuration is applicable to.
There can be multiple <Process /> elements in the same pvx file.
------------------------------------------------------
| Executable | Applicable Process Type |
| pvserver | server, dataserver, renderserver |
| pvrenderserver | server, renderserver |
| pvdataserver | server, dataserver |
------------------------------------------------------
-->
<Machine name="hostname"
Environment="DISPLAY=m1:0"
LowerLeft="-1.0 -1.0 -1.0"
LowerRight="1.0 -1.0 -1.0"
UpperRight="1.0 1.0 -1.0">
<!--
There can be multiple <Machine> elements in a <Process> element,
each one identifying the configuration for a process.
All attributes are optional.
name="hostname"
Environment: the environment for the process.
LowerLeft|LowerRight|UpperRight
-->
</Machine>
</Process>
</pvx>

Example Config

- The following example is for a six sided cave with origin at (0,0,0).

<?xml version="1.0" ?>
<pvx>
<Process Type="client" />
<Process Type="server">
<Machine Name="Front"
Environment="DISPLAY=:0"
LowerLeft=" -1 -1 -1"
LowerRight=" 1 -1 -1"
UpperRight=" 1 1 -1" />
<Machine Name="Right"
Environment="DISPLAY=:0"
LowerLeft=" 1 -1 -1"
LowerRight=" 1 -1 1"
UpperRight=" 1 1 1" />
<Machine Name="Left"
Environment="DISPLAY=:0"
LowerLeft=" -1 -1 1"
LowerRight="-1 -1 -1"
UpperRight="-1 1 -1"/>
<Machine Name="Top"
Environment="DISPLAY=:0"
LowerLeft=" -1 1 -1"
LowerRight=" 1 1 -1"
UpperRight=" 1 1 1"/>
<Machine Name="Bottom"
Environment="DISPLAY=:0"
LowerLeft=" -1 -1 1"
LowerRight=" 1 -1 1"
UpperRight=" 1 -1 -1"/>
<Machine Name="Back"
Environment="DISPLAY=:0"
LowerLeft=" 1 -1 1"
LowerRight="-1 -1 1"
UpperRight="-1 1 1"/>
</Process>
</pvx>

- A sample PVX is provided in ParaView/Documentation/cave.pvx. This can be used to play with different display configurations.

Things to keep in mind

- pvx file should be specified as the last command line argument for any of the server processes.

- Typically the pvx file is specified for all the executables whose environment is being changed using the pvx file. In case of data-server/render-server configuration, if you are setting up the environment for the two processes groups, then the pvx file must be passed as a command line option to both the executables: pvdataserver and pvrenderserver.

- When running in parallel the file is read on all nodes, hence must be present on all nodes.

Basic steps for a successful setup

- On a terminal run the server (6 processes each driving one display in the cave. Change the number of process according to the number of displays).

# PV_ICET_WINDOW_BORDERS=1 mpiexec -np 6 ./pvserver /path/to/ConfigFile.pvx

Note: PV_ICET_WINDOW_BORDERS=1 disables the full screen and instead opens up a 400x400 window. Remove this environment variable to work in full screen mode

- Open another terminal to run the client. (note: this client connects to the server which open up 6 windows according to the config given in cave.pvx). Change the stereo-type according to preference (see # ./paraview --help).

# ./paraview --stereo --stereo-type=Anaglyph --server=localhost --vrpn --vrpn-address=Tracker0@localhost

- Connect the client to the server.

- Enable the VRPN Plugin on client.