Parallel ParaView

Parallel ParaView

One of the main purposes of ParaView is to allow users to create visualizations of large data sets that reside on parallel systems without first collecting the data to a single machine. Transferring the data is often slow and wasteful of disk resources, and the visualization of large data sets can easily overwhelm the processing and especially memory resources of even high-performance workstations. This chapter first describes the concepts behind the parallelism in ParaView. We then discuss in detail the process of starting up ParaView's parallel server components. Lastly we explain explains how a parallel visualization session is initiated from within the user interface. Parallel rendering is an essential part of parallel ParaView. So essential that we've given it its own chapter in this version of the book.

The task of setting up a cluster for Visualization is unfortunately outside of the scope of this book. However there are several online resources that will help to get you started including:

- http://paraview.org/Wiki/Setting_up_a_ParaView_Server,

- http://www.iac.es/sieinvens/siepedia/pmwiki.php?n=HOWTOs.ParaviewInACluster

and

- http://paraview.org/Wiki/images/a/a1/Cluster09_PV_Tut_Setup.pdf

Parallel Structure

ParaView has three main logical components: client, data server, and render server. The client is responsibly for the GUI and is the interface between you the user and ParaView as a whole. The data server reads in files and processes the data through the pipeline. The render server takes the processed data and renders it to present the results to you.

Figure 1. Parallel Architecture

The three logical components can be combined in various different configurations. When ParaView is started, the client is connected to what is called the builtin server; in this case all three components exist within the same process. Alternatively, you can run the server as an independent program and connect a remote client to it, or run the server as a standalone parallel batch program without a GUI. In this case the server process contains both the data and render server components. The server can also be started as two separate programs: one for the data server and one for the render server. The server programs are data-parallel programs that can be run as a set of independent processes running on different CPUs. The processes use MPI to coordinate their activities as each works on different pieces of the data.

Figure 2. Common Configurations of the logical components

Client

The client is responsible for the user interface of the application. ParaView’s general-purpose client was written to make powerful visualization and analysis capabilities available from an easy-to-use interface. The client component is a serial program that controls the server components through the Server Manager API.

Data Server

The data server is primarily constructed from VTK readers, sources, and filters. It is responsible for reading and/or generating data, processing it, and producing geometric models that the render server and client will display. The data server exploits data parallelism by partitioning the data, adding ghost levels around the partitions as needed, and running synchronous parallel filters. Each data server process has an identical VTK pipeline, and each process is told which partition of the data it should load and process. By splitting the data ParaView is able to use the entire aggregate system memory and thus make large data processing possible.

Render Server

The render server is responsible for rendering the geometry. Like the data server, the render server can be run in parallel and has identical visualization pipelines (only the rendering portion of the pipeline) in all of its processes. Having the ability to run the render server separately from the data server allows for an optimal division of labor between computing platforms. Most large computing clusters are primarily used for batch simulations and do not have hardware rendering resources. Since it is not desirable to move large data files to a separate visualization system, the data server can run on the same cluster that ran the original simulation. The render server can be run on a separate visualization cluster that has hardware rendering resources.

It is possible to run the render server with fewer processes than the data server but never more. Visualization clusters typically have fewer nodes than batch simulation clusters, and processed geometry is usually significantly smaller than the original simulation dump. ParaView repartitions the geometric models on the data server before they are sent to the render server.

MPI Availability

Until recently, in order to use ParaView's parallel processing features, one needed to build ParaView from the source code as described in the Appendix. This was because there are many different versions of MPI, the library ParaView’s servers use internally for parallel communication, and for high performance computer users it is extremely important to use the version that is delivered with your networking hardware.

As of ParaView 3.10 however, we have begun to package MPI with our binary releases. If you have a multi-core workstation, you can now simply turn on the Use Multi-Core setting under ParaView's Settings to make use of all of them. This option makes parallel data server mode the default configuration, which can be very effective when you are working on computationally bound intensive processing tasks.

Otherwise, and when you need to run ParaView on an actual distributed memory cluster, you need to start up the various components and establish connections between them as is described in the next section.

Starting the Server(s)

Client / Server Mode

Client / Server Mode refers to a parallel ParaView session in which data server and render server components reside within the same set of processes, and the client is completely separate. The pvserver executable combines the two server components into one process.

You can run pvserver as a serial process on a single machine. If ParaView was compiled with parallel support, you can also run it as an MPI parallel program on a group of machines. Instructions for starting a program with MPI are implementation- and system-dependent, so contact your system administrator for information about starting an application with MPI. With the MPICH implementation of MPI, the command to start the server in parallel usually follows the format shown here.

mpirun –np number_of_processes path_to/pvserver arguments for pvserver

By default, pvserver will start and then wait for the client to connect to it. See the next section for a full description. Briefly, to make the connection, select Connection from the File menu, select (or make and then select) a configuration for the server, and click Connect. Note that you must start the server before the client attempts to connect to it.

If the computer running the server is behind a firewall, it is useful to have the server initiate the connection instead of the client. The -reverse-connection (or -rc) command-line option tells the server to do this. The server must know the name of the machine to which it should connect; this is specified with the --client-host (or -ch) argument. Note that when the connection is reversed, you must start the client and instruct it to wait for a connection before the server attempts to connect to it.

The client-to-server connection is made over TCP, using a default port of 11111. If your firewall puts restrictions on TCP ports, you may want to choose a different port number. In the client dialog, simply choose a port number in the Port entry of the Configure New Server dialog. Meanwhile, give pvserver the same port number by including --server-port (or –sp) in its command-line argument list.

An example command line to start the server and have it initiate the connection to a particular client on a particular port number is given below.

pvserver –rc –ch=magrathea –sp=26623

Render/Server Mode

The render server allows you to designate a separate group of machines (i.e., apart from the data server and the client) to perform rendering. This parallel mode lets you use dedicated rendering machines for parallel rendering rather than relying on the data server machines, which may have limited or no rendering capabilities. In ParaView, the number of machines (N) composing the render server must be no more than the number (M) composing the data server.

Note that although it is true that at some large installations with particularly high performing parallel rendering resources it can be very efficient to run ParaView in Render/Server Mode, we have found in practice that it is almost always the case that the data transfer time between the two servers overwhelms the speedup gained by rendering on the dedicated graphics cluster. For this reason we typically recommend that you combine the data and render servers together as one component and either render in software via Mesa on the data processing cluster or do all of the visualization processing directly on the GPU cluster.

If you still want to break up the data processing and rendering tasks, there are two sets of connections that must be made for ParaView to run in render-server mode. The first connection set is between the client and the first node of each of the data and render servers. The second connection set is between the nodes of the render server and the first N nodes of the data server. Once all of these connections are established, they are bi-directional. The diagram in Figure 5 depicts the connections established when ParaView is running in render server mode. Each double-ended arrow indicates a bi-directional TCP connection between pairs of machines. The dashed lines represent MPI connections between all machines within a server. In all the diagrams in this section, the render server nodes are denoted by RS 0, RS 1, …, RS N. The data server nodes are similarly denoted by DS 0, DS 1, …, DS N, …, DS M.

Figure 5. Connections required in render server mode

The establishment of connections between client and servers can either be forward (from client to servers) or reverse (from servers back to client). Likewise, the connections between render server and data server nodes can be established either from the data server to the render server or from the render server to the data server.

The main reason for reversing the direction of any of the initial connections is that machines behind firewalls are able to initiate connections to machines outside the firewall, but not vice versa. If the data server is behind a firewall, the data server should initiate the connection with the client, and the data server nodes should connect to the render server nodes. If the render server is behind the firewall instead, both servers should initiate connections to the client, but now the render server nodes should initiate the connections with the nodes of the data server.

In the remaining diagrams in this section, each arrow indicates the direction in which the connection is initially established. Double-ended arrows indicate bi-directional connections that have already been established. In the example command lines, optional arguments are enclosed in []’s. The rest of this section will be devoted to discussing the two connections required for running ParaView in render server mode.

Connection 1: Connecting the client and servers

The first connection that must be established is between the client and the first node of both the data and render servers. By default, the client initiates the connection to each server, as shown in Figure 6. In this case, both the data server and the render server must be running before the client attempts to connect to them.

Figure 6. Starting ParaView in render-server mode using standard connections

To establish the connections shown above, do the following. First, from the command line of the machine that will run the data server, enter “pvdataserver” to start it. Next, from the command line of the machine that will run the render server, enter “pvrenderserver” to start the render server. Now, from the machine that will run the client, start the client application, and connect to the running servers, as described in section 1.2 and summarized below.

Start ParaView and select Connect from the File menu to open the Choose Server dialog. Select Add Server to open the Configure New Server dialog. Create a new server connection with a Server Type of Client / Data Server / Render Server. Enter the machine names or IP addresses of the server machines in the Host entries. Select Configure, and then in the Configure Server dialog, choose Manual. Save the server configuration, and Connect to it. At this point ParaView will establish the two connections. This is similar to running ParaView in client/server mode but with the addition of a render server.

The connection between the client and the servers can also be initiated by the servers. As explained above, this is useful when the servers are running on machines behind a firewall. In this case, the client must be waiting for both servers when they start. The diagram indicating the initial connections is shown in Figure 7.

Figure 7. Reversing the connections between the servers and the client.

To establish the connections shown above, start by opening the Configure New Server dialog on the client. Choose Client / Data Server / Render Server (reverse connection) for the Server Type in the Configure New Server dialog. Next, add both --reverse-connection (or -rc) and --client-host (or -ch) to the command lines for the data server and render server. The value of the --client-host parameter is the name or IP address of the machine running the client. You can use the default port numbers for these connections, or you can specify ports in the client dialog by adding the --data-server-port (or –dsp) and --render-server-port (or –rsp) command-line arguments to the data server and render server command lines, respectively. The port numbers for each server must agree with the corresponding Port entries in the dialog, and they must be different from each other.

For the remainder of this chapter, -rc will be used instead of --reverse-connection when the connection between the client and the servers is to be reversed.

Connection 2: Connecting the render and data servers

After the connections are made between the client and the two servers, the servers will establish connections with each other. In parallel runs, this server-to-server connection is a set of connections between all N nodes of the render server and the first N nodes of the data server. By default the data server initiates the connection to the render server, but this can be changed with a configuration file. The format of this file is described below.

The server that initiates the connection must know the name of the machine running the other server and the port number it is using. In parallel runs, each node of the connecting server must know the name of the machine for the corresponding process in the other server to which it should connect. The port numbers are randomly assigned, but they can be assigned in the configuration file as described below.

The default set of connections is illustrated in Figure 8. To establish these connections, you must give the data server the connection information discussed above, which you specify within a configuration file. Use the --machines (or –m) command line argument to tell the data server the name of the configuration file. In practice, the same file should be given to all three ParaView components. This ensures that the client, the render server, and the data server all agree on the network parameters.

Figure 8. Initializing the connection from the data server to the render server.

An example network configuration file, called machines.pvx in this case, is given below.

<?xml version="1.0" ?>
<pvx>
<Process Type="client">>
</Process>
<Process Type="render-server">
<Option Name="render-node-port" Value="1357"/>
<Machine Name="rs_m1"
Environment="DISPLAY=rs_m1:0"/>
<Machine Name="rs_m2"
Environment="DISPLAY=rs_m2:0"/>
</Process>
<Process Host="data-server">
<Machine Name="ds_m1" />
<Machine Name="ds_m2" />
<Machine Name="ds_m3" />
<Machine Name="ds_m4" />
</Process>
</pvx>

Sample command-line arguments that use the configuration file above to initiate the network illustrated in Figure 8 are given below.

mpirun –np 2 pvdataserver –m=machines.pvx

mpirun –np 2 pvrenderserver -m=machines.pvx

paraview –m=machines.pvx

It should be noted that the machine configuration file discussed here is a distinct entity from, and has a different syntax from, the server configuration file discussed at the end of the next section. That file is read only by the client; the file discussed here will be given to the client and both servers.

In the machines file above, the render-node-port entry in the render server’s XML element tells the render server the port number on which it should listen for a connection from the data server, and it tells the data server what port number it should attempt to contact. This entry is optional, and if it does not appear in the file the port number will be chosen automatically. Note that it is not possible to assign port numbers to individual machines within the server; all will be given the same port number or use the automatically chosen one. Note also that each render server machine is given a display environment variable in this file. This is not required to establish the connections, but it is helpful if you need to assign particular X11 display names to the various render server nodes.

The initial connection between the nodes of the two servers is made from the data server to the render server. You can reverse this such that the render server nodes connect to the corresponding nodes of the data server instead as shown in Figure 9.

Figure 9. Reversing the connection between the servers and client and connecting the render server to the data server.

Typically when the server connection is reversed, the direction of the connection between the client and the servers is also reversed (e.g., if the render server is behind a firewall). In this case, the render server must have the machine names and a connection port number to connect to the data server. The same XML file is used for this arrangement as with the standard connection. The only difference in this case is that the render-node-port entry, if it is used, must appear in the data server’s XML element instead of the render server’s element. Example command-line arguments to initiate this type of network are given here.

paraview –m=machines.pvx

mpirun –np M pvdataserver –m=machines.pvx -rc -ch=client

mpirun –np N pvrenderserver –m=machines.pvx -rc -ch=client

Connecting to the Server

Connecting the Client

You establish connections between the independent programs that make up parallel ParaView from within the client’s user interface. The user interface even allows you to spawn the external programs and then automatically connect to them. Once you specify the information that ParaView needs to connect to, or spawn and connect to, the server components, ParaView saves it to make it easy to reuse the same server configuration at a later time. Some visualization centers in fact provide predefined configurations which makes it trivial to connect to a server that is tailored to that center by simply choosing one from a list in ParaView's GUI.

The Choose Server dialog shown below is the starting point for making and using server configurations. The Connect entry on the ParaView client’s File menu brings it up. The dialog shows the servers that you have previously configured. To connect to a server, click its name to select it; then click the Connect button at the bottom of the dialog box. To make changes to settings for a server, select it, and click Edit Server. To remove a server from the list, select it, and click Delete Server.

Figure 1. The Choose Server dialog establishes connections to configured servers.

To configure a new server connection, click the Add Server button to add it to the list. The dialog box shown below will appear. Enter a name in the first text entry box; this is the name that will appear in the Choose Server dialog (shown above).

Figure 2. Configuring a new server.

Next select the type of connection you wish to establish from the Server Type menu. The possibilities are as follows. The “reverse connection” entries mean that the server connects to the client instead of the client connecting to the server. This may be necessary when the server is behind a firewall. Servers are usually run with multiple processes and on a machine other than where the client is running.

- Client / Server: Attach the ParaView client to a server.

- Client / Server (reverse connection): Connect a server to the ParaView client.

- Client / Data Server / Render Server: Attach the ParaView client to separate data and render servers.

- Client / Data Server / Render Server (reverse connection): Attach both a data and a render server to the ParaView client.

In either of the client / server modes, you must specify the name or IP address of the host machine (node 0) for the server. You may also enter a port number to use, or you may use the default (11111). If you are running in client / data server / render server mode, you must specify one host machine for the data server and another for the render server. You will also need two port numbers. The default one for the data server is 11111; the default for the render server is 22221.

When all of these values have been set, click the Configure button at the bottom of the dialog. This will cause the Configure Server dialog box, shown below, to appear. You must first specify the startup type. The options are Command and Manual. Choose Manual to connect to a server that has been started or will be started externally, on the command line for instance, outside of the ParaView user interface. After selecting Manual, click the Save button at the bottom of the dialog.

Figure 3. Configure the server manually. It must be started outside of ParaView.

If you choose the Command option, in the text window labeled Execute an external command to start the server:, you must give the command(s) and any arguments for starting the server. This includes commands to execute a command on a remote machine (e.g., ssh) and to run the server in parallel (e.g., mpirun). You may also specify an amount of time to wait after executing the startup command(s) and before making the connection between the client and the server(s). (See the spin box at the bottom of the dialog.) When you have finished, click the Save button at the bottom of the dialog.

Figure 4. Enter a command that will launch a new server.

Clicking the Save button in the Configure Server dialog will return you to the Choose Server dialog (shown earlier in this section). The server you just configured will now be in the list of servers you may choose. Thereafter, whenever you run ParaView you can connect to any of the servers that you have configured. You can also give the ParaView client the –-server=server_config_name command-line argument to make it automatically connect to any of the servers from the list when it starts.

You can save and/or load server configurations to and/or from a file using the Save Servers and Load Servers buttons, respectively, on the Choose Server dialog. This is how some visualization centers provide system wide server configurations to allow the novice user to simply click his or her choice and connect to an already configured ParaView server. The format of the XML file for saving the server configurations is discussed online at http://paraview.org/Wiki/Server_Configuration.