Find Data breaks when running with mpiexec pvserver
I'm currently trying to find a node where U == max(U)
using the Find Data dialog. Paraview is connected to a pvserver instance running with np=4. The source filter has 3388 cells and 1794 nodes and is a polygonal mesh (comes from a slice of a structured grid).
Once I hit Run Selection Query, the UI freezes and the server emits the following error:
Error: Failed to evaluate Expression 'U == max(U)'. The following exception stack should provide additional developer specific information. This typically implies a malfor
med expression. Verify that the expression is valid.
( 38.351s) [pvserver.0 ] vtkPythonSelector.cxx:76 WARN| vtkPythonSelector (0x88ca890): Could not invoke 'python_selector.execute()'
Traceback (most recent call last):
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/paraview/detail/python_selector.py", line 79, in execute
Error: Failed to evaluate Expression 'U == max(U)'. The following exception stack should provide additional developer specific information. This typically implies a malfor
med expression. Verify that the expression is valid.
( 38.351s) [pvserver.3 ] vtkPythonSelector.cxx:76 WARN| vtkPythonSelector (0x51a89c0): Could not invoke 'python_selector.execute()'
Traceback (most recent call last):
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/paraview/detail/python_selector.py", line 79, in execute
maskArray = calculator.compute(inputs, query, ns=elocals)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/paraview/detail/calculator.py", line 143, in compute
maskArray = calculator.compute(inputs, query, ns=elocals)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/paraview/detail/calculator.py", line 143, in compute
retVal = eval(expression, globals(), mylocals)
File "<string>", line 1, in <module>
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/algorithms.py", line 361, in max
retVal = eval(expression, globals(), mylocals)
File "<string>", line 1, in <module>
Error: Failed to evaluate Expression 'U == max(U)'. The following exception stack should provide additional developer specific information. This typically implies[40/1921]
med expression. Verify that the expression is valid.
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/algorithms.py", line 361, in max
return _global_func(MaxImpl(), array, axis, controller)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/algorithms.py", line 188, in _global_func
return _global_func(MaxImpl(), array, axis, controller)
res = impl.op()(array, axis)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/algorithms.py", line 188, in _global_func
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/internal_algorithms.py", line 373, in max
( 38.351s) [pvserver.2 ] vtkPythonSelector.cxx:76 WARN| vtkPythonSelector (0x41a04c0): Could not invoke 'python_selector.execute()'
res = impl.op()(array, axis)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/internal_algorithms.py", line 373, in max
ans = numpy.max(narray, axis)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2505, in amax
Traceback (most recent call last):
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/paraview/detail/python_selector.py", line 79, in execute
ans = numpy.max(narray, axis)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2505, in amax
maskArray = calculator.compute(inputs, query, ns=elocals)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/paraview/detail/calculator.py", line 143, in compute
retVal = eval(expression, globals(), mylocals)
File "<string>", line 1, in <module>
initial=initial)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 84, in _wrapreduction
return reduction(axis=axis, out=out, **passkwargs)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/numpy/core/_methods.py", line 28, in _amax
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/algorithms.py", line 361, in max
return umr_maximum(a, axis, None, out, keepdims, initial)
initial=initial)ValueError: zero-size array to reduction operation maximum which has no identity
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 84, in _wrapreduction
return reduction(axis=axis, out=out, **passkwargs)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/numpy/core/_methods.py", line 28, in _amax
return umr_maximum(a, axis, None, out, keepdims, initial)
ValueError: zero-size array to reduction operation maximum which has no identity
return _global_func(MaxImpl(), array, axis, controller)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/algorithms.py", line 188, in _global_func
res = impl.op()(array, axis)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/internal_algorithms.py", line 373, in max
ans = numpy.max(narray, axis)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2505, in amax
Error: Failed to evaluate Expression 'U == max(U)'. The following exception stack should provide additional developer specific information. This typically implies a malfor
med expression. Verify that the expression is valid.
( 38.353s) [pvserver.1 ] vtkPythonSelector.cxx:76 WARN| vtkPythonSelector (0x81c39d0): Could not invoke 'python_selector.execute()'
Traceback (most recent call last):
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/paraview/detail/python_selector.py", line 79, in execute
maskArray = calculator.compute(inputs, query, ns=elocals)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/paraview/detail/calculator.py", line 143, in compute
initial=initial)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 84, in _wrapreduction
return reduction(axis=axis, out=out, **passkwargs)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/numpy/core/_methods.py", line 28, in _amax
retVal = eval(expression, globals(), mylocals)
File "<string>", line 1, in <module>
return umr_maximum(a, axis, None, out, keepdims, initial)
ValueError: zero-size array to reduction operation maximum which has no identity
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/algorithms.py", line 361, in max
return _global_func(MaxImpl(), array, axis, controller)
File "/home/shuhao/ParaView-5.7.0-RC4-osmesa-MPI-Linux-Python3.7-64bit/lib/python3.7/site-packages/vtkmodules/numpy_interface/algorithms.py", line 217, in _global_func
comm.Allreduce([res, mpi_type], [res_recv, mpi_type], impl.mpi_op())
File "mpi4py/MPI/Comm.pyx", line 714, in mpi4py.MPI.Comm.Allreduce
mpi4py.MPI.Exception: Message truncated, error stack:
PMPI_Allreduce(450)...............: MPI_Allreduce(sbuf=0x7fb35c8bd300, rbuf=0x4bf1c80, count=1, datatype=MPI_DOUBLE, op=MPI_MAX, comm=MPI_COMM_WORLD) failed
PMPI_Allreduce(436)...............:
MPIR_Allreduce_impl(293)..........:
MPIR_Allreduce_intra_auto(178)....:
MPIR_Allreduce_intra_auto(84).....:
MPIR_Bcast_impl(310)..............:
MPIR_Bcast_intra_auto(223)........:
MPIR_Bcast_intra_binomial(112)....:
MPIDI_CH3U_Receive_data_found(131): Message from rank 0 and ta
The only way to get out of the frozen UI on the client is to either force quit or shutdown the pvserver.
I can confirm this bug doesn't exist when the pvserver is not running under mpiexec and thus using a single core or when the same operation is performed on the builtin paraview instance, hinting at a parallelization bug.