Oscilators miniapp DIY deadlock
DIY enters a deadlock when the miniapp is run with more MPI ranks than blocks. I was using this as a test case to be sure the ADIOS adaptors can work under such conditions. Commands to reproduce below, followed by backtrace.
Note that the backtrace below shows that this issue is occurring in the destructor of the Autocorelation class. It is generally bad form to do anything but release resources in a destructor, and I suspect that this is the issue rather than a problem in DIY.
<sensei>
<analysis type="autocorrelation" array="data" association="cell"
window="10" k-max="3" enabled="1" />
</sensei>
mpiexec -np 2 ./bin/oscillator -b 1 -t 1 -f ./config.xml ../sensei/miniapps/oscillators/inputs/sample.osc
(gdb) bt
#0 0x00007fffc60c3b09 in mca_btl_vader_component_progress () at btl_vader_component.c:688
#1 0x00007fffd2678dea in opal_progress () at runtime/opal_progress.c:187
#2 0x00007fffd72862d5 in opal_condition_wait (c=<optimized out>, m=<optimized out>) at ../opal/threads/condition.h:78
#3 ompi_request_default_wait_all (count=2, requests=0x7fffffffba60, statuses=0x0) at request/req_wait.c:281
#4 0x00007fffc4e147a6 in ompi_coll_tuned_allreduce_intra_recursivedoubling (sbuf=<optimized out>, rbuf=0xd25650, count=10,
dtype=0x7fffd750d060 <ompi_mpi_float>, op=0x7fffd751db80 <ompi_mpi_op_sum>, comm=<optimized out>, module=0xc1d610) at coll_tuned_allreduce.c:245
#5 0x00007fffd7296efe in PMPI_Allreduce (sendbuf=0xd25860, recvbuf=0xd25650, count=10, e=0x7fffd750d060 <ompi_mpi_float>,
op=0x7fffd751db80 <ompi_mpi_op_sum>, comm=0x7fffd7514b20 <ompi_mpi_comm_world>) at pallreduce.c:109
#6 0x000000000052cff0 in diy::mpi::Collectives<float, add_vectors<float> >::all_reduce (comm=..., in=std::vector of length 10, capacity 10 = {...},
out=std::vector of length 10, capacity 10 = {...}) at /work/SENSEI/sensei/utils/diy/mpi/collectives.hpp:179
#7 0x000000000052cae6 in diy::mpi::all_reduce<float, add_vectors<float> > (comm=..., in=std::vector of length 10, capacity 10 = {...},
out=std::vector of length 10, capacity 10 = {...}, op=...) at /work/SENSEI/sensei/utils/diy/mpi/collectives.hpp:292
#8 0x000000000052c509 in diy::detail::AllReduceOp<std::vector<float, std::allocator<float> >, add_vectors<float> >::global (this=0x7fffb4000910, comm=...)
at /work/SENSEI/sensei/utils/diy/detail/collectives.hpp:26
#9 0x000000000051d8c0 in diy::Master::Collective::global (this=0x7fffb40009a0, c=...) at /work/SENSEI/sensei/utils/diy/master.hpp:350
#10 0x000000000051ed26 in diy::Master::process_collectives (this=0xc21ec0) at /work/SENSEI/sensei/utils/diy/master.hpp:1004
#11 0x000000000051eab8 in diy::Master::flush (this=0xc21ec0) at /work/SENSEI/sensei/utils/diy/master.hpp:974
#12 0x000000000051db31 in diy::Master::exchange (this=0xc21ec0) at /work/SENSEI/sensei/utils/diy/master.hpp:788
#13 0x0000000000566882 in sensei::Autocorrelation::PrintResults (this=0xc1d580, k_max=3) at /work/SENSEI/sensei/sensei/Autocorrelation.cxx:313
#14 0x00000000005654c6 in sensei::Autocorrelation::~Autocorrelation (this=0xc1d580, __in_chrg=<optimized out>)
at /work/SENSEI/sensei/sensei/Autocorrelation.cxx:210
#15 0x0000000000565524 in sensei::Autocorrelation::~Autocorrelation (this=0xc1d580, __in_chrg=<optimized out>)
at /work/SENSEI/sensei/sensei/Autocorrelation.cxx:212
#16 0x00007fffd8342820 in vtkObjectBase::UnRegisterInternal (this=0xc1d580, check=0) at /work/SENSEI/ParaView/VTK/Common/Core/vtkObjectBase.cxx:240
#17 0x00007fffd83426e6 in vtkObjectBase::UnRegister (this=0xc1d580, o=0x0) at /work/SENSEI/ParaView/VTK/Common/Core/vtkObjectBase.cxx:197
#18 0x00007fffd83796fb in vtkSmartPointerBase::~vtkSmartPointerBase (this=0xc1eb70, __in_chrg=<optimized out>)
at /work/SENSEI/ParaView/VTK/Common/Core/vtkSmartPointerBase.cxx:62
#19 0x0000000000549338 in vtkSmartPointer<sensei::AnalysisAdaptor>::~vtkSmartPointer (this=0xc1eb70, __in_chrg=<optimized out>)
at /work/SENSEI/ParaView/VTK/Common/Core/vtkSmartPointer.h:29
#20 0x000000000054a9e7 in std::_Destroy<vtkSmartPointer<sensei::AnalysisAdaptor> > (__pointer=0xc1eb70) at /usr/include/c++/5.3.1/bits/stl_construct.h:93
#21 0x000000000054a7d9 in std::_Destroy_aux<false>::__destroy<vtkSmartPointer<sensei::AnalysisAdaptor>*> (__first=0xc1eb70, __last=0xc1eb78)
at /usr/include/c++/5.3.1/bits/stl_construct.h:103
#22 0x000000000054a3f2 in std::_Destroy<vtkSmartPointer<sensei::AnalysisAdaptor>*> (__first=0xc1eb70, __last=0xc1eb78)
at /usr/include/c++/5.3.1/bits/stl_construct.h:126
#23 0x0000000000549e3f in std::_Destroy<vtkSmartPointer<sensei::AnalysisAdaptor>*, vtkSmartPointer<sensei::AnalysisAdaptor> > (__first=0xc1eb70,
__last=0xc1eb78) at /usr/include/c++/5.3.1/bits/stl_construct.h:151
#24 0x0000000000549877 in std::vector<vtkSmartPointer<sensei::AnalysisAdaptor>, std::allocator<vtkSmartPointer<sensei::AnalysisAdaptor> > >::~vector (
this=0xc21e90, __in_chrg=<optimized out>) at /usr/include/c++/5.3.1/bits/stl_vector.h:424
#25 0x00000000005493f6 in sensei::ConfigurableAnalysis::InternalsType::~InternalsType (this=0xc21e90, __in_chrg=<optimized out>)
at /work/SENSEI/sensei/sensei/ConfigurableAnalysis.cxx:99
#26 0x0000000000548728 in sensei::ConfigurableAnalysis::~ConfigurableAnalysis (this=0xc21e60, __in_chrg=<optimized out>)
at /work/SENSEI/sensei/sensei/ConfigurableAnalysis.cxx:501
#27 0x0000000000548770 in sensei::ConfigurableAnalysis::~ConfigurableAnalysis (this=0xc21e60, __in_chrg=<optimized out>)
at /work/SENSEI/sensei/sensei/ConfigurableAnalysis.cxx:502
#28 0x00007fffd8342820 in vtkObjectBase::UnRegisterInternal (this=0xc21e60, check=0) at /work/SENSEI/ParaView/VTK/Common/Core/vtkObjectBase.cxx:240
#29 0x00007fffd83426e6 in vtkObjectBase::UnRegister (this=0xc21e60, o=0x0) at /work/SENSEI/ParaView/VTK/Common/Core/vtkObjectBase.cxx:197
#30 0x00007fffd83796fb in vtkSmartPointerBase::~vtkSmartPointerBase (this=0x7fffffffc210, __in_chrg=<optimized out>)
at /work/SENSEI/ParaView/VTK/Common/Core/vtkSmartPointerBase.cxx:62
#31 0x00007fffd8379741 in vtkSmartPointerBase::operator= (this=0x8ceb58 <bridge::GlobalAnalysisAdaptor>, r=0x0)
at /work/SENSEI/ParaView/VTK/Common/Core/vtkSmartPointerBase.cxx:75
#32 0x00000000005313cd in vtkSmartPointer<sensei::ConfigurableAnalysis>::operator= (this=0x8ceb58 <bridge::GlobalAnalysisAdaptor>, r=0x0)
at /work/SENSEI/ParaView/VTK/Common/Core/vtkSmartPointer.h:58
#33 0x00000000005310c4 in bridge::finalize (k_max=3, nblocks=1) at /work/SENSEI/sensei/miniapps/oscillators/bridge.cpp:76
#34 0x00000000004eadeb in main (argc=8, argv=0x7fffffffd518) at /work/SENSEI/sensei/miniapps/oscillators/oscillator.cpp:267