Curiously this doesn't happen if the file has only one time step
Hello! I noticed that loading a state that involves an ADIOS2VTXReader fails to load the Data arrays of the file.
UPDATE: this only happens for a VTX file with multiple time steps for which the mesh was saved only once [as opposed to at every time step] AND if the saved state does not load the initial time step. Furthermore, if you do load a file with only one mesh for multiple time steps at a time step for which there is no mesh but then go to a time step at which there is a mesh, Paraview finds the Data arrays and everything starts working fine.
Step to reproduce:
Using Paraview latest compiled from source.
I am using ADIOS2 to save some time-dependent simulation results in VTX format and I noticed that the mesh is saved multiple times even though it does not change. Does VTX support reusing a mesh saved at a previous time step?
Fides recently added support for this functionality [see fides#15 ] so I was wondering if it would be possible to have it in VTX too.
I am not entirely sure whether this is a VTK issue, an ADIOS2 issue or something else entirely so apologies if I'm asking in the wrong place
Ok, I looked into the code and I found out what the problem was. And then I fixed it :D We can close this issue as the feature I was requesting is already implemented.
@williamfgc thanks a lot for the help, I am making progress. I tried adding a vtk.xml
file with which I exclude "vtkOriginalPointIds"
and Paraview reads the file as expected, so this is very good news.
I then compiled Paraview and ADIOS2 in debug mode and stepped through it. The problematic variable is, in fact, "vtkOriginalPointIds"
: when Paraview processes that variable it gets to ParaView/VTK/IO/ADIOS2/VTX/schema/VTXSchema.txx:95
which is
// linearized vector
if (variable.Count().size() == 2) // <-- here be dragons!
{
components = variable.Count().back();
}
and then it encounters an ADIOS2 exception at ADIOS2/source/adios2/core/Variable.tcc:46
, i.e.
auto MVI = m_Engine->MinBlocksInfo(*this, m_StepsStart);
if (MVI)
{
if (m_BlockID >= MVI->BlocksInfo.size()) // <-- here be dragons!
{
helper::Throw<std::invalid_argument>(
"Core", "Variable", "DoCount",
"blockID " + std::to_string(m_BlockID) +
" from SetBlockSelection is out of bounds for "
"available "
"blocks size " +
std::to_string(MVI->BlocksInfo.size()) + " for variable " + m_Name +
" for step " + std::to_string(m_StepsStart) +
", in call to Variable<T>::Count()");
}
where m_BlockID
is 7
and MVI->BlocksInfo.size()
is 0
.
This seems to indicate that the file is ill-formed somehow.
Curiously enough, the variable "vtkGhostType"
, which is also present in single copy, doesn't cause any issues to Paraview.
I understand this "vtkOriginalPointIds"
variable is created by the library I'm using so I will ask them what's up with it.
This is supposed to be just one function, or at least that was my intention. Are you referring to "TIME"
,"vtkOriginalPointIds"
etcetera as additional variables?
@williamfgc I tried: I created a file with two time steps and only one mesh; Paraview can load the first time step fine but crashes when moving to the second time step. I also created a regular file with two meshes; Paraview can read it fine. The way I see it, this can either be a bug in Paraview or the file I created is ill-formed. I attach the two files for reference. uL.oneMesh.bp.zip and uL.twoMeshes.bp.zip
The output of bpls
for the two files is
$ bpls -al uL.oneMesh.bp/
uint32_t NumberOfEntities {8} = 0 / 13
uint32_t NumberOfNodes {8} = 0 / 12
int64_t connectivity [8]*{__, 5} = -9223372036854775808 / 9223372036854775807
double geometry [8]*{__, 3} = -1.79769e+308 / 1.79769e+308
double step 2*scalar = 0 / 1
uint32_t types scalar = 71
double u 2*[8]*{__, 3} = -1.79769e+308 / 1.79769e+308
string vtk.xml attr =
<VTKFile type="UnstructuredGrid" version="0.1">
<UnstructuredGrid>
<Piece NumberOfPoints="NumberOfNodes" NumberOfCells="NumberOfCells">
<Points>
<DataArray Name="geometry" />
</Points>
<Cells>
<DataArray Name="connectivity" />
<DataArray Name="types" />
</Cells>
<PointData>
<DataArray Name="TIME">step</DataArray>
<DataArray Name="vtkOriginalPointIds" />
<DataArray Name="vtkGhostType" />
<DataArray Name="u" />
</PointData>
</Piece>
</UnstructuredGrid>
</VTKFile>
uint8_t vtkGhostType [8]*{__} = 0 / 255
int64_t vtkOriginalPointIds [8]*{__} = -9223372036854775808 / 9223372036854775807
and
$ bpls -al uL.twoMeshes.bp/
uint32_t NumberOfEntities 2*{8} = 0 / 13
uint32_t NumberOfNodes 2*{8} = 0 / 12
int64_t connectivity 2*[8]*{__, 5} = -9223372036854775808 / 9223372036854775807
double geometry 2*[8]*{__, 3} = -1.79769e+308 / 1.79769e+308
double step 2*scalar = 0 / 1
uint32_t types 2*scalar = 71 / 71
double u 2*[8]*{__, 3} = -1.79769e+308 / 1.79769e+308
string vtk.xml attr =
<VTKFile type="UnstructuredGrid" version="0.1">
<UnstructuredGrid>
<Piece NumberOfPoints="NumberOfNodes" NumberOfCells="NumberOfCells">
<Points>
<DataArray Name="geometry" />
</Points>
<Cells>
<DataArray Name="connectivity" />
<DataArray Name="types" />
</Cells>
<PointData>
<DataArray Name="TIME">step</DataArray>
<DataArray Name="vtkOriginalPointIds" />
<DataArray Name="vtkGhostType" />
<DataArray Name="u" />
</PointData>
</Piece>
</UnstructuredGrid>
</VTKFile>
uint8_t vtkGhostType 2*[8]*{__} = 0 / 255
int64_t vtkOriginalPointIds 2*[8]*{__} = -9223372036854775808 / 9223372036854775807
where the differences are in rows 1-4, 6 and the last two. Is this correct?
Thanks for any help!
I am using ADIOS2 to save some time-dependent simulation results in VTX format and I noticed that the mesh is saved multiple times even though it does not change. Does VTX support reusing a mesh saved at a previous time step?
Fides recently added support for this functionality [see fides#15 ] so I was wondering if it would be possible to have it in VTX too.
I am not entirely sure whether this is a VTK issue, an ADIOS2 issue or something else entirely so apologies if I'm asking in the wrong place
In order to compute the surface area of a mesh I've used Cell Size and Integrate Variables many times. Today I edited the name of the array produced by Cell Size and to my surprise the Integrate Variables filter gives the wrong value for its integral. I attach a clip showing the issue.
Thanks for the quick reply, unfortunately that filter doesn't do anything
@kmorel thanks for the explanation! It does, of course, make perfect sense that
\left| \bigcup K \right| = \int_{\bigcup K} 1 \ne \int_{\bigcup K} |K|.
I have to say I was always misled by the fact that "Integrate Variables" adds an "Area" field which computes the correct area of the domain and it really makes it look like "Integrate Variables" is adding up the "Area" field produced by "Cell Size". While in fact it is not: the original "Area" field is silently shadowed.
I can confirm that if I add a "Calculator" with Result equal to 1 and then integrate that with "Integrate Variables" then I do get the correct area for the domain.
I have a file [tried with VTU and Fides] written in parallel [MPI]. When I open it I can see internal surfaces on the borders between the parallel partitions of the mesh. The Extract Surface filter has two options, "Piece Invariant" and "Remove Ghost Interfaces" which from their description more or less sound like they should get rid of the internal surfaces but they actually don't [nor does any combination of the two].
Is this a bug? Or am I misunderstanding what these options do?
In order to compute the surface area of a mesh I've used Cell Size and Integrate Variables many times. Today I edited the name of the array produced by Cell Size and to my surprise the Integrate Variables filter gives the wrong value for its integral. I attach a clip showing the issue.
It must be that your bug-demolishing skills are transcending space, time and branches :D
In any case, I didn't use master but one of your free branches.
Thanks! A small update: I recompiled with the VTK version from your branch https://gitlab.kitware.com/caitlin.ross/vtk/-/commits/fides-bp5-support and tested with the files I attached previously. The VTX-BP4/5 files work, the Fides-BP5 file works but the Fides-BP4 file crashes with error message
what(): ERROR: neither vtk.xml file or bp attribute was found in /dev/shm/testFiles/fides.bp4
However, if I rename it from fides.bp4
to fides.bp
, then Paraview can read it fine.
I noticed that if I create a Line1 source, set the point coordinates to something, then create a Line2 source, right click on Line1, Copy, right click on Line2, Paste, nothing happens while I would expect the coordinates from Line1 to be copied over to Line2.
Ah, the mystery is finally solved! That also explains why sometimes I had to delete the whole folder to make it work. Your guess is entirely correct and that's some superhero-level clairvoyance, Caitlin -- or shall I call you Captain Rossvel? :D
I have some more insight, I finally found what works and what doesn't.
ADIOS2 latest, compiled by me; Paraview with VTK from your branch, compiled with the above ADIOS2; my code, same ADIOS2.
I'm attaching some test files for your convenience. testFiles.tar.gz
I tried your revision, I fed a BP4 file to Paraview and it crashes with
what(): [Mi Apr 26 17:14:16 2023] [ADIOS2 EXCEPTION] <Engine> <BP5Reader> <ParseMetadataIndex> : ADIOS2 BP5 Engine only supports bp format version 5, found 4 version
I can't guarantee 100% for the library that is creating these files but I would give it the benefit of the doubt for now.
[I am filing with Paraview but this might as well be a VTK problem].
With the release of ADIOS2 2.9.0, BP5 is now the default file engine [cfr. https://github.com/ornladios/ADIOS2/releases/tag/v2.9.0 ]. I tried to open BP5-VTX files created through my code that uses the newest version of ADIOS2 and I get a BP4Reader
error, whereas running with a debugger I verified that the file is created by a BP5Writer
.
I suspect Paraview should instead be using a BP5Reader
.