Use SetSelection instead of SetBlockSelection for getting ADIOS data to read
There's an issue when visualizing certain datasets, like Gray Scott; we get a gap between each rank's data:
With the bpls output looking like
$ bpls gs.bp U -D
double U 100*{64, 64, 64}
step 0:
block 0: [ 0:63, 0:31, 0:31]
block 1: [ 0:63, 32:63, 0:31]
block 2: [ 0:63, 0:31, 32:63]
block 3: [ 0:63, 32:63, 32:63]
(Output from only 2 ranks is shown in the image because the clip filter was used.)
In Fides's DataSource
we are always using SetBlockSelection()
. In this case, we should instead use SetSelection()
with the appropriate Start and Count indices we want to read. However, there are still some cases we need to use SetBlockSelection()
, when local arrays are used. And in the case of GTC and XGC special classes, we should probably still use SetBlockSelection
. See the docs for info on global vs local arrays.
I'll give my current thoughts on how to go about this, but if you see a better way of doing things, feel free to change it as you see fit!
- to keep
ReadVariableBlocksInternal()
as is - add another method that can use
SetSelection
-
DataSource::ReadVariable
can make the decision of which one to use depending on the situation
Another thing to consider is the fides::keys::BLOCK_SELECTION()
key. I was thinking we might need another key allowing the user to make selections similarly to ADIOS with start and count, but then this would require changing vtkFidesReader. I'm not sure that's necessary though. Maybe we could continue to use the block selection in this case. Then Fides can check the ShapeID of the variable. If it's a local array, it'll just do what it's already doing. If it's a global array, it could check the dimensions of the blocks to see if it needs to adjust the start/count and then call SetSelection. According to the docs, ADIOS already does part of this if you call SetBlockSelection
on a global array. It will automatically set the appropriate Start/Count selection. So we'd just want to take it a step further and update the dimensions appropriately.
For instance, if vtkFidesReader is reading the above dataset on 2 ranks. Rank 0 will request blocks 0 and 1, Rank 1 will request blocks 2 and 3. So then ultimately Fides could do one of two things:
Option 1, within a rank combine all blocks into a single SetSelection:
- Rank 0, Fides calls SetSelection for Start = {0, 0, 0}, Count = {64, 64, 32}
- Rank 1, Fides calls SetSelection for Start = {0, 0, 31}, Count = {64, 64, 33}
Option 2, Still calling SetSelection, but still process each block on a rank separately:
- Rank 0, Fides calls SetSelection for block 0: Start = {0, 0, 0}, Count = {64, 32, 32}, then calls SetSelection for block 1: Start = {0, 31, 0}, Count = {64, 33, 33}
- Rank 1, Fides calls SetSelection for block 2: Start = {0, 0, 31}, Count = {64, 32, 32}, then calls SetSelection for block 3: Start = {0, 31, 31}, Count = {64, 33, 33}