Skip to content

WIP: adding XGC support

Caitlin Ross requested to merge caitlin.ross/fides:xgc-support into master

Some details on the XGC support:

  • Requires VTK-m be built with MPI enabled (only if ADIOS2 and ADIS are built with it - if they aren't, then MPI not needed in VTK-m either). This is because the EnvironmentTracker in VTK-m is used for getting/setting MPI communicator info in various classes in ADIS.
  • The number of planes is needed by the cell set, coordinate system, and the fields. Instead of requiring the user to specify the details on reading the number of planes for all of these in the JSON file, there has been a new (optional) section added to the JSON for AppMetaData (see xgc-reader.json), so this only has to be specified once. I did it this way in case there's other simulations that we add support for in the future that also need to grab similar types of details. I think this should only be used for grabbing some scalar value, because ADIOS will make a scalar value available immediately when calling Get().
  • XGC VTK-m files: It turns out that the XGC support is in VTK-m, with the various files having "Extrude" in the name and are used when possible. I did discover some bugs in part of the code related to saving field data, so those files are in this repo under adis/xgc/. (I also plan to submit the fix to VTK-m, but I think I should also write a test for that part before I do.) One other change I did make to handling XGC fields in VTK-m is related to reading data when not running ADIS in parallel. The way the ArrayHandles work for XGC fields is that it is expecting that all planes have the same data. Which is fine in parallel, if we force one MPI rank per plane then a rank knows the data is only valid for its plane. But I also supported reading with just one process, so there's support in StorageXGC.h to handle that case. If we don't want to support that situation, then that can be removed and once my bug fix is in VTK-m, then we can just get rid of the files under adis/xgc altogether.
  • There are two examples under examples/xgc (along with a readme explaining what's needed for the JSON file when reading XGC data). xgc.cxx reads all data from files, while xgc-sst.cxx reads mesh from file but the 3d field data using SST (which has been tested out with the VM). I wasn't sure if I could generate a small XGC data set to put in the repo in order to have a test. If so it will be easy to convert xgc.cxx into a test, otherwise I'll work on just making a test that manually creates some data for testing.

Some other changes that were made that aren't only applicable to XGC:

  • Added ADIS_USE_MPI. Previously in some places ADIOS2_HAVE_MPI or USE_MPI were used. So now there's just one to use consistently across the code.
  • Cleaned up most compilation warnings. This also includes adding an adisNotUsed(param_name) macro.

@berkgeveci Hopefully this all makes sense, but let me know if you want to meet up and talk about any of it.

Edited by Caitlin Ross

Merge request reports