WIP: Add xgc runs given to me by Jong Choi.

This is a very large (327 MB) addition of XGC test data. @caitlin.ross : Could you clone and tell me if it's annoying to have this much in the repo? Does kitware have limits on it's lfs server?

(Note: I still need to check that this data is actually relevant to us, but if it's too large, then it's certainly not!)

Errors:

• Failed to run the checks: hosting service error: host error: communication with gitlab: error sending request for url (https://gitlab.kitware.com/api/v4/projects/vtk%2Fadis/merge_requests/58/notes): operation timed out.

Not sure about size limits. Will have to ask about that, but I've been looking through the data and have some questions.

• Is this data generated using ADIOS2? (I ask because of the next question)
• What are the different files for a given type of data? For instance the 3d data has xgc.3d.13000.bp, xgc.3d.13100.bp, and so on. Are they different time steps? If so, that may be a problem at the moment. When I first started adding XGC support, I apparently had data generated from ADIOS 1, which did do time steps in different files. So I was given a VM where I could run XGC with ADIOS2 (which is where our current data in the repo comes from) and that had all time steps in one file.

If the different time steps are indeed stored in different files, that isn't supported at the moment. And the original plan if we need to support one file per timestep was to actually add file series support to ADIOS, not ADIS.

Is this data generated using ADIOS2? (I ask because of the next question)

No, it was generated in ADIOS1. Norbert maintains a reader.

What are the different files for a given type of data? For instance the 3d data has xgc.3d.13000.bp, xgc.3d.13100.bp, and so on. Are they different time steps? If so, that may be a problem at the moment. When I first started adding XGC support, I apparently had data generated from ADIOS 1, which did do time steps in different files. So I was given a VM where I could run XGC with ADIOS2 (which is where our current data in the repo comes from) and that had all time steps in one file.

Yes, they are different timesteps. I got them from ADIOS1->ADIOS2 via

$ adios_reorganize xgc.3d.13000.bp test.bp BPFile "" BPFile ""

Now I need to see if I can assemble them into steps.

@caitlin.ross : Sadly it looks like we can't do this easily, see here:

https://github.com/ornladios/ADIOS2/issues/2189

Okay, I talked to Berk and as far as data size goes, it's fine since we're using LFS. But we're not supporting reading ADIOS 1 data, so we only want ADIOS2 data in the repo.

closed