Memory growth with restarts with the IOSS reader
Note - read my comments way below. This is not technically a memory leak, but rather unneeded memory growth.
We have a really wicked memory leak at very large scale with restart files with the IOSS reader. My user has 7200 files that are 20 TBytes total, with 250 million cells and points. Obviously, we can't use his dataset to debug. I believe I have voodoo'd up an example that works. Here is how to replicate:
- We are going to compare memory footprint of 5.9.1 and 5.10.1. I bet this replicates builtin server, but I am having disk problems and can only get at high performace disks on clusters.
- Download the attached tarball. It consists of the datasets and script to create a dataset that shows the issue. We use can.exo as our base. I have 4 versions of can.exo, each with the time offset. Thus, we have restarts. Now, we spread stuff to try to show the issue. run "source spreader".
- 5.10.1, remote server (16 ranks), Linux.
- Load can.e.32. Note that we have restarts. Turn off all of the variables. Apply.
- View/ Memory Inspector. Watch the server memory size.
- Now, play. Keep watching the memory size. It will grow over time.
What I am seeing may actually be different, I don't know. The big dataset has a HUGE memory jump, this one doesn't seem to. With permission of Stefan, I could show someone this dataset/ memory increase in a Teams meeting.
Edited by W. Alan Scott