Selecting datasets is slow
We have a significant performance regression selecting large datasets. I dug deep enough that I know what is wrong even if Kitware can't replicate it. The reason it is hard to replicate is that if you have a fast disk (such as an ssd) it won't show, and the second time you run it, due to disk caching, it won't show. The way I found the issue is comparing 5.9.1 (which is blazingly fast) with 5.11.1 (which is molasses slow). Here is how to replicate it and superficially what the problem is. Fixing it/ rolling back changes are over my head - I don't have a clue.
- I am going to compare and contrast debug builds of 5.9.1 and 5.11.1. Start with 5.11.1. Linux. Builtin server.
- Untar the attached tarball. This will create a directory with a script.
- Copy can.ex2 into this directory.
- Edit the script to make as many copies of can as you like. As I said above, you probably can't replicate the issue, so creating 8 should be fine. Run the script.
- Run ParaView in a debugger.
- File/ Open. Go to the directory created above. Don't do anything yet.
- Put a break in vtkPVFileInformation::CopyFromObject line 424. This is in file vtkPVFileInformation.cxx. The code looks about like this:
std::string path = helper->GetPath();
this->SetName(path.c_str());
- Now, select the dataset canLotsO.e.8. After some period (should be instantaneous, and is for 5.9.1, isn't for 5.11.1), canLotsO.e.8 will be moved into the text entry box below, where you can then say OK.
When you do this with 5.9.1, the path variable will be "/pathToTheMagicDirectory/canLotsO.e.8". It will be called once (I believe). When you do this with 5.11.1, the path variable will be "/pathToTheMagicDirectory/canLotsO.e.8.[0-7]". It will be called 8 times.
The issue is that each call is slow in a system call to access, which is killing us. Some datasets have hundreds of thousands or millions of files. These will never open. Here is the trace stack.
vtksys::SystemTools::FileExists,
vtksys::SystemTools::FileExists,
vtkPVFileInformationGetType,
vtkPVFileInformation::CopyFromObject,