Paraview fails to load big VTU file on Windows (A solution is included)
The problem is that the Windows version of Paraview can not load big VTU file (>2.7GB), while the Linux version works well. This phenomenon is confirmed using following versions:
- v5.11.1
- v5.10.1
- v5.9.1
This issue contains 3 sections: Problem replication, a verified solution and some existing issues about the same problem.
1. Problem replication
1.1 sample .vtu file to replicate the problem.
Google drive: https://drive.google.com/file/d/1T_1uHtZJz1CalxV1oKWsg9US-QqI2hOR/view?usp=sharing
Dropbox (just in case the above link expires): https://www.dropbox.com/scl/fi/qwbpmhhli9axoj9p0by63/cfd_hlcrm_medium_volume.vtu?dl=0&rlkey=tlhzctrc00gawuz8ec0fhvzql
1.2 error message of loading the sample .vtu file
2. A verified solution
The solution is: add the following line in the $ParaviewSrc\VTK\IO\XMLParser\CMakeLists.txt
add_definitions(-DXML_LARGE_SIZE)
Basically the root of the problem is due to the expat
XML parser used by the VTK library. It uses the macro XML_LARGE_SIZE to determine whether a type to be long
or long long
. However, for MSVC, long
is 32-bit integer, but for Linux, long
is 64-bit integer, which causes this inconsistent behavior between Windows and Linux version of Paraview. Of course, the ultimate solution is to change the expat library by using some accurate types, e.g. int32_t or int64_t, but that may be too much. Also, turning on XML_LARGE_SIZE as the default option also makes sense nowadays. A 3GB VTU file could be treated as "large" 10 years ago, but in the current era of big data, it is not large anymore.
After the above modification was applied, I can successfully load the sample vtu data into my modified paraview.exe, shown below:
Before the above solution, I also tried to turn on the EXPAT_LARGE_SIZE option in the expat's CMakeLists.txt at https://gitlab.kitware.com/vtk/vtk/-/blob/master/ThirdParty/expat/vtkexpat/CMakeLists.txt#L151, but it didn't fix the problem.
Although this solution is applied to the VTK repository, I think it is proper to post the issue here because the problem is exposed in the Paraview. If any developer of Paraview wants, I can open another issue to VTK's gitlab repository and paste the link of the current issue.
3. Some existing issues about the same problem
4. solution of changing expat source code directly
In this file: https://gitlab.kitware.com/vtk/vtk/-/blob/master/ThirdParty/expat/vtkexpat/lib/expat_external.h#L160 Remove the if..else statement to let XML_Index and XML_Size always use "long long" should also fix this problem (not verified.)
Hopefully the above issue could be fixed before the next release of Paraview. If you need more information, just reply under this ticket. Thank @Todd in the Paraview Forum (Link), since the above solution is actually from @Todd. Thank you!