Make vtkWrapHierarchy 5x faster on Windows

When VTK's Python wrapper is used at application level, vtkWrapHierarchy takes several minutes to create hierarchy file for each larger library, because there are hundreds of include directories (all the directories of ITK, VTK, and other dependencies) and hundreds of header files to locate.

It turns out that the root cause is that on Windows, checking existence of a random file using stat function (or any Window-specific API) is very slow.

It is magnitudes faster to iterate through all the files in a folder. For example, iterating through a total of 17000 files in 600 include folders just takes 1-2 seconds. Therefore, checking existence of a file can be made about 5x faster by collecting a list of all the files in include directories and use that list to check if the file exists.

Co-authored-by: @lassoan

added lang:python label

Errors:

• commit 2be6b130 is not allowed because the following files are not formatted according to the 'clang-format' check: Wrapping/Tools/vtkParsePreprocess.c. Post a comment ending in the line Do: reformat to rewrite the MR source branch automatically.

Warnings:

• please consider adding a changelog entry in a file ending with .md in Documentation/release/dev.
• the merge request is marked as a work-in-progress.

The warnings do not need to be fixed, but it is recommended to do so.

Please rewrite commits to fix the errors listed above (adding fixup commits will not resolve the errors) and force-push the branch again to update the merge request.

added 2 commits

• d848a12d - Wrapper hash table to track existing headers
• 2992eb32 - Manage the cleanup of wrapper file system cache

Compare with previous version

Errors:

• commit 2be6b130 is not allowed because the following files are not formatted according to the 'clang-format' check: Wrapping/Tools/vtkParsePreprocess.c. Post a comment ending in the line Do: reformat to rewrite the MR source branch automatically.
• commit d848a12d is not allowed because the following files are not formatted according to the 'clang-format' check: Wrapping/Tools/vtkParsePreprocess.c. Post a comment ending in the line Do: reformat to rewrite the MR source branch automatically.

Warnings:

• please consider adding a changelog entry in a file ending with .md in Documentation/release/dev.
• the merge request is marked as a work-in-progress.

The warnings do not need to be fixed, but it is recommended to do so.

Please rewrite commits to fix the errors listed above (adding fixup commits will not resolve the errors) and force-push the branch again to update the merge request.

Do: reformat

This topic has been reformatted and pushed; please fetch from the source repository and reset your local branch to continue with further development on the reformatted commits.

added 3 commits

• b13731d1 - Make vtkWrapHierarchy 5x faster on Windows
• eb1b70ae - Wrapper hash table to track existing headers
• 97dc8ca9 - Manage the cleanup of wrapper file system cache

Compare with previous version

Warnings:

• please consider adding a changelog entry in a file ending with .md in Documentation/release/dev.
• the merge request is marked as a work-in-progress.

The warnings do not need to be fixed, but it is recommended to do so.

added 2 commits

• 7b8d6e2f - Wrapper hash table to track existing headers
• 142cf7be - Manage the cleanup of wrapper file system cache

Compare with previous version

Warnings:

• please consider adding a changelog entry in a file ending with .md in Documentation/release/dev.
• the merge request is marked as a work-in-progress.

The warnings do not need to be fixed, but it is recommended to do so.

added 3 commits

• 83583b39 - Wrapper hash table to track existing headers
• 05470314 - Manage the cleanup of wrapper file system cache
• 6b5b0847 - Handle filename case-insensitivity issue in wrappers

Compare with previous version

Warnings:

• please consider adding a changelog entry in a file ending with .md in Documentation/release/dev.
• the merge request is marked as a work-in-progress.

The warnings do not need to be fixed, but it is recommended to do so.

added 1 commit

• 91add508 - Make a list of headers that cannot be found

Compare with previous version

Warnings:

• please consider adding a changelog entry in a file ending with .md in Documentation/release/dev.
• the merge request is marked as a work-in-progress.

The warnings do not need to be fixed, but it is recommended to do so.

added 154 commits

• 91add508...71085392 - 150 commits from branch vtk:master
• 0102a662 - Wrapper hash table to track existing headers
• df5918d8 - Manage the cleanup of wrapper file system cache
• 31c4c538 - Handle filename case-insensitivity issue in wrappers
• e13acc12 - Make a list of headers that cannot be found

Compare with previous version

Warnings:

• the merge request is marked as a work-in-progress.

The warnings do not need to be fixed, but it is recommended to do so.

marked this merge request as ready

changed the description

@ben.boeckel @lassoan this relates to our discussion on discourse.

+1 A quick glance over the code LGTM. CI seems OK too.

Thanks a lot for this. I've tested it on building 3D Slicer and it worked correctly and fast.

That's good to hear. I'll merge tonight.

added 4 commits

• 9debd225 - Wrapper hash table to track existing headers
• c7f36038 - Manage the cleanup of wrapper file system cache
• a25a7c75 - Handle filename case-insensitivity issue in wrappers
• 204175b2 - Make a list of headers that cannot be found

Compare with previous version

Do: merge

merged

mentioned in commit fbcdb5d8

mentioned in merge request !8039 (merged)

mentioned in merge request !10624 (merged)