Tests: Avoid leaving behind non-readable directories
Since !8908 (merged) we create some non-readable directories.
CMake 3.28 and below do not know how to delete them, so some nightly
builds fail ctest_empty_binary_directory
. Add read permission to those
directories when we are finished with them.