macOS: open() returning file descriptors 1024 or higher causes cmake_link_script to fail

(Opening issue as requested in !6909 (closed).)

See https://trac.macports.org/ticket/64368: MacPorts has a “trace” mode which ensures a build’s processes only have access to resources explicitly requested by a port. It relies in part on intercepting open() calls, and for some reason would sometimes return file descriptors with unusually large values (1024 and higher). For some builds using CMake, it was observed that cmake -E cmake_link_script would fail with Error running link command: Invalid argument, although sometimes the error was corrupted, e.g.

Error running link command: Invalid `rgument

or

Error running link command: In^Valid argument

where ^V is the ASCII SYN character.

The issue in CMake is due to code from KWSys which currently does not handle file descriptors 1024 or higher properly on macOS:

On macOS (reportedly for 10.5 and later), if _DARWIN_UNLIMITED_SELECT is not defined, then the select() call in kwsysProcessWaitForPipe() fails with EINVAL when (max + 1) > 1024. And, by default, FD_SETSIZE is 1024; passing file descriptors not less than FD_SETSIZE to FD_CLR(), FD_SET(), and FD_ISSET() causes out-of-bounds accesses, at least for an fd_set allocated on the heap (as is the case for cp->PipeSet, allocated by kwsysProcess_New(); one observed effect of this is corruption of the Invalid argument error message copied from strerror(errno) into cp->ErrorMessage, since cp->ErrorMessage is located shortly after cp->PipeSet). Defining _DARWIN_UNLIMITED_SELECT and defining FD_SETSIZE to OPEN_MAX (10240) avoids these issues.

However I am not aware whether CMake or other programs using KWSys ever use enough file descriptors to encounter this issue outside of MacPorts’ trace mode.