Skip to content

ProcessUNIX.c: fix select(), FD_SETSIZE on macOS

Christopher Chavez requested to merge (removed):fix-select-macos into master

On macOS (reportedly for 10.5 and later), if _DARWIN_UNLIMITED_SELECT is not defined, then the select() call in kwsysProcessWaitForPipe() fails with EINVAL when (max + 1) > 1024. And, by default, FD_SETSIZE is 1024; passing file descriptors not less than FD_SETSIZE to FD_CLR(), FD_SET(), and FD_ISSET() causes out-of-bounds accesses, at least for an fd_set allocated on the heap (as is the case for cp->PipeSet, allocated by kwsysProcess_New(); one observed effect of this is corruption of the Invalid argument error message copied from strerror(errno) into cp->ErrorMessage, since cp->ErrorMessage is located shortly after cp->PipeSet). Defining _DARWIN_UNLIMITED_SELECT and defining FD_SETSIZE to OPEN_MAX (10240) avoids these issues.

Background: see https://trac.macports.org/ticket/64368

MacPorts has a “trace” mode which ensures a build’s processes only have access to resources explicitly requested by a port. It relies in part on intercepting open() calls, and for some reason would sometimes return file descriptors with unusually large values (1024 and higher). For some builds using CMake, it was observed that cmake -E cmake_link_script would fail with Error running link command: Invalid argument, although sometimes the error was corrupted, e.g.

Error running link command: Invalid `rgument

or

Error running link command: In^Valid argument

where ^V is the ASCII SYN character.

I am sharing this change in case it is found desirable to include upstream. I am not aware whether CMake or other program using KWSys currently ever use enough file descriptors to encounter this issue naturally.

Merge request reports