ProcessUNIX.c: fix select(), FD_SETSIZE on macOS
On macOS (reportedly for 10.5 and later), if _DARWIN_UNLIMITED_SELECT
is
not defined, then the select()
call in kwsysProcessWaitForPipe()
fails
with EINVAL
when (max + 1) > 1024
. And, by default, FD_SETSIZE
is 1024;
passing file descriptors not less than FD_SETSIZE
to FD_CLR()
, FD_SET()
,
and FD_ISSET()
causes out-of-bounds accesses, at least for an fd_set
allocated on the heap (as is the case for cp->PipeSet
, allocated by
kwsysProcess_New()
; one observed effect of this is corruption of the
Invalid argument
error message copied from strerror(errno)
into
cp->ErrorMessage
, since cp->ErrorMessage
is located shortly after
cp->PipeSet
). Defining _DARWIN_UNLIMITED_SELECT
and defining FD_SETSIZE
to OPEN_MAX
(10240) avoids these issues.
Background: see https://trac.macports.org/ticket/64368
MacPorts has a “trace” mode which ensures a build’s processes only have
access to resources explicitly requested by a port. It relies in part on
intercepting open()
calls, and for some reason would sometimes return
file descriptors with unusually large values (1024 and higher). For some
builds using CMake, it was observed that cmake -E cmake_link_script
would fail with Error running link command: Invalid argument
, although
sometimes the error was corrupted, e.g.
Error running link command: Invalid `rgument
or
Error running link command: In^Valid argument
where ^V
is the ASCII SYN character.
I am sharing this change in case it is found desirable to include upstream. I am not aware whether CMake or other program using KWSys currently ever use enough file descriptors to encounter this issue naturally.