Deterministic builds of static libraries
Hi,
I'm working on a project where we're attempting to make our lib<name>.a
output archive (which we distribute) deterministic, which is to say we have a requirement that the lib<name>.a
is byte-for-byte identical across builds, across machines.
I'm finding that one of the places where this isn't true is related to the ar/ranlib tool on macOS including timestamps in the symbols table of the .a
file.
There's a flag in the GNU binutils ar
(-D
), GNU binutils ranlib
(-D
), and a similar one in Apple/Xcode's libtool
(can't find online documentation) (-D
) that is meant for this purpose, where it sets the timestamps of the .o
inputs to zero within the symbols table.
Here's the description of the flag in GNU binutils ar
:
[D] - use zero for timestamps and uids/gids (default)
and
‘D’ Operate in deterministic mode. When adding files and the archive index use zero for UIDs, GIDs, timestamps, and use consistent file modes for all files. When this option is used, if ar is used with identical options and identical input files, multiple runs will create identical output files regardless of the input files’ owners, groups, file modes, or modification times.
Here's the description of the flag in GNU binutils ranlib
:
-D Use zero for symbol map timestamp (default)
and
-D Operate in deterministic mode. The symbol map archive member’s header will show zero for the UID, GID, and timestamp. When this option is used, multiple runs will produce identical output files.
Here's the description of the flag in Apple's libtool
:
-D When building a static library, set archive contents' user ids, group ids, dates, and file modes to reasonable defaults. This allows libraries created with identical input to be identical to each other, regardless of time of day, user, group, umask, and other aspects of the environment.
It would be really nice to see deterministic build support in CMake. There are 3 issues that I've found related to supporting this that are inter-related:
- Support for adding
RANLIBFLAGS
to the linking step. This would be needed because even thoughar
is invoked and could have flags set usingSTATIC_LIBRARY_OPTIONS
,ranlib
is invoked right after, which in my testing seems to add the timestamps, sinceranlib
is generally the tool that is adding the symbols table (even thoughar
could be configured to add it instead). This is somewhat moot on certain Linux distros (Ubuntu, for example) since thear
andranlib
tools on those platforms enable the-D
flag by default. - Support within CMake for using
libtool
on macOS. Xcode'sar
/ranlib
tools don't seem to support the-D
flag that the GNU binutils version supports. In the documentation for these tools, Apple states that theirlibtool
is intended to be a replacement forar
andranlib
. Since Xcode'slibtool
appears to support the-D
flag in the same manner as GNU binutilsar
/ranlib
, it would be acceptable for CMake to invokelibtool
instead ofar
andranlib
when building with the Xcode toolchain. This would requirelibtool
support on CMake's end, as well as a way to specifyLIBTOOLFLAGS
. (Note: GNUlibtool
doesn't seem to support the-D
flag, so it's unclear to me what other incompatibilities the Xcode and GNU versions might have, which, to me, suggests that on non-Apple platforms, thear
andranlib
tools might be preferred). - Native CMake support for building determinstic static libraries (w.r.t. the symbols table). This would add the
-D
to the appropriate tools when enabled.
Even if native CMake support for deterministic static libraries was not added, it would be nice to see CMake use libtool
when using the Xcode toolchain. That coupled with the ability to add flags to libtool
on macOS platforms and ranlib
on Linux platforms, would enable me to add support for creating deterministic symbols table within static libraries via the -D
flag within my own project.
My current workaround is to set CMAKE_<lang>_CREATE_STATIC_LIBRARY
on macOS platforms to something along the lines of /usr/bin/xcrun libtool -static -D -o <TARGET> <LINK_FLAGS> <OBJECTS>
, which seems to have the same effect, but it would be preferable if CMake had native libtool
support instead of relying on xcrun
.