GitLab

Hi,

I'm working on a project where we're attempting to make our lib<name>.a output archive (which we distribute) deterministic, which is to say we have a requirement that the lib<name>.a is byte-for-byte identical across builds, across machines.

I'm finding that one of the places where this isn't true is related to the ar/ranlib tool on macOS including timestamps in the symbols table of the .a file.

There's a flag in the GNU binutils ar (-D), GNU binutils ranlib (-D), and a similar one in Apple/Xcode's libtool (can't find online documentation) (-D) that is meant for this purpose, where it sets the timestamps of the .o inputs to zero within the symbols table.

Here's the description of the flag in GNU binutils ar:

[D] - use zero for timestamps and uids/gids (default)

and

‘D’ Operate in deterministic mode. When adding files and the archive index use zero for UIDs, GIDs, timestamps, and use consistent file modes for all files. When this option is used, if ar is used with identical options and identical input files, multiple runs will create identical output files regardless of the input files’ owners, groups, file modes, or modification times.

Here's the description of the flag in GNU binutils ranlib:

-D Use zero for symbol map timestamp (default)

and

-D Operate in deterministic mode. The symbol map archive member’s header will show zero for the UID, GID, and timestamp. When this option is used, multiple runs will produce identical output files.

Here's the description of the flag in Apple's libtool:

-D When building a static library, set archive contents' user ids, group ids, dates, and file modes to reasonable defaults. This allows libraries created with identical input to be identical to each other, regardless of time of day, user, group, umask, and other aspects of the environment.

It would be really nice to see deterministic build support in CMake. There are 3 issues that I've found related to supporting this that are inter-related:

• Support for adding RANLIBFLAGS to the linking step. This would be needed because even though ar is invoked and could have flags set using STATIC_LIBRARY_OPTIONS, ranlib is invoked right after, which in my testing seems to add the timestamps, since ranlib is generally the tool that is adding the symbols table (even though ar could be configured to add it instead). This is somewhat moot on certain Linux distros (Ubuntu, for example) since the ar and ranlib tools on those platforms enable the -D flag by default.
• Support within CMake for using libtool on macOS. Xcode's ar/ranlib tools don't seem to support the -D flag that the GNU binutils version supports. In the documentation for these tools, Apple states that their libtool is intended to be a replacement for ar and ranlib. Since Xcode's libtool appears to support the -D flag in the same manner as GNU binutils ar/ranlib, it would be acceptable for CMake to invoke libtool instead of ar and ranlib when building with the Xcode toolchain. This would require libtool support on CMake's end, as well as a way to specify LIBTOOLFLAGS. (Note: GNU libtool doesn't seem to support the -D flag, so it's unclear to me what other incompatibilities the Xcode and GNU versions might have, which, to me, suggests that on non-Apple platforms, the ar and ranlib tools might be preferred).
• Native CMake support for building determinstic static libraries (w.r.t. the symbols table). This would add the -D to the appropriate tools when enabled.

Even if native CMake support for deterministic static libraries was not added, it would be nice to see CMake use libtool when using the Xcode toolchain. That coupled with the ability to add flags to libtool on macOS platforms and ranlib on Linux platforms, would enable me to add support for creating deterministic symbols table within static libraries via the -D flag within my own project.

My current workaround is to set CMAKE_<lang>_CREATE_STATIC_LIBRARY on macOS platforms to something along the lines of /usr/bin/xcrun libtool -static -D -o <TARGET> <LINK_FLAGS> <OBJECTS>, which seems to have the same effect, but it would be preferable if CMake had native libtool support instead of relying on xcrun.