Add an option to generate Ninja with a "phased" dependency graph
Introduction
I am interested in writing a patch that adds a boolean option to the CMake Ninja generator, tentatively called CMAKE_NINJA_PHASED_DEPENDENCY_GRAPH
, which modifies the Ninja generation such that the rule that currently runs a C/C++ build step is split into two rules: one that computes the dependencies Makefile using gcc -M
or similar, and one that actually runs the build.
The motivation for this is that I want to take the output of the CMake Ninja generator and use it to generate reproducible, deterministic, incremental builds via the Nix package manager.
Background on Nix
Despite the marketing, Nix is actually better described as a build system; it's just that the way we currently use it means that each target in the build graph is a full package, rather than, say, an object file. By munging the Ninja output of CMake, I can generate a Nix expression containing a build graph at object file granularity.
The only issue with this is that Nix sandboxes every build rule such that it only has precisely its dependencies in scope, which means that I need a 100% correct dependency graph. One easy way of getting a correct dependency graph would be to add every file in the source tree as a dependency to every target. Unfortunately, this destroys the incrementalism we were trying to get in the first place; Nix's caching mechanism involves hashing the complete description of how to build something, including the build script, its arguments, and the hashes of (how to build) any dependencies. In the case of a potentially nondeterministic build rule (e.g.: curl
ing a source tarball), Nix also requires you to specify the hash of the result after executing the rule, and this hash is included in the hash of how to build that rule. Nix rules that depend on a local file work in a similar way, although the hash is automatically computed by the Nix runtime.
The problem with the Ninja generator
The issue with using the CMake Ninja output for this purpose is that it generates a Ninja rule and build edge that looks like this:
rule CXX_COMPILER__foobar-baz
depfile = $DEP_FILE
deps = gcc
command = /path/to/bin/g++ $DEFINES $INCLUDES $FLAGS -MD -MT $out -MF $DEP_FILE -o $out $IN_ABS
description = Building CXX object $out
build baz/CMakeFiles/foobar-baz.dir/src/quux.cpp.o: CXX_COMPILER__foobar-baz ../baz/src/quux.cpp || cmake_order_depends_target_foobar-baz
DEP_FILE = baz/CMakeFiles/foobar-baz.dir/src/quux.cpp.o.d
... other stuff ...
The problem with this is that the same rule that generates the depfile
is also the rule that actually does the build. This is a useful feature of Ninja in most contexts, since it reduces the size of the build graph, but once I've added depfile
support to ninja2nix, I can almost guarantee that this will destroy incrementalism. This is because, if I even want to compute the build graph, I need to build all files that occur in a depfile = ...
attribute, and this will happen every time any file changes. If computing the depfile
s is just as expensive as actually building the project, there will be no incrementalism.
Thus, we'd need to add a CMake option that causes the rules to be generated like this:
rule CXX_COMPILER_DEPFILE__foobar-baz
command = /path/to/bin/g++ $DEFINES $INCLUDES $FLAGS -E -MD -MT $out -MF $DEP_FILE $IN_ABS
description = Computing dependencies of $out
rule CXX_COMPILER__foobar-baz
depfile = $DEP_FILE
deps = gcc
command = /path/to/bin/g++ $DEFINES $INCLUDES $FLAGS -o $out $IN_ABS
description = Building CXX object $out
build baz/CMakeFiles/foobar-baz.dir/src/quux.cpp.o.d: CXX_COMPILER_DEPFILE__foobar-baz ../baz/src/quux.cpp
build baz/CMakeFiles/foobar-baz.dir/src/quux.cpp.o: CXX_COMPILER__foobar-baz ../baz/src/quux.cpp baz/CMakeFiles/foobar-baz.dir/src/quux.cpp.o.d || cmake_order_depends_target_foobar-baz
DEP_FILE = baz/CMakeFiles/foobar-baz.dir/src/quux.cpp.o.d
... other stuff ...
I haven't tested the above, but I think that's what would be needed.
Other stuff
I also raised this related issue on Ninja: https://github.com/ninja-build/ninja/issues/1303