Ninja: Swift: Remodeling Swift with separate object build and link steps
I've started looking at re-doing how CMake models Swift compilations in the Ninja generator to match how CMake models other languages. CMake usually breaks the build into two parts, the build that produces object files, and the link which builds the executable, shared library, or static archive.
┌───────────┐┌───────────┐
│ foo.c ││ bar.c │
└───────────┘└───────────┘
build │ │
▼ ▼
┌───────────┐┌───────────┐
│ foo.c.o ││ bar.c.o │
└───────────┘└───────────┘
│ │
link └──────┬─────┘
▼
┌────────────────────────┐
│ liblib.a │
└────────────────────────┘
Swift builds and links in one step. Swift source files are passed to the swift driver as though they were object files, along with any other object files in the target.
┌───────────┐ ┌───────────┐
│ foo.c │ │ foo.swift │
└───────────┘ └───────────┘
build │ │
▼ │
┌───────────┐ │
│ foo.c.o │ │ build/link
│ clang -c │ │
└───────────┘ │
│ │
└────────┬──────┘
▼
┌─────────────────────────────────┐
│ liblib.a │
│ swiftc -emit-library -static │
└─────────────────────────────────┘
While this works, it has several disadvantages. First, we can't get Swift object libraries because there are no object targets created.
Second, adding rules to compile_commands.json
is done during the WriteObjectStatement
step for the source file, so we don't get LSP support for Swift in CMake projects. The other issue figuring out where to generate headers for interop with other languages. Swift can generate C and C++ headers, but there isn't a place to tell the compiler to emit that header in this build graph, so projects have to explicitly add a separate call to the Swift compiler in a custom command to generate that header if they want to transparently mix C++ and Swift sources in a single library.
The fix is to split the Swift model, from "linking" the Swift files to building the Swift files into objects, and then linking those into a library. We can emit swiftmodules, swift interface, and bridging headers (waiting on driver flags and not just frontend flags to emit the C++ bridging header before we can do that) during the Swift object build time. C and C++ libraries in the same target can then depend on those headers to use the exposed Swift functions.
┌───────────┐ ┌───────────┐ ┌───────────┐
│ foo.c │ │ foo.swift │ │ bar.swift │
└───────────┘ └───────────┘ └───────────┘
build │ │ │
│ └──────┬──────┘
▼ build ▼
┌───────────┐ ┌───────────────────────────┐
│ foo.c.o │ │┌───────────┐ ┌───────────┐│
│ clang -c │ ││foo.swift.o│ │bar.swift.o││
└───────────┘ │└───────────┘ └───────────┘│
│ │ swiftc -c │
│ └───────────────────────────┘
│ │
└──────────────┬───────┘
│ link
▼
┌────────────────────────────────────────────┐
│ liblib.a │
│ swiftc -emit-library -static │
└────────────────────────────────────────────┘
This is how Swift Package Manager and Xcode model Swift compilation as well. The whole-module-optimization bug in the old driver is masked by this model. The WMO bug causes the old driver to pass all inputs to the swift-frontend invocation, so the frontend tries parsing static archives and objects, which doesn't work. In the split model, we only need to pass -wmo
to the object build step, where the only inputs are Swift source files. The link step does not need -wmo
so we can call swiftc
without it and avoid trying to parse objects and static archives as source.
I have most of an initial implementation working, but want to get opinions on the approach taken before doing the other half of the remaining work. The biggest question I have is about modeling units of work that span multiple files. The translation units in C/C++ only contain a single file (at least until C++ modules), so WriteObjectBuildStatement
and may of the API it calls taking a single cmSourceFile
works fine, but Swift modules (and swiftc invocations) take all of the source files in the module at once, so we need to pass all of the source files in the module into the object statement. CMake isn't really set up for this.
I created a WriteSwiftObjectBuildStatement
function that is effectively WriteObjectBuildStatement
, but takes a vector reference of source files instead of a single cmSourceFile
:
void cmNinjaTargetGenerator::WriteSwiftObjectBuildStatement(
std::vector<cmSourceFile const*> const & sources,
std::string const &config,
std::string const &fileConfig,
bool firstForConfig);
Then the implementation mostly does what WriteObjectBuildStatment
does, but separating the combined statement work from the per-file work. Unfortunately, the work is somewhat interleaved. Getting things like flags and module names is the same for all files since there is a single compiler invocation, but we have to collect separate output-file-map information for each file in the module, before emitting the single output-file-map for that module.
I can probably factor things a little better if we decide this is the right path, but wanted to get something mostly working first. I've seen work for dyndeps and C++ modules with multi-file modules/translation-units, so it might be interesting trying to align the two if it makes sense.
I look forward to hearing thoughts, improvements, and responses. Thanks.