Skip to content

draft: CPack/DEB: parallel parameter generation

Russell Greene requested to merge russelltg/cmake:cpackdeb_parallel into master

This is a rough idea of how we could potentially parallelize CPackDeb.

This works by running CPackDeb.cmake in a separate cmake -P process, which allows us to run as many in parallel as we want.

It currently works by spawning a fixed number, then waiting on them in the same order that they are spawned. If we go with something similar to this, we would obviously want to make the amount of parallelism configurable (CPACK_THREADS does already exist, but the docs explicitly say that that's for compression....), as well as not wait in serial to allow for more parallelism when tasks don't all take the same duration. From my testing, they very much do not all take a similar amount of time.

Concerns/work to do:

  • CMake policies--cmake -P and cpack seem to have different cmake policies enabled? I needed to add CMP0012 and CMP0007 to get it to work, and it's possible I missed some that break edgecases.
  • Test cases--still doesn't pass a few.
  • Make parallelism configurable
  • Don't wait in order

Some performance numbers for a large project on a 24-core machine: Before:

real    17m50.839s
user    15m45.538s
sys     2m56.566s

After:

real    2m48.680s
user    23m0.374s
sys     4m12.677s

How does this approach seem? Wanted to get some feedback before putting more work into this.

Issue: #23152

Edited by Brad King

Merge request reports