Skip to content

CPack: Avoid adding duplicate files to archive when combining components

In our monorepo, for every target we create a component for executable/shared library itself and another component for its runtime dependencies. We are doing runtime dependency installation via install(RUNTIME_DEPENDENCY_SET). This is great for us as it brings us a great granularity, we can easily mix and match different components into a cpack script and using ALL_COMPONENTS_IN_ONE we can generate one big release file. However downside of this is most of our projects use mostly the same dependencies, and CPacks's Archive Generator doesn't differentiate same files of different components, thus our archive generation takes unnecessary long and the release file gets unnecessary big.

This proposal aims to introduce a file deduplication mechanism exclusively for CPack's archive generation (.tgz, .7z, etc.) when merging multiple components into a single archive. The deduplication feature is not extended to installer generation methods like qt, nsis, etc. After components are installed, in package creation phase, system will track every path added to the archive and if a path that matches one already in the archive, the content is checked against the stored version, if contents differ then should raise an error, otherwise we do not add the path to the archive again. After talks in the #25280 (closed), it is decided to be this a bug fix and should work without an option or a policy. Thanks to this system, our packaging times are down from 1m30,359s to 0m23,434s. The file sizes shrank too, from 850 MB to 124 MB.

Fixes: #25280 (closed)
Topic-rename: cpack-archive-dedup

Edited by Atilhan Emre Dursunoglu

Merge request reports