Incorrect Makefile dependencies generated for a custom target
Dear All,
I came across this issue just this week, after we observed some (very elusive) build failures in our software. Unfortunately the issue is a bit convoluted, so bear with me...
Our software is split into "packages", which are just subdirectories with one CMakeLists.txt
file each. The packages can do a lot of things, including "installing files".
In our build setup "installing" a file actually means that:
- During the build of the project the file would get soft linked into the build directory into some particular place. (Depending on the file type.)
- During the installation of the project it gets installed using the
install(...)
command.
And for python files we do one more thing: We byte-compile them during the build, putting the .pyc
files into the build directory, and then installing them along with our project during the installation step.
The code we use for this, is here: https://gitlab.cern.ch/atlas/atlasexternals/blob/master/Build/AtlasCMake/modules/AtlasInstallFunctions.cmake But I rather attached a much simplified demonstrator of this issue, which will hopefully make it easier to debug it.
InstallProblemDemonstrator.tar.bz2
The demonstrator showcases two "packages", Package1
and Package2
. They both install dummy python files, with a slightly different directory layout. Package1 looks like:
Package1/
Package1/CMakeLists.txt
Package1/python
Package1/python/module1.py
Package1/python/module2.py
Package1/python/__init__.py
While Package2 looks like:
Package2/
Package2/CMakeLists.txt
Package2/python
Package2/python/Package2
Package2/python/Package2/module1.py
Package2/python/Package2/module2.py
Package2/python/Package2/__init__.py
I.e. it has its python files "one directory deeper".
The demonstrator defines an install_python_modules(...)
function, which the two "packages" call like:
install_python_modules( PACKAGE Package1
python/__init__.py python/module1.py python/module2.py )
, and:
install_python_modules( PACKAGE Package2
python/Package2/__init__.py python/Package2/module1.py
python/Package2/module2.py )
Now... The install_python_modules(...)
function defines two targets. One responsible for setting up soft links in the build directory that point at these files (in the source directory). And another one responsible for generating .pyc
files for them. So in total I have 4 custom targets in the demonstrator project.
The following are some of the valid targets for this Makefile:
... all (the default if no target is provided)
... clean
... depend
... rebuild_cache
... edit_cache
... Package1PythonBytecodeInstall
... Package1PythonInstall
... Package2PythonBytecodeInstall
... Package2PythonInstall
And now comes the issue... The makefile code generated for Package1 looks fine. The Package1/CMakeFiles/Package1PythonInstall.dir/build.make
file only describes how to create the soft links, and Package1/CMakeFiles/Package1PythonBytecodeInstall.dir/build.make
only describes how to make the .pyc
files. The way I intended it.
But Package2 ends up being very strange. The Package2/CMakeFiles/Package2PythonInstall.dir/build.make
file still looks fine, but Package2/CMakeFiles/Package2PythonBytecodeInstall.dir/build.make
ends up containing rules both for generating the .pyc
files and the soft links. The soft links for the python files held in Pakcage2 end up being generated by both the Package2PythonInstall
and the Package2PythonBytecodeInstall
targets!
And in some rare cases this can lead to a "build" failure. When two make processes decide to try to set up a soft link through the two targets at the same time. Resulting in something like:
At this point I'll stop with the description for now. (It's very long already...) While writing the demonstrator, I did find ways of avoiding the problem, which I'll implement for our main code. But since this very much smells like some bug in the CMake code somewhere, I wanted to let people know about it.
Cheers, Attila
P.S. I found this behaviour with every CMake version that I tried. Including 3.11.1, and yesterday's state of the master branch.