[ENH] Compilation: Source file dependencies managment (#21321) · Issues · CMake / CMake

[ENH] Compilation: Source file dependencies managment

Currently, the management of source files dependencies (which enable automatic recompilation of sources when some dependencies are changed) is quite different depending of the generator used.
Currently, Ninja generator handles correctly the dependencies because it relies on the compiler itself to identify the dependencies. For Makefiles generators family, the dependencies are identified by CMake itself. Unfortunately, this scanner is quite primitive and does not identify correctly all the dependencies (see, for example #19573, #19071, #11770).

I think it is possible, for Makefiles generators, to adopt a strategy similar to the Ninja generator: relying on the compiler to compute the dependencies.
For that purpose, the following pattern can be used (ref: Auto Dependency Generation):

objdir/file.c.o : srcdir/file.c objdir/file.c.d
	gcc ... -c srcdir/file.c -o objdir/file.c.o ... -MT objdir/file.c.o -MD -MP -MF objdir/file.c.d


file.c.d:

include file.c.d

and the dependency file (file.c.d) should have the following information:

objdir/file.c.o: srcdir/file.c somedir/header.h

somedir/header.h:

To apply this approach, the compiler must have some capabilities regarding the dependencies generation: it is required to generate, as part of the dependencies file, a phony target for each dependency file (i.e. header.h: in the example) to avoid error during re-compilation if this file is non-longer available. For gcc, the option -MP generates these targets.

Currently, the following compilers support this capability: gcc, clang, nvcc (version >= 11.1), armclang (version >= 6.4), qcc (not fully sure about this one). So the most popular compilers are supported.
More compilers are able to generate dependencies but without the phony targets. For these compilers, a post-processing step can be included, as part of the compilation step, to add the phony targets.

So, I propose to enhance the Makefiles generators to adopt this approach when possible (but, in a first round, without post-processing) and continue to rely on the current behavior for the other cases.
And, to ensure maximum safety, an easy way to de-activate this capability and fall-back to the current behavior will be part of the new approach.

Currently, the management of source files dependencies (which enable automatic recompilation of sources when some dependencies are changed) is quite different depending of the generator used.  
Currently, `Ninja` generator handles correctly the dependencies because it relies on the compiler itself to identify the dependencies. For `Makefiles` generators family, the dependencies are identified by `CMake` itself. Unfortunately, this scanner is quite primitive and does not identify correctly all the dependencies (see, for example #19573, #19071, #11770).

I think it is possible, for `Makefiles` generators, to adopt a strategy similar to the `Ninja` generator: relying on the compiler to compute the dependencies.  
For that purpose, the following pattern can be used (ref: [Auto Dependency Generation](http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/#norule)):
```
objdir/file.c.o : srcdir/file.c objdir/file.c.d
	gcc ... -c srcdir/file.c -o objdir/file.c.o ... -MT objdir/file.c.o -MD -MP -MF objdir/file.c.d

file.c.d:

include file.c.d
```

and the dependency file (file.c.d) should have the following information:
```
objdir/file.c.o: srcdir/file.c somedir/header.h

somedir/header.h:
```

To apply this approach, the compiler must have some capabilities regarding the dependencies generation: it is required to generate, as part of the dependencies file, a phony target for each dependency file (i.e. `header.h:` in the example) to avoid error during re-compilation if this file is non-longer available. For `gcc`, the option `-MP` generates these targets.

Currently, the following compilers support this capability: gcc, clang, nvcc (version >= 11.1), armclang (version >= 6.4), qcc (not fully sure about this one). So the most popular compilers are supported.  
More compilers are able to generate dependencies but **without** the phony targets. For these compilers, a post-processing step can be included, as part of the compilation step, to add the phony targets.

So, I propose to enhance the `Makefiles` generators to adopt this approach when possible (but, in a first round, without post-processing) and continue to rely on the current behavior for the other cases.  
And, to ensure maximum safety, an easy way to de-activate this capability and fall-back to the current behavior will be part of the new approach.

To upload designs, you'll need to enable LFS and have admin enable hashed storage. More information