Utilities/Sphinx: Fix parallel documentation builds
In !7822 (merged) we declared the domain as parallel-safe without actually implementing the required merge_domaindata
method.
Since then, !8305 (merged) changed how we store resolved references, such that the fix for 3.26 does not work with 3.27. Apply a separate fix for each version.
Original Description
After updating to the latest version of the moderncmakedomain and it looks like some changes need to happen to make it parallel safe unfortunately. Here is the sanitized error log:
# Sphinx version: 4.5.0
# Python version: 3.8.10 (CPython)
# Docutils version: 0.17.1 release
# Jinja2 version: 3.0.3
# Last messages:
# reading sources... [...clipped log...]
#
# waiting for workers...
# Loaded extensions:
# [...clipped log...]
# sphinxcontrib.moderncmakedomain (unknown version) from /home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinxcontrib/moderncmakedomain/__init__.py
Traceback (most recent call last):
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/util/parallel.py", line 93, in join
if not self._join_one():
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/util/parallel.py", line 117, in _join_one
self._result_funcs.pop(tid)(self._args.pop(tid), result)
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 445, in merge
self.env.merge_info_from(docs, env, self.app)
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 309, in merge_info_from
domain.merge_domaindata(docnames, other.domaindata[domainname])
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/domains/__init__.py", line 295, in merge_domaindata
raise NotImplementedError('merge_domaindata must be implemented in %s '
NotImplementedError: merge_domaindata must be implemented in <class 'sphinxcontrib.moderncmakedomain.cmake.CMakeDomain'> to be able to do parallel builds!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
app.build(args.force_all, filenames)
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/application.py", line 330, in build
self.builder.build_update()
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 286, in build_update
self.build(to_build,
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 300, in build
updated_docnames = set(self.read())
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 405, in read
self._read_parallel(docnames, nproc=self.app.parallel)
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 456, in _read_parallel
tasks.join()
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/util/parallel.py", line 97, in join
self.terminate()
File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/util/parallel.py", line 103, in terminate
self._result_funcs.pop(tid)
KeyError: 3
The key error being:
NotImplementedError: merge_domaindata must be implemented in <class 'sphinxcontrib.moderncmakedomain.cmake.CMakeDomain'> to be able to do parallel builds!
I looking into it a little and found some other domain extensions that are parallel safe to compare their implementations, here is the PHP domain as an example. I used that as an inspiration to make this PR.
Issue: #24076 (closed)
Co-authored-by: @matthew-woehlke
Backport: release
Topic-rename: sphinx-domain-parallel