Skip to content

Utilities/Sphinx: Fix parallel documentation builds

In !7822 (merged) we declared the domain as parallel-safe without actually implementing the required merge_domaindata method.

Since then, !8305 (merged) changed how we store resolved references, such that the fix for 3.26 does not work with 3.27. Apply a separate fix for each version.

Original Description

After updating to the latest version of the moderncmakedomain and it looks like some changes need to happen to make it parallel safe unfortunately. Here is the sanitized error log:

# Sphinx version: 4.5.0
# Python version: 3.8.10 (CPython)
# Docutils version: 0.17.1 release
# Jinja2 version: 3.0.3
# Last messages:
#   reading sources... [...clipped log...]
#   
#   waiting for workers...
# Loaded extensions:
#   [...clipped log...]
#   sphinxcontrib.moderncmakedomain (unknown version) from /home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinxcontrib/moderncmakedomain/__init__.py
Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/util/parallel.py", line 93, in join
    if not self._join_one():
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/util/parallel.py", line 117, in _join_one
    self._result_funcs.pop(tid)(self._args.pop(tid), result)
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 445, in merge
    self.env.merge_info_from(docs, env, self.app)
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/environment/__init__.py", line 309, in merge_info_from
    domain.merge_domaindata(docnames, other.domaindata[domainname])
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/domains/__init__.py", line 295, in merge_domaindata
    raise NotImplementedError('merge_domaindata must be implemented in %s '
NotImplementedError: merge_domaindata must be implemented in <class 'sphinxcontrib.moderncmakedomain.cmake.CMakeDomain'> to be able to do parallel builds!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app.build(args.force_all, filenames)
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/application.py", line 330, in build
    self.builder.build_update()
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 286, in build_update
    self.build(to_build,
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 300, in build
    updated_docnames = set(self.read())
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 405, in read
    self._read_parallel(docnames, nproc=self.app.parallel)
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 456, in _read_parallel
    tasks.join()
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/util/parallel.py", line 97, in join
    self.terminate()
  File "/home/user/.cache/pypoetry/virtualenvs/user-5_SF86Eq-py3.8/lib/python3.8/site-packages/sphinx/util/parallel.py", line 103, in terminate
    self._result_funcs.pop(tid)
KeyError: 3

The key error being:

NotImplementedError: merge_domaindata must be implemented in <class 'sphinxcontrib.moderncmakedomain.cmake.CMakeDomain'> to be able to do parallel builds!

I looking into it a little and found some other domain extensions that are parallel safe to compare their implementations, here is the PHP domain as an example. I used that as an inspiration to make this PR.

Issue: #24076 (closed)
Co-authored-by: @matthew-woehlke
Backport: release
Topic-rename: sphinx-domain-parallel

Edited by Brad King

Merge request reports