Rendering/Matplotlib: missing GIL lock

dear experts

we built Paraview with external dependencies (system ones) like e.g system matplotlib, system freetype, etc.

Unfortunately with 5.11.0, if I want to label a variable with latex fonts, Paraview crashes with attached backtrace.

Before investigating this further, I can see that similar issues have been already reported but not further discussed.

I welcome hints or suggestions.

thanks

pv-5.11.txt

changed the description

What version of matplotlib are you building against?

Before investigating this further, I can see that similar issues have been already reported but not further discussed.

Could you link those to those in the issue description?

Hi Cory, indeed I experience this with several versions for matplotlib, e.g.

• Ubuntu 22.04: python3-matplotlib/jammy,now 3.5.1-2build1 amd64
• Fedora 32: matplolib 3.2.1-1.fc32

It used to work with Paraview 5.9.

I setup VTK_MATPLOTLIB_DEBUG=1 to see some useful messages but this did not report any issue about the import of matplotlib which seems to be OK, since I can read:

(  40.371s) [paraview        ]vtkMatplotlibMathTextUt:83    WARN| Initializing Python, if not already.
(  41.672s) [paraview        ]vtkMatplotlibMathTextUt:85    WARN| Attempting to import matplotlib.
(  42.398s) [paraview        ]vtkMatplotlibMathTextUt:112   WARN| Successfully imported matplotlib.

Moving forward, I ran with a debugger step by step and I can see that an exception is raised at this line:

vtkSmartPyObject mplBase(PyImport_ImportModule("matplotlib"));

It seems that the conversion of the PyObject* vtkSmartPyObject somehow fails.

Regards.

Thanks for the update.

@dgobbi?

Though the traceback says it crashes under PyImport_ImportModule, so vtkSmartPyObject isn't in the picture yet. Any chance we can see what libc function it is that is failing?

/cc @ben.boeckel

The issue is due to this import:

PyImport_ImportModule("matplotlib")

which strangely works in vtkMatplotlibMathTextUtilities::CheckMPLAvailability() but raises an exception if called in:

vtkMatplotlibMathTextUtilities::SetMathTextFont(vtkTextProperty* tprop)

I must confess that I am stuck and don't see what I can do next and welcome inputs.

Hi, can you point me to where I can find the logs of official tarbals to check which versions are used and check against our log files ? Is this available somewhere on Paraview gitlab ? thanks

Any of the logs here should be sufficient: https://gitlab.kitware.com/paraview/paraview-superbuild/-/pipelines?page=1&scope=all&ref=release

Hi, thanks to the sharp eye of a colleague of mine, the issue is understood. A call to vtkPythonScopeGilEnsurer was missing in method SetMathTextFont.

--- ParaView_ref/VTK/Rendering/Matplotlib/vtkMatplotlibMathTextUtilities.cxx    2022-12-01 10:35:02.476305685 +0100
+++ ParaView_dev/VTK/Rendering/Matplotlib/vtkMatplotlibMathTextUtilities.cxx    2022-12-01 10:35:41.811632384 +0100
@@ -790,6 +790,7 @@
 //------------------------------------------------------------------------------
 bool vtkMatplotlibMathTextUtilities::SetMathTextFont(vtkTextProperty* tprop)
 {
+  vtkPythonScopeGilEnsurer gilEnsurer;
   vtkSmartPyObject mplBase(PyImport_ImportModule("matplotlib"));
   if (this->CheckForError(mplBase))
   {

Strangely VTK_PYTHON_FULL_THREADSAFE cmake flag (ON by default) is not suffisant here. We are currently testing it on several platforms and will report about additional issues here.

tested OK on Fedora 32, 34, CentOS 7, Ubuntu 22.04 and Windows 10

closed

paraview-5.11.0-issueID-21676.patch

The issue is not fixed yet, please do not closed unfixed issues just because a patch exists (as others will need this). Moving to VTK where the actual problem exists.

reopened

moved from paraview/paraview#21676 (moved)

changed title from Paraview 5.11 and Latex fonts to Rendering/Matplotlib: missing GIL lock

added lang:python triage:haspatch labels

mentioned in merge request !9721 (closed)

I've drafted !9721 (closed) so that we can get this merged on our side.

We will have to adjust the testing configurations so that CI can catch missing GIL locks like this one.

Indeed, which confi is missing in your opinion ?

The one that is missing is that one that would catch the error! Some detective work is needed to find out which one that is.

I can reproduce the crash with these settings (on linux, though it probably applies to other platforms as well).

VTK_MODULE_ENABLE_VTK_RenderingMatplotlib=WANT
VTK_PYTHON_FULL_THREADSAFE=ON

VTK_PYTHON_FULL_THREADSAFE is problematic because, as noted in #18102 (closed), it is undocumented and doesn't even appear in any of VTK's cmake files. All follow-ups will be in that issue.

Can you reproduce even with !9722 (merged) applied ?

The crash does not occur when !9722 (merged) is applied. But we need CI builds with VTK_PYTHON_FULL_THREADSAFE=ON to catch similar GIL issues in the future.

Added to the list: #18100

mentioned in merge request !9722 (merged)

closed with commit d77900f8