Skip to content

Fixed undefined ptr overflow

Sean McBride requested to merge seanm/vtk:UB-ptroffset into master

Fixed the same UB in 3 tests:

vtkViewsInfovisCxx-TestHierarchicalGraphView vtkViewsInfovisCxx-TestIconGlyphFilter vtkViewsInfovisCxx-TestSpanTreeLayoutStrategy

Where GetBuffer() returned 0 and MaxId was -1, resulting in a pointer to 0xFFFFFFFF. Even though 1 is added right after, it's apparently UB to even have this temporary overflowed pointer. Crazy. Anyway, calculating the offset first seems no worse for readability and correctness.

Found by clang 5 UBSan.

Merge request reports