undefined behavior in vtkCellArray::Allocate
We recently started using the undefined behavior sanitzier (UBSan,
-fsanitize=undefined in clang) and ran into UB in
vtkCellArray (which we use extensively).
The error reported by UBSan is:
VTK/Common/DataModel/vtkCellArray.h:59:23: runtime error: member call on address 0x611000009140 which does not point to an object of type 'vtkGenericDataArray<vtkAOSDataArrayTemplate<long long>, long long>' 0x611000009140: note: object is of type 'vtkIdTypeArray' 08 00 80 70 60 75 cf 06 01 00 00 00 be be be be 01 00 00 00 00 00 00 00 00 00 00 00 00 be be be ^~~~~~~~~~~~~~~~~~~~~~~ vptr for 'vtkIdTypeArray' SUMMARY: AddressSanitizer: undefined-behavior VTK/Common/DataModel/vtkCellArray.h:59:23
IIUC, and without fully understanding
vtkAbstractArray, the problem seems to be that
vtkAbstractArray:: virtual Allocate(vtkIdType, vtkIdType) is set in the vtable to dispatch to the override in
vtkGenericDataArray:: Allocate(vtkIdType, vtkIdType), which is marked
override (so this seems intended), but the type of the actual object it is being invoked on (the type of
decltype(*this)) is actually a
vtkIdTypeArray, which seems to be a completely unrelated type.
a:b mean "a inherits b", then:
vtkIdTypeArray: vtkDataArray: vtkAbstractArray and
vtkGenericDataArray: vtkDataArray: vtkAbstractArray. Note how
vtkGenericDataArray are not related. Also note that neither
vtkAbstractArray::Allocate, which is pure virtual. So actually, calling
vtkIdTypeArray->Allocate on a
vtkIdTypeArray should always fail. Why does
vtkGenericDataArray->Allocate get called instead? No idea yet.