vtkIntersectionPolyDataFilter with tolerance=0 writes to invalid memory causing segfault
The recently rewritten vtkIntersectionPolyDataFilter::TriangleTriangleIntersection
causes a segfault by writing to a C array out of bounds because of the initial values assigned to local ts1/ts2 variables of 50. When within the loop the condition for setting these to a value index of 0, 1, or 2, the next swap(t1[ts1], t1[2])
and/or swap(t2[ts2], t2[2])
writes to memory address t1[50]
and/or t2[50]
, which may cause undefined behaviour for the program (or luckily in my case a segfault).
This issue comes up with a tolerance=0
(in order to have the same conditions as before the change) when the t
line parameter returned by vtkPlane::IntersectWithLine
is either 0 or 1. The condition
if (t < 1+tolerance && t > 1-tolerance)
{
ts1 = index1;
}
will then never be true
. One (or both) of the comparison operators should probably include the equal sign.
In my application, setting tolerance to the default used by vtkIntersectionPolyDataFilter
, i.e., 1e-6, resolved the issue.