Fix/workaround ASan stack buffer overrun
@ken-martin , when running with ASan on OS X 10.11.5, there are a bunch of tests that fail due to a stack buffer overrun (ex vtkRenderingCoreCxx-TestOpacity):
=================================================================
==22130==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff569198a0 at pc 0x000114d0e064 bp 0x7fff569196b0 sp 0x7fff56918e70
READ of size 256 at 0x7fff569198a0 thread T0
#0 0x114d0e063 in wrap_memmove (libclang_rt.asan_osx_dynamic.dylib+0x40063)
#1 0x7fff8b09d1ae in glBufferData_Exec (GLEngine+0x441ae)
#2 0x10b8ea205 in vtkOpenGLBufferObject::UploadInternal(void const*, unsigned long, vtkOpenGLBufferObject::ObjectType) vtkOpenGLBufferObject.cxx:149
#3 0x10bcc7b5e in bool vtkOpenGLBufferObject::Upload<float [4]>(float const (*) [4], unsigned long, vtkOpenGLBufferObject::ObjectType) vtkOpenGLBufferObject.h:140
#4 0x10bcc7123 in vtkOpenGLRenderUtilities::PrepFullScreenVAO(vtkOpenGLVertexArrayObject*, vtkShaderProgram*) vtkOpenGLRenderUtilities.cxx:176
#5 0x10bf93044 in vtkDualDepthPeelingPass::CopyOpaqueDepthBuffer() vtkDualDepthPeelingPass.cxx:589
#6 0x10bf8dcae in vtkDualDepthPeelingPass::Prepare() vtkDualDepthPeelingPass.cxx:515
#7 0x10bf8c64c in vtkDualDepthPeelingPass::Render(vtkRenderState const*) vtkDualDepthPeelingPass.cxx:79
#8 0x10bcf3361 in vtkOpenGLRenderer::DeviceRenderTranslucentPolygonalGeometry() vtkOpenGLRenderer.cxx:359
#9 0x10bceff1b in vtkOpenGLRenderer::UpdateGeometry() vtkOpenGLRenderer.cxx:231
#10 0x10bcedd2f in vtkOpenGLRenderer::DeviceRender() vtkOpenGLRenderer.cxx:163
#11 0x10ed173f1 in vtkRenderer::Render() vtkRenderer.cxx:325
#12 0x10ed10448 in vtkRendererCollection::Render() vtkRendererCollection.cxx:51
#13 0x10ed602bb in vtkRenderWindow::DoStereoRender() vtkRenderWindow.cxx:768
#14 0x10ed5fd37 in vtkRenderWindow::DoFDRender() vtkRenderWindow.cxx:737
#15 0x10ed5e420 in vtkRenderWindow::DoAARender() vtkRenderWindow.cxx:616
#16 0x10ed5b263 in vtkRenderWindow::Render() vtkRenderWindow.cxx:432
#17 0x10bccfb91 in vtkOpenGLRenderWindow::Render() vtkOpenGLRenderWindow.cxx:661
#18 0x10937423f in TestOpacity(int, char**) TestOpacity.cxx:168
#19 0x1092f1d31 in main vtkRenderingCoreCxxTests.cxx:522
#20 0x7fff8639f5ac in start (libdyld.dylib+0x35ac)
#21 0x7 (<unknown module>)
Address 0x7fff569198a0 is located in stack of thread T0 at offset 96 in frame
#0 0x10bcc6e6f in vtkOpenGLRenderUtilities::PrepFullScreenVAO(vtkOpenGLVertexArrayObject*, vtkShaderProgram*) vtkOpenGLRenderUtilities.cxx:158
This frame has 9 object(s):
[32, 96) 'verts'
[128, 129) 'endl' <== Memory access at offset 96 partially underflows this variable
[144, 176) 'vtkmsg' <== Memory access at offset 96 partially underflows this variable
[208, 232) '' <== Memory access at offset 96 partially underflows this variable
[272, 273) 'endl1' <== Memory access at offset 96 partially underflows this variable
[288, 320) 'vtkmsg2' <== Memory access at offset 96 partially underflows this variable
[352, 376) ''
[416, 417) 'endl3'
[432, 464) 'vtkmsg4'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow (libclang_rt.asan_osx_dynamic.dylib+0x40063) in wrap_memmove
Shadow bytes around the buggy address:
0x1fffead232c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1fffead232d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1fffead232e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1fffead232f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1fffead23300: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00
=>0x1fffead23310: 00 00 00 00[f2]f2 f2 f2 01 f2 00 00 00 00 f2 f2
0x1fffead23320: f2 f2 00 00 00 f2 f2 f2 f2 f2 01 f2 00 00 00 00
0x1fffead23330: f2 f2 f2 f2 00 00 00 f2 f2 f2 f2 f2 01 f2 00 00
0x1fffead23340: 00 00 f3 f3 f3 f3 f3 f3 00 00 00 00 00 00 00 00
0x1fffead23350: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1fffead23360: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==22130==ABORTING
=========================================================
Process id 22130 Caught SIGABRT
Program Stack:
0x7fff84f3052a : _sigtramp [(libsystem_platform.dylib) ???:-1]
0x0 : ??? [(???) ???:-1]
0x7fff85cc86e7 : abort [(libsystem_c.dylib) ???:-1]
0x114d35876 : __sanitizer::Abort() [(libclang_rt.asan_osx_dynamic.dylib) ???:-1]
0x114d0e086 : wrap_memmove [(libclang_rt.asan_osx_dynamic.dylib) ???:-1]
0x7fff8b09d1af : glBufferData_Exec [(GLEngine) ???:-1]
0x10b8ea206 : vtkOpenGLBufferObject::UploadInternal(void const*, unsigned long, vtkOpenGLBufferObject::ObjectType) [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x10bcc7b5f : bool vtkOpenGLBufferObject::Upload<float [4]>(float const (*) [4], unsigned long, vtkOpenGLBufferObject::ObjectType) [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x10bcc7124 : vtkOpenGLRenderUtilities::PrepFullScreenVAO(vtkOpenGLVertexArrayObject*, vtkShaderProgram*) [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x10bf93045 : vtkDualDepthPeelingPass::CopyOpaqueDepthBuffer() [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x10bf8dcaf : vtkDualDepthPeelingPass::Prepare() [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x10bf8c64d : vtkDualDepthPeelingPass::Render(vtkRenderState const*) [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x10bcf3362 : vtkOpenGLRenderer::DeviceRenderTranslucentPolygonalGeometry() [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x10bceff1c : vtkOpenGLRenderer::UpdateGeometry() [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x10bcedd30 : vtkOpenGLRenderer::DeviceRender() [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x10ed173f2 : vtkRenderer::Render() [(libvtkRenderingCore-7.1.1.dylib) ???:-1]
0x10ed10449 : vtkRendererCollection::Render() [(libvtkRenderingCore-7.1.1.dylib) ???:-1]
0x10ed602bc : vtkRenderWindow::DoStereoRender() [(libvtkRenderingCore-7.1.1.dylib) ???:-1]
0x10ed5fd38 : vtkRenderWindow::DoFDRender() [(libvtkRenderingCore-7.1.1.dylib) ???:-1]
0x10ed5e421 : vtkRenderWindow::DoAARender() [(libvtkRenderingCore-7.1.1.dylib) ???:-1]
0x10ed5b264 : vtkRenderWindow::Render() [(libvtkRenderingCore-7.1.1.dylib) ???:-1]
0x10bccfb92 : vtkOpenGLRenderWindow::Render() [(libvtkRenderingOpenGL2-7.1.1.dylib) ???:-1]
0x109374240 : TestOpacity(int, char**) [(vtkRenderingCoreCxxTests) ???:-1]
0x1092f1d32 : main [(vtkRenderingCoreCxxTests) ???:-1]
0x7fff8639f5ad : start [(libdyld.dylib) ???:-1]
=========================================================I don't actually see a problem with the code, but if I use heap memory ASan does not complain, as per this hack.
Any thoughts?