Restored remainder operator in vtkSelectPolyData::IsBoundaryEdge

This mostly reverts 926ec960 which incorrectly removed the remainder operator while fixing a buffer under-run due to a negative index.

Restored the remainder operator, but make sure to never give it a negative number.