Implement Test Image Comparison
Reference to kmorel/ecp-vtk-m-project-management#19
Test Image Comparison
Testing visualization algorithms on more than trivial problems can be tricky due in part to the difficulty in generating correct output that can be validated in a reasonable amount of time. Instead of directly comparing the values associated with these algorithms, it should be possible to instead directly compare rendered images against one another. This requires having an available "correct" baseline image and then generating a render of a specific filter/algorithm at test time. The differences of these images can be calculated using the pixel values, and thresholds can be set to determine when an image is close enough to the baseline to pass the test.
Borrowing From VTK
VTK supports the comparison of rendered images with stored images in PNG format using an ImageDifference algorithm that directly compares two rgb unsigned char images pixel by pixel. It supports checking for pixel shift differences, as well as setting allowable thresholds for error on each individual pixel. The implementation takes advantage of additional VTK functionality to retrieve extent information related to the provided images, crop the images to fit a bounding box, and read PNGs from a source. While VTKm may not have all the rendering functionality required to directly mimic the process that VTK uses to compare images, there are enough facilities in place with which image comparison can be performed with minimal change to the rendering capabilities available.
Additional Features
In order to implement an image comparison capability in VTKm similar to the one used in VTK, the following features need to be implemented and updated. Example code stubs are provided when possible/useful:
- Capability to read image files (ppm/png supported) and store into a useful vtkm internal type
-
ColorBufferType::PortalConstControl
, ieArrayHandle<vtkm::Vec4f_32>
- Some other vector of floats/shorts/whatever type works
-
// basic image class
class ImageBase
{
/// Constructs an ImageBase, filling in the ImageDataSet from the supplied
/// canvas's colorBuffer.
///
explicit ImageBase(vtkm::rendering::Canvas canvas);
/// Reads image data from a file. Meant to be implemented in overriden
/// image-specific classes
///
virtual void ReadFromFile(const std::string& fileName);
/// Writes an store ImageDataSet to afile. Meant to be implemented in
/// overriden image-specific classes
///
virtual void WriteToFile(const std::string& fileName) const;
void FillImageDataSet(vtkm::cont::ArrayHandle<vtkm::Vec4f_32> colorBuffer);
void CreateImageDataSet(vtkm::Id width, vtkm::Id height)
{
vtkm::cont::DataSetBuilderUniform dsb;
constexpr vtkm::Id2 dimensions(width, height);
this->ImageDataSet = dsb.Create(dimensions);
}
vtkm::cont::DataSet ImageDataSet;
};
class PNG : public ImageBase
{
PNG(std::string filename)
// concrete read function
// concrete write function
};
class PPM : public ImageBase
{
PPM(std::string filename)
// concrete read function
// concrete write function
}
- Capability to Ensure correct image bounding/sizing via extent comparison (or via size comparison of the vtkm internal type)
vtkm::rendering::CanvasRayTracer canvasTest(300, 300);
auto baselineImageData = vtkm::io::PPM("ContourTest.ppm");
vtkm::filter::compareExtents(canvasTest, canvasBaseline);
- Capability to perform image comparison though error thresholding of rgb/intensity values while allowing pixel shift (algorithm work). Will work via a filter.
double thresholdError = 0.0;
// Loop through x axis
// Loop through y axis
// optionally loop over pixel shift range
// Grab RGB values
// Perform value comparison
thresholdError = // some math
- Helper test functions to easily call image comparison code and verify imaging thresholds
vtkmCompareImagesThreshold(char* FileName, vtkm::rendering::View view)
- Assist in debugging by displaying useful log messages and outputting image differences as a usable file
- New features are Documented in the Users-Guide
Optional but Useful Features
The following features are not necessary, but would greatly improve the usability of the image comparison framework:
- Capability to write rendered views as PNGs (currently only support writing rendered views as ppm).
- Additional CMake functions to assist in passing in file names/directories for test images via run-time parameters
- Implement a fall-back system similar to VTK that parses additional back-up images for testing on initial image failure.