Did a few things to speed things up:
- eliminated some double<->integer conversions.
- repeated the last colour in the lut as a 4th "special" colour, eliminating need for a test in very hot function. Improved shift & scale calculations against pathological cases.
- use memcpy() (with fixed size) instead of manual copying byte by byte. Modern compilers optimize this well, as verified with godbolt.

Did some other cleanup while investigating my performance issues: added more const safety, fixed comments, etc.