ColorTable now converts to and from double less often.

Previously we would convert to double to perform some operations instead of always staying in float space.