This also allows us to add a more efficient code path, using memcpy, when only the X array is generated.