librhash: Avoid signed left-shift overflow

Fix `rhash_md5_final` to use unsigned integers for left shifting to
avoid the possibility of undefined overflow behavior.