hdf5: fix long long format detection

The code does a loop itself, so just run it once and use its output as
the result.

In fact, this code would always return "ll" because if the file had a
right answer, it would return 0 for it, but then think it was "ll" that
passed, not one of the formats in the file.