Wishlist: Flaky Test Tag
For tests with non-deterministic results, like when using threads, we might find the idea of tagging a test flaky useful. The TRI/MIT project Drake uses this feature with Bazel. So in Bazel the idea is the following, it runs a flaky test at most
n number of times and declares it a failure iff it fails all the
n times. 3 is their default value of
To get some ballpark figures, consider a test with 60% passing rate, the probability of it failing would be 6.4% if declared flaky in default settings.