1. 24 Jun, 2020 1 commit
  2. 10 Apr, 2020 1 commit
  3. 27 Mar, 2020 1 commit
    • Ryan Thornton's avatar
      GoogleTest: Parameterize tests to check PRE_TEST/POST_BUILD discovery mode · 1ba4cb56
      Ryan Thornton authored
      Now, the unit tests are ran twice -- once with POST_BUILD (i.e. default mode)
      and again with PRE_TEST (i.e. new discovery mode).
      Both modes of setting gtest discovery mode are also tested:
      1. Using the global override (i.e. CMAKE_GTEST_DISCOVER_TESTS_DISCOVERY_MODE)
      2. Explicitly passing DISCOVERY_MODE in calls to gtest_discover_tests (in GoogleTestDiscoveryTimeout.cmake)
      The goal is to show that the new PRE_TEST discovery mode does not break existing behavior
      (i.e. should not break POST_BUILD mode) and should also pass the same tests
      in the same way.
      A few non trivial implementation details worth noting:
      1. Refactoring discovery_timeout_test into own project
      Originally, I tried doing:
      Without changing the internal structure of run_GoogleTest.
      But since discovery_timeout_test is part of the same project as the other tests,
      and CTest include files always get evaluated and that's where test discovery occurs,
      this means every other test now notices the timeout problem when running in PRE_TEST mode.
      As a result, keeping the existing test structure meant that each existing test
      (and any new test) would need to have its own PRE_TEST / POST_BUILD variant for stderr and stdout
      in order to handle the case where discovery_timeout_test timed out.
      This exponential increase in test output files introduced unnecessary complexity
      and made it more cumbersome to work on test cases.
      Why should an unrelated test case care about discovery_timeout_test?
      So, to fix that issue, the tests were broken apart into two main groups:
      1. run_GoogleTest_discovery_timeout (the test dealing with discovery_timeout_test)
      2. run_GoogleTest (everything else)
      This isolates the PRE_TEST / POST_BUILD timeout variants to a single test case.
      And the other test cases remain unchanged -- further driving home the point that
      DISCOVERY_MODE shouldn't change existing behavior.
      2. Different number of PRE_TEST / POST_BUILD file variants
      On the PRE_TEST path, different build systems / compilers (i.e. MSBuild and ninja/gcc)
      produces different build output when building discovery_timeout_test,
      but we don't actually care what it is, just as long as it builds
      This the fundamental difference in behavior between POST_BUILD (which would have failed)
      and PRE_TEST (which doesn't) and is the reason why we don't need
      a GoogleTest-discovery-build-result.txt or GoogleTest-discovery-build-stdout.txt
      3. Fix flaky discovery timeout test
      The test expects to see:
      > Output:
      >  timeout
      >    case.
      But sometimes, the test would only produce:
      > Output:
      >  timout
      In certain environments, specifically when built with OpenWatcom 1.4,
      and while the build server was under heavy load (i.e. running many tests in parallel),
      std::endl behaves inconsistently and doesn't completely
      flush std::cout when the program is terminated due to timeout.
      This results in inconsistent test failures because the actual output
      doesn't fully match what's expected.
      At first we tried adding an additional:
          std::cout << std::flush
      That didn't work. But using C-style printf() and fflush() appears to do
      the trick:
      > This time I managed to get on the machine while it was still busy doing other nightly builds
      > and could reproduce the problem reliably. With that I was finally able to find a fix.
      > It turns out my earlier hypothesis that C++ stream flushing was not working on the old compiler was correct,
      > but even .flush() is not enough.
      > I changed it to use C-style printf() and fflush() and now the test passes on that build.
      > -- Brad King <brad.king@kitware.com>
      Co-authored-by: Ryan Thornton's avatarRyan Thornton <ThorntonRyan@JohnDeere.com>
      Co-authored-by: default avatarKevin Puetz <PuetzKevinA@JohnDeere.com>
  4. 15 Mar, 2018 1 commit
    • Craig Scott's avatar
      GoogleTest: Rename TIMEOUT parameter to avoid clash · 96fdde26
      Craig Scott authored and Brad King's avatar Brad King committed
      In gtest_discover_tests(), the TIMEOUT keyword was making it
      impossible to set the TIMEOUT test property via the PROPERTIES
      keyword. This would be a frequent case, but it doesn't complain
      and instead silently does something different to what would
      normally be expected. The TIMEOUT keyword has been renamed
      to DISCOVERY_TIMEOUT, thereby removing the clash.
      This is a breaking change. 3.10.1 and 3.10.2 were the only versions
      that supported the TIMEOUT keyword and uses of it were likely
      not working as intended.
      Fixes: #17801
  5. 06 Dec, 2017 1 commit
    • Matthew Woehlke's avatar
      GoogleTest: Add timeout to discovery · 29731d89
      Matthew Woehlke authored and Brad King's avatar Brad King committed
      Add a TIMEOUT option to gtest_discover_tests. This provides a
      work-around in case a test goes out to lunch, rather than causing the
      build to simply hang. (Although this is still a problem with the user's
      project, hanging the build is not cool, especially in the case of
      automatically running CI builds. It is much preferred that the build
      should actively fail in this case, and it is trivially easy for us to
      implement that.)
  6. 21 Nov, 2017 1 commit
    • Matthew Woehlke's avatar
      GoogleTest: Fix multiple discovery on same target · 70f9f62d
      Matthew Woehlke authored and Brad King's avatar Brad King committed
      According to the documentation, tests can be discovered for a target
      multiple times by using a different prefix and/or suffix to ensure name
      uniqueness. However, while this worked for gtest_add_tests, it did not
      work with gtest_discover_tests because the generated file that sets up
      the tests was named based only on the target name, and so subsequent
      discovery from the same target would clobber earlier discovery.
      Fix this by introducing a counter that records how many times discovery
      has been used on a target, and use this to generate unique names of the
      generated test list files.
  7. 27 Jul, 2017 1 commit
    • Matthew Woehlke's avatar
      Add dynamic test discovery for for Google Test · bfcda401
      Matthew Woehlke authored
      Add a new gtest_discover_tests function to GoogleTest.cmake,
      implementing dynamic test discovery (i.e. tests are discovered by
      actually running the test executable and asking for the list of
      available tests, which is used to dynamically declare the tests) rather
      than the source-parsing approach used by gtest_add_tests. Compared to
      the source-parsing approach, this has the advantage of being robust
      against users declaring tests in unusual ways, and much better support
      for advanced features such as parameterized tests.
      A unit test, modeled after the TEST_INCLUDE_DIR[S] test, is also
      included. Note that the unit test does not actually require that Google
      Test is available. The new functionality does not actually depend on
      Google Test as such; it only requires that the test executable lists
      tests in the expected format when invoked with --gtest_list_tests, which
      the unit test can fake readily.