Re-executing failed tests

When some tests in your test pack execution run fail, you may want to re-execute them, but without the complete test pack execution.

Each test pack execution consists of iterations, so you are able to re-execute any iteration and by that create a new one.

Strategy selection

When re-executing an iteration, Suitest offers a selection of strategy. Strategies should help you to select the right combinations of test executions to be re-executed.

Suitest considers the following execution statuses as failed:

  • fatal
  • failed
  • aborted

Therefore, the following execution statuses are considered as success:

  • passed
  • passed with warnings
  • exited

You can select from the strategies below.

Only failed

Re-executing only those tests on those devices where the original execution failed.

"Only failed" scheme

Flaky test

Under the expectation that failing tests are "flaky" (unreliable), re-executing the tests that failed across all devices defined inside the test pack.

"Flaky test" scheme

Flaky device

Under the expectation that failing devices are "flaky" (unreliable), re-executing all tests on devices where some tests failed.

"Flaky device" scheme

Flaky test or device

Under the expectation that both tests and devices may be "flaky", re-executing all tests that failed at least once on all devices where at least one test failed.

"Flaky test or device" tests scheme

Whole execution

Re-executing the whole iteration, no matter which tests failed in the original one.

"Whole execution" scheme