Configuring unattended automated tests

In Suitest there are two primary modes of using a device: automated and interactive. Automated mode is designed for unattended execution of many test automation scenarios on many devices. Interactive mode is designed for operating a single device and to experiment with a test on that device.


Setup a test pack

To configure execution of many tests on many devices in Suitest you'll need create a test pack. A test pack can contain concrete test scenarios created in the Test Editor or serve as a container for executions of JavaScript API based tests.

Test pack for test cases created in the Test Editor

It is often desired to run tests in several groups dedicated to a specific topic, for example: Player tests, Do-The-Basics-Work tests, etc. Before creating a test pack, you should decide what will be its purpose and choose a descriptive tag. Then you can add a given tag to all test scenarios that are related to the purpose. Then head to the Test packs page, press Add a test pack button. Suitest will ask you to specify the following:

Settings tab

Settings tab test pack example

  • Test pack tags - Combination of tags to choose a correct set of tests. You can use the following logical operators on tags:
    • negation (not tag1)
    • union (tag1 or tag2)
    • intersection (tag1 and tag2)
    • set difference (tag1 and not tag2)

You may need to use brackets in order to define more complex combinations. Every time you update tags inside the test, affected test packs are automatically updated to include (or exclude) this test scenario. The already planned executions will however remain unaffected.

  • App configuration - select app configuration with which you wish to run these tests.

Schedule tab

Schedule tab test pack example

  • Devices - select devices on which the tests should be executed. The devices need to support the platform specified by the selected configuration.

  • Execution schedule - select the time when to start executing these tests. Please note, however, that the tests will be planned for execution at the specified time. If some of the target devices will be busy or offline the execution on those devices will resume after they will become operational.

  • Notifications - select the type of notification and specify email recipients to where to send the notification. Additionally you can send a notification to a predefined Slack channel.

Important

Scheduled test packs use only the applied changed assets. All unapplied changes in the version are ignored by the scheduled test packs.

Executions tab

Test run executions tab

This tab contains the test pack executions. It gives an overview of how well the test pack runs have been executing. It also allows you to download an overview in PDF or Excel format.

You can cancel the entire Test pack execution while it is in progress on this tab as well.

Cancel Test pack run

Execution round results page

You can navigate to the execution round results page by clicking on the specific round on the Executions tab of the Test Pack.

Execution round page provides a better overview of the round result. You can also navigate to the specific test result directly from here.

Execution round page

Test pack for tests created with Suitest API

Test packs for tests created outside Test Editor have a bit different concept since the test definition is unknown to Suitest.

To create the test pack for JavaScript API based tests perform the same steps as above and then copy the test pack ID and submit it as a value for testpackId parameter in the Suitest launcher command.

Example launching test pack with Suitest launcher:

suitest automated \
    -k <token id> \
    -p <token password> \
    -t <test pack id> \
    npm run e2e

For JavaScript API based tests the test pack tag specified in the test pack configuration is ignored and all the tests launched by your test command will be included in the test pack. Similarly the test pack's configured execution schedule has no effect.

Since for JavaScript API based tests the control flow of the execution remains with your test runner, Suitest cannot tell when one test begins and the other ends. Consider the following example of a test suite consisting of two tests:

describe('my test suite', function() {
    it ('test 1', function() {
        //suitest api call 1
        //suitest api call 2
    });
    it('test 3', function() {
        //suitest api call 3
        //suitest api call 4
    });
})

Since Suitest makes no assumptions about the test runner you are about to use, it would not know that the API calls 1 and 2 belong to one test while 3 and 4 belong to the other. You can hint Suitest that you begin execution of a new test in your test suite with startTest() API call.

describe('my test suite', function() {
    it ('test 1', function() {
        // suitest api start test
        // suitest api call 1
        // suitest api call 2
    });

    it('test 2', function() {
        // suitest api start test
        // suitest api call 3
        // suitest api call 4
    });
})

Currently the execution results for test cases created with Suitest API are not stored on Suitest servers but this will be implemented in the nearest future.