Configuring unattended automated tests

In Suitest there are two primary modes of using a device: automated and interactive. Automated mode is designed for unattended execution of many test automation scenarios on many devices. Interactive mode is designed for operating a single device and to experiment with a test on that device.


Setup a test pack

To configure execution of many tests on many devices in Suitest you'll need create a test pack. A test pack can contain concrete test scenarios created in the Test Editor or serve as a container for executions of JavaScript API based tests.

Example test pack with 3 tests created in the Test Editor
Example test pack with 3 tests created in the Test Editor

Test pack for test cases created in the Test Editor

It is often desired to run tests in several groups dedicated to a specific topic, for example: Player tests, Do-The-Basics-Work tests, etc. Before creating a test pack decide what will be its purpose and choose a descriptive tag. Then add this tag to all test scenarios that you would like to see in this test pack. Then head to the Test Packs area of your application, press Create a new test pack. Suitest will ask you to specify the following:

  1. Test tags - One or more tags to look up the scenarios. Scenarios containing any of the mentioned labels will be included into this test pack.

  2. App configuration - select app configuration with which you wish to run these tests.

  3. Target devices - select devices on which the tests should be executed. The devices need to support the platform specified by the selected configuration.

  4. Execution schedule - select the time when to start executing these tests. Please note, however, that the tests will be planned for execution at the specified time. If some of the target devices will be busy or offline the execution on those devices will resume after they will become operational.

  5. Notification preferences - select the type of notification and specify email recipients to where to send the notification. Additionally you can send a notification to a predefined Slack channel.

Every time you add or remove tags in a test scenario the test packs are automatically updated to include (or exclude) this test scenario. The already planned executions will however remain unaffected.

Test pack for tests created with Suitest API

Test packs for tests created outside Test Editor have a bit different concept since the test definition is unknown to Suitest.

To create the test pack for JavaScript API based tests perform the same steps as above and then copy the test pack ID and submit it as a value for testpackId parameter in the Suitest launcher command.

Example launching test pack with Suitest launcher:

suitest automated \
    -k <token key> \
    -p <token password> \
    -t <test pack id> \
    npm run e2e

For JavaScript API based tests the test pack tag specified in the test pack configuration is ignored and all the tests launched by your test command will be included in the test pack. Similarly the test pack's configured execution schedule has no effect.

Since for JavaScript API based tests the control flow of the execution remains with your test runner, Suitest cannot tell when one test begins and the other ends. Consider the following example of a test suite consisting of two tests:

describe('my test suite', function() {
    it ('test 1', function() {
        //suitest api call 1
        //suitest api call 2
    });
    it('test 3', function() {
        //suitest api call 3
        //suitest api call 4
    });
})

Since Suitest makes no assumptions about the test runner you are about to use, it would not know that the API calls 1 and 2 belong to one test while 3 and 4 belong to the other. You can hint Suitest that you begin execution of a new test in your test suite with startTest() API call.

describe('my test suite', function() {
    it ('test 1', function() {
        // suitest api start test
        // suitest api call 1
        // suitest api call 2
    });

    it('test 2', function() {
        // suitest api start test
        // suitest api call 3
        // suitest api call 4
    });
})

Currently the execution results for test cases created with Suitest API are not stored on Suitest servers but this will be implemented in the nearest future.