Configuring unattended automated tests

In Suitest there are two primary modes of using a device: automated and interactive. Automated mode is designed for unattended execution of many test automation scenarios on many devices. Interactive mode is designed for operating a single device and to experiment with a test on that device.

Setup a test pack

To configure execution of many tests on many devices in Suitest you'll need create a test pack. A test pack can contain concrete test scenarios created in the Test Editor.

If you want to use our Suitest JavaScript API, please see related documentation.

Test pack for test cases created in the Test editor

It is often desired to run tests in several groups dedicated to a specific topic, for example: Player tests, Do-The-Basics-Work tests, etc. Before creating a test pack, you should decide what will be its purpose and choose a descriptive tag. Then you can add a given tag to all test scenarios that are related to the purpose. Then head to the Test packs page, press Add a test pack button. Suitest will ask you to specify the following:

Settings tab

Settings tab test pack example

  • Test pack tags - Combination of tags to choose a correct set of tests. You can use the following logical operators on tags:

Tag operators explanation

You may need to use brackets in order to define more complex combinations. Every time you update tags inside the test, affected test packs are automatically updated to include (or exclude) this test scenario. The already planned executions will however remain unaffected.

  • App configuration - select app configuration with which you wish to run these tests.

Schedule tab

Schedule tab test pack example

  • Devices - select devices on which the tests should be executed. The devices need to support the platform specified by the selected configuration. Every test included in the test pack will be executed on each selected device.

    TIP: If you would like to speed up the execution of a set of tests by running them in parallel, you can split test packs into smaller, separate test packs and select individual devices to execute these smaller test packs on.

  • Execution schedule - select the time when to start executing these tests. Please note, however, that the tests will be planned for execution at the specified time. If some of the target devices will be busy or offline the execution on those devices will resume after they will become operational.

  • Notifications - select the type of notification and specify email recipients to where to send the notification. Additionally you can send a notification to a predefined Slack channel.

Important

Scheduled test packs use only the applied changed assets. All unapplied changes in the version are ignored by the scheduled test packs.

Executions tab

Test run executions tab

This tab contains the test pack executions. It gives an overview of how well the test pack runs have been executing. It also allows you to download an overview in PDF or Excel format.

You can cancel the entire Test pack execution while it is in progress on this tab as well.

Cancel Test pack run

Execution round results page

You can navigate to the execution round results page by clicking on the specific round on the Executions tab of the Test Pack.

Execution round page provides a better overview of the round result. Each round consists of iterations. Those can be useful, when you want to re-execute only failed tests without executing whole test pack. You can also navigate to the specific test result directly from here.

Execution round page

Running tests created with Suitest API

Suitest JavaScript API tests do not use Test packs. You can simply define the combinations of devices and configurations directly inside your JavaScript API configuration file.

If you want to use several combinations, please take a look at the presets.

Example launching test with Suitest launcher:

suitest run \
    -k <your Token ID> \
    -p <your Token password> \
    -c <your Configuration ID> \
    -d <your Device ID>
    node ./test.js
Results not stored in Suitest Web UI

The execution results for test cases created with Suitest API are not stored on Suitest servers.

Since for JavaScript API based tests the control flow of the execution remains with your test runner, Suitest cannot tell when one test begins and the other ends. Consider the following example of a test suite consisting of two tests:

describe('my test suite', function() {
    it ('test 1', function() {
        //suitest api call 1
        //suitest api call 2
    });
    it('test 2', function() {
        //suitest api call 3
        //suitest api call 4
    });
})