Suitest Test Launcher¶
This part of user documentation is describing an obsolete version 2.5.* (or older) of Suitest JavaScript API and is meant only for users who have not yet switched to the version 3. If you are new to Suitest JavaScript API, please use the version 3 instead.
Suitest Test Launcher is a CLI tool, designed to ease the configuration and usage of Suitest JavaScript API. You can still use JS API without the test launcher, but you'll have to handle the API bootstrapping process on your own. See Advanced usage for details.
NPM/Yarn Prefixes
Examples outside this box do not include yarn or npm / npx prefixes for brevity. Add
the prefixes based on your installation method to examples. We recommend setting
up those commands inside package.json to avoid using the prefixes every time.
For example, you can add into the scripts part of package.json file the following structure:
"interactive": "suitest interactive node test.js"
And then execute it from the terminal as:
npm run interactive
What does the launcher do:
- Makes it easier to submit configuration values to the API calls.
- Manages simultaneous test execution on multiple devices.
- Runs authentication requests and generates temporary tokens for the test runs.
Usage¶
CLI signature for the test launcher:
suitest [run mode] [run mode options] [test command] [test command options]
- run mode - one of
interactiveorautomated- sets the Suitest test run mode. - run mode options - modify the behavior of the execution.
- test command - command to start the test run. For example
node ./test.js. - test command options - options passed to the test command.
Launcher options¶
Suitest Test Launcher operates in two execution modes:
Interactive execution mode¶
suitest interactive reserves a single Suitest device for use in interactive
mode and triggers execution of the test command. Interactive mode is
intended only for test authoring and application debugging.
suitest interactive \
-u <your@e.mail> \
-p <your password> \
-o <organization id> \
-d <device id> \
-c <app config id> \
node ./test.js
Interactive mode options¶
| In CLI | In .suitestrc |
Mandatory | Description |
|---|---|---|---|
| --app-config-id, -c | appConfigId | yes | App configuration id to launch the app with. |
| --config-file | configFile | no | Path to JSON config file to override .suitestrc config |
| --continue-on-fatal-error | continueOnFatalError | no, defaults to false | If the Suitest server responds with a fatal error to any command made by Suitest API, Suitest API will by default attempt to kill the process it is running in. You can turn this off by passing --continue-on-fatal-error and handle the shutdown yourself. Please note, that after a fatal server error occurs, all subsequent Suitest commands will fail until the connection is reinitialized with startTest(). |
| --default-timeout | defaultTimeout | no | Default timeout for assertion line. Default set to 2000ms. Value set in ms. |
| --device-id, -d | deviceId | yes | Device to which to connect interactively. |
| --disallow-crash-reports | disallowCrashReports | no, defaults to false | If a crash occurs an automatic report will be submitted to Suitest. These reports help Suitest to improve and update faster. |
| --help, -h | - | no | Show CLI usage help. |
| --log-dir | logDir | no | If defined, logs will be stored inside specified folder. |
| --log-level | logLevel | no, defaults to normal | Logger verbosity level. Values: silent, normal, verbose, debug |
| --org-id, -o | orgId | yes | Id of the organization you want to login into. |
| --password, -p | password | yes | Suitest password. Same that you're using to login to Suitest. |
| --timestamp | timestamp | no, defaults to YYYY.MM.DD HH:mm:ss format |
Console log timestamp in YYYY.MM.DD HH:mm:ss format unless other format is specified. Supported formats can be found in time-stamp package. Timestamp can be turned off by passing 'none'. |
| --username, -u | username | yes | Suitest username. Same that you're using to login to Suitest. |
| --version, -v | - | no | Show Suitest JavaScript API library version. |
Automated execution mode¶
suitest automated executes a test pack on multiple devices in parallel.
For every device involved in this test pack - a new instance of test command is started in a separate process.
Every test included in the test pack will be executed on each selected device. Automated mode is
intended for periodic test executions (for example when used in a CI process).
suitest automated \
-k <token id> \
-p <token password> \
-t <test pack id> \
node ./tests
TIP: If you would like to speed up the execution of a set of tests by running them in parallel, you can split test packs into smaller, separate test packs and select individual devices to execute these smaller test packs on.
Automated mode options¶
| In CLI | In .suitestrc |
Mandatory | Description |
|---|---|---|---|
| --concurrency | concurrency | no, defaults to 0 | Amount of simultaneously running tests. I.e: how many devices may be simultaneously busy with running this Test Pack. 0 = unlimited. Normally you should not specify this parameter. |
| --config-file | configFile | no | Path to JSON config file to override .suitestrc config |
| --continue-on-fatal-error | continueOnFatalError | no, defaults to false | If the Suitest server responds with a fatal error to any command made by Suitest API, Suitest API will by default attempt to kill the process it is running in. You can turn this off by passing --continue-on-fatal-error and handle the shutdown yourself. Please note, that after a fatal server error occurs, all subsequent Suitest commands will fail until the connection is reinitialized with startTest(). |
| --default-timeout | defaultTimeout | no | Default timeout for assertion line. Default set to 2000ms. Value set in ms. |
| --disallow-crash-reports | disallowCrashReports | no, defaults to false | If a crash occurs an automatic report will be submitted to Suitest. These reports help Suitest to improve and update faster. |
| --help, -h | - | no | Show CLI usage info. |
| --log-dir | logDir | no | If defined, logs will be stored inside specified folder. |
| --log-level | logLevel | no, defaults to normal | Logger verbosity level. Values: silent, normal, verbose, debug |
| --test-pack-id, -t | testPackId | yes | Id of the test pack to launch. |
| --timestamp | timestamp | no, defaults to YYYY.MM.DD HH:mm:ss format |
Console log timestamp in YYYY.MM.DD HH:mm:ss format unless other format is specified. Supported formats can be found in time-stamp package. Timestamp can be turned off by passing 'none'. |
| --token-key, -k, --token-id | tokenId | yes | Suitest token id. |
| --token-password, -p | tokenPassword | yes | Suitest token password. |
| --version, -v | - | no | Show Suitest JavaScript API library version. |
Concurrency - simultaneously running test limit
The amount of tests allowed to run simultaneously varies per subscription. You can check the current limit in your profile summary. Suitest Test Launcher would however launch as many processes as there are devices in the Test Pack (unless limited by the --concurrency flag). If beginning another parallel test would cause exceeding your subscription limits, the corresponding process will idly wait for a free execution slot.
Launcher child processes¶
Suitest Test Launcher will always execute the test command in a separate process (in both interactive and automated run modes).
In automated mode the running instances don't know about one another and run completely independently.
The environment variable used internally by the launcher and it's child processes:
SUITEST_CHILD_PROCESS
You should not rely on using this variable.
Enable E2E test debugging¶
To enable debugging in your End-To-End tests using the Test Launcher in interactive mode please refer to the E2E test debugging article.