Testing¶
Questions you may encounter while testing with Suitest. If you cannot find an answer to your question please contact us.
Do you have a continuous integration API?¶
Yes, check the Suitest Network API for more details. Add your API token in your profile.
Can I run my tests in my test pack in order?¶
You cannot, however, this is by design.
Tests should not rely on each other, for example, let's say you have 10 tests in your test pack, 5th test run on the device and then between test 5 and 6 somebody connects to the device in interactive mode and changes the state of the app, well if the 6th test depended on test 5 then test would most likely fail.
Couple things to keep in mind:
- Have test be actually end-to-end, should not rely on the state of the app or another test to change the state.
- If you have designed two tests that depend on each other, then use one master test and add the tests in order using the run test operation.
Why are my tests unstable when running via Test Packs (automated mode)?¶
This can be due to many factors related to device speed and the way test scenarios are created.
-
Specify a timeout for assert lines.
It is almost always preferred to specify a max timeout for the assert line based on the acceptable waiting time for the condition to be fulfilled. This is mainly due to varying device speed (rendering, handling inputs, etc.). Read more about assertions.
-
Use press until condition instead of just press lines.
When navigating within your app, it is better to use the press until condition. Each device handles the user inputs at different speeds, meaning that it's always better to adjust your test to navigate using the press until some condition is met. Read more about press until condition.
-
Avoid using the sleep line at all cost.
The sleep line should not be used if the app is initialized and instrumentation library ready. The sleep line is simply "do nothing for x seconds", therefore, it should be used only when the application has not yet opened. When testing apps use assert line with timeout as you will be saving time on faster devices because you are not setting a constant time of waiting such as with the sleep line, and with the added benefit of verifying some condition.
-
Make sure the app is completely initialized before starting test cases.
Before starting with your test case, you should always verify that the application is loaded correctly. This is mainly to avoid to start sending inputs when the app or device is not ready to receive them yet. You can check if the application is loaded completely by checking that certain elements are loaded and that the default element is focused.
What is the time limit for a single test execution?¶
Single test execution is limited to 60 minutes by default. If you wish for longer time limit, please contact our sales team.
Can I write cross-platform tests?¶
In general, yes. The only condition would be using commands that are available for all tested platforms (e.g. button presses). Tests containing platform-specific commands (e.g. tap gesture on mobile devices) would have to be still written separately per platform.
Can I write data-driven tests in Suitest?¶
The best option to use data-driven tests would be using our JavaScript API together with some test runner that supports data driven tests out of the box. As an example, you can use our Gauge Demo application - read more in official Gauge docs.
With our Test editor, it is also possible but with limitations. When launching a Test Pack over our Network API (look for POST /test-packs/{testPackId}, configVariables property), there is an option to provide configuration override, which also includes Configuration variables. That way you can set up your CI server to send a set of variables together with the request to start the test.
Can Suitest compare screenshots with reference images?¶
Unfortunately, Suitest currently allows users to only compare screenshots visually. There is no assertion on images based on a reference one at the moment.
How can we scale different tests on different devices (Same Test on different devices at the same time, Different Test Packs on different devices at the same time).¶
To run a test in the interactive mode (debugging), every user can test only on one device at a time. In case of automated mode (test pack), you can run any tests (no matter from which test packs) on as many devices as your current subscription plan allows - if you are unsure, look for parallel test executions on our pricing page.
How can I test two different versions (builds) of my application (e.g. dev and production)?¶
If you do not need to have both installed on the device at the same time, you can simply use 2 different configurations in Suitest. In such case, Suitest always installs the one which is chosen to be tested. If you need to have both packages installed on the device at the same time, you can name those application packages differently.
Can I use Suitest on a production build?¶
No. We are forbidding testing on production builds as it could affect the end users.