Surprises in Suitest and how to avoid them¶

Sometimes you will create a test that looks perfectly in order and after being very proud of yourself for doing such a good job, you will be taken aback by the fact that the test fails randomly at the same device where you have created it.

There can be a couple of reasons for that. Let's briefly check them out here.

Not waiting for the app to initialize¶

This is the most common and straightforward mistake that you will most definitely make in one of your first Suitest tests. When you create the test you know instinctively that you have to wait for the application to load data before making the next step in your test (like making an assertion or navigate to a page referenced by the data).

For example, consider the following test scenario:

  0. Open app WatchMe at homepage
  1. Assert element MyTeaser has properties ... timeout 0

This is an excellent candidate for a randomly failing test. You inspect a view element immediately after the application has been opened without considering the fact that the application may be still loading data for the view element you have included into the assertion.

If the network and the device are fast enough, the test will go through, otherwise the test will fail. Just for the record, the tests that produce random results bring more damage than value.

The Suitest Rule No. 1. - Don't run assertions immediately after startup. Allow the app to load the content first.

Armed with this rule, we know that we should always use a Assert line with timeout after the Open App line, for example like this:

  0. Open app WatchMe at homepage
  1. Assert element MyVideoTeaser has properties ... timeout 10s

Now the only bad thing that can happen to our test is that the 10 seconds timeout will not be enough for the app to load the data. Experiment with different timeouts to establish the clean difference between the real world timeout and the wrong test scenario. Typically, timeout values less then 7 minutes should be fine. The timeout value is the maximum time Suitest will wait for the condition to be fulfilled. If the app happens to respond sooner, Suitest will move on to the next line.

Not waiting for the app to process user's input¶

A careful observer will quickly notice that the problem of data loading is not bound to the initial start but precautions must be taken every time user input is dispatched to the application. Continuing with our test, consider the following scenario:

  0. Open app WatchMe at homepage
  1. Assert MyVideoTeaser has properties ... timeout 10s
  2. Press OK
  3. Assert video is playing timeout 0

In this instance on line 3 you made an assertion that the video is playing immediately after the OK key is dispatched to the MyVideoTeaser. But that is not what happens in real world, is it?

In real world situations like this countless other things happen before video actually starts playing; The application destroys the previous view, loads data for the player view, negotiates the video playback URL, initializes the video component, buffers a piece of video contents and if all of that went through successfully it finally starts playing the video.

Suitest Rule No. 2 - After sending user input to the app, wait for the app to process it before running assertions.

To obey this rule, you can use the Assert line with timeout in most cases:

  0. Open app WatchMe at homepage
  1. Assert element teaser 1 has properties ... timeout 10s
  2. Press OK
  3. Assert video is playing timeout 5s

From considerations similar to the above we can further derive the No. 3 rule.

Suitest Rule No. 3 - Before sending user input to the app, wait for the view to be ready.

This one is kind of similar to the No. 1 and we have branched it out to the separate rule just for clarity. Consider the following example:

  0. Open app WatchMe at homepage
  1. Assert element teaser 1 has properties ... timeout 10s
  2. Press OK
  3. Press BACK

The BACK is sent into the wild right after the OK and it has no conception of the state of the app. A well written app will mostly be able to process this consistently, however in certain situations pressing BACK too fast could cause the app to "ignore" the OK key and process the BACK key on the homepage instead on the player page. This should be reported as a bug in the first place, however this test will tend to produce random results and if such situation were to occur in a test with a larger amount of lines its random failures would surely cause lots of confusion.

When using Suitestify the app looks broken¶

There can be several causes for this. So you should start by eliminating the "device issue". In the configuration of your app select "Insert instrumentation library manually" but do not include the instrumentation library physically. Then try launching the application from Suitest.

Option one: App does not function correctly - you have a problem with the app or device not relating to Suitest.
Option two: App functions as intended - the basic configuration of Suitestify is not enough. Read about the advanced configuration of Suitestify.

Network request condition never matches¶

Network requests are captured with Suitestify. So if you don't use Suitestify in your app, the network log will always be empty and the assertions will never be fulfilled. If you do use Suitestify, then check if the request which you are trying to match comes from one of the domains specified in the Suitestify configuration.

If you are still receiving Network request matching given conditions was not made error in interactive mode, try to disconnect the device and then connect again.

Getting out of Suitest app boundary¶

A Suitest application is a virtual thing defined by the Suitest app id. If you include the instrumentation library with the same id on two different HTML pages (or apps) both will be treated as part of the same app. If during test execution the app will open a page (or app) without instrumentation library or with a different Suitest app id, Suitest will consider this an error and report it too.

Depending on the type of testing you are to perform it may be beneficial to split your project into multiple Suitest apps. For example if you have a portal to test where each topic is an app (like Videos, News, etc) and the interconnection between these is weak, you may be under circumstances better off creating separate Suitest apps for every topic.

Previous:
Locations of ID's and Tokens Next:
TV specific keys in browser