Testing

Questions you may encounter while testing with Suitest. If you cannot find an answer to your question please contact us.

Do you have a continuous integration API?

Yes, check the Suitest Network API for more details. Add your API token in your profile.

How would I check if an element is in a focused state using the test editor?

This greatly depends on the platform you are testing and the implementation of your application.

HTML-based

There are several ways to check if an element is in a focused state:

  • Contents of class attribute
  • Comparing elements visual attributes (text color, border color etc.)

To check for a button in a focused state using the contents of the class attribute, the element of the button should have something along the lines of "Focused" or "ButtonFocused" in the elements class attribute (CSS Selector).

  1. Navigate to the element with the Virtual Remote Control (VRC) or your normal TV control so that the element is in a focused state.

  2. Go to the Element Repository (Elements tab) select the element that is in a focused state and save it. More information about selecting elements.

  3. Switch to the test editor (Tests tab).

  4. Use either an Assert or Wait until line based on your use case.

  5. Select element as the subject.

  6. Select the element you wish to do the assertions on. In the class attribute, switch to the comparator to contains and for example "ButtonFocused" or keep it as it is if you need it to exactly equal the class.

The second option is comparing elements visual attributes to see if the state of the element has changed.

  1. Repeat steps 1 to 5 as seen above. Additionally, you can save the element in normal (unfocused state) to the element repository to be able to tell what properties have changed.

  2. Select the element you wish to do the assertions on. In the properties of the element compare background color, for example, the color was rgba(0,0,0,1.0) in unfocused state and once focused the color is rgba(0,0,255,0). Depending on the style changes of your element there could be a multitude of properties to compare when the element is in a focused state. Find what attributes work for your particular element.

Xbox One Native

Xbox One Native has Focus properties which can be used to check if a particular element is in a focused state. Learn more about Focus properties.

When I export a test into a different application are my elements exported as well?

No. At the moment when you export a test to a different app the elements saved within the repository are only for that particular app, they are not tied to the test.

Can I run my tests in my test pack in order?

You cannot, however, this is by design.

Tests should not rely on each other, for example, let's say you have 10 tests in your test pack, 5th test run on the device and then between test 5 and 6 somebody connects to the device in interactive mode and changes the state of the app, well if the 6th test depended on test 5 then test would most likely fail.

Couple things to keep in mind:

  • Have test be actually end-to-end, should not rely on the state of the app or another test to change the state.
  • If you have designed two tests that depend on each other, then use one master test and add the tests in order using the run test operation.
Workaround for ordering tests
Workaround for ordering tests

Why are my tests unstable when running via Test Packs (automated mode)?

This can be due to many factors related to device speed and the way test scenarios are created.

  1. Use wait until lines instead of assert lines.

    It is almost always preferred to use a wait until line with a max timeout which would be acceptable for you to wait for the condition to fulfill instead of asserts. This is mainly due to varying device speed (rendering, handling inputs, etc.). Read more about assertions.

  2. Use press until condition instead of just press lines.

    When navigating within your app, it is better to use the press until condition. Each device handles the user inputs at different speeds, meaning that it's always better to adjust your test to navigate using the press until some condition is met. Read more about press until condition.

  3. Avoid using the sleep line at all cost.

    The sleep line should not be used if the app is initialized and instrumentation library ready. The sleep line is simply "do nothing for x seconds", therefore, it should be used only when the application has not yet opened. When testing apps use the wait until line as you will be saving time on faster devices because you are not setting a constant time of waiting such as with the sleep line, and with the added benefit of verifying some condition.

  4. Make sure the app is completely initialized before starting test cases.

    Before starting with your test case, you should always verify that the application is loaded correctly. This is mainly to avoid to start sending inputs when the app or device is not ready to receive them yet. You can check if the application is loaded completely by checking that certain elements are loaded and that the default element is focused.