Use suite retry

A suite tends to include several test modules and can reach quite a large test corpus size. For example, the Android Compatibility Test Suite (CTS) includes hundreds of modules and hundreds of thousands test cases.

It becomes possible for a large amount of tests to fail due to poor isolation or devices going into a bad state.

The suite retry feature is meant to address those cases: It lets you retry the failures only instead of the full suites in order to rule out flakiness and poor isolation. If a test is consistently failing, the retry also fails; and you get a much stronger signal that there's a real issue.

Implement suite retry

The retry of results involves reading the previous results and rerunning the previous invocation.

The main interface driving the retry is ITestSuiteResultLoader, which lets you load a previous result, and the previous command line.

RetryRescheduler then uses this information to recreate the previous command and populate some filters in order to rerun only the previous failures or not executed tests.

Example suite retry: CTS

The retry configuration in CTS is:

<configuration description="Runs a retry of a previous CTS session.">
    <object type="previous_loader" class="com.android.compatibility.common.tradefed.result.suite.PreviousResultLoader" />
    <test class="com.android.tradefed.testtype.suite.retry.RetryRescheduler" />

    <logger class="com.android.tradefed.log.FileLogger">
        <option name="log-level-display" value="WARN" />
    </logger>
</configuration>

This is applicable to most of the suites that extend it, for example VTS.

It would be invoked with:

cts-tradefed run retry --retry <session>

The session would be found by listing the previous results in the CTS console:

cts-tf > l r
Session  Pass  Fail  Modules Complete  Result Directory     Test Plan  Device serial(s)  Build ID   Product
0        2092  30    148 of 999        2018.10.29_14.12.57  cts        [serial]          P          Pixel

The exact original command will be reloaded and rerun with extra filters. This means that if your original command included some options, they are also part of the retry.

For example:

cts-tradefed run cts-dev -m CtsGestureTestCases

The retry of the above is always bound to CtsGestureTestCases because we are retrying a command that involved only it.

Configure retry for CTS-style suite

In order for the retry to work, the previous results need to be exported in proto format. The following needs to be added:

<result_reporter class="com.android.compatibility.common.tradefed.result.suite.CompatibilityProtoResultReporter" />

This needs to be added to the XML configuration of the main command, and it results in a test-record.pb file being created in the result folder.

The CTS retry then loads data from a combination of the test-record.pb and the existing test_result.xml to prepare the retry invocation.