A suite tends to include several test modules and can reach quite a large test corpus size. For example, the Android Compatibility Test Suite (CTS) includes hundreds of modules and hundreds of thousands test cases.
It becomes possible for a large amount of tests to fail due to poor isolation or devices going into a bad state.
The suite retry feature is meant to address those cases: It lets you retry the failures only instead of the full suites in order to rule out flakiness and poor isolation. If a test is consistently failing, the retry also fails; and you get a much stronger signal that there's a real issue.
Implement suite retry
The retry of results involves reading the previous results and rerunning the previous invocation.
The main interface driving the retry is
which lets you load a previous result, and the previous command line.
then uses this information to recreate the previous command and populate some
filters in order to rerun only the previous failures or not executed tests.
Example suite retry: CTS
The retry configuration in CTS is:
<configuration description="Runs a retry of a previous CTS session."> <object type="previous_loader" class="com.android.compatibility.common.tradefed.result.suite.PreviousResultLoader" /> <test class="com.android.tradefed.testtype.suite.retry.RetryRescheduler" /> <logger class="com.android.tradefed.log.FileLogger"> <option name="log-level-display" value="WARN" /> </logger> </configuration>
This is applicable to most of the suites that extend it, for example VTS.
It would be invoked with:
cts-tradefed run retry --retry <session>
The session would be found by listing the previous results in the CTS console:
cts-tf > l r Session Pass Fail Modules Complete Result Directory Test Plan Device serial(s) Build ID Product 0 2092 30 148 of 999 2018.10.29_14.12.57 cts [serial] P Pixel
The exact original command will be reloaded and rerun with extra filters. This means that if your original command included some options, they are also part of the retry.
cts-tradefed run cts-dev -m CtsGestureTestCases
The retry of the above is always bound to
we are retrying a command that involved only it.
Configure retry for CTS-style suite
In order for the retry to work, the previous results need to be exported in proto format. The following needs to be added:
<result_reporter class="com.android.compatibility.common.tradefed.result.suite.CompatibilityProtoResultReporter" />
This needs to be added to the XML configuration of the main command, and it
results in a
test-record.pb file being created in the result folder.
The CTS retry then loads data from a combination of the
test_result.xml to prepare the retry invocation.