Running CTS tests

Trade Federation is a continuous test framework for running tests on Android devices. To run the Compatibility Test Suite (CTS), first read the Trade Federation Overview for an explanation of Tradefed test framework.

To run a test plan:

Set up Devices Under Test (DUTs) as per Android device setup.
Download the released CTS builds onto your Linux host machine and unzip the build to host location.
Connect at least one device. To prepare the DUT:
- Press the home button to set the device to the home screen.
- Don't use the DUT for any other tasks.
- Keep the DUT in a stationary position to avoid triggering sensor activity.
- Point the device camera at an object that can be focused.
- Do not press any keys on the device while the CTS is running. Pressing keys or touching the screen of a DUT interferes with the running tests and may lead to test failures.
Launch the CTS console cts-tradefed script from the folder where the CTS package has been unzipped. On the host's command-line shell, run:
```
./android-cts/tools/cts-tradefed
```
Run the default test plan (contains all test packages):
```
cts-tradefed > run cts
```
- If you want to improve test execution time, you can shard the tests across multiple devices. Sharding requires the host to connect at least two devices, but six or more devices are recommended for efficiency. When sharding more than 1 device:
  - For Android 9 and higher, use the command option
```
--shard-count number_of_shards
```
  - For Android 8.1 and lower, use the command option
```
--shards number_of_shards
```
- If you don't want to run the whole test suite, you can run the CTS plan of your choice from the command line:
```
run cts --plan test_plan_name
```
  To find the test plan name:
  - For Android 7.0 and higher, to see a list of test modules, enter
```
list modules
```
  - For Android 6.0 and lower, to view a list of test plans in the repository, enter
```
list plans
```
  - For Android 6.0 and lower, to view a list of test packages in the repository, enter
```
list packages
```
- For additional command options depending on CTS versions, refer to the console command references below, or under "help all" in the Tradefed console.
  Note: You may save run time in Android 7.0 (Nougat) and higher, by using the run cts-dev command instead of run cts. This command skips preconditions, device-information collection, and all system status checkers. It also runs the tests on only a single ABI. For device validation, avoid this optimization and include all preconditions and checks.
Run multiple Retry sessions until all test modules are completed and the test failure numbers are the same in the last two retry sessions.
- For Android 9 and higher, use
```
run retry --retry session_number --shard-count number_of_shards
```
- For Android 7.0–8.1, use
```
run cts --retry session_number --shards number_of_shards
```
- For additional Retry command options depending on CTS version, see the Using the CTS v2 console section below.
- fTo understand implementation details for CTS retry, see Trade Federation Suite Retry.
View test progress and results reported on the console.

Using the CTS v2 console

For Android 7.0 or higher, use CTS v2.

Selecting plans

Available test plans include the following:

cts—Runs CTS from an pre-existing CTS installation.
cts-camera— Runs CTS-camera from a pre-existing CTS installation.
cts-java— Runs Core Java Tests from a pre-existing CTS installation.
cts-pdk— Runs Tests useful on validating a PDK fusion build.
everything— Common config for Compatibility suites.

Other available configurations include the following:

basic-reporters— Configuration with basic CTS reporters.
collect-tests-only—Runs CTS from a pre-existing CTS installation.
common-compatibility-config— Common config for Compatibility suites.
cts-filtered-sample— Common config for Compatibility suites.
cts-known-failures— Configuration with CTS known failures.
cts-preconditions— CTS precondition configs.
host— Runs a single host-based test on an existing device.
instrument— Runs a single Android instrumentation test on an existing device.
native-benchmark— Runs a native stress test on an existing device.
native-stress— Runs a native stress test on an existing device.
recharge— A fake test that waits for nearly-discharged devices and holds them for charging.
testdef— Runs tests contained in test_def.xml files on an existing device.
util/wifi— Utility config to configure Wi-Fi on device.
util/wipe— Wipes user data on device.

All of these plans and configs can be executed with the run cts command.

CTS v2 console command reference

Table 1. This table summarizes the CTS V2 console commands for various uses.

Host	Description
`help`	Display a summary of the most commonly used commands
`help all`	Display the complete list of available commands
`version`	Show the version.
`exit`	Gracefully exit the CTS console. Console closes when all currently running tests are finished.
Run	Description
`run cts`	In Android 10, run the default CTS plan and CTS-Instant together (that is, the full CTS invocation). For Android 9 and lower, run the default CTS plan only. Use this comprehensive option (including preconditions) for device validation. See cts.xml for inclusions. The CTS console can accept other commands while tests are in progress. If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests. If more than one device is connected, the CTS host will choose a device automatically.
`run cts-instant`	For Android 9, run the default CTS-Instant plan.
`run cts --module-parameter INSTANT_APP`	In Android 10, run the default CTS-Instant plan.
`run cts --module-parameter INSTANT_APP --module/-m test_module_name`	In Android 10, run the specified CTS-Instant test module or modules.
`run retry`	For Android 9 and higher only. Retry all the tests that failed or weren't executed from the previous sessions. For example, `run retry --retry -s` or `run retry --retry --shard-count` with TF sharding. `run cts --retry` isn't allowed for Android 9 and higher.
`--device-token`	For Android 8.1 and lower versions. Specifies that a given device has the given token. For example, `--device-token 1a2b3c4d:sim-card`.
`--enable-token-sharding`	For Android 10 only. Automatically matches the test that requires respective SIM type. No need to provide a device serial number to execute SIM-related test cases. Supported SIMs: `SIM_CARD`, `UICC_SIM_CARD`, and `SECURE_ELEMENT_SIM_CARD`.
`run cts-dev`	Run the default CTS plan (that is, the full CTS invocation) but skip preconditions to save run time for iterative development of a new test. This bypasses verification and setup of the device's configuration, such as pushing media files or checking for Wi-Fi connection, as is done when the `--skip-preconditions` option is used. This command also skips device-information collection, and all system status checkers. It also runs the tests on only a single ABI. For device validation, avoid this optimization and include all preconditions and checks. See cts-dev.xml for exclusions. The CTS console can accept other commands while tests are in progress. If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests. If more than one device is connected, the CTS host will choose a device automatically.
`run retry`	For Android 9. Retry all tests that failed or were not executed from the previous sessions. For example, `run retry --retry session id -sdevice serial`, or `run retry --retry session id --shard-count` with TF sharding. `run cts --retry` is not allowed for Android 9.
`--plan test_plan_name`	Run the specified test plan.
`--module/-m test_module_name [--module/-m test_module2...]`	Run the specified test module or modules. For example, `run cts --module CtsGestureTestCases` executes the gesture test module (this can be shortened to `run cts -m Gesture`). `run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes` runs the specific package, class, or test.
`--subplan subplan_name`	Run the specified subplan.
`-- module/-m test_module_name -- test test_name`	Run the specified module and test. For example, `run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes` runs the specific package, class, or test.
`--retry`	Retry all tests that failed or were not executed from the previous sessions. Use `list results` to get the session id.
`--retry-type not_executed`	Retry only tests that were not executed from the previous sessions. Use `list results` to get the session id.
`--shards number_of_shards`	For Android 8.1 and lower versions. Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel.
`--shard-count number_of_shards`	For Android 9. Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel.
`--serial/-s deviceID`	Run CTS on the specific device.
`--include-filter module_name [--include-filter module2...]`	Run only with the specified modules.
`--exclude-filter module_name [--exclude-filter module2...]`	Exclude the specified modules from the run.
`--log-level-display/-l log_level`	Run with the minimum specified log level displayed to `STDOUT`. Valid values: [`VERBOSE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `ASSERT`].
`--abi abi_name`	Force the test to run on the given ABI, 32 or 64. By default CTS runs a test once for each ABI the device supports.
`--logcat`, `--bugreport`, and `--screenshoot-on-failure`	Give more visibility into failures and can help with diagnostics.
`--device-token`	Specifies a given device has the given token, such as `--device-token 1a2b3c4d:sim-card`.
`--skip-device-info`	Skips collection of information about the device. Caution: Don't use this option when running CTS for approval.
`--skip-preconditions`	Skip preconditions to save run time for iterative development of a new test. This bypasses verification and setup of the device's configuration, such as pushing media files or checking for Wi-Fi connection.
List	Description
`list modules`	List all available test modules in the repository.
`list plans` or `list configs`	List all available test plans (configs) in the repository.
`list subplans`	List all available subplans in the repository.
`list invocations`	List 'run' commands currently being executed on devices.
`list commands`	List all 'run' commands currently in the queue waiting to be assigned to devices.
`list results`	List CTS results currently stored in repository.
`list devices`	List currently connected devices and their state. 'Available' devices are functioning, idle devices, available for running tests. 'Unavailable' devices are devices visible via adb, but are not responding to adb commands and won't be allocated for tests. 'Allocated' devices are devices currently running tests.
Dump	Description
`dump logs`	Dump the tradefed logs for all running invocations.
Add	Description
`add subplan --name/-n subplan_name --result-type [pass \| fail \| timeout \| notExecuted] [--session/-s session_id]`	Create a subplan derived from previous session; this option generates a subplan that can be used to run a subset of tests. The only required option is `--session`. Others are optional but, when included, must be followed by a value. The `--result-type` option is repeatable; for example `add subplan --session 0 --result-type passed --result-type failed` is valid.

Using the CTS v1 console

For Android 6.0 or lower, refer to CTS v1 Command Console Reference.