CTS FAQ

The Android Compatibility Program is the key driver to sustain the positive feedback for Android ecosystem. CTS is the key tool to ensure the quality of compatibility in the scale. Android team continues to improve CTS tool and test coverage.The regular addition of test cases has significant improvement on the quality of compatible devices.

This article provides FAQs to run the CTS tests more efficiently.

What is the difference between CTS Sharding and TF Sharding?

CTS Sharding and TF Sharding are totally different test plans powered by different test infrastructure codebase. While the run command is the same across different versions, the sharding result behaves differently. CTS Sharding statically assigns test cases to Devices Under Test (DUTs) as follows:

TF Sharding dynamically assigns test cases to available DUTs as follows:

What is expected from a device supporting multiple ABIs?

The device has to pass all CTS/Verifier for each ABI mode it claims to support. Therefore, it is necessary to execute an app for the particular ABIs. The guidelins for multiple ABIs are as follows:

  • For CTS/Verifier, there are ARM and x86 releases for each architecture. Each of them can support to 32- or 64-bit mode.
  • For CTS tests, if a device supports both ARM and x86, it has to run and pass both ARM and x86 CTS tests respectively.

See CDD 3.3.1. Application Binary Interfaces for CDD requirements on ABI.

Is it sufficient to run a test only on the primary ABI (e.g. 64 bits) to reduce test execution time?

No. An Android app runs on its own 32 -bit or 64-bit runtimes. The actual machine code, code path, and state are different between 32 and 64. If you skip one mode, you are only covering 50% of the device ABI.

Why are there so many test cases reported as Not Executed?

You should check the Module Done number instead of Not Executed number.

In the older versions, CTS modules were reported as Module Done too aggressively before being completed. Therefore, a Modules Done number was reported without all the test case complete even when some devices had problems. The new test harness is more conservative and reports a higher number of Not Executed tests when a problem occurs.

A module run to completion reports Module Not Done in the most recent invocation (done="false") in the report during the following:

  • A test run for the module was interrupted by a device connection issue.
  • Not all expected test runs for the module were performed.

  • Retried (using option -r/--retry) with additional filtering options, such as:

    • --include-filter
    • --exclude-filter
    • -t/--test (Option not yet supported on retry)
    • --retry-type failed
    • --subplan

To obtain a status of Module Done (done="true") for these modules, retry the following for the most recent invocation:

run retry --retry <session_id> for Android 9 and later versions

run cts --retry <session_id> for Android 8.1 and previous versions

A module executed without any of the problems mentioned above (even with 0 remaining tests) is marked Module Done in the new report.

Exceptions

  • CtsNNAPITestCases has a known issue due to linux/OS limitation of args. The module can be rerun in isolation through run cts -m CtsNNAPITestCases directly.

How can I avoid test preparation failing behind corporate firewall?

All automated test suites try to download either the CTS media files or the business logic files during runtime. In many corporate environments, a firewall/proxy is typical, which makes the test preparation fail. Execute the following line or add it to .profile (on Ubuntu).

export JAVA_TOOL_OPTIONS='-Djava.net.useSystemProxies=true'

Do I need a SIM card for CTS for Secure Element?

Whether a SIM card is needed for the test depends on the understanding if the feature is supported in the test device.

  • If your device DOES NOT need to support the Android apps accessing secure elements—either in the UICC (e.g., a SIM card) distributed by the mobile network operators (carriers) or embedded in the device—you can configure the HIDL manifest to not include the android.hardware.secure_element HAL element. In this case, the android.se.omapi.SEService.getReaders() API will report an empty list and the CTS test will automatically pass and report a pass for the CTS.
  • If your device DOES need to support Android apps accessing either secure elements—either in the UICC (e.g., a SIM card) distributed by the mobile network operators (carriers) or embedded in the device—you need to implement secure element properly and test it in-house. CTS Test for Secure Element outlines how to prepare to run the CTS tests that ensure android.se.omapi API package added in Android 9 is functional. We also recommend performing additional testing on your own since the CTS test coverage is minimal.

Where can I get the SIM cards for CTS for Secure Element?

You can reach out to your preferred SIM vendor.

Why is Orange SIM on the lock screen during CTS execution with token sharding?

The test case does not start because testing the SIM card is locked. Disable "Lock SIM card" option in "SIM card lock settings" before executing the CTS with token sharding.