Initialize your Repo client
Follow the instructions from Downloading the Source to get
and build the Android source code. When issuing the repo init
command, specify a
specific CTS branch using -b
. This ensures that your CTS changes are included
in subsequent CTS releases.
The following example code shows how to use repo init
.
mkdir android11-tests-dev && cd android11-tests-dev && repo init -c -u https://android.googlesource.com/platform/manifest -b android11-tests-dev --use-superproject --partial-clone --partial-clone-exclude=platform/frameworks/base --clone-filter=blob:limit=10M && repo sync -c -j8
Build and run CTS
Execute the following commands to build CTS and start the interactive CTS console:
cd /path/to/android/root
make cts -j32 TARGET_PRODUCT=aosp_arm64
cts-tradefed
In the CTS console, enter:
tf> run cts --plan CTS
Write CTS tests
CTS tests use JUnit and the Android testing APIs. Review the
Test
your app
tutorial and existing tests under the cts/tests
directory.
CTS tests mostly follow the same conventions used in other Android tests.
CTS runs across many production devices, so the tests must follow these rules:
- Take into account varying screen sizes, orientations, and keyboard layouts.
- Use only public API methods. In other words, avoid all classes, methods, and fields
with the
hide
annotation. - Avoid using view layouts or relying on the dimensions of assets that may not be on some devices.
- Don't rely on root privileges.
Add Java annotation
If your test verifies an API behavior, annotate your test code with @ApiTest
and list
all APIs involved in the apis
field. Use the appropriate format from among the
following examples:
API type | Annotation format | Notes |
---|---|---|
Method | android.example.ClassA#methodA |
The most common use case. |
Method with key values | android.example.ClassB#methodB(KeyA) |
Use only when your test uses an API method to validate a field, as in this example. |
Field | android.example.ClassC#FieldA | Use only when your test validates an API field directly, as in this example. |
If your test verifies a CDD requirement, annotate the requirement ID (including CDD Section
ID and Requirement ID) with @CddTest
in the CTS test code as shown in the
following example. In your commit message, mention which CDD requirement is tested by your
test by referring to CDD requirement IDs. CDD requirement IDs are a combination of section ID
and requirement ID, connected by a slash (/) as in 7.3.1/C-1-1.
/**
* Verify Passpoint configuration management APIs for a Passpoint
* @throws Exception
*/
@CddTest(requirement="7.4.2.3/C-1-1,C-2-1")
public void testAddPasspointConfigWithUserCredential() throws Exception {
if (!WifiFeature.isWifiSupported(getContext())) {
// skip the test if WiFi is not supported
return;
} testAddPasspointConfig(generatePasspointConfig(generateUserCredential()));
}
For CTS Verifier, annotate each Activity in your AndroidManifest.xml
with the
relevant CDD ID. The formats for value fields are similar to formats of Java annotations in
CTS. In the commit message, mention which CDD requirement is enforced by referencing the CDD
requirement ID.
<activity>
......
<!-- OPTIONAL: Add a meta data attribute to indicate CDD requirements. -->
<meta-data android:name="cdd_test" android:value="7.4.1/C-4-1" />
<!-- OPTIONAL: Add a meta data attribute to indicate APIs being tested. -->
<meta-data android:name="api_test"
android:value="com.example.MyClass#myMethod" />
<!-- OPTIONAL: Add a metadata attribute to indicate the reason why the test doesn't enforce any CDD requirement but still useful in CTS-V. -->
<meta-data android:name="non_compliance_test"
android:value="detailed reasons" />
</activity>
In the commit message
Clearly mention why your test needs to be added, and add relevant links for support. For CTS-D tests, include a link to the test proposal that you created in Google Issue Tracker as part of the CTS-D submission process.
Create a subplan
As an example, you can add a SubPlan.xml file in
android-cts/subplans
as follows:
<?xml version="1.0" encoding="utf-8" standalone="no"?> <SubPlan version="2.0"> <Entry include="CtsSystemIntentTestCases" /> <Entry include="CtsSystemUiHostTestCases" /> <Entry include="CtsSecurityHostTestCases android.security.cts.SELinuxHostTest#testAospFileContexts" /> <Entry include="CtsSecurityHostTestCases android.security.cts.SELinuxHostTest#testAospServiceContexts" /> </SubPlan>
To run the subplan:
run cts --subplan aSubPlan
The subplan entry format is:
Include a module name as follows: <Entry include="MODULE_NAME" /> Include a package: <Entry include="MODULE_NAME PACKAGE_NAME" /> Include a class: <Entry include="MODULE_NAME PACKAGE_NAME.CLASS_NAME" /> Include an individual test: <Entry include="MODULE_NAME PACKAGE_NAME.CLASS_NAME#TEST_NAME" />
Test naming and location
Most CTS test cases target a specific class in the Android API. These tests
have Java package names with a cts
suffix and class names with a
Test
suffix. Each test case consists of multiple tests, where each
test typically exercises a specific method of the class being tested.
These tests are arranged in a directory structure where tests are grouped into
different categories such as "widgets" or "views".
For example, the CTS test for the Java package
android.widget.TextView
is
android.widget.cts.TextViewTest
with its Java package name as
android.widget.cts
and its class name as
TextViewTest
.
- Java package name
The Java package name for the CTS tests is the package name of the class that the test is testing, followed by.cts
. For our example, the package name would beandroid.widget.cts
. - Class name
The class name for CTS tests is the name of the class being tested with "Test" appended. For example, if a test is targetingTextView
, the class name should beTextViewTest
. - Module name (CTS v2 only)
CTS v2 organizes tests by module. The module name is usually the second string of the Java package name (in our example,widget
).
The directory structure and sample code depend on whether you are using CTS v1 or CTS v2.
CTS v1
For Android 6.0 or lower, use CTS v1. For CTS v1, the sample code is at
cts/tests/tests/example
.
The directory structure in CTS v1 tests looks like this:
cts/ tests/ tests/ package-name/ Android.mk AndroidManifest.xml src/ android/ package-name/ SampleDeviceActivity.java cts/ SampleDeviceTest.java
CTS v2
For Android 7.0 or higher, use CTS v2. For details, see the sample test in Android Open Source Project (AOSP).
The CTS v2 directory structure looks like this:
cts/ tests/ module-name/ Android.mk AndroidManifest.xml src/ android/ package-name/ SampleDeviceActivity.java cts/ SampleDeviceTest.java
New sample packages
When adding new tests, there might not be an existing directory to place your test. In those cases, you need to create the directory and copy the appropriate sample files.
CTS v1
If you're using CTS v1, refer to the example under
cts/tests/tests/example
and create a new directory. Also,
make sure to add your new package's module name from its Android.mk
to CTS_COVERAGE_TEST_CASE_LIST
in
cts/CtsTestCaseList.mk
. build/core/tasks/cts.mk
uses this makefile
to combine all the tests and create the final CTS package.
CTS v2
Use the sample test
/cts/tests/sample/
to quick start your new test module with the following steps:
- To create the test directory and copy sample files, run:
mkdir cts/tests/module-name && cp -r cts/tests/sample/* cts/tests/module-name
- Navigate to
cts/tests/module-name
and substitute all instances of "[Ss]ample" with the recommended naming convention from above. - Update
SampleDeviceActivity
to exercise the feature you're testing. - Update
SampleDeviceTest
to ensure that the activity succeeds or logs its errors.
Additional directories
Other Android directories such as assets
, jni
,
libs
, and res
can also be added.
To add JNI code,
create a directory in the root of the project next to src
with the native
code and an Android.mk
makefile in it.
The makefile typically contains the following settings:
LOCAL_PATH := $(call my-dir) include $(CLEAR_VARS) LOCAL_MODULE := libCtsSample_jni # don't include this package in any target LOCAL_MODULE_TAGS := optional LOCAL_SRC_FILES := list of source code files LOCAL_C_INCLUDES := $(JNI_H_INCLUDE) # Tag this module as a cts test artifact LOCAL_COMPATIBILITY_SUITE := cts LOCAL_SHARED_LIBRARIES := libnativehelper LOCAL_SDK_VERSION := current include $(BUILD_SHARED_LIBRARY)
Android.mk file
Finally, modify the Android.mk
file in the root of the
project to build the native code and depend on it, as shown below:
# All tests should include android.test.runner. LOCAL_JAVA_LIBRARIES := android.test.runner # Includes the jni code as a shared library LOCAL_JNI_SHARED_LIBRARIES := libCtsSample_jni # Include for InstrumentationCtsTestRunner LOCAL_STATIC_JAVA_LIBRARIES := ctstestrunner... LOCAL_SDK_VERSION := currentinclude $(BUILD_CTS_PACKAGE) #Tells make to look in subdirectories for more make files to include include $(call all-makefiles-under,$(LOCAL_PATH))
Fix or remove tests
In addition to adding new tests, you can fix or
remove tests annotated with BrokenTest
or KnownFailure
.
Submit your changes
When submitting CTS or VTS patches in AOSP, choose your development branch based on which API levels the patch applies to.
-
For changes that apply to multiple API levels, first
develop a patch in
aosp/main
and then cherry-pick to the most upstream test branch. Allow the automerger to merge the changes downstream in AOSP test branches. See Release schedule and branch information for the list of branches and automerge path information. - For changes that are specific to a specific API level, develop or cherry-pick the changes to the correct test branch with DO NOT MERGE or RESTRICT AUTOMERGE in the commit message.
Follow the Submitting patches workflow to contribute changes to CTS. A reviewer will be assigned to review your change accordingly.
Release schedule and branch information
CTS releases follow this schedule.
Version | API level | Branch | Frequency |
---|---|---|---|
15 | 35 | android15-tests-dev | Quarterly |
14 | 34 | android14-tests-dev | Quarterly |
13 | 33 | android13-tests-dev | Quarterly |
12L | 32 | android12L-tests-dev | Quarterly |
12 | 31 | android12-tests-dev | Quarterly |
Important dates during the release
- End of the first week: Code freeze. Changes merged in the branch until the code freeze are considered for the upcoming version of CTS. Submissions to the branch after the code freeze, or after a candidate for release is chosen, are considered for the subsequent release.
- Second or third week: CTS is published in AOSP.
Automerge flow
CTS development branches have been set up so that changes submitted to each branch automatically merge to higher branches.
For changes directly to an AOSP test dev branch, the automerge path is:
android11-tests-dev
>
android12-tests-dev
>
android12L-tests-dev
>
android13-tests-dev
>
android14-tests-dev
>
android15-tests-dev
>
aosp-main
For changes only to the next Android version, the automerge path is:
aosp-main
>
<Internal git_main>
.
If a changelist (CL) fails to merge correctly, the author of the patch is sent an email with instructions on how to resolve the conflict. In most of the cases, the author of the patch can use the instructions to skip the automerge of the conflicting CL.
If an older branch requires the change, then the patch needs to be cherry-picked from the newer branch.