This is a brief introduction of Test Mapping and an explanation of how to get started configuring tests easily in the Android Open Source Project (AOSP).
What is Test Mapping?
Test Mapping is a Gerrit-based approach that allows developers to create pre- and post-submit test rules directly in the Android source tree and leave the decisions of branches and devices to be tested to the test infrastructure itself. Test Mapping definitions are JSON files named TEST_MAPPING that can be placed in any source directory.
Atest can use the TEST_MAPPING files to run presubmit tests in the associated directories. With Test Mapping, you can add the same set of tests to presubmit checks with a simple change inside the Android source tree.
See these examples:
Add presubmit tests to TEST_MAPPING for services.core
Add presubmit tests to TEST_MAPPING for tools/dexter using imports
Test Mapping relies on the Trade Federation (TF) Test Harness for tests execution and results reporting.
Defining test groups
Test Mapping groups tests via a test group. The name of a test group can be any string. For example, presubmit can be for a group of tests to run when validating changes. And postsubmit tests can be used to validate the builds after changes are merged.
Packaging build script rules
In order for the Trade Federation Test Harness to run Test Mapping's test modules for a given build, these modules must have test_suite set for Soong or LOCAL_COMPATIBILITY_SUITE set for Make to one of these two suites:
- general-tests - tests that don't depend on device-specific functionality (such as vendor-specific hardware that most devices don't have). Most tests should be in the general-tests suite, even if they're specific to one ABI or bitness or hardware features like HWASan (there's a separate test_suites target for each ABI), and even if they have to run on a device.
- device-tests - tests that depend on device-specific functionality.
Typically these tests will be found under
vendor/
. Because "device-specific" does not refer to ABI or SoC functionality that other devices might or might not have, but only to functionality that's unique to a device, this applies to JUnit tests every bit as much as GTest native tests (which should usually begeneral-tests
even if they're ABI-specific).
Examples:
Android.bp: test_suites: ["general-tests"],
Android.mk: LOCAL_COMPATIBILITY_SUITE := general-tests
Configuring tests to run in a test suite
For a test to run inside of a test suite, the test:
- must not have any build provider.
- must clean up after it's finished, for example, by deleting any temporary files generated during the test.
- change system settings to default or original value.
- should not assume a device in a certain state, e.g., root ready. Most
tests don't require root privilege to run. If a test must require root, it
should specify that with a
RootTargetPreparer
in itsAndroidTest.xml
, as in the following example:
<target_preparer class="com.android.tradefed.targetprep.RootTargetPreparer"/>
Creating Test Mapping files
For the directory requiring test coverage, simply add a TEST_MAPPING JSON file resembling the example below. These rules will ensure the tests run in presubmit checks when any files are touched in that directory or any of its subdirectories.
Following an example
Here is a sample TEST_MAPPING
file (it's in JSON format but with comments
supported):
{
"presubmit": [
// JUnit test with options and file patterns.
{
"name": "CtsWindowManagerDeviceTestCases",
"options": [
{
"include-annotation": "android.platform.test.annotations.RequiresDevice"
}
],
"file_patterns": ["(/|^)Window[^/]*\\.java", "(/|^)Activity[^/]*\\.java"]
},
// Device-side GTest with options.
{
"name" : "hello_world_test",
"options": [
{
"native-test-flag": "\"servicename1 servicename2\""
},
{
"native-test-timeout": "6000"
}
]
}
// Host-side GTest.
{
"name" : "net_test_avrcp",
"host" : true
}
],
"postsubmit": [
{
"name": "CtsWindowManagerDeviceTestCases"
}
],
"imports": [
{
"path": "frameworks/base/services/core/java/com/android/server/am"
}
]
}
Setting attributes
In the above example, presubmit
and postsubmit
are the names of each test
group. See
Defining test groups for more information about test
groups.
The name of the test module or Trade Federation integration test
name (resource path to the test XML file, e.g.,
uiautomator/uiautomator-demo)
can be set in the value of the name
attribute. Note the name field cannot
use class name
or test method name
. To narrow down the tests to run, you can
use options such as include-filter
here. See
(include-filter sample usage).
The host setting of a test indicates whether the test is a deviceless test running on host or not. The default value is false, meaning the test requires a device to run. The supported test types are HostGTest for GTest binaries and HostTest for JUnit tests.
The file_patterns attribute allows you to set a list of regex strings for
matching the relative path of any source code file (relative to the directory
containing the TEST_MAPPING file). In above example, test CtsWindowManagerDeviceTestCases
will run in presubmit only when any java file starts with Window or Activity,
which exists in the same directory of the TEST_MAPPING file or any of its sub
directories, is changed. Backslashes \ need to be escaped as they are in a
JSON file.
The imports attribute allows you to include tests in other TEST_MAPPING files without copying the content. Note that the TEST_MAPPING files in the parent directories of the imported path will also be included. Test Mapping allows nested imports; this means two TEST_MAPPING files can import each other, and Test Mapping is able to properly merge the included tests.
The options attribute contains additional TradeFed command line options.
To get a complete list of available options for a given test, run:
tradefed.sh run commandAndExit [test_module] --help
Refer to TradeFed Option Handling for more details about how options work.
Running tests with Atest
To execute the presubmit test rules locally:
- Go to the directory containing the TEST_MAPPING file.
- Run the command:
atest
All presubmit tests configured in the TEST_MAPPING files of the current directory and its parent directories are run. Atest will locate and run two tests for presubmit (A and B).
This is the simplest way to run presubmit tests in TEST_MAPPING files in the current working directory (CWD) and parent directories. Atest will locate and use the TEST_MAPPING file in CWD and all of its parent directories.
Structuring source code
The following example shows how TEST_MAPPING files can be configured across the source tree.
src
├── project_1
│ └── TEST_MAPPING
├── project_2
│ └── TEST_MAPPING
└── TEST_MAPPING
Content of src/TEST_MAPPING
:
{
"presubmit": [
{
"name": "A"
}
]
}
Content of src/project_1/TEST_MAPPING
:
{
"presubmit": [
{
"name": "B"
}
],
"postsubmit": [
{
"name": "C"
}
],
"other_group": [
{
"name": "X"
}
]}
Content of src/project_2/TEST_MAPPING
:
{
"presubmit": [
{
"name": "D"
}
],
"import": [
{
"path": "src/project_1"
}
]}
Specifying target directories
You can specify a target directory to run tests in TEST_MAPPING files in that directory. The following command will run two tests (A, B).
atest --test-mapping src/project_1
Running postsubmit test rules
You can also use this command to run the postsubmit test rules defined in
TEST_MAPPING in src_path
(default to CWD)
and its parent directories:
atest [--test-mapping] [src_path]:postsubmit
Running only tests that require no device
You can use option --host for Atest to only run tests configured against the host that require no device. Without this option, Atest will run both tests, the ones requiring device and the ones running on host and require no device. The tests will be run in two separate suites.
atest [--test-mapping] --host
Identifying test groups
You can specify test groups in the Atest command. The following command will run all postsubmit tests related to files in directory src/project_1, which contains only one test (C).
Or you can use :all to run all tests regardless of group. The following command runs four tests (A, B, C, X):
atest --test-mapping src/project_1:all
Including subdirectories
By default, running tests in TEST_MAPPING with Atest will run only presubmit
tests configured in the TEST_MAPPING file in CWD (or
given directory) and its parent directories. If you want to run tests in all
TEST_MAPPING files in the sub-directories, use the option --include-subdir
to
force Atest to include those tests too.
atest --include-subdir
Without the --include-subdir
option, Atest will run only test A. With the
--include-subdir
option, Atest will run two tests (A, B).
Line-level comment is supported
You can add a line-level //
-format comment to flesh out the TEST_MAPPING file
with a description of the setting that follows. ATest and Trade Federation will
preprocess the TEST_MAPPING to a valid JSON format without comments. To keep
the JSON file clean and easy to read, only line-level //
-format comment is
supported.
Example:
{
// For presubmit test group.
"presubmit": [
{
// Run test on module A.
"name": "A"
}
]
}