Test Mapping

This is a brief introduction of Test Mapping and an explanation of how to get started configuring tests easily in the Android Open Source Project (AOSP).

What is Test Mapping?

Test Mapping is a Gerrit-based approach that allows developers to create pre- and post-submit test rules directly in the Android source tree and leave the decisions of branches and devices to be tested to the test infrastructure itself. Test Mapping definitions are JSON files named TEST_MAPPING that can be placed in any source directory.

Atest can use the TEST_MAPPING files to run presubmit tests in the associated directories. With Test Mapping, you can add the same set of tests to presubmit checks with a simple change inside the Android source tree.

See these examples:

Add presubmit tests to TEST_MAPPING for services.core

Add presubmit tests to TEST_MAPPING for tools/dexter using imports

Test Mapping relies on the Trade Federation (TF) Test Harness for tests execution and results reporting.

Defining test groups

Test Mapping groups tests via a test group. The name of a test group can be any string. For example, presubmit can be for a group of tests to run when validating changes. And postsubmit tests can be used to validate the builds after changes are merged.

Packaging build script rules

In order for the Trade Federation Test Harness to run Test Mapping's test modules for a given build, these modules must have test_suite set for Soong or LOCAL_COMPATIBILITY_SUITE set for Make to one of these two suites:

  • general-tests - tests that don't depend on device-specific functionality (such as vendor-specific hardware that most devices don't have). Most tests should be in the general-tests suite, even if they're specific to one ABI or bitness or hardware features like HWASan (there's a separate test_suites target for each ABI), and even if they have to run on a device.
  • device-tests - tests that depend on device-specific functionality. Typically these tests will be found under vendor/. Because "device-specific" does not refer to ABI or SoC functionality that other devices might or might not have, but only to functionality that's unique to a device, this applies to JUnit tests every bit as much as GTest native tests (which should usually be general-tests even if they're ABI-specific).

Examples:

Android.bp: test_suites: ["general-tests"],
Android.mk: LOCAL_COMPATIBILITY_SUITE := general-tests

Configuring tests to run in a test suite

For a test to run inside of a test suite, the test:

  • must not have any build provider.
  • must clean up after it's finished, for example, by deleting any temporary files generated during the test.
  • change system settings to default or original value.
  • should not assume a device in a certain state, e.g., root ready. Most tests don't require root privilege to run. If a test must require root, it should specify that with a RootTargetPreparer in its AndroidTest.xml, as in the following example:
<target_preparer class="com.android.tradefed.targetprep.RootTargetPreparer"/>

Creating Test Mapping files

For the directory requiring test coverage, simply add a TEST_MAPPING JSON file resembling the example below. These rules will ensure the tests run in presubmit checks when any files are touched in that directory or any of its subdirectories.

Following an example

Here is a sample TEST_MAPPING file (it's in JSON format but with comments supported):

{
  "presubmit": [
    // JUnit test with options and file patterns.
    {
      "name": "CtsWindowManagerDeviceTestCases",
      "options": [
        {
          "include-annotation": "android.platform.test.annotations.RequiresDevice"
        }
      ],
      "file_patterns": ["(/|^)Window[^/]*\\.java", "(/|^)Activity[^/]*\\.java"]
    },
    // Device-side GTest with options.
    {
      "name" : "hello_world_test",
      "options": [
        {
          "native-test-flag": "\"servicename1 servicename2\""
        },
        {
          "native-test-timeout": "6000"
        }
      ]
    }
    // Host-side GTest.
    {
      "name" : "net_test_avrcp",
      "host" : true
    }
  ],
  "postsubmit": [
    {
      "name": "CtsWindowManagerDeviceTestCases"
    }
  ],
  "imports": [
    {
      "path": "frameworks/base/services/core/java/com/android/server/am"
    }
  ]
}

Setting attributes

In the above example, presubmit and postsubmit are the names of each test group. See Defining test groups for more information about test groups.

The name of the test module or Trade Federation integration test name (resource path to the test XML file, e.g., uiautomator/uiautomator-demo) can be set in the value of the name attribute. Note the name field cannot use class name or test method name. To narrow down the tests to run, you can use options such as include-filter here. See (include-filter sample usage).

The host setting of a test indicates whether the test is a deviceless test running on host or not. The default value is false, meaning the test requires a device to run. The supported test types are HostGTest for GTest binaries and HostTest for JUnit tests.

The file_patterns attribute allows you to set a list of regex strings for matching the relative path of any source code file (relative to the directory containing the TEST_MAPPING file). In above example, test CtsWindowManagerDeviceTestCases will run in presubmit only when any java file starts with Window or Activity, which exists in the same directory of the TEST_MAPPING file or any of its sub directories, is changed. Backslashes \ need to be escaped as they are in a JSON file.

The imports attribute allows you to include tests in other TEST_MAPPING files without copying the content. Note that the TEST_MAPPING files in the parent directories of the imported path will also be included. Test Mapping allows nested imports; this means two TEST_MAPPING files can import each other, and Test Mapping is able to properly merge the included tests.

The options attribute contains additional TradeFed command line options.

To get a complete list of available options for a given test, run:

tradefed.sh run commandAndExit [test_module] --help

Refer to TradeFed Option Handling for more details about how options work.

Running tests with Atest

To execute the presubmit test rules locally:

  1. Go to the directory containing the TEST_MAPPING file.
  2. Run the command:
atest

All presubmit tests configured in the TEST_MAPPING files of the current directory and its parent directories are run. Atest will locate and run two tests for presubmit (A and B).

This is the simplest way to run presubmit tests in TEST_MAPPING files in the current working directory (CWD) and parent directories. Atest will locate and use the TEST_MAPPING file in CWD and all of its parent directories.

Structuring source code

The following example shows how TEST_MAPPING files can be configured across the source tree.

src
├── project_1
│   └── TEST_MAPPING
├── project_2
│   └── TEST_MAPPING
└── TEST_MAPPING

Content of src/TEST_MAPPING:

{
  "presubmit": [
    {
      "name": "A"
    }
  ]
}

Content of src/project_1/TEST_MAPPING:

{
  "presubmit": [
    {
      "name": "B"
    }
  ],
  "postsubmit": [
    {
      "name": "C"
    }
  ],
  "other_group": [
    {
      "name": "X"
    }
  ]}

Content of src/project_2/TEST_MAPPING:

{
  "presubmit": [
    {
      "name": "D"
    }
  ],
  "import": [
    {
      "path": "src/project_1"
    }
  ]}

Specifying target directories

You can specify a target directory to run tests in TEST_MAPPING files in that directory. The following command will run two tests (A, B).

atest --test-mapping src/project_1

Running postsubmit test rules

You can also use this command to run the postsubmit test rules defined in TEST_MAPPING in src_path (default to CWD) and its parent directories:

atest [--test-mapping] [src_path]:postsubmit

Running only tests that require no device

You can use option --host for Atest to only run tests configured against the host that require no device. Without this option, Atest will run both tests, the ones requiring device and the ones running on host and require no device. The tests will be run in two separate suites.

atest [--test-mapping] --host

Identifying test groups

You can specify test groups in the Atest command. The following command will run all postsubmit tests related to files in directory src/project_1, which contains only one test (C).

Or you can use :all to run all tests regardless of group. The following command runs four tests (A, B, C, X):

atest --test-mapping src/project_1:all

Including subdirectories

By default, running tests in TEST_MAPPING with Atest will run only presubmit tests configured in the TEST_MAPPING file in CWD (or given directory) and its parent directories. If you want to run tests in all TEST_MAPPING files in the sub-directories, use the option --include-subdir to force Atest to include those tests too.

atest --include-subdir

Without the --include-subdir option, Atest will run only test A. With the --include-subdir option, Atest will run two tests (A, B).

Line-level comment is supported

You can add a line-level //-format comment to flesh out the TEST_MAPPING file with a description of the setting that follows. ATest and Trade Federation will preprocess the TEST_MAPPING to a valid JSON format without comments. To keep the JSON file clean and easy to read, only line-level //-format comment is supported.

Example:

{
  // For presubmit test group.
  "presubmit": [
    {
      // Run test on module A.
      "name": "A"
    }
  ]
}