Android Test Station

Android Test Station is a testing tool that Android developers and test engineers can use to employ a user interface for running standard Android test suites, such as the Android Compatibility Test Suite (CTS). This tool acts as a web interface for Trade Federation (TF), allowing you to easily run CTS on a set of test devices with minimal setup, as well as establish a schedule to continuously run tests.

Setting up Android Test Station

This section explains how to install and set up Android Test Station.

Android Test Station uses source code from these locations:

• Android Test Station source code
• TradeFed Cluster source code

Installing Android Test Station

Note: The links provided here are for installation on an Ubuntu Linux machine, which is recommended. This tool may work on other operating systems, but may need different steps for each requirement.

Follow any hardware and software requirements for the tests suites you wish to run.

The requirements for CTS are found on source.android.com.

There are no additional hardware requirements, but we recommend using a machine with 100 GB free hard drive space and 8 GB memory, which is enough to store the output files from several runs of the test suite.

There are two ways to install Android Test Station:

• Run the installer program.
• Install it manually, which requires installing multiple programs and resources.

Installing with the installer program

On Ubuntu 20.04+, the installer program installs and configures all the programs and resources needed to run Android Test Station.

To use the install program:

1. Run the installer program:

   curl https://storage.googleapis.com/android-mtt.appspot.com/prod/install.sh | bash
   

2. Run mtt version to check the installed version of Android Test Station CLI.

Installing manually

Installing Docker

1. Follow the instructions for installing Docker Community Edition (CE) on your Linux machine.

2. Follow the post-installation steps to manage Docker as a non-root user.

3. You might need to restart your terminal window or log out and log in again for the permission changes to take effect.

Note: ATS data is stored in the /var/lib/docker folder by default. To change the storage location used, set the data-root field to a new storage path by modifying the docker daemon config file.

Installing Python 3

The Android Test Station CLI is verified against Python versions 3.7 through 3.10.

For Ubuntu 16.04 or earlier, first add the repository for Python 3 by doing one of the following:

• Run this command:

  sudo add-apt-repository ppa:deadsnakes/ppa
  

• Build and install the repository from the source.

To install Python 3, run these commands:

sudo apt-get update
sudo apt install python3 python3-distutils

To install a specific Python 3 version (for example, 3.10), run these commands instead:

sudo apt-get update
sudo apt install python3.10 python3.10-distutils

Getting the Android Test Station CLI

Download the command line interface (CLI) package here.

Starting Android Test Station

Start Android Test Station with the following command:

mtt start

The first time the UI is started, it may take a few minutes to appear. The CLI displays a web URL to access the UI in a browser. By default, the web URL is localhost:8000. If needed, you can change the default port upon startup with the --port flag.

If a newer version is available, you can update to the current version. You can check the Release Notes for the latest releases.

Note: If you are updating from a version older than R5, back up your data before upgrading so that you can safely roll back to the previous version without losing data. See Backing up and restoring the database for backup and restore instructions.

To update to the current version, run:

mtt start --force_update

To stop the application, run:

mtt stop

To view a list of other commands, use:

mtt --help

Backing up and restoring the database

To back up the ATS database, stop the application and run the following command, which backs up the current database into a TAR file named mtt-backup.tar in your home directory:

docker run --rm --mount source=mtt-data,target=/data -v ~:/out ubuntu bash -c "cd /data && tar cvf /out/mtt-backup.tar ."

To restore, run the following command before starting the application:

docker run --rm --mount source=mtt-data,target=/data -v ~:/out ubuntu bash -c "cd /data && tar xvf /out/mtt-backup.tar"

Setup Wizard

After you install and run Android Test Station for the first time, the Setup Wizard will take you through a few steps to help you customize the tool for your environment. Any changes you make here can be reconfigured later through the Settings page.

Restoring a configuration backup

If you have a configuration file backed up from another Android Test Station host, you can upload the file to copy any configurations modified from that host by clicking the Upload File button.

Figure 1. Restoring a Configuration Backup

Setting the default service account

You can set a service account that Android Test Station uses by default when accessing your resources (for example, Google Cloud Storage, Google Drive). To authenticate your service account, click Upload Service Account Key and select your service account's JSON key file.

Figure 2. Setting the service account

When the service account is authenticated successfully, the account email address appears in the top right corner of the page. To change the service account, click the account name, remove the current default account, and upload a new service account key.

Figure 3. Changing the service account

Importing config sets

A config set is a bundle of configs for running test suites, including related device actions, build channels, etc. Config sets are hosted in a specific Google Cloud Storage (GCS) bucket. After authenticating the GCS build channel with your Google account, you will see a list of all config sets that are available to you.

Note: Currently there are no publicly available config sets, so this list may be empty for some users.

Select any config sets you would like to add to your Test Station host and click Import Selected.

Figure 4. Importing a Config Set

Including Wi-Fi settings

Some CTS tests require your device to connect to a Wi-Fi hotspot. To select your Wi-Fi network, enter the WiFi SSID and optional WiFi PSK.

Figure 5. Wi-Fi hotspot settings

After completing the Setup Wizard, the page will reload with the new settings applied.

Connecting a device

USB debugging must be enabled to use a device for testing. To enable debugging:

1. Follow the instructions in Enable developer options and debugging.

2. If you plan to use test Android builds preloaded with custom ADB keys, put the custom .adb_key files under the ~/.android/ directory.

   The files are loaded automatically and passed to ADB to auto-enable USB debugging after the device is flashed for devices running those builds.

   Note: To add custom ADB keys to your Android debuggable builds, see the change to add PRODUCT_ADB_KEYS.

3. Connect the device to the host machine using USB.

   The device will appear in the Android Test Station Devices tab within one minute after refreshing the web interface. You can also view the state of the devices on this tab.

   

   Figure 6. Connecting a device

The different device states are:

• Available - The device is connected and ready to run a test.
• Allocated - The device is connected and currently running a test. Each device can run only one test at a time, so the device must finish its current test before running a new one.

Important: Don't use ADB in a terminal outside of Android Test Station while the device is connected. The app conflicts with the Android Test Station ADB instance and causes the device to disconnect from it.

Running a test

Selecting a test

Android Test Station comes with a set of prebundled CTS configurations. To run one of these tests, go to the Test Suites tab and click Run test for the desired test.

Figure 7. Selecting a test

To edit or add new tests, see Adding tests.

Configuring test run

Edit the parameters to use for this specific test run. Most parameters are prepopulated with values defined in the selected test configuration.

This step can be completed using the default values, but you can change any of the parameters, such as Max Retry and Command, to suit your needs.

Note: Any changes made here affect only this test run and don't change the saved test configuration. To change a test configuration, see Editing a Test.

Figure 8. Configuring a test run

The test run parameters are:

• Name - Name of the test suite you wish to run.
• Run Count - Number of times this test run should be executed when scheduled. Test runs are scheduled using Trade Federation, which runs up to 20 test runs in parallel if there's capacity to do so.
• Max Retry - Maximum number of times to retry a test run if at least one test fails. This is usually set to 4–6 retries for a full CTS run to handle flaky tests.
• Queue Timeout - If a test run remains in the Queued state for too long, it's automatically canceled. Specify the amount of time to wait before cancellation here. The default is 24 hours.

• Command - The command to run the test suite. You can enter additional command line arguments here. For example, run a specific module in CTS 8.1 with:

  cts-suite -m ShortModuleName
  

• Retry Command - The command for retrying a test suite. You can add additional command line arguments here. For example, to retry only a specific module in CTS 8.1, use:

  cts --retry 0 -m ShortModuleName
  

  Retry arguments may differ from those available with the initial command, so check the supported parameters on the official site for the selected test suite.

• Previous Test Run - If you wish to re-run a previous test run:

  • Local - If the run was started on the current host, enter the test run ID seen when viewing the details of the test run.

    

    Figure 9. Local previous test run

  • Remote - If the run was started on a different host, upload the test results file by selecting Remote, clicking Upload Test Results File, and selecting a file from your local storage.

    

    Figure 10. Remote previous test run

Selecting devices

Click the checkboxes to select the devices to allocate for running the test suite. The shard count should automatically change to match the number of devices selected.

Figure 11. Selecting devices

To select devices by attributes other than device serials, you can manually enter "Device Specs". For example, to select 3 devices whose product name is "bramble", enter the following:

product:bramble;product:bramble;product:bramble

The supported attributes are:

• build_id
• device_serial
• device_type
• hostname
• product
• product_variant
• sim_state

All selected devices must be in the Available state to execute the test run, and they all switch to the Allocated state when the test run is executed. A test run is in the Queued state while it's waiting for devices to become available.

Note: To be used with sharding, tests must implement the IShardableTest interface, or they fail on invocation. Currently, all CTS and GTS tests support sharding.

Adding device actions

Device actions are scripts that can be executed before each test run. Some device actions come already configured, such as flashing and rebooting. To create new device actions, see Create a new device action.

Figure 12. Device actions

To add a device action to a test run, click Add new action, select the checkboxes for the actions to add, and click Add Action(s). Device actions are performed sequentially. You can reorder the actions by dragging them.

Figure 13. Reordering actions

Setting test resources

Test resources are files required to execute a test run. For example, running CTS requires an android-cts*.zip file, and flashing a device requires you to provide the build image.

The download URL for the test suite zip file should default to the Google Drive links given to partners. You can select a different file by clicking browse. In the popup window, you can enter a file download link, use a file from an authenticated build channel, or upload a file to use from local storage.

Figure 14. Test resources

Below is the popup window for selecting a test resource by a web URL. You can simply enter the download url link, and click Select button to confirm the selection.

Figure 15. Test Resource Selector - Web URL

If you have uploaded resources to Google Grive, Google Cloud Storage (GCS), or other channels, you can also navigate to the specific channel's tab and select resources there. Here is an example for selecting a resource from google drive.

Figure 16. Test Resource Selector - Google Drive

In addition to just selecting files, wildcard characters are also supported in Filename field. The documentation can be found here.

Note: When wildcards are entered, the most recent file that matches the pattern will be used to run the test.

Figure 17. Test Resource Selector - Wildcard pattern support

You can also select a file from Android Test Station's local file storage. You can upload files into this storage, or directly use local files and directories.

Figure 18. Test Resource Selector - Local file store

Adding rerun configs

You can schedule reruns which will start after the primary run is complete and load its results, but can use different device, actions, or resources.

Figure 19. Adding Rerun Configs

Starting a test run

After you enter the information needed for the test run, click Start Test Run. If all of the information is valid, the test run starts, and you're redirected to a page to view the details and progress of the test run.

Figure 20. Starting a test run

Creating a test plan

Test plans are used to create test runs on a periodic schedule. For example, running CTS 9.0 every day at 5 PM. To create a new test plan, click Create a new test plan.

Figure 21. Creating a test plan

Configure test plan

Enter the name of the test plan and any labels you want to add. Then select a schedule to use.

• Manual - The test plan will only create test runs when a user clicks on Run test plan in the test plan list page.
• Periodic - The test plan will automatically schedule test runs on the periodic schedule selected. For example, scheduling a test run every day at 5:00pm.
• Custom - The test plan will autmatically schedule test runs based on the cron expression entered. For example, to schedule a test run every day at 5:00pm, the cron expression would be 0 17 * * *.

Figure 22. Configuring a test plan

Add test suites

Add tests suites you want to be scheduled by the test plan by clicking + Add test run configuration. Select a test suite from the Name dropdown and click Next step. Then select the devices you would like to run the test on and click Add Configuration. You can add multiple configurations for each test plan.

Figure 23. Configuring a test run

Add device actions

Add the device actions you want to be executed before each test run. See Adding device actions for more details.

Figure 24. Adding device actions

Set test resources

Adding test resources to test plans is the same as adding them to individual test runs. See Setting test resources for more details.

Figure 25. Setting test resources

Viewing test runs

Test run list

View the list of scheduled test runs on the Test Runs page. Click View to see more details about a test run.

You can also filter the list by entering a string into the filter bar and pressing the Enter key. You can use multiple filters by separating them with a comma. The filter returns all rows that contain the exact text (no substring matching) in any column, excluding Status and Created.

An empty filter returns all rows. There's currently no way to filter for rows with empty values.

Figure 26. Test run list

Test run details

You can view the details of a test run here, such as the status, logs, and results.

Figure 27. Test run details

Test run status

The progress for a test run is shown in the Status section. If there's a related message, such as download progress, cancellation reason, or error message, it's shown here as well.

Figure 28. Test run status

The test run states are:

• Pending - Required resources are being downloaded.
• Queued - The test is ready to be run when a device becomes available.
• Running - The test is running on an allocated device.
• Completed - The test has completed and reported its results.
• Canceled - The test was canceled by the user or timed out while trying to find available devices.
• Error - An error occurred that prevented the test from running.

Canceling a test run

If the test run hasn't completed, you can cancel it by clicking Cancel and then clicking Yes in the confirmation dialog. Test runs are also automatically canceled if they remain in the Queued state longer than the queue_timeout_seconds field. Canceling a test run while in a Running state may take a few minutes to take effect.

Figure 29. Canceling a test run

Test run results

After a test run has finished, results will be collected and displayed. You can view additional details by clicking the arrow for each run. Click View Output Files to see the test artifacts collected, such as test_result.xml and test_result_failures.html.

Figure 30. Test run results

You can view live host and Tradefed logs in the Logs tab.

Figure 31. Logs tab

The results for individual modules are on the Test Results tab.

Figure 32. Test Results tab

You can download the files used as test resources by clicking Open in the Test Resources tab.

Figure 33. Test Resources tab

To see the details of the test run, such as create_time, go to the Config tab.

Figure 34. Config tab

Advanced features

Managing config files

Android Test Station uses configuration files written in YAML to load predefined options such as tests, build channels, and device actions. An example config file for some options is shown below.

// example_file.yaml
tests:
- id : android.cts.9_0.arm
  name: CTS 9.0 (ARM)
  test_resource_defs:
  - name: android-cts.zip
    default_download_url: https://dl.google.com/dl/android/cts/android-cts-9.0_r7-linux_x86-arm.zip
    test_resource_type: TEST_PACKAGE
  command: cts
  env_vars:
  - name: TF_PATH
    value: ${TF_WORK_DIR}/android-cts/tools:${TF_WORK_DIR}/android-cts/testcases
  - name: LD_LIBRARY_PATH
    value: ${TF_WORK_DIR}/android-cts/lib:${TF_WORK_DIR}/android-cts/lib64
  setup_scripts:
  output_file_patterns:
  - android-cts/logs/latest/.*
  - android-cts/results/latest/.*\.html
  - android-cts/results/latest/compatibility_result\..*
  - android-cts/results/latest/logo.png
  - android-cts/results/latest/test_result.xml
  result_file: test_result.xml
  java_properties:
  - name: CTS_ROOT
    value: ${TF_WORK_DIR}
  context_file_dir: android-cts/results/
  context_file_pattern: '[\d_\.]+\.zip'
  retry_command_line: retry --retry 0
  runner_sharding_args: --shard-count ${TF_SHARD_COUNT}

build_channels:
- id: google_drive
  name: Google Drive
  provider_name: Google Drive

device_actions:
- id: flash
  name: Flash
  test_resource_defs:
  - name: bootloader.img
    test_resource_type: DEVICE_IMAGE
  - name: radio.img
    test_resource_type: DEVICE_IMAGE
  - name: img.zip
    test_resource_type: DEVICE_IMAGE
  tradefed_target_preparers:
  - class_name: com.android.tradefed.targetprep.RunHostCommandTargetPreparer
    option_values:
    - name: work-dir
      values:
      - ${TF_WORK_DIR}
    - name: host-setup-command
      values:
      - adb -s $SERIAL reboot-bootloader
      - fastboot -s $SERIAL flash bootloader bootloader.img
      - fastboot -s $SERIAL flash radio radio.img
      - fastboot -s $SERIAL reboot-bootloader
      - fastboot -s $SERIAL -w update img.zip
      - adb -s $SERIAL wait-for-device
    - name: host-cmd-timeout
      values:
      - 10m

When you set up your Android Test Station instance, you can share your configuration with other users by exporting it as a file. To do this, go to the Settings page and click Export in the top right.

Figure 35. Configuration file management

After your configuration file is downloaded, share the file with other users. They can add the configuration file to their Android Test Station instance by clicking Import and selecting the config file.

Creating a new device action

Device actions are used for automating the device setup process. Actions are scripts executed on each device the test is running on before each test run, including before retries. To view a list of available device actions, go to the Settings page and click the Device Actions tab. Several device actions come already configured, such as rebooting and flashing.

Figure 36. Device Actions tab

Adding a new device action

1. Click New device action.

   

   Figure 37. New device action button

2. Enter a name and description.

   

   Figure 38. Device action name

3. Click Add Target Preparer.

4. Enter the Trade Federation Target Preparer full class name, for example, com.android.tradefed.targetprep.RunHostCommandTargetPreparer.

   

   Figure 39. Adding a target preparer

   A list of available target preparers can be found in the com.android.tradefed.targetprep reference.

   

   Figure 40. Target Preparer list

5. Add any options to use with the target preparer. To view available options, check targetprep for the source code for each target preparer in AOSP:

   

   Figure 41. Action option example

6. To add an option, click Add Target Preparer Option and enter the required values.

   

   Figure 42. Action command example

7. Define the test resources that are needed to execute the device action, for example, build images for flashing. To add a resource definition, click Add Test Resource and fill in the required fields. If you know where your files are located, you can provide a default download URL by clicking browse. If the target preparers accept directory as test resource, select Decompress. Then specify the relative Destination directory under the temporary working directory and the File Names to be decompressed. If no file names are given, all files are decompressed from the test resource.

   

   Figure 43. Action test resources

8. Click Update.

   

   Figure 44. Action save changes

Managing tests

Editing a test

To edit a saved test, go to the Tests page and click Edit on the row of the test you want to modify. After changing the test configuration, click Update.

Figure 45. Editing a test

Note: To modify one of the prebundled tests, create a new variant instead with the Make a Copy feature described in Adding a new test. Any changes to the prebundled test configurations are reverted when Android Test Station is restarted.

Adding a new test

To add a new test, go to the Tests page and click Create a New Test. Enter the appropriate information and click Create.

Figure 46. Creating a test

Note: To prefill data from a saved test, go to the Tests page, click the three dots at the end of the row for the test you want to copy, and click Copy.

Figure 47. Copying a test

Exporting host configurations

After you configure a host, you can export the host's configurations to a file. You can upload this file to other hosts to copy the saved configurations.

To export a host's configurations, go to the Settings page and click Export in the top right corner.

Figure 48. Exporting a host config

To import a host configuration file, go to the Settings page and click Import in the top right corner.

Figure 49. Importing a host config

Using local files and directories

Starting in version R11, files in the $HOME/.ats_storage directory are automatically accessible in Android Test Station. Copy or move a file into that directory, then you can select it from the Local File tab when scheduling a test run.

cp /path/to/file $HOME/.ats_storage

Figure 50. Selecting a file from the $HOME/.ats_storage directory

You can mount additional directories into the local file store with the --mount_local_path flag.

mtt start --mount_local_path=/path/to/dir1 --mount_local_path=/path/to/dir2:renamed_dir2

Figure 51. Additional directories mounted in the local file store

Enable multi-host mode

With Multi-host mode, users can use a single ATS controller host to manage the devices and tests on multiple ATS worker hosts.

Figure 52. Multi-host mode architecture

1. To start the ATS controller, use the following command:

   mtt start --operation_mode=ON_PREMISE
   

2. Check controller is accessible at http://${CONTROLLER_HOSTNAME}:8000.

3. To start the workers, use the following command:

   mtt start --control_server_url=http://CONTROLLER_HOSTNAME:8000 --operation_mode=ON_PREMISE
   

If your network doesn't allow hosts to communicate with each other, you will need to follow the more advanced setup instructions below on ATS worker.

1. Connect the two hosts using SSH tunnels. Select ports for the primary and file server ports, e.g. 9000 and 9006.

   ssh -L ATS_PORT:localhost:8000 -L FS_PORT:localhost:8006 CONTROLLER_HOSTNAME
   

2. Configure and start ATS.

   DOCKER_GATEWAY_IP_ADDRESS=$(ip -4 addr show dev docker0 | grep -Eo 'inet [.0-9]+/' | grep -Eo '[.0-9]+')
   socat tcp-listen:ATS_PORT,bind="${DOCKER_GATEWAY_IP_ADDRESS}",reuseaddr,fork tcp-connect:127.0.0.1:ATS_PORT &
   socat tcp-listen:FS_PORT,bind="${DOCKER_GATEWAY_IP_ADDRESS}",reuseaddr,fork tcp-connect:127.0.0.1:FS_PORT &
   mtt start --control_server_url=http://${DOCKER_GATEWAY_IP_ADDRESS}:ATS_PORT \
                   --control_file_server_url=http://${DOCKER_GATEWAY_IP_ADDRESS}:FS_PORT \
                   --operation_mode=ON_PREMISE
   

Support

Bug Reporting

Your contribution to Android Test Station helps improve the development of the tool, and we want your input! See the ATS release notes for details on the latest release. To report bugs or offer suggestions, please file a bug report. Partners should report bugs or suggestions via their partner channels.