Android Test Station is a testing tool that Android developers and test engineers can use to employ a user interface for running standard Android test suites, such as the Android Compatibility Test Suite (CTS). This tool acts as a web interface for Trade Federation (TF), allowing you to easily run CTS on a set of test devices with minimal setup, as well as establish a schedule to continuously run tests.
Setting up Android Test Station
This section explains how to install and set up Android Test Station.
Android Test Station uses source code from these locations:
- Android Test Station source code
- TradeFed Cluster source code
Installing Android Test Station
Follow any hardware and software requirements for the tests suites you wish to run.
The requirements for CTS are found on source.android.com.
There are no additional hardware requirements, but we recommend using a machine with 100 GB free hard drive space and 8 GB memory, which is enough to store the output files from several runs of the test suite.
There are two ways to install Android Test Station:
- Run the installer program.
- Install it manually, which requires installing multiple programs and resources.
Installing with the installer program
On Ubuntu 20.04+, the installer program installs and configures all the programs and resources needed to run Android Test Station.
To use the install program:
Run the installer program:
curl https://storage.googleapis.com/android-mtt.appspot.com/prod/install.sh | bash
Run
mtt version
to check the installed version of Android Test Station CLI.
Installing manually
Installing Docker
Follow the instructions for installing Docker Community Edition (CE) on your Linux machine.
Follow the post-installation steps to manage Docker as a non-root user.
You might need to restart your terminal window or log out and log in again for the permission changes to take effect.
Installing Python 3
The Android Test Station CLI is verified against Python versions 3.7 through 3.10.
For Ubuntu 16.04 or earlier, first add the repository for Python 3 by doing one of the following:
Run this command:
sudo add-apt-repository ppa:deadsnakes/ppa
Build and install the repository from the source.
To install Python 3, run these commands:
sudo apt-get update
sudo apt install python3 python3-distutils
To install a specific Python 3 version (for example, 3.10), run these commands instead:
sudo apt-get update
sudo apt install python3.10 python3.10-distutils
Getting the Android Test Station CLI
Download the command line interface (CLI) package here.
Starting Android Test Station
Start Android Test Station with the following command:
mtt start
The first time the UI is started, it may take a few minutes to appear. The CLI
displays a web URL to access the UI in a browser. By default, the web URL is
localhost:8000
. If needed, you can change the default port upon startup with
the --port
flag.
If a newer version is available, you can update to the current version. You can check the Release Notes for the latest releases.
To update to the current version, run:
mtt start --force_update
To stop the application, run:
mtt stop
To view a list of other commands, use:
mtt --help
Backing up and restoring the database
To back up the ATS database, stop the application and run the following command,
which backs up the current database into a TAR file named mtt-backup.tar
in
your home directory:
docker run --rm --mount source=mtt-data,target=/data -v ~:/out ubuntu bash -c "cd /data && tar cvf /out/mtt-backup.tar ."
To restore, run the following command before starting the application:
docker run --rm --mount source=mtt-data,target=/data -v ~:/out ubuntu bash -c "cd /data && tar xvf /out/mtt-backup.tar"
Setup Wizard
After you install and run Android Test Station for the first time, the Setup Wizard will take you through a few steps to help you customize the tool for your environment. Any changes you make here can be reconfigured later through the Settings page.
Restoring a configuration backup
If you have a configuration file backed up from another Android Test Station host, you can upload the file to copy any configurations modified from that host by clicking the Upload File button.
Figure 1. Restoring a Configuration Backup
Setting the default service account
You can set a service account that Android Test Station uses by default when accessing your resources (for example, Google Cloud Storage, Google Drive). To authenticate your service account, click Upload Service Account Key and select your service account's JSON key file.
Figure 2. Setting the service account
When the service account is authenticated successfully, the account email address appears in the top right corner of the page. To change the service account, click the account name, remove the current default account, and upload a new service account key.
Figure 3. Changing the service account
Importing config sets
A config set is a bundle of configs for running test suites, including related device actions, build channels, etc. Config sets are hosted in a specific Google Cloud Storage (GCS) bucket. After authenticating the GCS build channel with your Google account, you will see a list of all config sets that are available to you.
Select any config sets you would like to add to your Test Station host and click Import Selected.
Figure 4. Importing a Config Set
Including Wi-Fi settings
Some CTS tests require your device to connect to a Wi-Fi hotspot. To select your Wi-Fi network, enter the WiFi SSID and optional WiFi PSK.
Figure 5. Wi-Fi hotspot settings
After completing the Setup Wizard, the page will reload with the new settings applied.
Connecting a device
USB debugging must be enabled to use a device for testing. To enable debugging:
Follow the instructions in Enable developer options and debugging.
If you plan to use test Android builds preloaded with custom ADB keys, put the custom
.adb_key
files under the~/.android/
directory.The files are loaded automatically and passed to ADB to auto-enable USB debugging after the device is flashed for devices running those builds.
Connect the device to the host machine using USB.
The device will appear in the Android Test Station Devices tab within one minute after refreshing the web interface. You can also view the state of the devices on this tab.
Figure 6. Connecting a device
The different device states are:
- Available - The device is connected and ready to run a test.
- Allocated - The device is connected and currently running a test. Each device can run only one test at a time, so the device must finish its current test before running a new one.
Running a test
Selecting a test
Android Test Station comes with a set of prebundled CTS configurations. To run one of these tests, go to the Test Suites tab and click Run test for the desired test.
Figure 7. Selecting a test
To edit or add new tests, see Adding tests.
Configuring test run
Edit the parameters to use for this specific test run. Most parameters are prepopulated with values defined in the selected test configuration.
This step can be completed using the default values, but you can change any of the parameters, such as Max Retry and Command, to suit your needs.
Figure 8. Configuring a test run
The test run parameters are:
- Name - Name of the test suite you wish to run.
- Run Count - Number of times this test run should be executed when scheduled. Test runs are scheduled using Trade Federation, which runs up to 20 test runs in parallel if there's capacity to do so.
- Max Retry - Maximum number of times to retry a test run if at least one test fails. This is usually set to 4–6 retries for a full CTS run to handle flaky tests.
- Queue Timeout - If a test run remains in the Queued state for too long, it's automatically canceled. Specify the amount of time to wait before cancellation here. The default is 24 hours.
Command - The command to run the test suite. You can enter additional command line arguments here. For example, run a specific module in CTS 8.1 with:
cts-suite -m ShortModuleName
Retry Command - The command for retrying a test suite. You can add additional command line arguments here. For example, to retry only a specific module in CTS 8.1, use:
cts --retry 0 -m ShortModuleName
Retry arguments may differ from those available with the initial command, so check the supported parameters on the official site for the selected test suite.
Previous Test Run - If you wish to re-run a previous test run:
Local - If the run was started on the current host, enter the test run ID seen when viewing the details of the test run.
Figure 9. Local previous test run
Remote - If the run was started on a different host, upload the test results file by selecting Remote, clicking Upload Test Results File, and selecting a file from your local storage.
Figure 10. Remote previous test run
Selecting devices
Click the checkboxes to select the devices to allocate for running the test suite. The shard count should automatically change to match the number of devices selected.
Figure 11. Selecting devices
To select devices by attributes other than device serials, you can manually enter "Device Specs". For example, to select 3 devices whose product name is "bramble", enter the following:
product:bramble;product:bramble;product:bramble
The supported attributes are:
- build_id
- device_serial
- device_type
- hostname
- product
- product_variant
- sim_state
All selected devices must be in the Available state to execute the test run, and they all switch to the Allocated state when the test run is executed. A test run is in the Queued state while it's waiting for devices to become available.
Adding device actions
Device actions are scripts that can be executed before each test run. Some device actions come already configured, such as flashing and rebooting. To create new device actions, see Create a new device action.
Figure 12. Device actions
To add a device action to a test run, click Add new action, select the checkboxes for the actions to add, and click Add Action(s). Device actions are performed sequentially. You can reorder the actions by dragging them.
Figure 13. Reordering actions
Setting test resources
Test resources are files required to execute a test run. For example, running
CTS requires an android-cts*.zip
file, and flashing a device requires you to
provide the build image.
The download URL for the test suite zip file should default to the Google Drive links given to partners. You can select a different file by clicking browse. In the popup window, you can enter a file download link, use a file from an authenticated build channel, or upload a file to use from local storage.
Figure 14. Test resources
Below is the popup window for selecting a test resource by a web URL. You can simply enter the download url link, and click Select button to confirm the selection.
Figure 15. Test Resource Selector - Web URL
If you have uploaded resources to Google Grive, Google Cloud Storage (GCS), or other channels, you can also navigate to the specific channel's tab and select resources there. Here is an example for selecting a resource from google drive.
Figure 16. Test Resource Selector - Google Drive
In addition to just selecting files, wildcard characters are also supported in Filename field. The documentation can be found here.
Figure 17. Test Resource Selector - Wildcard pattern support
You can also select a file from Android Test Station's local file storage. You can upload files into this storage, or directly use local files and directories.
Figure 18. Test Resource Selector - Local file store
Adding rerun configs
You can schedule reruns which will start after the primary run is complete and load its results, but can use different device, actions, or resources.
Figure 19. Adding Rerun Configs
Starting a test run
After you enter the information needed for the test run, click Start Test Run. If all of the information is valid, the test run starts, and you're redirected to a page to view the details and progress of the test run.
Figure 20. Starting a test run
Creating a test plan
Test plans are used to create test runs on a periodic schedule. For example, running CTS 9.0 every day at 5 PM. To create a new test plan, click Create a new test plan.
Figure 21. Creating a test plan
Configure test plan
Enter the name of the test plan and any labels you want to add. Then select a schedule to use.
- Manual - The test plan will only create test runs when a user clicks on Run test plan in the test plan list page.
- Periodic - The test plan will automatically schedule test runs on the periodic schedule selected. For example, scheduling a test run every day at 5:00pm.
- Custom - The test plan will autmatically schedule test runs based on the
cron expression entered. For example,
to schedule a test run every day at 5:00pm, the cron expression would be
0 17 * * *
.
Figure 22. Configuring a test plan
Add test suites
Add tests suites you want to be scheduled by the test plan by clicking + Add test run configuration. Select a test suite from the Name dropdown and click Next step. Then select the devices you would like to run the test on and click Add Configuration. You can add multiple configurations for each test plan.
Figure 23. Configuring a test run
Add device actions
Add the device actions you want to be executed before each test run. See Adding device actions for more details.
Figure 24. Adding device actions
Set test resources
Adding test resources to test plans is the same as adding them to individual test runs. See Setting test resources for more details.
Figure 25. Setting test resources
Viewing test runs
Test run list
View the list of scheduled test runs on the Test Runs page. Click View to see more details about a test run.
You can also filter the list by entering a string into the filter bar and pressing the Enter key. You can use multiple filters by separating them with a comma. The filter returns all rows that contain the exact text (no substring matching) in any column, excluding Status and Created.
An empty filter returns all rows. There's currently no way to filter for rows with empty values.
Figure 26. Test run list
Test run details
You can view the details of a test run here, such as the status, logs, and results.
Figure 27. Test run details
Test run status
The progress for a test run is shown in the Status section. If there's a related message, such as download progress, cancellation reason, or error message, it's shown here as well.
Figure 28. Test run status
The test run states are:
- Pending - Required resources are being downloaded.
- Queued - The test is ready to be run when a device becomes available.
- Running - The test is running on an allocated device.
- Completed - The test has completed and reported its results.
- Canceled - The test was canceled by the user or timed out while trying to find available devices.
- Error - An error occurred that prevented the test from running.
Canceling a test run
If the test run hasn't completed, you can cancel it by clicking Cancel and then clicking Yes in the confirmation dialog. Test runs are also automatically canceled if they remain in the Queued state longer than the queue_timeout_seconds field. Canceling a test run while in a Running state may take a few minutes to take effect.
Figure 29. Canceling a test run
Test run results
After a test run has finished, results will be collected and displayed. You can
view additional details by clicking the arrow for each run. Click View Output
Files to see the test artifacts collected, such as test_result.xml
and
test_result_failures.html
.
Figure 30. Test run results
You can view live host and Tradefed logs in the Logs tab.
Figure 31. Logs tab
The results for individual modules are on the Test Results tab.
Figure 32. Test Results tab
You can download the files used as test resources by clicking Open in the Test Resources tab.
Figure 33. Test Resources tab
To see the details of the test run, such as create_time, go to the Config tab.
Figure 34. Config tab
Advanced features
Managing config files
Android Test Station uses configuration files written in YAML to load predefined options such as tests, build channels, and device actions. An example config file for some options is shown below.
// example_file.yaml
tests:
- id : android.cts.9_0.arm
name: CTS 9.0 (ARM)
test_resource_defs:
- name: android-cts.zip
default_download_url: https://dl.google.com/dl/android/cts/android-cts-9.0_r7-linux_x86-arm.zip
test_resource_type: TEST_PACKAGE
command: cts
env_vars:
- name: TF_PATH
value: ${TF_WORK_DIR}/android-cts/tools:${TF_WORK_DIR}/android-cts/testcases
- name: LD_LIBRARY_PATH
value: ${TF_WORK_DIR}/android-cts/lib:${TF_WORK_DIR}/android-cts/lib64
setup_scripts:
output_file_patterns:
- android-cts/logs/latest/.*
- android-cts/results/latest/.*\.html
- android-cts/results/latest/compatibility_result\..*
- android-cts/results/latest/logo.png
- android-cts/results/latest/test_result.xml
result_file: test_result.xml
java_properties:
- name: CTS_ROOT
value: ${TF_WORK_DIR}
context_file_dir: android-cts/results/
context_file_pattern: '[\d_\.]+\.zip'
retry_command_line: retry --retry 0
runner_sharding_args: --shard-count ${TF_SHARD_COUNT}
build_channels:
- id: google_drive
name: Google Drive
provider_name: Google Drive
device_actions:
- id: flash
name: Flash
test_resource_defs:
- name: bootloader.img
test_resource_type: DEVICE_IMAGE
- name: radio.img
test_resource_type: DEVICE_IMAGE
- name: img.zip
test_resource_type: DEVICE_IMAGE
tradefed_target_preparers:
- class_name: com.android.tradefed.targetprep.RunHostCommandTargetPreparer
option_values:
- name: work-dir
values:
- ${TF_WORK_DIR}
- name: host-setup-command
values:
- adb -s $SERIAL reboot-bootloader
- fastboot -s $SERIAL flash bootloader bootloader.img
- fastboot -s $SERIAL flash radio radio.img
- fastboot -s $SERIAL reboot-bootloader
- fastboot -s $SERIAL -w update img.zip
- adb -s $SERIAL wait-for-device
- name: host-cmd-timeout
values:
- 10m
When you set up your Android Test Station instance, you can share your configuration with other users by exporting it as a file. To do this, go to the Settings page and click Export in the top right.
Figure 35. Configuration file management
After your configuration file is downloaded, share the file with other users. They can add the configuration file to their Android Test Station instance by clicking Import and selecting the config file.
Creating a new device action
Device actions are used for automating the device setup process. Actions are scripts executed on each device the test is running on before each test run, including before retries. To view a list of available device actions, go to the Settings page and click the Device Actions tab. Several device actions come already configured, such as rebooting and flashing.
Figure 36. Device Actions tab
Adding a new device action
Click New device action.
Figure 37. New device action button
Enter a name and description.
Figure 38. Device action name
Click Add Target Preparer.
Enter the Trade Federation Target Preparer full class name, for example,
com.android.tradefed.targetprep.RunHostCommandTargetPreparer
.Figure 39. Adding a target preparer
A list of available target preparers can be found in the com.android.tradefed.targetprep reference.
Figure 40. Target Preparer list
Add any options to use with the target preparer. To view available options, check targetprep for the source code for each target preparer in AOSP:
Figure 41. Action option example
To add an option, click Add Target Preparer Option and enter the required values.
Figure 42. Action command example
Define the test resources that are needed to execute the device action, for example, build images for flashing. To add a resource definition, click Add Test Resource and fill in the required fields. If you know where your files are located, you can provide a default download URL by clicking browse. If the target preparers accept directory as test resource, select Decompress. Then specify the relative Destination directory under the temporary working directory and the File Names to be decompressed. If no file names are given, all files are decompressed from the test resource.
Figure 43. Action test resources
Click Update.
Figure 44. Action save changes
Managing tests
Editing a test
To edit a saved test, go to the Tests page and click Edit on the row of the test you want to modify. After changing the test configuration, click Update.
Figure 45. Editing a test
Adding a new test
To add a new test, go to the Tests page and click Create a New Test. Enter the appropriate information and click Create.
Figure 46. Creating a test
Figure 47. Copying a test
Exporting host configurations
After you configure a host, you can export the host's configurations to a file. You can upload this file to other hosts to copy the saved configurations.
To export a host's configurations, go to the Settings page and click Export in the top right corner.
Figure 48. Exporting a host config
To import a host configuration file, go to the Settings page and click Import in the top right corner.
Figure 49. Importing a host config
Using local files and directories
Starting in version R11, files in the $HOME/.ats_storage
directory are
automatically accessible in Android Test Station. Copy or move a file into that
directory, then you can select it from the Local File tab when scheduling a
test run.
cp /path/to/file $HOME/.ats_storage
Figure 50. Selecting a file from the $HOME/.ats_storage
directory
You can mount additional directories into the local file store with the
--mount_local_path
flag.
mtt start --mount_local_path=/path/to/dir1 --mount_local_path=/path/to/dir2:renamed_dir2
Figure 51. Additional directories mounted in the local file store
Enable multi-host mode
With Multi-host mode, users can use a single ATS controller host to manage the devices and tests on multiple ATS worker hosts.
Figure 52. Multi-host mode architecture
To start the ATS controller, use the following command:
mtt start --operation_mode=ON_PREMISE
Check controller is accessible at
http://${CONTROLLER_HOSTNAME}:8000
.To start the workers, use the following command:
mtt start --control_server_url=http://CONTROLLER_HOSTNAME:8000 --operation_mode=ON_PREMISE
If your network doesn't allow hosts to communicate with each other, you will need to follow the more advanced setup instructions below on ATS worker.
Connect the two hosts using SSH tunnels. Select ports for the primary and file server ports, e.g. 9000 and 9006.
ssh -L ATS_PORT:localhost:8000 -L FS_PORT:localhost:8006 CONTROLLER_HOSTNAME
Configure and start ATS.
DOCKER_GATEWAY_IP_ADDRESS=$(ip -4 addr show dev docker0 | grep -Eo 'inet [.0-9]+/' | grep -Eo '[.0-9]+')
socat tcp-listen:ATS_PORT,bind="${DOCKER_GATEWAY_IP_ADDRESS}",reuseaddr,fork tcp-connect:127.0.0.1:ATS_PORT &
socat tcp-listen:FS_PORT,bind="${DOCKER_GATEWAY_IP_ADDRESS}",reuseaddr,fork tcp-connect:127.0.0.1:FS_PORT &
mtt start --control_server_url=http://${DOCKER_GATEWAY_IP_ADDRESS}:ATS_PORT \ --control_file_server_url=http://${DOCKER_GATEWAY_IP_ADDRESS}:FS_PORT \ --operation_mode=ON_PREMISE
Support
Bug Reporting
Your contribution to Android Test Station helps improve the development of the tool, and we want your input! See the ATS release notes for details on the latest release. To report bugs or offer suggestions, please file a bug report. Partners should report bugs or suggestions via their partner channels.