Complete Automotive Tests in a Box

Complete Automotive Tests in a Box (CATBox) is an open source package that provides the frameworks and tools required to streamline and execute automotive tests with minimal configuration. CATBox is scalable and compatible with the tools, infrastructure, and tests you use to test and validate your Android Automotive OS (AAOS) implementations. As a result, you can provide high-quality products to the marketplace.

CATBox design and implementation

The CATBox test suite provides the tools, frameworks, and tests needed to execute functional and performance tests on an automotive target device and while using a companion device.

The following image illustrates a high level design of a CATBox package.

Figure 1. CATBox package

Tradefed

Tradefed is an open source continuous testing framework used to execute tests on Android devices. For details, see Trade Federation Overview.

Automotive test framework

CATBox is based on an Automotive test framework called Spectatio, which provides APIs for testing various apps on automotive devices. It is built on top of UI Automator, an open source testing framework, that provides a set of APIs to build UI tests that interact with user and system applications.

Note: Passing CATBox tests doesn’t ensure the Android system compatibility. Instead, CATBox tests ensure the functionality and performance for reference applications on an automotive device. To ensure compatibility with Android, we recommend executing test suites such as CTS and VTS. For details on Android compatibility test suites, see Compatibility Test Suite.

CATBox test suite downloads

Android 14

Android Automotive 14 is the release of the development milestone code-named U. Use the following links to download CATBox test suites for Android 14:

• Android 14 (arm64)
• Android 14 (x86_64)

Android 12

Android 12 is the release of the development milestone code-named S. Use the following links to download CATBox test suites for Android 12:

• Android 12 (arm64)
• Android 12 (x86_64)

Android 11

Android 11 is the release of the development milestone code-named R. Use the following links to download CATBox test suites for Android 11:

• Android 11 (arm64)
• Android 11 (x86_64)

Build CATBox

Use the catbox build command once the Android AOSP code is available on the local workstation.

To build the CATBox package locally:

1. Select the Device Target based on the device architecture and run the following command:

   lunch <target>
   

2. To build CATBox, run the following command:

   make catbox
   

   This command creates the android-catbox.zip file in the /out/host/linux-x86/catbox directory, which can then be used to run tests.

Set up the environment

The environment setup required to execute CATBox tests is similar to Setting up CTS. To set up the environment, complete each of the following tasks:

• Install ADB and AAPT.

• Install JDK. See Java Development Kit for Ubuntu.

• Increase memory limit. See Storage requirements.

Set up and run CATBox

Use the Tradefed test framework to run the tests on Android devices. Before you set up CATBox, see the Trade Federation Overview.

To set up and run CATBox, see the sections below.

Set up the device

Before running a test plan, complete the following steps to set up your device:

1. Ensure that you have installed the recent versions of both the Android Debug Bridge (adb) and the Android Asset Packaging Tool (AAPT), as well as added the location of these tools to the system path of your machine. For details, see ADB and AAPT.

2. Connect at least one device and prepare the Device Under Test (DUT) as follows:

   1. To factory reset the automotive device, go to Settings > System > Reset Options > Erase all data.

   2. Wait for the device to automatically restart.

   3. Make sure adb is enabled. To confirm the device is available, run the following command:

      adb devices
      

   4. To enable root, run the following command:

      adb -s DEVICE_SERIAL root
      

      Replace DEVICE_SERIAL with the Serial ID for the automotive device.

   5. Connect to Wi-Fi.

Extract the CATBox package

After setting up the device, unzip the CATBox package and go to the extracted folder:

unzip android-catbox.zip
cd android-catbox

Execute tests in CATBox

Note: Some tests, such as those for Dialer, Bluetooth audio, and Contacts list tests, require additional device setup.

To run CATBox with the test plans in the CATBox package:

./tools/catbox-tradefed run commandAndExit TEST_PLAN_NAME --serial DEVICE_SERIAL

Replace DEVICE_SERIAL with the Serial ID for the automotive device. If you have multiple devices connected to the host machine, use DEVICE_SERIAL to distinguish them. Replace TEST_PLAN_NAME with the name of the test plan you want to run.

To get the most complete list of available plans, use the command line:

./tools/catbox-tradefed list plans | grep -i catbox

Types of test plans

The following table displays the available functional test plans.

Functional test plans	Description
catbox-functional	Functional tests.
catbox-functional-app-info-setting	App info setting functional tests.
catbox-functional-admin-user-grant-permissions	Admin user grant permissions functional tests.
catbox-functional-app-info-setting-ui-elements	App info setting functional tests.
catbox-functional-appgrid	App grid functional tests.
catbox-functional-base	List of functional tests.
catbox-functional-bluetooth-audio	Bluetooth audio Functional tests.
catbox-functional-bluetooth-palette	Status bar functional tests.
catbox-functional-bluetooth-tests	Bluetooth tests using Mobly.
catbox-functional-brightness-palette	Status bar functional tests.
catbox-functional-btmoped	Bluetooth Moped-based functional tests.
catbox-functional-date-time-setting	Date and time setting functional tests.
catbox-functional-dial	Dial functional tests.
catbox-functional-display-setting	Display setting functional tests.
catbox-functional-driving-optimized-apps	UX restriction functional tests.
catbox-functional-enable-developers-option	Enable developers option functional tests.
catbox-functional-home	Home functional tests.
catbox-functional-lock-screen	Lock screen functional tests.
catbox-functional-mediacenter	Media Center functional tests.
catbox-functional-microphone-recent-apps	Microphone setting functional tests.
catbox-functional-microphone-setting	Microphone setting functional tests.
catbox-functional-multiuser	Multiuser functional tests.
catbox-functional-multiuser-system-user	Multiuser functional tests.
catbox-functional-navigation-bar	Navigation bar functional tests.
catbox-functional-network-palette	Status bar functional tests.
catbox-functional-network-setting	Network setting functional tests.
catbox-functional-notification	Notification functional tests.
catbox-functional-privacy-permission-manager	Privacy App Permission setting functional tests.
catbox-functional-privacy-setting	Microphone setting functional tests.
catbox-functional-privacy-setting-ui-elements	Privacy setting functional tests.
catbox-functional-profile-icon	Profile Icon Bar functional tests.
catbox-functional-profiles-icon-list	Profiles icon list tests.
catbox-functional-security-setting	Security setting functional tests.
catbox-functional-setting	Setting functional tests.
catbox-functional-settings-location	Setting location Tests.
catbox-functional-sound-setting	Sound setting functional tests.
catbox-functional-status-bar	Status bar functional tests.
catbox-functional-storage-setting	Storage setting functional tests.
catbox-functional-system-setting	System setting functional tests.
catbox-functional-systemui	System UI tests.
catbox-functional-ux-restriction	UX restriction functional tests.
catbox-functional-ux-restriction-facet-bar	UX restriction functional tests.

The following table lists CATBox performance test plans.

Performance test plans	Description
catbox-performance-cold-app-start-up-dialer	Cold app start-up performance test for Dialer and Phone.
catbox-performance-cold-app-start-up-mediacenter	Cold app start-up performance test for Media Center.
catbox-performance-cold-app-start-up-settings	Cold app start-up performance test for settings.
catbox-performance-create-and-switch-to-new-guest	Measure latency to switch to a new guest.
catbox-performance-create-and-switch-to-new-user	Measure latency to switch to a new non-admin user.
catbox-performance-hot-app-start-up-dialer	Hot app start-up performance test for Dialer and Phone.
catbox-performance-hot-app-start-up-mediacenter	Hot app start-up performance test for Media Center.
catbox-performance-hot-app-start-up-settings	Hot app start-up performance test for Settings.
catbox-performance-jank-appgrid	Performance test for Appgrid.
catbox-performance-jank-contact-list	Performance test for Contacts.
catbox-performance-jank-media	Performance tests for Media.
catbox-performance-jank-media-switch-playback	Performance test for media switch playback.
catbox-performance-jank-notifications	Performance test for Notifications.
catbox-performance-jank-settings	Performance test for Settings.
catbox-performance-switch-to-existing-user	Measure latency to switch to a existing user.

Run the multidevice host side test

The execution of multidevice host side tests uses CATBox to run the test in one session. For example, BTDiscoveryTest:

make catbox
./tools/catbox-tradefed run commandAndExit catbox-functional-bluetooth-tests --{phone}serial <serial> --{auto}serial <serial> --mobly-host:mobly-par-file-name BTDiscoveryTest

CATBox installs the apk and then the test is run in the correct Python execution environment.

Review CATBox test results

Results of functional and performance test runs are saved in android-catbox/results/latest.

Result type	Location
Test results	test_result.xml
Failure tests details	test_result_failures_suite.html
Performance metrics results	/report-log-files/CatboxPerformanceTests.reportlog.json
Functional and performance test runs	/android-catbox/logs/latest