Spectatio is an open source test framework developed for testing Android Automotive OS (AAOS) on real and virtual devices. Spectatio provides APIs for testing apps on an automotive device and is an extendable and scalable solution used for verifying the functionality and performance of AAOS and its apps.
High-level design
The Spectatio framework is adaptable and expandable for various AAOS UI implementations. It's used for testing the functionality and performance of AAOS on device hardware, emulators, and virtualized environments.
The following figure explains the high-level design of the Spectatio framework.
Figure 1. Spectatio framework high-level design
Built on top of UI Automator, the Spectatio framework provides a set of APIs to build UI tests that interact with user and system apps on AAOS. Automotive tests use the APIs provided by the Spectatio framework for testing, which makes these tests independent of the device under test (DUT) and scalable to test varied devices, if supported.
Figure 1 shows that the Spectatio framework is modularized based on reference apps such as Dialer, Medicenter, and Settings using app-specific interfaces and helpers, making it easily extendable for new apps. The Spectatio framework reuses the common standard and utility helper classes. The standard helper class is the parent class for all the application helper functions and provides standard functions that are device specific or applicable across apps. The utility helper classes provide utilities such as reading or writing files from the device.
Architecture
In order to provide a set of APIs to build UI tests, the Spectatio framework implements app-specific interfaces and helpers while extending the existing standard helper class and importing the utility helper classes.
The following figure explains the high-level architecture of the Spectatio framework and all the entities involved in implementing APIs for testing an app.
Figure 2. Spectatio framework high-level architecture
The application helper interface provides a blueprint for the implementation of
an application helper. It consists of various helper functions that are needed
for testing apps. Each app has its own interface, such as IAutoSettingHelper
and IAutoDialHelper
.
For more information and a list of interface functions, see the application helper interface functions on AOSP.
The standard helper class consists of standard attributes and functions that are
required for device setup but aren’t specific to any app, such as pressHome
and scroll
. The standard helper class is defined in AbstractAutoStandardAppHelper.java
.
The utility helper classes are used by the framework. For
example, AutoJsonUtility.java
is a
utility class that loads the given device JSON configuration file and updates
the framework configurations at runtime.
The application helper implementation module is the core of the Spectatio
framework. It contains the implementation for the helper functions defined in
the application helper interface, which are required for testing apps on an
automotive device. Each app has its own implementation, such as SettingHelperImpl
and
DialHelperImpl
, used by
the Automotive tests for testing the apps. For more information and a list of
implementations, see the application helper implementation functions
'on AOSP.
Automotive Tests
use the application helper implementation functions to test various operations
related to the app. Use the HelperAccessor
class
to gain access to the application helper implementation functions.
The following code shows the setup, cleanup, and execution of a sample automotive test.
@RunWith(AndroidJUnit4.class)
public class AutoApplicationTest {
static HelperAccessor<IAutoApplicationHelper> autoApplicationHelper =
new HelperAccessor<>(IAutoApplicationHelper.class);
public AutoApplicationTest() {
// constructor
// Initialize any attributes that are required for the test execution
}
@Before
public void beforeTest() {
// Initial setup before each test
// For example - open the application
autoApplicationHelper.open();
}
@After
public void afterTest() {
// Cleanup after each test.
// For example - exit the application
autoApplicationHelper.exit();
}
@Test
public void testApplicationFeature() {
// Test
// For example - Test if application is open
assertTrue("Application is not open.", autoApplicationHelper.isOpen());
}
}
Customization
The Spectatio framework is independent of the device UI, so it’s scalable for
testing devices with varied UIs and hardware. To achieve this scalability,
Spectatio uses default device configurations based on the reference device.
To support nondefault device configurations, the framework uses a JSON
configuration file at runtime to set the desired UI changes for the device. A
JSON configuration file supports UI elements like TEXT
, DESCRIPTION
, and
RESOURCE_ID
, along with path
settings and must contain
only the information about the UI changes for the DUT. The rest of the UI
elements use the default configuration values provided in the framework.
Default device configurations
The following sample JSON configuration file shows the available device configurations and their default values.
Click the plus sign to see a sample JSON configuration file
{ "SETTINGS": { "APPLICATION_CONFIG": { "SETTINGS_TITLE_TEXT": "Settings", "SETTINGS_PACKAGE": "com.android.car.settings", "SETTINGS_RRO_PACKAGE": "com.android.car.settings.googlecarui.rro", "OPEN_SETTINGS_COMMAND": "am start -a android.settings.SETTINGS", "OPEN_QUICK_SETTINGS_COMMAND": "am start -n com.android.car.settings/com.android.car.settings.common.CarSettingActivity" }, "QUICK_SETTINGS": { "OPEN_MORE_SETTINGS": { "TYPE": "RESOURCE_ID", "VALUE": "toolbar_menu_item_1", "PACKAGE": "com.android.car.settings" }, "NIGHT_MODE": { "TYPE": "TEXT", "VALUE": "Night mode" } }, "DISPLAY": { "PATH": "Settings > Display", "OPTIONS": [ "Brightness level" ], "BRIGHTNESS_LEVEL": { "TYPE": "RESOURCE_ID", "VALUE": "seekbar", "PACKAGE": "com.android.car.settings" } }, "SOUND": { "PATH": "Settings > Sound", "OPTIONS": [ "Media volume", "Alarm volume" ] }, "NETWORK_AND_INTERNET": { "PATH": "Settings > Network & internet", "OPTIONS": [ ], "TOGGLE_WIFI": { "TYPE": "RESOURCE_ID", "VALUE": "master_switch", "PACKAGE": "com.android.car.settings" } }, "BLUETOOTH": { "PATH": "Settings > Bluetooth", "OPTIONS": [ ], "TOGGLE_BLUETOOTH": { "TYPE": "RESOURCE_ID", "VALUE": "car_ui_toolbar_menu_item_switch", "PACKAGE": "com.android.car.settings" } }, "APPS_AND_NOTIFICATIONS": { "PATH": "Settings > Apps & notifications", "OPTIONS": [ ], "SHOW_ALL_APPS": { "TYPE": "TEXT", "VALUE": "Show all apps" }, "ENABLE_DISABLE_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "car_ui_toolbar_menu_item_text", "PACKAGE": "com.android.car.settings" }, "DISABLE_BUTTON_TEXT": { "TYPE": "TEXT", "VALUE": "Disable" }, "ENABLE_BUTTON_TEXT": { "TYPE": "TEXT", "VALUE": "Enable" }, "DISABLE_APP_BUTTON": { "TYPE": "TEXT", "VALUE": "DISABLE APP" }, "FORCE_STOP_BUTTON": { "TYPE": "TEXT", "VALUE": "Force stop" }, "OK_BUTTON": { "TYPE": "TEXT", "VALUE": "OK" }, "PERMISSIONS_MENU": { "TYPE": "TEXT", "VALUE": "Permissions" }, "ALLOW_BUTTON": { "TYPE": "TEXT", "VALUE": "Allow" }, "DENY_BUTTON": { "TYPE": "TEXT", "VALUE": "Deny" }, "DENY_ANYWAY_BUTTON": { "TYPE": "TEXT", "VALUE": "Deny anyway" } }, "DATE_AND_TIME": { "PATH": "Settings > Date & time", "OPTIONS": [ "Automatic date & time", "Automatic time zone" ], "AUTOMATIC_DATE_AND_TIME": { "TYPE": "TEXT", "VALUE": "Automatic date & time" }, "AUTOMATIC_TIME_ZONE": { "TYPE": "TEXT", "VALUE": "Automatic time zone" }, "SET_DATE": { "TYPE": "TEXT", "VALUE": "Set date" }, "SET_TIME": { "TYPE": "TEXT", "VALUE": "Set time" }, "SELECT_TIME_ZONE": { "TYPE": "TEXT", "VALUE": "Select time zone" }, "USE_24_HOUR_FORMAT": { "TYPE": "TEXT", "VALUE": "Use 24-hour format" }, "OK_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "toolbar_menu_item_0", "PACKAGE": "com.android.car.settings" }, "NUMBER_PICKER_WIDGET": { "TYPE": "CLASS", "VALUE": "android.widget.NumberPicker" }, "EDIT_TEXT_WIDGET": { "TYPE": "CLASS", "VALUE": "android.widget.EditText" } }, "USERS": { "PATH": "Settings > Users", "OPTIONS": [ "Guest" ] }, "ACCOUNTS": { "PATH": "Settings > Accounts", "OPTIONS": [ "Automatically sync data" ], "ADD_ACCOUNT": { "TYPE": "TEXT", "VALUE": "ADD ACCOUNT" }, "ADD_GOOGLE_ACCOUNT": { "TYPE": "TEXT", "VALUE": "Google" }, "SIGN_IN_ON_CAR_SCREEN": { "TYPE": "TEXT", "VALUE": "Sign in on car screen" }, "GOOGLE_SIGN_IN_SCREEN": { "TYPE": "TEXT", "VALUE": "Sign in to your Google Account" }, "ENTER_EMAIL": { "TYPE": "CLASS", "VALUE": "android.widget.EditText" }, "ENTER_PASSWORD": { "TYPE": "CLASS", "VALUE": "android.widget.EditText" }, "NEXT_BUTTON": { "TYPE": "TEXT", "VALUE": "Next" }, "DONE_BUTTON": { "TYPE": "TEXT", "VALUE": "Done" }, "REMOVE_BUTTON": { "TYPE": "TEXT", "VALUE": "Remove" }, "REMOVE_ACCOUNT_BUTTON": { "TYPE": "TEXT", "VALUE": "Remove Account" } }, "SYSTEM": { "PATH": "Settings > System", "OPTIONS": [ "About", "Legal information" ], "ABOUT_MENU": { "TYPE": "TEXT", "VALUE": "About" }, "RESET_OPTIONS_MENU": { "TYPE": "TEXT", "VALUE": "Reset options" }, "LANGUAGES_AND_INPUT_MENU": { "TYPE": "TEXT", "VALUE": "Languages & input" }, "DEVICE_MODEL": { "TYPE": "TEXT", "VALUE": "Model" }, "ANDROID_VERSION": { "TYPE": "TEXT", "VALUE": "Android version" }, "ANDROID_SECURITY_PATCH_LEVEL": { "TYPE": "TEXT", "VALUE": "Android security patch level" }, "KERNEL_VERSION": { "TYPE": "TEXT", "VALUE": "Kernel version" }, "BUILD_NUMBER": { "TYPE": "TEXT", "VALUE": "Build number" }, "RECYCLER_VIEW_WIDGET": { "TYPE": "CLASS", "VALUE": "androidx.recyclerview.widget.RecyclerView" }, "RESET_NETWORK": { "TYPE": "TEXT", "VALUE": "Reset network" }, "RESET_SETTINGS": { "TYPE": "TEXT", "VALUE": "RESET SETTINGS" }, "RESET_APP_PREFERENCES": { "TYPE": "TEXT", "VALUE": "Reset app preferences" }, "RESET_APPS": { "TYPE": "TEXT", "VALUE": "RESET APPS" }, "LANGUAGES_MENU": { "TYPE": "TEXT", "VALUE": "Languages" }, "LANGUAGES_MENU_IN_SELECTED_LANGUAGE": { "TYPE": "TEXT", "VALUE": "Idiomas" } }, "SECURITY": { "PATH": "Settings > Security", "OPTIONS": [ ], "TITLE": { "TYPE": "RESOURCE_ID", "VALUE": "car_ui_toolbar_title", "PACKAGE": "com.android.car.settings.googlecarui.rro" }, "CHOOSE_LOCK_TYPE": { "TYPE": "TEXT", "VALUE": "Choose a lock type" }, "LOCK_TYPE_PASSWORD": { "TYPE": "TEXT", "VALUE": "Password" }, "LOCK_TYPE_PIN": { "TYPE": "TEXT", "VALUE": "PIN" }, "LOCK_TYPE_NONE": { "TYPE": "TEXT", "VALUE": "None" }, "CONTINUE_BUTTON": { "TYPE": "TEXT", "VALUE": "Continue" }, "CONFIRM_BUTTON": { "TYPE": "TEXT", "VALUE": "Confirm" }, "ENTER_PASSWORD": { "TYPE": "CLASS", "VALUE": "android.widget.EditText" }, "PIN_PAD": { "TYPE": "RESOURCE_ID", "VALUE": "pin_pad", "PACKAGE": "com.android.car.settings" }, "ENTER_PIN_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "key_enter", "PACKAGE": "com.android.car.settings" }, "REMOVE_BUTTON": { "TYPE": "TEXT", "VALUE": "Remove" } } }, "PHONE": { "APPLICATION_CONFIG": { "DIAL_PACKAGE": "com.android.car.dialer", "PHONE_ACTIVITY": "com.android.car.dialer/.ui.TelecomActivity", "OPEN_DIAL_PAD_COMMAND": "am start -a android.intent.action.DIAL" }, "IN_CALL_VIEW": { "DIALED_CONTACT_TITLE": { "TYPE": "RESOURCE_ID", "VALUE": "user_profile_title", "PACKAGE": "com.android.car.dialer" }, "DIALED_CONTACT_NUMBER": { "TYPE": "RESOURCE_ID", "VALUE": "user_profile_phone_number", "PACKAGE": "com.android.car.dialer" }, "END_CALL": { "TYPE": "RESOURCE_ID", "VALUE": "end_call_button", "PACKAGE": "com.android.car.dialer" }, "MUTE_CALL": { "TYPE": "RESOURCE_ID", "VALUE": "mute_button", "PACKAGE": "com.android.car.dialer" }, "SWITCH_TO_DIAL_PAD": { "TYPE": "RESOURCE_ID", "VALUE": "toggle_dialpad_button", "PACKAGE": "com.android.car.dialer" }, "CHANGE_VOICE_CHANNEL": { "TYPE": "RESOURCE_ID", "VALUE": "voice_channel_view", "PACKAGE": "com.android.car.dialer" }, "VOICE_CHANNEL_CAR": { "TYPE": "TEXT", "VALUE": "Car speakers" }, "VOICE_CHANNEL_PHONE": { "TYPE": "TEXT", "VALUE": "Phone" } }, "DIAL_PAD_VIEW": { "DIAL_PAD_MENU": { "TYPE": "TEXT", "VALUE": "Dial Pad" }, "DIAL_PAD_FRAGMENT": { "TYPE": "RESOURCE_ID", "VALUE": "dialpad_fragment", "PACKAGE": "com.android.car.dialer" }, "DIALED_NUMBER": { "TYPE": "RESOURCE_ID", "VALUE": "title", "PACKAGE": "com.android.car.dialer" }, "MAKE_CALL": { "TYPE": "RESOURCE_ID", "VALUE": "call_button", "PACKAGE": "com.android.car.dialer" }, "DELETE_NUMBER": { "TYPE": "RESOURCE_ID", "VALUE": "delete_button", "PACKAGE": "com.android.car.dialer" } }, "CONTACTS_VIEW": { "CONTACTS_MENU": { "TYPE": "TEXT", "VALUE": "Contacts" }, "CONTACT_INFO": { "TYPE": "RESOURCE_ID", "VALUE": "call_action_id", "PACKAGE": "com.android.car.dialer" }, "CONTACT_NAME": { "TYPE": "RESOURCE_ID", "VALUE": "title", "PACKAGE": "com.android.car.dialer" }, "CONTACT_DETAIL": { "TYPE": "RESOURCE_ID", "VALUE": "show_contact_detail_id", "PACKAGE": "com.android.car.dialer" }, "ADD_CONTACT_TO_FAVORITE": { "TYPE": "RESOURCE_ID", "VALUE": "contact_details_favorite_button", "PACKAGE": "com.android.car.dialer" }, "SEARCH_CONTACT": { "TYPE": "RESOURCE_ID", "VALUE": "menu_item_search", "PACKAGE": "com.android.car.dialer" }, "CONTACT_SEARCH_BAR": { "TYPE": "RESOURCE_ID", "VALUE": "car_ui_toolbar_search_bar", "PACKAGE": "com.android.car.dialer" }, "SEARCH_RESULT": { "TYPE": "RESOURCE_ID", "VALUE": "contact_name", "PACKAGE": "com.android.car.dialer" }, "CONTACT_SETTINGS": { "TYPE": "RESOURCE_ID", "VALUE": "menu_item_setting", "PACKAGE": "com.android.car.dialer" }, "CONTACT_ORDER": { "TYPE": "TEXT", "VALUE": "Contact order" }, "SORT_BY_FIRST_NAME": { "TYPE": "TEXT", "VALUE": "First name" }, "SORT_BY_LAST_NAME": { "TYPE": "TEXT", "VALUE": "Last Name" }, "CONTACT_TYPE_WORK": { "TYPE": "TEXT", "VALUE": "Work" }, "CONTACT_TYPE_MOBILE": { "TYPE": "TEXT", "VALUE": "Mobile" }, "CONTACT_TYPE_HOME": { "TYPE": "TEXT", "VALUE": "Home" } }, "CALL_HISTORY_VIEW": { "CALL_HISTORY_MENU": { "TYPE": "TEXT", "VALUE": "Recents" }, "CALL_HISTORY_INFO": { "TYPE": "RESOURCE_ID", "VALUE": "call_action_id", "PACKAGE": "com.android.car.dialer" } }, "FAVORITES_VIEW": { "FAVORITES_MENU": { "TYPE": "TEXT", "VALUE": "Favorites" } } }, "NOTIFICATIONS": { "APPLICATION_CONFIG": { "OPEN_NOTIFICATIONS_COMMAND": "service call statusbar 1" }, "EXPANDED_NOTIFICATIONS_SCREEN": { "NOTIFICATION_VIEW": { "TYPE": "RESOURCE_ID", "VALUE": "notification_view", "PACKAGE": "com.android.systemui" }, "CLEAR_ALL_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "clear_all_button", "PACKAGE": "com.android.systemui" }, "STATUS_BAR": { "TYPE": "RESOURCE_ID", "VALUE": "car_top_navigation_bar_container", "PACKAGE": "com.android.systemui" }, "APP_ICON": { "TYPE": "RESOURCE_ID", "VALUE": "app_icon", "PACKAGE": "com.android.systemui" }, "APP_NAME": { "TYPE": "RESOURCE_ID", "VALUE": "header_text", "PACKAGE": "com.android.systemui" }, "NOTIFICATION_TITLE": { "TYPE": "RESOURCE_ID", "VALUE": "notification_body_title", "PACKAGE": "com.android.systemui" }, "NOTIFICATION_BODY": { "TYPE": "RESOURCE_ID", "VALUE": "notification_body_content", "PACKAGE": "com.android.systemui" }, "CARD_VIEW": { "TYPE": "RESOURCE_ID", "VALUE": "card_view", "PACKAGE": "com.android.systemui" } } }, "MEDIA_CENTER": { "APPLICATION_CONFIG": { "MEDIA_CENTER_PACKAGE": "com.android.car.media", "MEDIA_ACTIVITY": "com.android.bluetooth/com.android.bluetooth.avrcpcontroller.BluetoothMediaBrowserService" }, "MEDIA_CENTER_SCREEN": { "PLAY_PAUSE_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "play_pause_stop", "PACKAGE": "com.android.car.media" }, "NEXT_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "skip_next", "PACKAGE": "com.android.car.media" }, "PREVIOUS_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "skip_prev", "PACKAGE": "com.android.car.media" }, "SHUFFLE_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "overflow_on", "PACKAGE": "com.android.car.media" }, "PLAY_QUEUE_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "play_queue", "PACKAGE": "com.android.car.media" }, "MINIMIZED_MEDIA_CONTROLS": { "TYPE": "RESOURCE_ID", "VALUE": "minimized_playback_controls", "PACKAGE": "com.android.car.media" }, "TRACK_NAME": { "TYPE": "RESOURCE_ID", "VALUE": "title", "PACKAGE": "com.android.car.media" }, "TRACK_NAME_MINIMIZED_CONTROL": { "TYPE": "RESOURCE_ID", "VALUE": "minimized_control_bar_title", "PACKAGE": "com.android.car.media" }, "BACK_BUTTON": { "TYPE": "DESCRIPTION", "VALUE": "Back" } }, "MEDIA_CENTER_ON_HOME_SCREEN": { "PLAY_PAUSE_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "play_pause_stop", "PACKAGE": "com.android.car.carlauncher" }, "NEXT_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "skip_next", "PACKAGE": "com.android.car.carlauncher" }, "PREVIOUS_BUTTON": { "TYPE": "RESOURCE_ID", "VALUE": "skip_prev", "PACKAGE": "com.android.car.carlauncher" }, "TRACK_NAME": { "TYPE": "RESOURCE_ID", "VALUE": "title", "PACKAGE": "com.android.car.carlauncher" } } } }
Nondefault device configurations
The following code sample shows an example of the JSON configuration file where default settings are overridden by the settings on the DUT. In this example:
- Internet settings are named
Network & internet
on reference devices andConnectivity
on the DUT. - The date and time settings are available at
Settings > Date and time
for reference devices and atSettings > System > Date and time
for the DUT.
# JSON configuration file for Reference Device
{
....
"NETWORK_AND_INTERNET_SETTINGS": {
"path": "Settings > Network & internet"
},
"DATE_AND_TIME_SETTINGS": {
"path": "Settings > Date and time"
},
....
}
# JSON configuration file for non-reference device
{
....
"NETWORK_AND_INTERNET_SETTINGS": {
"path": "Settings > Connectivity"
},
"DATE_AND_TIME_SETTINGS": {
"path": "Settings > System > Date and time"
},
....
}
When the JSON configuration file is ready, it’s provided at runtime as shown in the following code block.
# Push The JSON configuration file to the device
adb -s DEVICE-SERIAL push PATH-OF-JSON-FILE /data/local/tmp/jsonFile.json
In these commands:
- PATH-TO-JSON-FILE: The path of the JSON file on the host machine.
- DEVICE-SERIAL: The serial ID of the DUT. This parameter isn’t required if only one device is connected to the host.
Building and execution
The Spectatio framework is automatically built as part of the test APK. To build the test APK, the AOSP codebase must reside on the local workstation. After the test APK is built, the user must install the APK on the device and execute the test.
The following code sample shows the building, installation, and execution of a test APK.
# Build Test APK
make TEST-APK-NAME
# Install Test APK
adb -s DEVICE-SERIAL install -r PATH-FOR-BUILT-TEST-APK
# Execute Test with the JSON file
adb -s DEVICE-SERIAL shell am instrument -w -r -e debug false -e config-file-path /data/local/tmp/jsonFile.json -e class TEST-PACKAGE.TEST-CLASSNAME TEST-PACKAGE/androidx.test.runner.AndroidJUnitRunner
In these commands:
TEST-APK-NAME: The name of the app to be tested. For example, set TEST-APK-NAME to
AndroidAutomotiveSettingsTests
to test the Wi-Fi settings as specified in theAndroid.bp
file. The name of the APK can be found in the respectiveAndroid.bp
file for the Automotive test.DEVICE-SERIAL: The serial ID of the DUT. This parameter isn’t required if only one device is connected to the host.
config-file-path
: Optional parameter that is required only to provide nondefault device UI configurations as specified in the JSON configuration file. If not provided, the framework uses default values for executing the tests.PATH-FOR-BUILT-TEST-APK: The path where the test APK is built when the
make
command is executed.TEST-PACKAGE: The name of the test package.
TEST-CLASSNAME: The name of the test class. For example, for the Wifi Settings test, the test package is
android.platform.tests
and the test class name isWifiSettingTest
.