The Android Compatibility Test Suite Verifier (CTS Verifier) supplements the Compatibility Test Suite (CTS). While CTS checks APIs and functions that can be automated, CTS Verifier provides tests for APIs and functions that can't be tested on a stationary device without manual input, such as audio quality, touchscreen, accelerometer, and camera.
Before running CTS Verifier, make sure that you have the following equipment:
- Android device that has verified Android API compatibility by successfully passing the CTS. This is the device under test (DUT).
- Linux computer with USB 2.0 compatible port. All connections to the DUT go through this port.
- Second Android device with a known compatible Bluetooth, Wi-Fi direct, and NFC host card emulation (HCE) implementation.
- A Wi-Fi router configured with access point name and password. The router should have the ability to disconnect from the internet, but not powered off.
UICC Requirements for NFC Tests
CTS Verifier has the following NFC test cases:
- Field-off (uses transaction data from 0x54)
- De-select (uses transaction data from 0x52)
- HCI command (0025000000) (uses transaction data from 0x02)
Transaction event tests require two devices, one of which must have SecureElement UICC with the following access rules:
- CtsVerifier APK hash: 61:ED:37:7E:85:D3:86:A8:DF:EE:6B:86:4B:D8:5B:0B:FA:A5:AF:81
- Authorized Application Identifier (AID) for NFC event access: 0xA000000476416E64726F696443545341
On the emulator side, we expect the transaction event to push the corresponding activity into the foreground.
Change the system navigation mode of the test device to three-button mode as follows:
- Open Settings.
- Navigate to System > Gesture > System Navigation.
- Select any button-based navigation mode, three-button mode preferred, if available.
To setup the CTS Verifier testing environment:
On the Linux computer:
Connect the DUT to the Linux computer.
From a terminal on the Linux computer, install
CtsVerifier.apkon the DUT.
adb install -r -g CtsVerifier.apk
For Android 10 and higher, run the following command to give permission to the app to create the report.
adb shell appops set com.android.cts.verifier android:read_device_identifiers allow
For Android 11 and higher, run the following command to allow reports to be saved in a self-defined directory under the device's external top-level directory.
adb shell appops set com.android.cts.verifier MANAGE_EXTERNAL_STORAGE 0
Ensure that the DUT has the system data and time set correctly.
Launch the CTS Verifier application by tapping the CTS Verifier icon on the DUT.
Figure 1. CTS Verifier icon
The app displays several test sets available for manual verification.
Figure 2. CTS Verifier menu of tests
Each test contains a set of common elements (Info, Pass, Fail) at the bottom of the screen.
Figure 3. Test elements
- Info (?). Tap to display test instructions. Also appears automatically the first time a test is opened.
- Pass (✓). Tap if the DUT meets the test requirements per the Info instructions.
- Fail (!). Tap if the DUT doesn't meet the test requirements per the Info instructions.
Some tests, such as the USB accessory mode and camera calibration test, require additional test setup and instructions as detailed in the following sections.
Testing USB accessory mode for 8.0 and above
Figure 4. USB accessory test prcedure for 8.0 and above
Figure 5. Testing USB accessory mode for 8.0 and above
Testing USB accessory mode for 7.x and below
The USB Accessory test requires a Linux computer to run the USB desktop machine (host) program.
- Connect the DUT to the Linux computer.
On the computer, execute the
cts-usb-accessoryprogram from the CTS Verifier package:
Wait for a popup message to appear on the DUT, then select OK.
Figure 6. USB accessory test
Go to the USB Accessory Test in the CTS Verifier application on DUT.
On the computer, review the output from the console. Example output:
CTS USB Accessory Tester Found possible Android device (413c:2106) - attempting to switch to accessory mode... Failed to read protocol versionfigure3 Found Android device in accessory mode (18d1:2d01)... [RECV] Message from Android device #0 [SENT] Message from Android accessory #0 [RECV] Message from Android device #1 [SENT] Message from Android accessory #1 [RECV] Message from Android device #2 [SENT] Message from Android accessory #2 [RECV] Message from Android device #3 [SENT] Message from Android accessory #3 [RECV] Message from Android device #4 [SENT] Message from Android accessory #4 [RECV] Message from Android device #5 [SENT] Message from Android accessory #5 [RECV] Message from Android device #6 [SENT] Message from Android accessory #6 [RECV] Message from Android device #7 [SENT] Message from Android accessory #7 [RECV] Message from Android device #8 [SENT] Message from Android accessory #8 [RECV] Message from Android device #9 [SENT] Message from Android accessory #9 [RECV] Message from Android device #10 [SENT] Message from Android accessory #10
Calibrating camera field of view
Use the field of view calibration procedure to quickly determine the device field of view with moderate accuracy.
Set up the test environment:
- Print the calibration-pattern.pdf target file on 11” x 17” or A3 size paper.
- Mount the printed pattern on a rigid backing.
Orient the camera device and the printed target as shown below.
Figure 7. Camera printed target
Set the target width:
- Measure the distance (in centimeters) between the solid lines on the target pattern to account for printing inaccuracies (~38 cm).
- Start the calibration application.
- Press the setup button and select Marker distance.
- Measure and enter the distance to the target pattern (~100 cm).
- Press the back button to return to the calibration preview.
Verify that the device and target are placed as shown in the figure and the correct distances have been entered into the setup dialog. The preview displays the image with a vertical line overlaid onto it; this line should align with the center line of the target pattern. The transparent grid can be used with the other vertical lines to ensure that the optical axis is orthogonal to the target.
Run the calibration test:
- Select an image resolution (using selector at the bottom left), then tap the screen to take a photo. The test enters calibration mode and displays the photo with two vertical lines overlaid onto the image.
- Determine accuracy:
- If the lines align with the vertical lines on the target pattern within a few centimeters, the reported field of view for the selected resoultion is accurate.
- If the lines don't align, the reported field of view is inaccurate. To correct this, adjust the slider at the bottom of the screen until the overlay aligns with the target pattern as closely as possible. When the overlay and the target pattern image are aligned, the displayed field of view is a close approximation to the correct value. The reported field of view should be within +/-1 degree of the calibration value.
- Press back button and repeat the calibration test for all image resolutions supported by the DUT.
Running CTS Verifier for Alternate Modes
Starting CTS 10 R6 and CTS 11 R2 releases, CTS Verifier supports the tests required for devices with alternate modes, or for devices with more than one screen mode.
At the top of the main list view in CTS Verifier a button is shown to enable users to toggle between unfolded and folded display mode. CTS Verifier will show the necessary tests for the selected display mode. To execute the alternate modes in CTS Verifier, one should switch the button to the appropriate display mode and run the list of tests shown.
The results of folded tests will be recorded in the same report with the unfolded tests. To identify each test result from which the test set is from, a suffix is added to each test name under the folded mode.
<Test result="fail" name="com.android.cts.verifier.deskclock.DeskClockTestsActivity[folded]"> <RunHistory subtest="CREATE_ALARM[folded]"> <Run start="1594176663973" end="1594176665841" isAutomated="false" /> </RunHistory> </Test>
After all tests complete, you can save the results as a report and download them to a computer. Report names are automatically time-stamped based on the DUT system time.
Tap the Save (disk) icon.
Figure 8. CTS Verifier Save icon
Wait for the popup message to display the path to the saved report (for example,
/sdcard/verifierReports/ctsVerifierReport-date-time.zip), then record the path.
Figure 9. CTS Verifier path to saved report
Connect the DUT to the Linux computer.
From the Android SDK installation on the Linux computer, download reports from the connected device using
adb shell content reador
adb pull CTSVerifierReportPath.
For Android 7.x and higher, download all reports using:
adb pull /sdcard/verifierReports
For Android 6.0 and lower, download all reports using:
adb pull /mnt/sdcard/ctsVerifierReports/
For Android 10 and higher, implementing Automotive and devices running as a secondary user, download the latest report using:
adb shell content read --user CURRENT_USER --uri content://com.android.cts.verifier.testresultsprovider/reports/latest > report.zip
To list all reports available in the connected device:
adb shell content query --user CURRENT_USER --uri content://com.android.cts.verifier.testresultsprovider/reports
To download a report from the list, you can specify the Row ID or filename. For example:
adb shell content read --user CURRENT_USER --uri content://com.android.cts.verifier.testresultsprovider/reports/0 > report.zip adb shell content read --user CURRENT_USER --uri content://com.android.cts.verifier.testresultsprovider/reports/ctsVerifierReport-date-time.zip > report.zip
To clear pass/fail results, select the results in the CTS Verifier app and select Menu > Clear.