This document lists all tests available for evaluating the Android camera hardware abstraction layer (HAL). It is intended for original equipment manufacturers (OEMs) and application processor (AP) vendors so they may ensure proper implementation of the camera HAL with minimum defects. Although this is a voluntary addition to the Android Compatibility Test Suite (CTS), it greatly increases camera test coverage and will certainly identify potential bugs.
By passing these tests, OEMs validate
whether they have properly integrated the Android
abstraction layer (HAL) 3 interfaces. When conforming with all items in the
checklist, a device implementation may be considered full with respect
to the Android Camera HAL interfaces. This will in turn enable a device to
properly support the
android.hardware.camera2 package that
camera apps build upon.
Camera HAL3 specification
The Android Camera HAL3 specification is the authoritative source of information on what devices must satisfy; this page provides a summary of all tests that can be used as a checklist. Camera HAL implementers (e.g. AP vendors) should go through the Camera HAL3 specification line-by-line and ensure their devices conform to it.
The current HAL specification is defined in these files within the Android 5.0 and later generic Android Platform Development Kit (PDK):
- Camera HAL 3.x interface and spec:
- Camera HAL 3.x metadata spec:
- HAL pixel format interface and spec:
Camera test types
Here are the primary types of tests available for the latest Android camera along with references to associated instructions below:
- Vendor Test Suite (VTS): Tests that directly test the camera HAL interface
- Compatibility Test Suite (CTS): Standard, automated Android tests to ensure device compatibility - see the CTS introduction and the Trade Federation Overview
- Image Test Suite (ITS): Manually run
tests to ensure image correctness - see the top-level and test-specific
tutorial.pyfor setup instructions
- Manual TestingCam tests:
Run from the source in
- Manual TestingCam2.1
tests: Run from the source in
All of these test types are described in detail below. These tests are presented in the chronological order in which OEMs are expected to execute them.
For instance, if a device fails the native tests, it will assuredly fail the subsequent Compatibility Test Suite (CTS) tests. And if a device fails CTS, there is little use in proceeding to the Image Test Suite (ITS). We recommend addressing failures in each test type before proceeding to the next set of tests.
Vendor Test Suite (VTS) tests
The Android Vendor Test Suite (VTS) is a testing suite that works on the HIDL interface level. For more information on using VTS, see Vendor Test Suite.
Compatibility Test Suite (CTS) tests
Camera Android Compatibility Test Suite (CTS) tests focus upon device compatibility. For information on setting up a test environment, see Setting up CTS.
The starting path for camera CTS tests is:
When running camera CTS for devices that support external cameras (such as USB webcams), you must have a device plugged in when running CTS or the tests will automatically fail. Examples of external cameras include: Logitech HD Pro Webcam C920 and the Microsoft LifeCam HD-3000.
See the CTS introduction and its subpages for general instructions on running CTS.
CTS tests for
Find these camera tests under
CTS tests for
Find these camera tests under
CTS Verifier camera tests
Find these camera tests under:
Image Test Suite (ITS) tests
The Camera Image Test Suite (ITS) tests focus on image correctness. To perform the tests, run the Python scripts on a workstation with the Android device connected over USB.
The Camera ITS infrastructure and tests are located in the
cts/apps/CameraITS directory. Each test resides in a
adb install -r CtsVerifier.apk
For more information on how to set up and run the tests, see the
CameraITS PDF file in
tutorial.py in the
tests subdirectory for a
guide on how to use the scripts.
The ITS static tests (scenes 0-5) can run with any operating
system with the required Python 2.7 environment. However, the
test with the Sensor
Fusion Box must be run with the
Linux operating system.
To run ITS manually, prepare a simple physical environment with a specific, reusable target such as a white wall, a grey card, and a desk lamp. Mount the Android device on a tripod and run the scripts to test the camera functions. Most tests are pass or fail but some offer metrics.
These scripts test scenarios that are not tested in CTS and are an important component of the HAL 3.2 test plan.
ITS tests either pass or fail. All mandated
tests in each scene folder must pass. Tests that are not mandated can
fail and still count as a pass in
Scene 0 to scene 4 tests
These scenes represent a large part of ITS testing and are included as PDF
files in the
scene folder. To automate these tests, use the Camera ITS-in-a-box system.
- Scene 0: Requires no setup.
- Scene 1: Requires a grey card.
- Scene 2: Requires a face scene.
- Scene 3: ISO12233 chart.
- Scene 4: Custom scene with a circle inside a square.
Scene 5 tests
Scene 5 tests require a diffuser to be placed on top of the camera.
Sensor fusion tests
Sensor fusion tests require specific camera motion to test the timestamp
difference between the camera and the gyroscope for AR and VR applications. This
test is skipped if no gyroscope is included or if the
parameter is not enabled. The
sensor_fusion test can be automated
with the Sensor Fusion Box.
Media Framework tests
Pass all of the camera-related media tests in MediaFrameworkTest. Please
note, these tests require the mediaframeworktest.apk be installed on the Android
device. You will need to
make mediaframeworktest and then use adb
to install the resulting .apk. Example commands are included below.
The starting path for Camera-related media framework tests is:
Find the source code for the tests here:
To set up these tests:
adb install out/target/product/name/data/app/mediaframeworktest.apk
Where the name variable represents the directory containing the vendor's product.
Find all of the tests in the following directory or its subdirectories:
Each subdirectory represents a class of tests:
Running Media Framework tests
To see all of the available tests::
adb shell pm list instrumentation
This will yield results resembling:
instrumentation:com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner (target=com.android.mediaframeworktest) instrumentation:com.android.mediaframeworktest/.MediaRecorderStressTestRunner (target=com.android.mediaframeworktest) instrumentation:com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner (target=com.android.mediaframeworktest) instrumentation:com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner (target=com.android.mediaframeworktest)
Identify and extract the component (between
(target=com.android.mediaframeworktest) from each test line.
The component is composed of the target package name
com.android.mediaframeworktest) and the test runner name
com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner com.android.mediaframeworktest/.MediaRecorderStressTestRunner com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner
You may then pass each component to
adb shell am instrument like
adb shell am instrument -w component.name
component.name equals the extracted value
adb shell am instrument -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
Please note, while the class path is the Java package + class name, the instrumentation package isn't necessarily the same as the Java package. Make sure you use the AndroidManifest.xml package when concatenating the component name, not the Java package in which the test runner class resides.
To run a single class of tests, pass the -e class
adb shell am instrument -e class com.android.mediaframeworktest.integration.CameraBinderTest -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
To run only a single method in a test class, append a pound (#) sign and the
method name (in this case,
testConnectPro) to the class name, like so:
adb shell am instrument -e class 'com.android.mediaframeworktest.integration.CameraBinderTest#testConnectPro' -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
Media settings functional tests
Here is an example run of a functional test. This test verifies the basic functionality of different combinations of camera settings. (ie, Flash, exposure, WB, scene, picture size and geoTag)
Run the test command:
adb shell am instrument -w -r -e delay_msec 15 -e log true -e class com.android.mediaframeworktest.functional.camera.CameraPairwiseTest com.android.mediaframeworktest/com.android.mediaframeworktest.CameraStressTestRunner
Media integration tests
Here is an example run of an integration test, in this case mediaframeworktest/integration/CameraBinderTest.java and mediaframeworktest/CameraStressTestRunner.java:
adb shell am instrument -e class \ 'com.android.mediaframeworktest.integration.CameraBinderTest' -w \ 'com.android.mediaframeworktest/.CameraStressTestRunner'
If successful, this results in output resembling:
----- com.android.mediaframeworktest.integration.CameraBinderTest:........... Test results for CameraStressTestRunner=........... Time: 3.328 OK (11 tests) -----
Media performance tests
This preview memory test will open and release the camera preview for 200 times. In each 20 iterations, the snapshot of ps mediaserver will be recorded and it will compare the memory usage different after 200 iterations. Test will fail If the difference is greater than 150kM.
Run the test command:
adb shell am instrument -w -r -e class com.android.mediaframeworktest.performance.MediaPlayerPerformance#testCameraPreviewMemoryUsage com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner
More detailed output can be found in:
Media unit tests
The commands to run unit tests are all similar. For example, for CameraMetadataTest.java, the command would be:
adb shell am instrument -e class 'com.android.mediaframeworktest.unit.CameraMetadataTest' -w 'com.android.mediaframeworktest/.CameraStressTestRunner'
Media stress tests
This test is to stress out the camera image capture and video recording.
Run the test command:
adb shell am instrument -w com.google.android.camera.tests/com.android.camera.stress.CameraStressTestRunner
All tests should pass.
Manual TestingCam tests
The TestingCam app should be run manually with the following checks performed.
The source for TestingCam is here:
Infinity focus with camera tilt
Start TestingCam, turn on preview, and ensure that autofocus mode is set to infinity. Using the Take picture button, capture shots of distant subjects (at least 10m away) with the camera pointed horizontally, upwards (close to vertical), and downwards (close to vertical); an example of the upwards shot could be high leaves/branches of a tree from beneath and an example of the downwards shot could be the street as seen from the roof of a building. In all cases, the distant subject should be sharp and in focus. Save and view the shots in the gallery view so that you can zoom in and inspect the sharpness more easily.
Note that for a camera with a VCM actuator to pass this test, it will require either a closed-loop AF control system, or it will need some sort of SW correction based on using accelerometer data to determine camera orientation. Reliable factory calibration of the lens infinity position will also be needed.
Manual TestingCam2 tests
The TestingCam2 app should be run manually, with the following checks
performed. The source for TestingCam2 is here:
Start TestingCam2, and press the JPEG button. The image that appears to the right of the viewfinder image should appear the same as the viewfinder, including having the same orientation.