Fulfill commands

This page describes how to fulfill commands with voice interaction.

Fulfill media commands

Media-related command can be split into three different groups:

• External media sources (such as Spotify installed in AAOS).
• Backend media sources (such as music streamed through the VIA).
• Local media sources (such as car radio).

Handle external media source commands

External media sources are defined as Android apps that support MediaSessionCompat and MediaBrowseCompat APIs (refer to Build media apps for cars for a detailed explanation on the use of these APIs).

Important: For an assistant app to connect to the MediaBrowseService of all installed media apps in the system, it must:

1. Be installed as system-signed (see Media Application Development guidelines for AAOS and the sample PackageValidator code).
2. Hold android.permission.MEDIA_CONTENT_CONTROL system-privileged permission (see Grant system-privileged permissions).

In addition to MediaBrowserCompat and MediaControllerCompat, AAOS provides the following:

• CarMediaService provides centralized information on the currently selected media source. This is also used to resume a previously playing media source after car shutdown-restart.
• car-media-common provides convenient methods to list, connect, and interact with media apps.

Provided below are guidelines specific to the implementation of common voice interaction commands.

Get a list of installed media sources

Media sources can be detected using PackageManager, and filtering for services matching the MediaBrowserService.SERVICE_INTERFACE. In some cars there might be some special media browser service implementations, which should be excluded. Here is an example of this logic:

private Map<String, MediaSource> getAvailableMediaSources() {
    List<String> customMediaServices =
        Arrays.asList(mContext.getResources()
            .getStringArray(R.array.custom_media_packages));
    List<ResolveInfo> mediaServices = mPackageManager.queryIntentServices(
            new Intent(MediaBrowserService.SERVICE_INTERFACE),
            PackageManager.GET_RESOLVED_FILTER);
    Map<String, MediaSource> result = new HashMap<>();
    for (ResolveInfo info : mediaServices) {
        String packageName = info.serviceInfo.packageName;
        if (customMediaServices.contains(packageName)) {
            // Custom media sources should be ignored, as they might have a
            // specialized handling (e.g., radio).
            continue;
        }
        String className = info.serviceInfo.name;
        ComponentName componentName = new ComponentName(packageName,
            className);
        MediaSource source = MediaSource.create(mContext, componentName);
        result.put(source.getDisplayName().toString().toLowerCase(),
            source);
    }
    return result;
}

Be aware that media sources might be installed or uninstalled at any time. In order to maintain an accurate list, it is recommended to implement a BroadcastReceiver instance for the intent actions ACTION_PACKAGE_ADDED, ACTION_PACKAGE_CHANGED, ACTION_PACKAGE_REPLACED, and ACTION_PACKAGE_REMOVED.

Connect to currently playing media source

CarMediaService provides methods to get the currently selected media source, and when this media source changes. These changes could happen because the user interacted with the UI directly, or due the use of hardware buttons in the car. On the other hand, car-media-common library offers convenient ways to connect to a given media source. Here is a simplified snippet on how to connect to the currently selected media app:

public class MediaActuator implements
        MediaBrowserConnector.onConnectedBrowserChanged {
    private final Car mCar;
    private CarMediaManager mCarMediaManager;
    private MediaBrowserConnector mBrowserConnector;

    …

    public void initialize(Context context) {
        mCar = Car.createCar(context);
        mBrowserConnector = new MediaBrowserConnector(context, this);
        mCarMediaManager = (CarMediaManager)
            mCar.getCarManager(Car.CAR_MEDIA_SERVICE);
        mBrowserConnector.connectTo(mCarMediaManager.getMediaSource());
        …
    }

    @Override
    public void onConnectedBrowserChanged(
            @Nullable MediaBrowserCompat browser) {
        // TODO: Handle connected/disconnected browser
    }

    …
}

Control playback of currently playing media source

With a connected MediaBrowserCompat it's easy to send transport control commands to the target app. Here is a simplified example:

public class MediaActuator …  {
    …
    private MediaControllerCompat mMediaController;

    @Override
    public void onConnectedBrowserChanged(
            @Nullable MediaBrowserCompat browser) {
        if (browser != null && browser.isConnected()) {
            mMediaController = new MediaControllerCompat(mContext,
                browser.getSessionToken());
        } else {
            mMediaController = null;
        }
    }

    private boolean playSongOnCurrentSource(String song) {
        if (mMediaController == null) {
            // No source selected.
            return false;
        }
        MediaControllerCompat.TransportControls controls =
            mMediaController.getTransportControls();
        PlaybackStateCompat state = controller.getPlaybackState();
        if (state == null || ((state.getActions() &
                PlaybackStateCompat.ACTION_PLAY_FROM_SEARCH) == 0)) {
            // Source can't play from search
            return false;
        }
        controls.playFromSearch(query, null);
        return true;
    }

    …
}

Handle local media source commands (radio, CD player, Bluetooth, USB)

Local media sources expose their functionality to the system using the same MediaSession and MediaBrowse APIs detailed above. To accommodate the particularities of each type of hardware, these MediaBrowse services use specific conventions to organize their information and media commands.

Handle radio

Radio MediaBrowseService can be identified by the ACTION_PLAY_BROADCASTRADIO intent filter. They are expected to follow the playback controls and media browse structure described in Implement radio. AAOS offers the car-broadcastradio-support library containing constants and methods to help OEMs create MediaBrowseService implementations for their own radio services that follow the defined protocol, and provides support for apps consuming their browse tree (for example, VIAs).

Handle auxiliary input, CD audio, and USB media

There is no default implementation of these media sources as part of AOSP. The suggested approach is to:

• Have OEMs implement media services for each of them. For details, see Build media apps for cars.
• These MediaBrowseService implementations would be identified and responded to in the intent actions defined at General play intents.
• These services would expose a browse tree following the guidelines described at Other source types.

Handle Bluetooth

Bluetooth media content is exposed through the AVRCP Bluetooth profile. In order to facilitate access to this functionality, AAOS includes a MediaBrowserService and MediaSession implementation that abstracts out the communication details (see packages/apps/Bluetooth).

The respective media browser tree structure is defined at BrowseTree class. Playback control commands can be delivered similarly to any other app, by using its MediaSession implementation.

Handle streaming media commands

To implement server side media streaming, the VIA must become itself a media source, implementing MediaBrowse and MediaSession API. Refer to Build media apps for cars. By implementing these APIs, a voice control app would be able to (among other things):

• Participate seamlessly in the media source selection
• Be automatically resumed after car restart
• Provide playback and browsing control using Media Center UI
• Receive standard hardware media button events

Fulfill navigation commands

There is no standardized way of interacting with all navigation apps. For integrations with Google Maps, see Google Maps for Android Automotive Intents. For integrations with other apps, contact the app developers directly. Before launching an intent to any app (including Google Maps), verify that the intent can be resolved (see Intent requests). This creates the opportunity to inform the user in case the target app is not available.

Fulfill vehicle commands

Access to vehicle properties for both read and write is provided through CarPropertyManager. Vehicle properties types, its implementation and other details are explained in Property configurations. For an accurate description of the properties supported by Android, it is best to refer directly to hardware/interfaces/automotive/vehicle/2.0/types.hal. The VehicleProperty enum defined there contains both standard and vendor specific properties, data types, change mode, units and read/write access definition.

To access these same constants from Java, you can use VehiclePropertyIds and its companion classes. Different properties have different Android permissions controlling their access. These permissions are declared in the CarService manifest, and the mapping between properties and permissions described in the VehiclePropertyIds Javadoc and enforced in PropertyHalServiceIds.

Read a vehicle property

The following is an example showing how to read the vehicle speed:

public class CarActuator ... {
    private final Car mCar;
    private final CarPropertyManager mCarPropertyManager;
    private final TextToSpeech mTTS;

    /** Global VHAL area id */
    public static final int GLOBAL_AREA_ID = 0;

    public CarActuator(Context context, TextToSpeech tts) {
        mCar = Car.createCar(context);
        mCarPropertyManager = (CarPropertyManager) mCar.getCarManager(Car.PROPERTY_SERVICE);
        mTTS = tts;
        ...
    }

    @Nullable
    private void getSpeedInMetersPerSecond() {
        if (!mCarPropertyManager.isPropertyAvailable(VehiclePropertyIds.PERF_VEHICLE_SPEED,
                GLOBAL_AREA_ID)) {
            mTTS.speak("I'm sorry, but I can't read the speed of this vehicle");
            return;
        }
        // Data type and unit can be found in
        // automotive/vehicle/2.0/types.hal
        float speedInMps = mCarPropertyManager.getFloatProperty(
                VehiclePropertyIds.PERF_VEHICLE_SPEED, GLOBAL_AREA_ID);
        int speedInMph = (int)(speedInMetersPerSecond * 2.23694f);
        mTTS.speak(String.format("Sure. Your current speed is %d miles "
                + "per hour", speedInUserUnit);
    }

    ...
}

Set a vehicle property

The following is an example showing how to turn on and off the front AC.

public class CarActuator … {
    …

    private void changeFrontAC(boolean turnOn) {
        List<CarPropertyConfig> configs = mCarPropertyManager
                .getPropertyList(new ArraySet<>(Arrays.asList(
                    VehiclePropertyIds.HVAC_AC_ON)));
        if (configs == null || configs.size() != 1) {
            mTTS.speak("I'm sorry, but I can't control the AC of your vehicle");
            return;
        }

        // Find the front area Ids for the AC property.
        int[] areaIds = configs.get(0).getAreaIds();
        List<Integer> areasToChange = new ArrayList<>();
        for (int areaId : areaIds) {
            if ((areaId & (VehicleAreaSeat.SEAT_ROW_1_CENTER
                        | VehicleAreaSeat.SEAT_ROW_1_LEFT
                        | VehicleAreaSeat.SEAT_ROW_1_RIGHT)) == 0) {
                continue;
            }
            boolean isACInAreaAlreadyOn = mCarPropertyManager
                    .getBooleanProperty(VehiclePropertyIds.HVAC_AC_ON, areaId);
            if ((!isACInAreaAlreadyOn && turnOn) || (isACInAreaAlreadyOn && !turnOn)) {
                areasToChange.add(areaId);
            }
        }
        if (areasToChange.isEmpty()) {
            mTTS.speak(String.format("The AC is already %s", turnOn ? "on" : "off"));
            return;
        }

        for (int areaId : areasToChange) {
            mCarPropertyManager.setBooleanProperty(
                VehiclePropertyIds.HVAC_AC_ON, areaId, turnOn);
        }
        mTTS.speak(String.format("Okay, I'm turning your front AC %s",
            turnOn ? "on" : "off"));
    }

    …
}

Fulfill communication commands

Handle messaging commands

VIAs must handle incoming messages following the "tap-to-read" flow described in Voice assistant Tap-to-Read, which can optionally handle sending replies back to the incoming message sender. Additionally, VIAs can use SmsManager (part of the android.telephony package) to compose and send SMS messages directly from the car or over Bluetooth.

Handle call commands

In a similar way, VIAs can use TelephonyManager to place phone calls and call to the user's voice mail number. In these cases, VIAs will interact with the telephony stack directly or with the Car Dialer app. In any case, the Car Dialer app should be the one displaying voice-call related UI to the user.

Fulfill other commands

For a list of other possible points of integration between the VIA and the system, check the list of well-known Android intents. Many user commands can be resolved server-side (for example, reading users emails and calendar events) and don't require any interactions with the system other than the voice interaction itself.

Immersive actions (display visual content)

Where it enhances user actions or understanding, a VIA can provide supplementary visual content on the car screen. To minimize driver distraction, keep such content simple, brief, and actionable. For details about UI/UX guidelines on immersive actions, see Preloaded Assistants: UX Guidance.

To enable customization and consistency with the rest of the head unit (HU) design, VIAs should use Car UI Library components for most of the UI elements. For details, see Customization.