Companion app streaming

In Android 13, the app streaming feature lets phones stream their apps to connected devices, and lets those devices interact with the apps. A new COMPANION_DEVICE_APP_STREAMING role lets an app create and manage proxies for connected remote devices to stream apps from the local device to the remote device. The app holding the COMPANION_DEVICE_APP_STREAMING role can obtain permissions to create a virtual display, launch an app on it, and then stream a video of the launched app to another device. The role holder app can also inject input and microphone events received from a remote device back into the local device, as if that device were connected as a peripheral.

Architecture

Beginning in Android 13, the new Virtual Device Manager and the Companion Device Manager (CDM) form the key building blocks to support app streaming and interaction with remote, connected devices.

Virtual Device Manager

App streaming works by taking advantage of virtual displays. The Virtual Device Manager creates a virtual display that's separate from the visible primary display. When the user consents to start streaming an app, the app is launched on, or transferred to, the virtual display. The contents of the virtual display are transferred in a video stream to the connected device to be displayed.

The Virtual Device Manager includes APIs that enable the creation, registration, and management of VirtualDevice instances.

A VirtualDevice instance is a proxy for the connected device and its capabilities. A VirtualDevice instance enables a connected device to receive, display and interact with an incoming app stream, by doing the following:

• Creating a VirtualDisplay instance intended to be displayed on a connected device's display.
• Injecting a remote audio stream, such as from the connected device's microphone, into the local device for playback.
• Injecting remote input events, such as the from the connected device's keyboard, into the local device for playback.

Companion Device Manager

The CDM manages the connectivity state and enforces the role requirements that must be met to enable app streaming.

The following figure illustrates the interactions between the local and remote device during app streaming:

Figure 1. Interactions between the local and remote devices during app streaming

Implementation of the companion app streaming role

OEMs can implement an app with the COMPANION_DEVICE_APP_STREAMING role to enable a rich and secure cross-device experience on their devices.

Note: Only preinstalled priv-apps are allowed to hold the COMPANION_DEVICE_APP_STREAMING role. OEMs must ensure that the app is submitted for review as part of the device verification process. Reach out to your Google point of contact for more information.

To create a virtual device to enable app streaming, an app must be a role holder of the COMPANION_DEVICE_APP_STREAMING role. When the role is granted, the app receives the CREATE_VIRTUAL_DEVICE permission, which enables the app to create a virtual device. Role holders are expected to implement app streaming by creating a VirtualDevice instance that acts as a proxy for a connected device. The VirtualDevice class provides methods that enable the extraction of a composited surface of a virtual display, suitable for streaming to another device. The VirtualDevice class also provides APIs for injecting input events that occur on a remote device into the local device, enabling an app that's running on a local device to appear as if it's running natively on a remote device.

Follow these steps to implement the companion app streaming role:

1. Create an app that requests the REQUEST_COMPANION_PROFILE_APP_STREAMING permission in the manifest.

2. Prompt the user to grant the app permission to perform app streaming.

3. Create a CDM AssociationRequest instance to request a COMPANION_DEVICE_APP_STREAMING role. The app receives the CREATE_VIRTUAL_DEVICE permission when the role is granted.

4. Call VirtualDeviceManager#create() to create a VirtualDevice instance. With a VirtualDevice instance, the app can create and manage virtual displays and virtual inputs.

5. Launch the notification's pending intent on the virtual display and create a video capture of that display.

6. Create a connection to the connected device and stream the virtual display over to the connected device.

7. Inject back input events from the connected device into the local device through the VirtualDevice APIs.

8. When the user closes the streaming app on the remote device, end the stream and tear down the VirtualDevice instance. At this point, the previously streaming app runs in the background on the local device and the connection is closed.

9. If needed, await more signals from the connected device to restart the app streaming.

The app is responsible for connecting to the remote device, reporting the connectivity state to CDM, and enforcing security requirements as described in the CDD.

Note: Google offers a closed-source implementation called Cross Device Service that enables app streaming from any Android phone running Android 13 or higher, to a Chromebook. OEMs can preload the Cross Device Service APK under the priv-app directory.

COMPANION_DEVICE_APP_STREAMING role holder requirements

The CDM grants the COMPANION_DEVICE_APP_STREAMING role when the user associates an app streaming app with a device. This role is associated with a device profile so there's some control over which apps can be added to the Play Store that match this profile. See Android roles for a list of the COMPANION_DEVICE_APP_STREAMING role requirements. Reach out to your Google point of contact for more information.

COMPANION_DEVICE_APP_STREAMING role holder capabilities

To perform app streaming, the COMPANION_DEVICE_APP_STREAMING role assumes that the role holder app has the following capabilities and behaviors:

• Create and manage connections to other devices.
• Create and manage trusted virtual displays, including unlocked displays, as follows:
  • Start activities on the virtual display.
  • Inject events that happen on a streamed app on a connected device back on the local device, such as playing a touch event on the tablet at the same coordinates on the phone.
  • Capture audio data from the streamed app.
  • Replace the local device's microphone stream with a connected device's microphone stream while a streamed app is using the microphone.
  • Replace the local device's camera stream with a connected device's camera stream while a streamed app is using the camera.
• Manage and stream notifications from the local device to the connected device, and take actions on notifications.
• Stream metadata from the local device, such as the list of apps available on the local device, to the connected device.
• Request device verification.