In Android 13, the app streaming feature lets phones
stream their apps to connected devices, and lets those devices interact with the
apps. A new COMPANION_DEVICE_APP_STREAMING
role lets an app create and manage proxies for connected remote devices to
stream apps from the local device to the remote device. The app holding the
COMPANION_DEVICE_APP_STREAMING
role can obtain permissions to create a virtual
display, launch an app on it, and then stream a video of the launched app to
another device. The role holder app can also inject input and microphone events
received from a remote device back into the local device, as if that device were
connected as a peripheral.
Architecture
Beginning in Android 13, the new Virtual Device Manager and the Companion Device Manager (CDM) form the key building blocks to support app streaming and interaction with remote, connected devices.
Virtual Device Manager
App streaming works by taking advantage of virtual displays. The Virtual Device Manager creates a virtual display that's separate from the visible primary display. When the user consents to start streaming an app, the app is launched on, or transferred to, the virtual display. The contents of the virtual display are transferred in a video stream to the connected device to be displayed.
The Virtual Device Manager includes APIs that enable the creation, registration,
and management of
VirtualDevice
instances.
A VirtualDevice
instance is a proxy for the connected device and its
capabilities. A VirtualDevice
instance enables a connected device to receive,
display and interact with an incoming app stream, by doing the following:
- Creating a
VirtualDisplay
instance intended to be displayed on a connected device's display. - Injecting a remote audio stream, such as from the connected device's microphone, into the local device for playback.
- Injecting remote input events, such as the from the connected device's keyboard, into the local device for playback.
Companion Device Manager
The CDM manages the connectivity state and enforces the role requirements that must be met to enable app streaming.
The following figure illustrates the interactions between the local and remote device during app streaming:
Figure 1. Interactions between the local and remote devices during app streaming
Implementation of the companion app streaming role
OEMs can implement an app with the COMPANION_DEVICE_APP_STREAMING
role to
enable a rich and secure cross-device experience on their devices.
To create a virtual device to enable app streaming, an app must be a role holder
of the COMPANION_DEVICE_APP_STREAMING
role. When the role is granted, the app
receives the
CREATE_VIRTUAL_DEVICE
permission, which enables
the app to create a virtual device. Role
holders are expected to implement app streaming by creating a VirtualDevice
instance that acts as a proxy for a connected device. The VirtualDevice
class
provides methods that enable the extraction of a composited surface of a virtual
display, suitable for streaming to another device. The VirtualDevice
class
also provides APIs for injecting input events that occur on a remote device into
the local device, enabling an app that's running on a local device to appear as
if it's running natively on a remote device.
Follow these steps to implement the companion app streaming role:
Create an app that requests the
REQUEST_COMPANION_PROFILE_APP_STREAMING
permission in the manifest.Prompt the user to grant the app permission to perform app streaming.
Create a CDM
AssociationRequest
instance to request aCOMPANION_DEVICE_APP_STREAMING
role. The app receives theCREATE_VIRTUAL_DEVICE
permission when the role is granted.Call
VirtualDeviceManager#create()
to create aVirtualDevice
instance. With aVirtualDevice
instance, the app can create and manage virtual displays and virtual inputs.Launch the notification's pending intent on the virtual display and create a video capture of that display.
Create a connection to the connected device and stream the virtual display over to the connected device.
Inject back input events from the connected device into the local device through the
VirtualDevice
APIs.When the user closes the streaming app on the remote device, end the stream and tear down the
VirtualDevice
instance. At this point, the previously streaming app runs in the background on the local device and the connection is closed.If needed, await more signals from the connected device to restart the app streaming.
The app is responsible for connecting to the remote device, reporting the connectivity state to CDM, and enforcing security requirements as described in the CDD.
COMPANION_DEVICE_APP_STREAMING role holder requirements
The CDM grants the COMPANION_DEVICE_APP_STREAMING
role when the user
associates an app streaming app with a device. This role is associated with a
device profile so there's some control over which apps can be added to the Play
Store that match this profile. See Android
roles
for a list of the COMPANION_DEVICE_APP_STREAMING
role requirements. Reach out
to your Google point of contact for more information.
COMPANION_DEVICE_APP_STREAMING role holder capabilities
To perform app streaming, the COMPANION_DEVICE_APP_STREAMING
role assumes that
the role holder app has the following capabilities and behaviors:
- Create and manage connections to other devices.
- Create and manage trusted virtual displays, including unlocked displays, as
follows:
- Start activities on the virtual display.
- Inject events that happen on a streamed app on a connected device back on the local device, such as playing a touch event on the tablet at the same coordinates on the phone.
- Capture audio data from the streamed app.
- Replace the local device's microphone stream with a connected device's microphone stream while a streamed app is using the microphone.
- Replace the local device's camera stream with a connected device's camera stream while a streamed app is using the camera.
- Manage and stream notifications from the local device to the connected device, and take actions on notifications.
- Stream metadata from the local device, such as the list of apps available on the local device, to the connected device.
- Request device verification.