SurfaceTexture is a combination of a surface and an OpenGL ES (GLES)
SurfaceTexture instances are used to provide surfaces that output to GLES
SurfaceTexture contains an instance of
BufferQueue for which
apps are the consumer. The
notifies apps when the producer queues a new buffer. Then, apps call
updateTexImage(), which releases the previously held buffer,
acquires the new buffer from the queue, and makes EGL calls to make the buffer available to GLES as an external texture.
External GLES textures
External GLES textures (
from traditional GLES textures (
GL_TEXTURE_2D) in the following
- External textures render textured polygons directly from data received
- External texture renderers are configured differently than traditional GLES texture renderers.
- External textures can't perform all traditional GLES texture activities.
The main benefit of external textures is
their ability to render directly from
instances set the consumer usage flags to
GRALLOC_USAGE_HW_TEXTURE when it creates
BufferQueue instances for external textures to ensure that the data in the buffer is
recognizable by GLES.
SurfaceTexture instances interact with an EGL context, an app can only call
its methods while the EGL context that owns the texture is current on the
calling thread. For more information see the
SurfaceTexture class documentation.
Timestamps and transformations
SurfaceTexture instances include the
getTimeStamp() method, which
retrieves a timestamp, and
getTransformMatrix() method, which
retrieves a transformation matrix. Calling
sets both the timestamp and the transformation matrix. Each buffer that
BufferQueue passes includes transformation parameters and a timestamp.
Transformation parameters are useful for efficiency. In some cases, source data might be in the incorrect orientation for the consumer. Instead of rotating the data before sending it to the consumer, send the data in its orientation with a transform that corrects it. The transformation matrix can be merged with other transformations when the data is used, minimizing overhead.
The timestamp is useful for buffer sources that are time dependent. For
the producer interface to the output of the camera, frames from the camera can
be used to create a video. Each frame needs to have a presentation
timestamp from when the frame was captured, not from when the app received the
frame. The camera code sets the timestamp provided with the buffer,
resulting in a more consistent series of timestamps.
Case study: Grafika's continuous capture
Grafika's continuous capture involves recording frames
from a device's camera and displaying those frames on screen.
To record frames, create a surface with the
createInputSurface() method and pass the surface to the camera. To
display frames, create an instance of
SurfaceView and pass the surface to
setPreviewDisplay(). Note that recording frames and displaying
them at the same time is a more involved process.
The continuous capture activity displays video from the camera as video is being recorded. In this case, encoded video is written to a circular buffer in memory that can be saved to disk at any time.
This flow involves three buffer queues:
App— The app uses a
SurfaceTextureinstance to receive frames from the camera, converting them to an external GLES texture.
SurfaceFlinger— The app declares a
SurfaceViewinstance to display the frames.
MediaServer— Configure a
MediaCodecencoder with an input surface to create the video.
In the figure below, the arrows indicate data propagation from the camera.
BufferQueue instances are in color (producers are teal, consumers are green).
Encoded H.264 video goes to a circular buffer in RAM in the app process.
When a user presses the capture button, the
writes the encoded video to an MP4 file on disk.
BufferQueue instances are handled with a single EGL context in the
app while the GLES operations are performed on the UI thread. The handling of
encoded data (managing a circular buffer and writing it to disk) is done
on a separate thread.
surfaceCreated()callback creates the
EGLSurfaceinstances for the display and the video encoder. When a new frame arrives,
SurfaceTextureperforms four activities:
- Acquires the frame.
- Makes the frame available as a GLES texture.
- Renders the frame with GLES commands.
- Forwards the transform and timestamp for each instance of
The encoder thread then pulls the encoded output from
MediaCodec and stashes
it in memory.
Secure texture video playback
Android supports GPU post-processing of protected video content. This lets apps use the GPU for complex, nonlinear video effects (such as warps), mapping protected video content onto textures for use in general graphics scenes (for example, using GLES), and virtual reality (VR).
Support is enabled using the following two extensions:
- EGL extension —
EGL_EXT_protected_content) Enables the creation of protected GL contexts and surfaces, which can both operate on protected content.
- GLES extension —
GL_EXT_protected_textures) Enables tagging textures as protected so they can be used as framebuffer texture attachments.
SurfaceTexture and ACodec
libstagefright.so) to send protected content even if
the window's surface doesn't queue to
and provides a protected video surface for use within a protected context. This
is done by setting the protected consumer bit
GRALLOC_USAGE_PROTECTED) on surfaces created in a protected
context (verified by ACodec).
Secure texture video playback sets the foundation for strong DRM implementation in the OpenGL ES environment. Without a strong DRM implementation, such as Widevine Level 1, many content providers don't allow rendering of their high-value content in the OpenGL ES environment, preventing important VR use cases such as watching DRM-protected content in VR.
AOSP includes framework code for secure texture video playback. Driver
support is up to OEMs. Device implementers must implement
GL_EXT_protected_textures extensions. When using your own codec
library (to replace
libstagefright), note the changes in
/frameworks/av/media/libstagefright/SurfaceUtils.cpp that allow
buffers marked with
GRALLOC_USAGE_PROTECTED to be sent to
ANativeWindow (even if
ANativeWindow doesn't queue directly to the window
composer) as long as the consumer usage bits contain
GRALLOC_USAGE_PROTECTED. For detailed documentation on implementing
the extensions, refer to the Khronos registries