HDR in Android screenshots

Following the introduction of high dynamic range (HDR) video, streaming services began streaming HDR video, focusing on full-screen experiences. Social media apps now support HDR video and Ultra HDR, which indicates growing interest in HDR adoption across various apps.

Android support for HDR

Android has supported HDR technology through several milestones:

Android 7

  • Initial support for HDR video decoding and display.
  • Continued advancements in HDR capabilities.

Android 13

  • End-to-end support for HDR video capture, encoding, and display.
  • Introduced Mixed SDR and HDR composition, which defines different displayable luminance ranges between SDR and HDR.

Android 14

HDR screenshot support has also evolved and changed.

Advancements in HDR screenshot capabilities

This section describes the progression of HDR screenshot capability in Android updates.

Android 9

SurfaceFlinger, Android's graphics compositor, introduces HDR video support. GPU rendering of HDR video and screenshots uses a complex polynomial tone mapper. This tone-mapping curve isn't always equivalent to the display tone-mapper, so screenshots differ from the on-screen content.

Android 13

SurfaceFlinger's GPU rendering block includes a tone mapping plugin, which lets the OEM provide a GPU shader to match their display's tone-mapping curve. Screenshots almost match the on-screen content, but with the following differences:

  • Screenshots remain in SDR format. Consequently, when viewed alongside an HDR scene, the HDR regions within the screenshot appear dimmer.
  • SDR luminance isn't managed, so SDR content within the screenshot appears as bright as HDR content.

In other words, the system converts any HDR video captured in the screenshot to SDR video.

Android 14

Ultra HDR poses a significant challenge to screenshotting. Unlike videos, the system typically renders images within the UI framebuffer, which has two main implications:

  • Images can't have image processing, including tonemapping, that differs from the surrounding UI.
  • Apps are responsible for source-based tone mapping when rendering their UI.

To alleviate this challenge, consider three potential screenshotting implementations:

  • Preserve the HDR details of an Ultra HDR image, resulting in a darkened app UI in the screenshot.
  • Preserve the app UI details, causing Ultra HDR image clipping.
  • Compromise by brightening the app UI while clipping HDR highlights.

Android 14 implements the third approach of brightening the app UI and clipping HDR highlights.

Android 15-QPR1

SurfaceFlinger includes a local tone-mapping algorithm for screenshots. This process involves:

  • Dividing the input image into smaller images.
  • Computing the maximum luminance in each image, and discarding low luminance values within each section.
  • Interpolating the computed luminances through blurring and resampling.
  • Applying a parameterized Reinhard tonemapper to the input image, based on the interpolated luminance values.

This algorithm significantly improves screenshots between Android 14 and Android 15-QPR1, as the following examples demonstrate:

  • Example 1 is a screenshot of an HDR video overlaid on top of a Chrome page containing Ultra HDR. The UI colors are mostly preserved in the new implementation, and the image is no longer clipped.

    Android 14 Android 15-QPR1
    Example 1 in Android 14 Example 1 in Android 15qpr

    Figure 1. Comparison of Android 14 and Android 15-QPR1 for Example 1.

  • Example 2 is a screenshot of an HDR video overlaid on top of Settings with subsequent screenshots. In Android 14, the screenshot colors are successively darker. In Android 15-QPR1, the tonemapper correctly replicates and preserves the UI colors.

    Android 14 Android 15-QPR1
    Example 2 in Android 14 Example 2 in Android 15qpr

    Figure 2. Comparison of Android 14 and Android 15-QPR1 for Example 2.

Android 16

Similar to Ultra HDR, HDR screenshots store a gainmap in the screenshot file to recover the HDR representation during rendering. However, unlike Ultra HDR, the screenshot remains in a PNG format for backward compatibility with systems that ingest PNG screenshots.

Screenshot generation involves the following:

  • When HDR content is displayed on the device, a screenshot is generated using FP16 pixels.
  • The local tone-mapper described in Android 15-QPR1 generates an 8-bit base SDR rendition.
  • An 8-bit gainmap is produced by combining the SDR base rendition with the HDR rendition.
  • The SDR base rendition and the gainmap are encoded into a single PNG file.

PNG encoding involves the following:

  • The gainmap is encoded as a PNG image, which includes a gmAP chunk, containing the ISO 21496-1 metadata for the gainmap.
  • The SDR base rendition is encoded as a PNG image, which includes a gmAP chunk, containing the version of the ISO 21496-1 metadata. This PNG image also includes a gdAT chunk, containing the entirety of the encoded gainmap PNG.

The following figure shows the layout of the PNG chunks:

Layout of the PNG chunks

Figure 3. Layout of the PNG chunks.

With Android 16, the PNG codec supports both encoding and decoding of these PNGs. Apps can display a PNG with a gainmap in the same manner as Ultra HDR.