Rozszerzenia aparatu

Producenci urządzeń mogą udostępniać rozszerzenia, takie jak bokeh, tryb nocny i HDR, niezależnym programistom za pośrednictwem interfejsu Camera Extensions udostępnianego przez bibliotekę dostawców OEM. Programiści mogą korzystać z interfejsów API rozszerzeń Camera2 i interfejsu API rozszerzeń CameraX, aby uzyskać dostęp do rozszerzeń zaimplementowanych w bibliotece dostawców OEM.

Aby uzyskać listę obsługiwanych rozszerzeń, które są takie same w Camera2 i CameraX, zobacz CameraX Extensions API . Jeśli chcesz dodać rozszerzenie, zgłoś błąd za pomocą narzędzia do śledzenia problemów .

Na tej stronie opisano, jak zaimplementować i włączyć bibliotekę dostawcy OEM na urządzeniach.

Architektura

Poniższy schemat opisuje architekturę interfejsu rozszerzeń kamery lub extensions-interface :

Rysunek 1. Diagram architektury rozszerzeń aparatu

Jak pokazano na diagramie, aby obsługiwać rozszerzenia aparatu, należy zaimplementować extensions-interface dostarczony przez bibliotekę dostawcy OEM. Twoja biblioteka dostawcy OEM udostępnia dwa interfejsy API: CameraX Extensions API i Camera2 Extensions API , które są używane odpowiednio przez aplikacje CameraX i Camera2 w celu uzyskiwania dostępu do rozszerzeń dostawców.

Zaimplementuj bibliotekę dostawców OEM

Aby zaimplementować bibliotekę dostawcy OEM, skopiuj pliki camera-extensions-stub do projektu biblioteki systemowej. Te pliki definiują interfejs Camera Extensions.

Uwaga: Pliki należy przechowywać w pakietach Java androidx.camera.extensions.impl . Aby uzyskać informacje, zobacz przykładowy projekt .

Pliki camera-extensions-stub są podzielone na następujące kategorie:

Niezbędne pliki interfejsu (nie modyfikuj)

• PreviewExtenderImpl.java
• ImageCaptureExtenderImpl.java
• ExtenderStateListener.java
• ProcessorImpl.java
• PreviewImageProcessorImpl.java
• CaptureProcessorImpl.java
• CaptureStageImpl.java
• RequestUpdateProcessorImpl.java
• ProcessResultImpl.java
• advanced/AdvancedExtenderImpl.java
• advanced/Camera2OutputConfigImpl.java
• advanced/Camera2SessionConfigImpl.java
• advanced/ImageProcessorImpl.java
• advanced/ImageReaderOutputConfigImpl.java
• advanced/ImageReferenceImpl.java
• advanced/MultiResolutionImageReaderOutputConfigImpl.java
• advanced/OutputSurfaceImpl.java
• advanced/RequestProcessorImpl.java
• advanced/SessionProcessorImpl.java
• advanced/SurfaceOutputConfigImpl.java

Implementacje obowiązkowe (dodaj swoją implementację)

• ExtensionVersionImpl.java
• InitializerImpl.java

Klasy przedłużaczy Bokeh (zaimplementuj je, jeśli obsługiwane jest rozszerzenie Bokeh)

• BokehImageCaptureExtenderImpl.java
• BokehPreviewExtenderImpl.java
• advanced/BokehAdvancedExtenderImpl.java

Klasy rozszerzeń nocnych (zaimplementuj je, jeśli obsługiwane jest rozszerzenie nocne)

• NightImageCaptureExtenderImpl.java
• NightPreviewExtenderImpl.java
• advanced/NightAdvancedExtenderImpl.java

Klasy automatycznych rozszerzeń (zaimplementuj je, jeśli obsługiwane jest automatyczne rozszerzenie)

• AutoImageCaptureExtenderImpl.java
• AutoPreviewExtenderImpl.java
• advanced/AutoAdvancedExtenderImpl.java

Klasy rozszerzeń HDR (zaimplementuj je, jeśli obsługiwane jest rozszerzenie HDR)

• HdrImageCaptureExtenderImpl.java
• HdrPreviewExtenderImpl.java
• advanced/HdrAdvancedExtenderImpl.java

Klasy rozszerzenia Face Retouch (zaimplementuj je, jeśli obsługiwane jest rozszerzenie Face Retouch)

• BeautyImageCaptureExtenderImpl.java
• BeautyPreviewExtenderImpl.java
• advanced/BeautyAdvancedExtenderImpl.java

Narzędzia (opcjonalnie, można usunąć)

• advanced/Camera2OutputConfigImplBuilder.java
• advanced/Camera2SessionConfigImplBuilder.java

Nie musisz dostarczać implementacji dla każdego rozszerzenia. Jeśli nie zaimplementujesz rozszerzenia, ustaw isExtensionAvailable() na zwracanie false lub usuń odpowiednie klasy Extender. Interfejsy API rozszerzeń Camera2 i CameraX zgłaszają aplikacji, że rozszerzenie jest niedostępne.

Uwaga: ponieważ biblioteka dostawcy OEM działa jako część procesu aplikacji innej firmy, biblioteka dostawcy NIE może wymagać żadnych specjalnych uprawnień (takich jak uprawnienia systemowe).

Przyjrzyjmy się, w jaki sposób interfejsy API rozszerzeń Camera2 i CameraX współdziałają z biblioteką dostawcy w celu włączenia rozszerzenia. Poniższy diagram ilustruje kompleksowy przepływ na przykładzie rozszerzenia Night:

Rysunek 2. Implementacja przedłużenia nocnego

Uwaga: dla wygody interfejsy API rozszerzeń Camera2 i CameraX określamy jako Camera2/X .

1. Weryfikacja wersji:

   Camera2/X wywołuje funkcję ExtensionVersionImpl.checkApiVersion() , aby upewnić się, że wersja extensions-interface zaimplementowana przez producenta OEM jest kompatybilna z wersjami obsługiwanymi przez Camera2/X.

2. Inicjalizacja biblioteki dostawcy:

   InitializerImpl ma metodę init() , która inicjuje bibliotekę dostawcy. Camera2/X kończy inicjalizację przed uzyskaniem dostępu do klas Extender.

3. Utwórz wystąpienia klasy Extender:

   Tworzy wystąpienia Extender klas dla rozszerzenia. Istnieją dwa typy urządzeń Extender: Basic Extender i Advanced Extender. Musisz zaimplementować jeden typ Extendera dla wszystkich rozszerzeń. Aby uzyskać więcej informacji, zobacz Basic Extender a Advanced Extender .

   Camera2/X tworzy instancje i wchodzi w interakcje z klasami Extender w celu pobrania informacji i włączenia rozszerzenia. Dla danego rozszerzenia Camera2/X może wielokrotnie tworzyć instancje klas Extender. W rezultacie nie wykonuj intensywnej inicjalizacji w konstruktorze ani w wywołaniu init() . Ciężkie prace wykonuj tylko wtedy, gdy ma się rozpocząć sesja kamery, na przykład gdy onInit() jest wywoływana w Basic Extender lub initSession() jest wywoływana w Advanced Extender.

   W przypadku rozszerzenia Night dla typu Basic Extender tworzone są instancje następujących klas Extender:

   • NightImageCaptureExtenderImpl.java
   • NightPreviewExtenderImpl.java

   A dla typu Advanced Extender:

   • NightAdvancedExtenderImpl.java

4. Sprawdź dostępność rozszerzenia:

   Przed włączeniem rozszerzenia isExtensionAvailable() sprawdza, czy rozszerzenie jest dostępne dla określonego identyfikatora kamery za pośrednictwem instancji Extender.

5. Zainicjuj ekstender z informacjami o aparacie:

   Camera2/X wywołuje init() w instancji Extender i przekazuje jej identyfikator kamery oraz CameraCharacteristics .

6. Informacje o zapytaniu:

   Wywołuje klasę Extender w celu pobrania informacji, takich jak obsługiwane rozdzielczości, nadal przechwytuje szacowane opóźnienie i przechwytuje klucze żądań z programu Extender w ramach przygotowań do włączenia rozszerzenia.

7. Włącz rozszerzenie na urządzeniu Extender:

   Klasa Extender zapewnia wszystkie interfejsy potrzebne do włączenia klasy. Oferuje mechanizm dołączania implementacji OEM do potoku Camera2, taki jak wstrzykiwanie parametrów żądania przechwytywania lub włączanie postprocesora.

   W przypadku typu Advanced Extender Camera2/X współdziała z SessionProcessorImpl w celu włączenia rozszerzenia. Camera2/X pobiera instancję SessionProcessorImpl przez wywołanie metody createSessionProcessor() na urządzeniu Extender.

W poniższych sekcjach opisano bardziej szczegółowo przepływ rozszerzenia.

Weryfikacja wersji

Podczas ładowania biblioteki dostawcy OEM z urządzenia w czasie wykonywania, Camera2/X sprawdza, czy biblioteka jest kompatybilna z wersją extensions-interface . extensions-interface używa wersjonowania semantycznego lub MAJOR.MINOR.PATCH, na przykład 1.1.0 lub 1.2.0. Jednak podczas weryfikacji wersji używane są tylko wersje główne i pomocnicze.

Aby zweryfikować wersję, Camera2/X wywołuje ExtensionVersionImpl.checkApiVersion() z obsługiwaną wersją extensions-interface . Camera2/X używa następnie wersji zgłoszonej przez bibliotekę OEM, aby określić, czy rozszerzenie może zostać włączone i jakie możliwości powinno wywołać.

Uwaga: funkcja checkApiVersion() nie powinna zwracać wersji extensions-interface nowszej niż wersja obsługiwana przez Camera2/X. Na przykład, jeśli Camera2/X obsługuje wersję 1.3.0, a biblioteka dostawcy OEM obsługuje wersję 1.4.0, powinna zwrócić wartość 1.3.0 zamiast 1.4.0.

Kompatybilność głównych wersji

Jeśli główne wersje interfejsu rozszerzenia różnią się między Camera2/X a biblioteką dostawcy, to jest to uważane za niekompatybilne i rozszerzenie jest wyłączone.

Kompatybilność wsteczna

Tak długo, jak główna wersja jest identyczna, Camera2/X zapewnia kompatybilność wsteczną z bibliotekami dostawców OEM zbudowanymi z wcześniejszymi wersjami extensions-interface . Na przykład, jeśli Camera2/X obsługuje extensions-interface 1.3.0, biblioteki dostawców OEM, które zaimplementowały wersje 1.0.0, 1.1.0 i 1.2.0, są nadal kompatybilne. Oznacza to również, że po zaimplementowaniu określonej wersji biblioteki dostawcy Camera2/X upewnia się, że biblioteka jest wstecznie kompatybilna z nadchodzącymi wersjami extension-interface .

Kompatybilność w przód

Kompatybilność w przód z bibliotekami dostawców nowszych extensions-interface zależy od Ciebie, producenta OEM. Jeśli potrzebujesz pewnych funkcji do zaimplementowania rozszerzeń, możesz włączyć rozszerzenia począwszy od określonej wersji. W takim przypadku możesz zwrócić obsługiwaną wersję extensions-interface gdy wersja biblioteki Camera2/X spełnia wymagania. Jeśli wersje Camera2/X nie są obsługiwane, możesz zwrócić niekompatybilną wersję, taką jak 99.0.0, aby wyłączyć rozszerzenia.

Inicjalizacja biblioteki dostawcy

Po sprawdzeniu wersji extensions-interface zaimplementowanej przez bibliotekę OEM, Camera2/X rozpoczyna proces inicjalizacji. Metoda InitializerImpl.init() sygnalizuje bibliotece OEM, że aplikacja próbuje użyć rozszerzeń.

Camera2/X nie wykonuje żadnych innych wywołań do biblioteki OEM (oprócz sprawdzania wersji), dopóki biblioteka dostawcy OEM nie wywoła OnExtensionsInitializedCallback.onSuccess() w celu powiadomienia o zakończeniu inicjalizacji.

Musisz zaimplementować InitializerImpl począwszy od extensions-interface 1.1.0. Camera2/X pomija etap inicjalizacji biblioteki, jeśli biblioteka dostawcy OEM implementuje extensions-interface 1.0.0.

Podstawowy przedłużacz a zaawansowany przedłużacz

Istnieją dwa typy implementacji extensions-interface : Basic Extender i Advanced Extender. Advanced Extender jest obsługiwany od wersji 1.2.0 extensions-interface .

Zaimplementuj Basic Extender dla rozszerzeń przetwarzających obrazy w warstwie HAL kamery lub użyj postprocesora zdolnego do przetwarzania strumieni YUV.

Zaimplementuj Advanced Extender dla rozszerzeń, które muszą dostosować konfigurację strumienia Camera2 i wysyłać żądania przechwytywania w razie potrzeby.

Zobacz poniższą tabelę dla porównania:

	Podstawowy przedłużacz	Zaawansowany przedłużacz
Konfiguracje strumieniowe	Naprawił Podgląd: PRIVATE lub YUV_420_888 (jeśli istnieje procesor) Przechwytywanie obrazu nieruchomego: JPEG lub YUV_420_888 (jeśli istnieje procesor)	Możliwość dostosowania przez OEM.
Wysyłanie żądania przechwycenia	Tylko Camera2/X może wysyłać żądania przechwytywania. Możesz ustawić parametry tych żądań. Gdy procesor jest przeznaczony do przechwytywania obrazu, Camera2/X może wysyłać wiele żądań przechwytywania i wysyłać wszystkie obrazy i wyniki przechwytywania do procesora.	Dostępna jest instancja RequestProcessorImpl w celu wykonania żądania przechwytywania kamery2 i uzyskania wyników oraz obrazu. Camera2/X wywołuje startRepeating i startCapture na SessionProcessorImpl , aby zasygnalizować producentowi OEM rozpoczęcie powtarzającego się żądania podglądu i odpowiednio rozpoczęcie sekwencji przechwytywania obrazu.
Haczyki w rurociągu kamery	onPresetSession zapewnia parametry sesji. onEnableSession wysyła pojedyncze żądanie zaraz po skonfigurowaniu CameraCaptureSession . onDisableSession wysyła pojedyncze żądanie przed zamknięciem CameraCaptureSession .	initSession inicjuje i zwraca dostosowaną konfigurację sesji Camera2 do tworzenia sesji przechwytywania. onCaptureSessionStart jest wywoływana zaraz po skonfigurowaniu CameraCaptureSession . onCaptureSessionEnd jest wywoływana przed zamknięciem CameraCaptureSession .
Nadaje się do	Rozszerzenia zaimplementowane w aparacie HAL lub w procesorze przetwarzającym obrazy YUV.	Posiada implementacje rozszerzeń oparte na Camera2. Wymaga niestandardowej konfiguracji strumienia, takiej jak strumień RAW. Wymaga interaktywnej sekwencji przechwytywania.
Obsługiwana wersja interfejsu API	Rozszerzenia Camera2: Android 13 lub nowszy Rozszerzenia CameraX: camera-extensions 1.1.0 lub nowsze	Rozszerzenia Camera2: Android 12L lub nowszy Rozszerzenia CameraX: camera-extensions 1.2.0-alpha03 lub nowsze

Uwaga: Jeśli ExtensionVersionImpl.isAdvancedExtenderImplemented() zwróci wartość true , należy zaimplementować wszystkie rozszerzenia w programie Advanced Extender. Jeśli isAdvancedExtenderImplemented() zwróci false , należy zaimplementować wszystkie rozszerzenia w programie Basic Extender.

Przepływy aplikacji

W poniższej tabeli przedstawiono trzy typy przepływów aplikacji i odpowiadające im wywołania API Camera Extensions. Chociaż Camera2/X udostępnia te interfejsy API, musisz odpowiednio zaimplementować bibliotekę dostawcy, aby obsługiwać te przepływy, które opisujemy bardziej szczegółowo w dalszej części.

	Rozszerzenia Camera2	Rozszerzenia CameraX
Dostępność rozszerzenia zapytania	CameraExtensionCharacteristics . getSupportedExtensions	ExtensionsManager. isExtensionAvailable
Informacje o zapytaniu	CameraExtensionCharacteristics. getExtensionSupportedSizes CameraExtensionCharacteristics. getEstimatedCaptureLatencyRangeMillis CameraExtensionCharacteristics. getAvailableCaptureRequestKeys CameraExtensionCharacteristics. getAvailableCaptureResultKeys	ExtensionsManager. getEstimatedCaptureLatencyRange CameraX obsługuje pozostałe informacje w bibliotece.
Podgląd i robienie zdjęć z włączonym rozszerzeniem	CameraDevice. createExtensionSession cameraExtensionsSession. setRepeatingRequest cameraExtensionsSession. capture	val cameraSelector = ExtensionsManager. getExtensionEnabledCameraSelector bindToLifecycle(właściciel cyklu życia, selektor kamery, podgląd, ...)

Podstawowy przedłużacz

Interfejs Basic Extender zapewnia zaczepy w kilku miejscach w potoku kamery. Każdy typ rozszerzenia ma odpowiednie klasy Extender, które muszą zaimplementować producenci OEM.

W poniższej tabeli wymieniono klasy rozszerzeń, które producenci OEM muszą zaimplementować dla każdego rozszerzenia:

	Klasy rozszerzeń do zaimplementowania
Noc	NightPreviewExtenderImpl.java NightImageCaptureExtenderImpl.java
HDR	HdrPreviewExtenderImpl.java HdrImageCaptureExtenderImpl.java
Automatyczny	AutoPreviewExtenderImpl.java AutoImageCaptureExtenderImpl.java
bokeh	BokehPreviewExtenderImpl.java BokehImageCaptureExtenderImpl.java
Retusz twarzy	BeautyPreviewExtenderImpl.java BeautyImageCaptureExtenderImpl.java

W poniższym przykładzie używamy PreviewExtenderImpl i ImageCaptureExtenderImpl jako symboli zastępczych. Zastąp je nazwami rzeczywistych plików, które wdrażasz.

Uwaga: Te klasy Extender mogą być tworzone kilka razy w zależności od potrzeb wewnątrz Camera2/X.

Basic Extender ma następujące możliwości:

• Wstaw parametry sesji podczas konfigurowania CameraCaptureSession ( onPresetSession ).
• Powiadamiaj o zdarzeniach rozpoczęcia i zamknięcia sesji przechwytywania i wyślij pojedyncze żądanie powiadomienia warstwy HAL ze zwróconymi parametrami ( onEnableSession , onDisableSession ).
• Wstaw parametry przechwytywania dla żądania ( PreviewExtenderImpl.getCaptureStage , ImageCaptureExtenderImpl.getCaptureStages ).
• Dodaj procesory do podglądu i przechwytywania, które są w stanie przetworzyć strumień YUV_420_888 .

Uwaga: w przypadku rozszerzeń, które chcesz włączyć, musisz zaimplementować zarówno podgląd, jak i przechwytywanie. Na przykład, jeśli chcesz włączyć rozszerzenie Night, musisz zaimplementować zarówno NightPreviewExtenderImpl.java , jak i NightCaptureExtenderImpl.java .

Zobaczmy, jak Camera2/X wywołuje extensions-interface , aby osiągnąć trzy wspomniane powyżej przepływy aplikacji.

Uwaga: Camera2/X zawsze inicjuje bibliotekę dostawcy ( InitializerImpl.init() ) przed uruchomieniem któregokolwiek z tych przepływów aplikacji.

Przepływ aplikacji 1: Sprawdź dostępność rozszerzenia

Rysunek 3. Przepływ aplikacji 1 w Basic Extender

W tym przepływie Camera2/X bezpośrednio wywołuje metodę isExtensionAvailable() zarówno PreviewExtenderImpl , jak i ImageCaptureExtenderImpl bez wywoływania init() . Obie klasy Extender muszą zwrócić true , aby włączyć rozszerzenia.

Często jest to pierwszy krok, który aplikacje sprawdzają, czy dany typ rozszerzenia jest obsługiwany dla danego identyfikatora kamery przed włączeniem rozszerzenia. Dzieje się tak, ponieważ niektóre rozszerzenia są obsługiwane tylko w przypadku niektórych identyfikatorów kamer.

Przepływ aplikacji 2: Zapytanie o informacje

Rysunek 4. Przepływ aplikacji 2 w Basic Extender

Po ustaleniu, czy rozszerzenie jest dostępne, aplikacje powinny zapytać o następujące informacje przed włączeniem rozszerzenia.

• Zakres opóźnienia przechwytywania nadal: ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange zwraca zakres opóźnienia przechwytywania, aby aplikacja mogła ocenić, czy należy włączyć rozszerzenie dla bieżącego scenariusza.

• Obsługiwane rozmiary powierzchni podglądu i przechwytywania: ImageCaptureExtenderImpl.getSupportedResolutions i PreviewExtenderImpl.getSupportedResolutions zwracają listę formatów obrazu i rozmiarów obsługiwanych dla formatu i rozmiaru powierzchni.

• Obsługiwane klucze żądań i wyników: Camera2/X wywołuje następujące metody w celu pobrania obsługiwanych kluczy żądań przechwytywania i kluczy wyników z Twojej implementacji:

  • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys
  • ImageCaptureExtenderImpl.getAvailableCapturetResultKeys

Camera2/X zawsze najpierw wywołuje init() na tych klasach Extender przed zapytaniem o więcej informacji.

Przepływ aplikacji 3: podgląd/zdjęcia z włączonym rozszerzeniem (implementacja HAL)

Rysunek 5. Przepływ aplikacji 3 w Basic Extender

Powyższy diagram ilustruje główny przepływ włączania podglądu i przechwytywania z rozszerzeniem bez żadnego procesora. Oznacza to, że warstwa HAL aparatu przetwarza rozszerzenie.

W tym przepływie Camera2/X najpierw wywołuje init() a następnie onInit , która powiadamia o rozpoczęciu sesji kamery z określonymi rozszerzeniami. Możesz wykonać ciężką inicjalizację w onInit() .

Podczas konfigurowania CameraCaptureSession , Camera2/X wywołuje onPresetSession , aby uzyskać parametry sesji. Po pomyślnym skonfigurowaniu sesji przechwytywania Camera2/X wywołuje onEnableSession zwracając instancję CaptureStageImpl zawierającą parametry przechwytywania. Camera2/X natychmiast wysyła pojedyncze żądanie z tymi parametrami przechwytywania, aby powiadomić warstwę HAL. Podobnie, przed zamknięciem sesji przechwytywania, Camera2/X wywołuje onDisableSession , a następnie wysyła pojedyncze żądanie ze zwróconymi parametrami przechwytywania.

Powtarzające się żądanie wywołane przez Camera2/X zawiera parametry żądania zwrócone przez PreviewExtenderImpl.getCaptureStage() . Ponadto żądanie przechwycenia obrazu nieruchomego zawiera parametry zwrócone przez ImageCaptureExtenderImpl.getCaptureStages() .

Wreszcie, Camera2/X wywołuje onDeInit() po zakończeniu sesji kamery. Możesz zwolnić zasoby w onDeinit() .

Uwaga: Camera2/X może otwierać i zamykać wiele sesji kamery między onInit() i onDeInit() , chociaż ten diagram pokazuje tylko jedną sesję kamery.

Podgląd procesora

Oprócz kamery HAL można również zaimplementować rozszerzenia w procesorze.

Zaimplementuj PreviewExtenderImpl.getProcessorType , aby określić typ procesora, jak wyjaśniono poniżej:

• PROCESSOR_TYPE_NONE : Brak procesora. Obrazy są przetwarzane w kamerze HAL.

• PROCESSOR_TYPE_REQUEST_UPDATE_ONLY : Typ procesora umożliwia aktualizację powtarzającego się żądania o nowe parametry żądania przechwytywania na podstawie najnowszego TotalCaptureResult .

  PreviewExtenderImpl.getProcessor musi zwrócić instancję RequestUpdateProcessorImpl , która przetwarza instancję TotalCaptureResult i zwraca instancję CaptureStageImpl , aby zaktualizować powtarzające się żądanie. PreviewExtenderImpl.getCaptureStage() powinien również odzwierciedlać wynik przetwarzania i zwracać najnowszy CaptureStageImpl .

• PROCESSOR_TYPE_IMAGE_PROCESSOR : Ten typ umożliwia zaimplementowanie procesora do przetwarzania obrazów YUV_420_888 i zapisywania danych wyjściowych na powierzchni PRIVATE .

  Musisz zaimplementować i zwrócić instancję PreviewImageProcessorImpl w PreviewExtenderImpl.getProcessor . Procesor jest odpowiedzialny za przetwarzanie obrazów wejściowych YUV_420_888 . Powinien zapisać dane wyjściowe w formacie PRIVATE podglądu. Camera2/X używa powierzchni YUV_420_888 zamiast PRIVATE do skonfigurowania CameraCaptureSession do podglądu.

  Zobacz poniższą ilustrację przepływu:

Rysunek 6. Podgląd przepływu z PreviewImageProcessorImpl

Interfejs PreviewImageProcessorImpl rozszerza ProcessImpl i ma trzy ważne metody:

• onOutputSurface(Surface surface, int imageFormat) ustawia powierzchnię wyjściową dla procesora. W przypadku PreviewImageProcessorImpl imageFormat jest formatem pikseli, takim jak PixelFormat.RGBA_8888 .

• onResolutionUpdate(Size size) ustawia rozmiar obrazu wejściowego.

• onImageFormatUpdate(int imageFormat) ustawia format obrazu obrazu wejściowego. Obecnie może to być tylko YUV_420_888 .

Procesor przechwytywania obrazu

W przypadku przechwytywania nieruchomego można zaimplementować procesor, zwracając instancję CaptureProcessorImpl przy użyciu ImageCaptureExtenderImpl.getCaptureProcessor . Procesor jest odpowiedzialny za przetworzenie listy przechwyconych obrazów YUV_420_888 i instancji TotalCaptureResult oraz zapisanie danych wyjściowych na powierzchni YUV_420_888 .

Możesz bezpiecznie założyć, że podgląd jest włączony i uruchomiony przed wysłaniem żądania przechwycenia obrazu.

Zobacz przepływ na poniższym diagramie:

Rysunek 7. Nadal przechwytuj przepływ za pomocą CaptureProcessorImpl

1. Camera2/X używa powierzchni formatu YUV_420_888 do przechwytywania obrazu w celu skonfigurowania sesji przechwytywania. Camera2/X przygotowuje CaptureProcessorImpl , wywołując:

   • CaptureProcessorImpl.onImageFormatUpdate() z YUV_420_888 .
   • CaptureProcessorImpl.onResolutionUpdate() z rozmiarem obrazu wejściowego.
   • CaptureProcessorImpl.onOutputSurface() z wyjściową powierzchnią YUV_420_888 .

2. ImageCaptureExtenderImpl.getCaptureStages zwraca listę CaptureStageImpl , gdzie każdy element jest mapowany na instancję CaptureRequest z parametrami przechwytywania, które są wysyłane przez Camera2/X. Na przykład, jeśli zwraca listę trzech instancji CaptureStageImpl , Camera2/X wysyła trzy żądania przechwytywania z odpowiednimi parametrami przechwytywania za pomocą interfejsu API captureBurst .

   Uwaga: getCaptureStages() może zwrócić więcej niż jedną instancję CaptureStageImpl tylko wtedy, gdy getCaptureProcessor() zwraca instancję.

3. Otrzymane obrazy i instancje TotalCaptureResult są pakowane razem i wysyłane do CaptureProcessorImpl w celu przetworzenia.

4. CaptureProcessorImpl zapisuje obraz wynikowy (format YUV_420_888 ) na powierzchnię wyjściową określoną przez wywołanie onOutputSurface() . Camera2/X konwertuje je w razie potrzeby na obrazy JPEG.

Obsługa kluczy żądań przechwytywania i wyników

Oprócz podglądu i robienia zdjęć z aparatu, aplikacje mogą ustawiać zoom, parametry lampy błyskowej lub uruchamiać funkcję „dotknij, aby ustawić ostrość”. Te parametry mogą nie być zgodne z implementacją Twojego rozszerzenia.

Następujące metody zostały dodane do extensions-interface 1.3.0, aby umożliwić wyeksponowanie parametrów obsługiwanych przez implementację:

• ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys() zwraca klucze żądania przechwytywania obsługiwane przez Twoją implementację.
• ImageCaptureExtenderImpl.getAvailableCaptureResultKeys() zwraca klucze wyniku przechwytywania, które są zawarte w wyniku przechwytywania.

Jeśli kamera HAL przetwarza rozszerzenie, Camera2/X pobiera wyniki przechwytywania w CameraCaptureSession.CaptureCallback . Jeśli jednak procesor jest zaimplementowany, Camera2/X pobiera wyniki przechwytywania w ProcessResultImpl , które są przekazywane do metody process() w PreviewImageProcessorImpl i CaptureProcessorImpl . Jesteś odpowiedzialny za zgłoszenie wyniku przechwytywania przez ProcessResultImpl do Camera2/X.

Zobacz definicję interfejsu CaptureProcessorImpl poniżej jako przykład. W extensions-interface w wersji 1.3.0 lub nowszej wywoływane jest drugie wywołanie process() :

Interface CaptureProcessorImpl extends ProcessorImpl {
    // invoked when extensions-interface version < 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results);
    // invoked when extensions-interface version >= 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results,
            ProcessResultImpl resultCallback, Executor executor);
}

Uwaga: Klucze CaptureRequest#JPEG_QUALITY i CaptureRequest#JPEG_ORIENTATION są zawsze obsługiwane, niezależnie od tego, czy zostały dodane do listy.

W przypadku typowych operacji aparatu, takich jak powiększanie, ustawianie ostrości przez dotknięcie, lampa błyskowa i kompensacja ekspozycji, zalecamy obsługę następujących klawiszy zarówno w przypadku żądania przechwycenia, jak i wyniku przechwycenia:

• Powiększenie:
  • CaptureRequest#CONTROL_ZOOM_RATIO
  • CaptureRequest#SCALER_CROP_REGION
• Dotknij, aby ustawić ostrość:
  • CaptureRequest#CONTROL_AF_MODE
  • CaptureRequest#CONTROL_AF_TRIGGER
  • CaptureRequest#CONTROL_AF_REGIONS
  • CaptureRequest#CONTROL_AE_REGIONS
  • CaptureRequest#CONTROL_AWB_REGIONS
• Błysk:
  • CaptureRequest#CONTROL_AE_MODE
  • CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
  • CaptureRequest#FLASH_MODE
• Kompensacja ekspozycji:
  • CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION

W przypadku Basic Extenders, które implementują wersje 1.2.0 lub wcześniejsze, interfejs CameraX Extensions API wyraźnie obsługuje wszystkie powyższe klucze. W przypadku extensions-interface 1.3.0 zarówno CameraX, jak i Camera2 honorują zwróconą listę i obsługują tylko zawarte w niej klucze. Na przykład, jeśli zdecydujesz się zwrócić tylko CaptureRequest#CONTROL_ZOOM_RATIO i CaptureRequest#SCALER_CROP_REGION w implementacji 1.3.0, oznacza to, że aplikacja obsługuje tylko powiększanie, a ustawianie ostrości przez dotknięcie, lampa błyskowa i kompensacja ekspozycji są niedozwolone.

Uwaga: Rozszerzenia Camera2 obsługują ustawianie tylko parametrów JPEG_QUALITY i JPEG_ORIENTATION w systemach Android 12 i Android 12L. Począwszy od Androida 13, interfejs Camera2 Extensions API obsługuje inne parametry, takie jak zoom lub lampa błyskowa, o ile te klucze są zwracane przez ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys w bibliotece dostawcy OEM, który implementuje extensions-interface 1.3.0 lub nowszej.

Zaawansowany przedłużacz

Advanced Extender to rodzaj implementacji dostawcy opartej na API Camera2. Ten typ Extendera został dodany w extensions-interface 1.2.0. W zależności od producenta urządzenia rozszerzenia mogą zostać zaimplementowane w warstwie aplikacji, co zależy od następujących czynników:

• Niestandardowa konfiguracja strumienia: Skonfiguruj niestandardowe strumienie, takie jak strumień RAW lub wiele strumieni dla różnych identyfikatorów fizycznych aparatów.

• Możliwość wysyłania żądań Camera2: obsługuje skomplikowaną logikę interakcji, która może wysyłać żądania przechwytywania z parametrami opartymi na wynikach poprzednich żądań.

Advanced Extender zapewnia opakowanie lub warstwę pośrednią, dzięki czemu można dostosować konfigurację strumienia i wysyłać żądania przechwytywania na żądanie.

Uwaga: API Camera2 Extensions obsługuje Advanced Extender od Androida 12L. W przypadku interfejsu CameraX Extensions API funkcja Advanced Extender jest obsługiwana od wersji CameraX 1.2.0-alpha03.

Pliki do wdrożenia

Aby przełączyć się na implementację Advanced Extender, metoda isAdvancedExtenderImplemented() w ExtensionVersionImpl musi zwrócić true . Dla każdego typu rozszerzenia producenci OEM muszą zaimplementować odpowiednie klasy rozszerzeń. Pliki implementacyjne Advanced Extender znajdują się w pakiecie zaawansowanym .

	Klasy rozszerzeń do zaimplementowania
Noc	advanced/NightAdvancedExtenderImpl.java
HDR	advanced/HdrAdvancedExtenderImpl.java
Automatyczny	advanced/AutoAdvancedExtenderImpl.java
bokeh	advanced/BokehAdvancedExtenderImpl.java
Retusz twarzy	advanced/BeautyAdvancedExtenderImpl.java

W poniższym przykładzie używamy AdvancedExtenderImpl jako symbolu zastępczego. Zastąp go nazwą pliku Extender dla implementowanego rozszerzenia.

Zobaczmy, jak Camera2/X wywołuje extensions-interface , aby osiągnąć trzy przepływy aplikacji.

Uwaga: Camera2/X zawsze inicjuje bibliotekę dostawcy ( InitializerImpl.init() ) przed uruchomieniem któregokolwiek z tych przepływów aplikacji.

Przepływ aplikacji 1: Sprawdź dostępność rozszerzeń

Rysunek 8. Przepływ aplikacji 1 w Advanced Extender

Najpierw aplikacja sprawdza, czy dane rozszerzenie jest obsługiwane.

Uwaga: aplikacja nie musi wywoływać AdvancedExtenderImpl.init() przed wywołaniem isExtensionAvailable() .

Przepływ aplikacji 2: Zapytanie o informacje

Rysunek 9. Przepływ aplikacji 2 w Advanced Extender

Po wywołaniu AdvancedExtenderImpl.init() aplikacja może zapytać o następujące informacje dotyczące AdvancedExtenderImpl :

• Szacowane opóźnienie przechwytywania nadal: AdvancedExtenderImpl.getEstimatedCaptureLatencyRange() zwraca zakres opóźnienia przechwytywania, aby aplikacja mogła ocenić, czy należy włączyć rozszerzenie dla bieżącego scenariusza.

• Obsługiwane rozdzielczości do podglądu i przechwytywania:

  • AdvancedExtenderImpl.getSupportedPreviewOutputResolutions() zwraca mapę formatu obrazu do listy rozmiarów obsługiwanych dla formatu i rozmiaru podglądu powierzchni. Producenci OEM muszą obsługiwać co najmniej format PRIVATE .

  • AdvancedExtenderImpl.getSupportedCaptureOutputResolutions() zwraca obsługiwany format i rozmiary nieruchomej powierzchni przechwytywania. Producenci OEM muszą obsługiwać wyjście w formacie JPEG i YUV_420_888 .

  • AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() zwraca obsługiwane rozmiary dla dodatkowego strumienia YUV_420_888 do analizy obrazu. Jeśli powierzchnia analizy obrazu YUV nie jest obsługiwana, getSupportedYuvAnalysisResolutions() powinna zwrócić null lub pustą listę.

• Dostępne klucze/wyniki żądań przechwytywania (dodane w extensions-interface 1.3.0): Camera2/X wywołuje następujące metody w celu pobrania obsługiwanych kluczy żądań przechwytywania i kluczy wyników z Twojej implementacji:

  • AdvancedExtenderImpl.getAvailableCaptureRequestKeys
  • AdvancedExtenderImpl.getAvailableCaptureResultKeys

Aby uzyskać więcej informacji, zobacz Klucze i wyniki żądań przechwytywania pomocy technicznej .

Przepływ aplikacji 3: podgląd/zdjęcia z włączonym rozszerzeniem

Rysunek 10. Przepływ aplikacji 3 w Advanced Extender

Powyższy diagram przedstawia główny przepływ uruchamiania podglądu i przechwytywania obrazu dla typu Advanced Extender. Przejdźmy przez każdy krok.

1. Instancja SessionProcessorImpl

   Podstawowa implementacja Advanced Extender znajduje się w SessionProcessorImpl , która jest odpowiedzialna za zapewnianie dostosowanej konfiguracji sesji i wysyłanie żądań przechwytywania w celu zainicjowania podglądu i nadal żądania przechwytywania. AdvancedExtenderImpl.createSessionProcessor() jest wywoływana w celu zwrócenia instancji SessionProcessorImpl .

   Uwaga: sesja kamery może wielokrotnie używać instancji SessionProcessorImpl . W rezultacie implementacja musi obsługiwać wiele initSession() i deInitSession() dla jednej instancji SessionProcessorImpl .

2. initSession

   SessionProcessorImpl.initSession() inicjuje sesję dla rozszerzenia. W tym miejscu przydzielasz zasoby i zwracasz konfigurację sesji w celu przygotowania CameraCaptureSession .

   W przypadku parametrów wejściowych Camera2/X określa konfiguracje powierzchni wyjściowej dla podglądu, przechwytywania obrazu nieruchomego i opcjonalnej analizy obrazu YUV. Ta konfiguracja powierzchni wyjściowej ( OutputSurfaceImpl ) zawiera powierzchnię, rozmiar i format obrazu, które są pobierane za pomocą następujących metod w AdvancedExtenderImpl :

   • getSupportedPreviewOutputResolutions()
   • getSupportedCaptureOutputResolutions()
   • getSupportedYuvAnalysisResolutions()

   Musisz zwrócić instancję Camera2SessionConfigImpl , która składa się z listy instancji Camera2OutputConfigImpl oraz parametrów sesji używanych do konfigurowania CameraCaptureSession . Jesteś odpowiedzialny za wyprowadzanie prawidłowych obrazów z kamery na powierzchnie wyjściowe przekazywane przez Camera2/X. Oto kilka opcji włączania danych wyjściowych:

   • Przetwarzanie w aparacie HAL: Możesz bezpośrednio dodać powierzchnie wyjściowe do CameraCaptureSession za pomocą implementacji SurfaceOutputConfigImpl . To konfiguruje dostarczoną powierzchnię wyjściową do potoku kamery i umożliwia HAL kamery na przetwarzanie obrazu.

   • Przetwarzanie pośredniej powierzchni ImageReader (RAW, YUV itp.): Dodaj pośrednie powierzchnie ImageReader do CameraCaptureSession za pomocą instancji ImageReaderOutputConfigImpl .

     Musisz przetworzyć obrazy pośrednie i zapisać obraz wynikowy na powierzchni wyjściowej.

   Uwaga: Camera2/X tworzy i zarządza instancjami ImageReader za Ciebie. Możesz ustawić instancję ImageProcessorImpl , aby pobrać obraz.
   • Użyj udostępniania powierzchni Camera2: Użyj udostępniania powierzchni z inną powierzchnią, dodając dowolną instancję Camera2OutputConfigImpl do metody getSurfaceSharingOutputConfigs() innej instancji Camera2OutputConfigImpl . Format i rozmiar powierzchni muszą być identyczne.

   Wszystkie Camera2OutputConfigImpl , w tym SurfaceOutputConfigImpl i ImageReaderOutputConfigImpl , muszą mieć unikalny identyfikator ( getId() ), który jest używany do określenia powierzchni docelowej i pobrania obrazu z ImageReaderOutputConfigImpl .

3. onCaptureSessionStart i RequestProcessorImpl

   Kiedy CameraCaptureSession zostaje uruchomiona, a środowisko Camera wywołuje onConfigured() , następnie Camera2/X wywołuje SessionProcessorImpl.onCaptureSessionStart() z opakowaniem żądania Camera2 RequestProcessImpl . Camera2/X implementuje RequestProcessImpl , który umożliwia wykonywanie żądań przechwytywania i pobieranie obrazów , jeśli używany jest ImageReaderOutputConfigImpl .

   Interfejsy API RequestProcessImpl są podobne do interfejsów API Camera2 CameraCaptureSession pod względem wykonywania żądań. Różnice są następujące:

   • Powierzchnia docelowa jest określona przez identyfikator instancji Camera2OutputConfigImpl .
   • Możliwość pobierania obrazu ImageReader .

   Możesz wywołać RequestProcessorImpl.setImageProcessor() z określonym identyfikatorem Camera2OutputConfigImpl , aby zarejestrować instancję ImageProcessorImpl w celu odbierania obrazów.

   Instancja RequestProcessImpl staje się nieważna po wywołaniu przez Camera2/X SessionProcessorImpl.onCaptureSessionEnd() .

4. Uruchom podgląd i zrób zdjęcie

   W implementacji Advanced Extender można wysyłać żądania przechwytywania za pośrednictwem interfejsu RequestProcessorImpl . Camera2/X powiadamia o konieczności rozpoczęcia powtarzającego się żądania podglądu lub sekwencji przechwytywania obrazu przez wywołanie odpowiednio SessionProcessorImpl#startRepeating i SessionProcessorImpl#startCapture . Powinieneś wysłać żądania przechwytywania, aby spełnić te żądania podglądu i przechwycenia.

   Camera2/X ustawia również parametry żądania przechwytywania za pomocą SessionProcessorImpl#setParameters . Należy ustawić te parametry żądania (jeśli parametry są obsługiwane) zarówno dla powtarzających się, jak i pojedynczych żądań.

   Musisz obsługiwać co najmniej CaptureRequest.JPEG_ORIENTATION i CaptureRequest.JPEG_QUALITY . extensions-interface 1.3.0 obsługuje klucze żądania i wyniku, które są udostępniane za pomocą następujących metod:

   • AdvancedExtenderImpl.getAvailableCaptureRequestKeys()
   • AdvancedExtenderImpl.getAvailableCaptureResultKeys()

   Gdy programiści ustawiają klucze na liście getAvailableCaptureRequestKeys , należy włączyć parametry i upewnić się, że wynik przechwytywania zawiera klucze z listy getAvailableCaptureResultKeys .

5. startTrigger

   SessionProcessorImpl.startTrigger() jest wywoływana w celu uruchomienia wyzwalacza, takiego jak CaptureRequest.CONTROL_AF_TRIGGER i CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER . Możesz zignorować wszystkie klucze żądania przechwytywania, które nie zostały anonsowane w AdvancedExtenderImpl.getAvailableCaptureRequestKeys() .

   startTrigger() jest obsługiwana od wersji 1.3.0 extensions-interface . Umożliwia aplikacjom wdrażanie funkcji „dotknij, aby ustawić ostrość” i flashowanie z rozszerzeniami.

6. Posprzątać

   Po zakończeniu sesji przechwytywania SessionProcessorImpl.onCaptureSessionEnd() jest wywoływana przed zamknięciem CameraCaptureSession . Po zamknięciu sesji przechwytywania deInitSession() wykonuje czyszczenie.

Obsługa podglądu, przechwytywania zdjęć i analizy obrazu

Powinieneś zastosować rozszerzenie zarówno do podglądu, jak i do przechwytywania przypadków użycia. Jeśli jednak opóźnienie jest zbyt duże, aby płynnie wyświetlić podgląd, możesz zastosować rozszerzenie tylko do przechwytywania nieruchomego obrazu.

Dla typu Basic Extender, niezależnie od włączenia rozszerzenia do podglądu, należy zaimplementować zarówno ImageCaptureExtenderImpl , jak i PreviewExtenderImpl dla danego rozszerzenia. Often, an app also uses a YUV stream to analyze the image content such as finding QR codes or text. To better support this use case , you should support the stream combination of preview, still capture, and a YUV_420_888 stream for configuring CameraCaptureSession . This means that if you implement a processor, then you have to support the stream combination of three YUV_420_888 streams.

For Advanced Extender, Camera2/X passes three output surfaces to the SessionProcessorImpl.initSession() call. These output surfaces are for preview , still capture, and image analysis, respectively. You must ensure that preview and still capture output surfaces show the valid output. However, for the image analysis output surface, ensure it's working only when it's non-null. If your implementation can't support the image analysis stream, you can return an empty list in AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() . This ensures the image analysis output surface is always null in SessionProcessorImpl.initSession() .

Note: You aren't required to apply the extension on the image analysis stream. Furthermore, the Camera2 Extensions API only allows for configuring two surfaces: preview and still capture.

Support video capture

The current Camera Extension architecture supports only the preview and still capture use cases. We don't support enabling the extension on the MediaCodec or MediaRecorder surfaces for recording the video. However, it's possible for apps to record the preview output.

Supporting MediaCodec and MediaRecorder surfaces is under investigation.

Extensions interface version history

The following table shows the Camera Extension interface version history. You should always implement the vendor library with the latest version.

Version	Added features
1.0.0	Version verification ExtensionVersionImpl Basic Extender PreviewExtenderImpl ImageCaptureExtenderImpl Processor PreviewImageProcessorImpl CaptureProcessorImpl RequestUpdateProcessorImpl
1.1.0	Library initialization InitializerImpl Expose supported resolutions PreviewExtenderImpl.getSupportedResolutions ImageCaptureExtenderImpl.getSupportedResolutions
1.2.0	AdvancedExtender AdvancedExtenderImpl SessionProcessorImpl Get estimated capture latency ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange
1.3.0	Expose supported capture request keys/results keys ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys AdvancedExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys New process() call that takes ProcessResultImpl in PreviewImageProcessorImpl and CaptureProcessorImpl Support trigger type request AdvancedExtenderImpl.startTrigger

Reference implementation

The following reference OEM vendor library implementations are available in frameworks/ex .

• advancedSample : A basic implementation of Advanced Extender.

• sample : A basic implementation of Basic Extender.

• service_based_sample : An implementation that demonstrates how to host Camera Extensions in a Service . This implementation contains the following components:

  • oem_library : A Camera Extensions OEM library for Camera2 and CameraX Extensions APIs that implements Extensions-Interface . This acts as a passthrough that forwards calls from Extensions-Interface to the service. This library also provides AIDL files and wrapper classes to communicate with the service.

    Advanced Extender is enabled by default. To enable the Basic Extender, change ExtensionsVersionImpl#isAdvancedExtenderImplemented to return false .

  • extensions_service : A sample implementation of the Extensions Service. Add your implementation here. The interface to implement in the service is similar to the Extensions-Interface . For example, implementing the IAdvancedExtenderImpl.Stub performs the same operations as AdvancedExtenderImpl . ImageWrapper and TotalCaptureResultWrapper are required to make Image and TotalCaptureResult parcelable.

Set up the vendor library on a device

The OEM vendor library isn't built into an app; it's loaded from the device at runtime by Camera2/X. In CameraX, the <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file of the camera-extensions library, is a dependency of CameraX and must be loaded at runtime. In Camera2, the framework loads an extensions service that also declares that the <uses-library> loads the same androidx.camera.extensions.impl library at runtime.

This allows third-party apps using extensions to automatically load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device. Camera2/X handles this behavior automatically when an app tries to use a camera extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ ANY_FILENAME .xml . For example, /etc/permissions/camera_extensions.xml . The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

2. Use the example below to add the required information to the file.

   • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
   • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar ).

   <?xml version="1.0" encoding="utf-8"?>
   <permissions>
       <library name="androidx.camera.extensions.impl"
                file="OEM_IMPLEMENTED_JAR" />
   </permissions>
   

In Android 12 or higher, devices supporting CameraX extensions must have the ro.camerax.extensions.enabled property set to true , which allows for querying whether a device supports extensions. To do this, add the following line in the device make file:

PRODUCT_VENDOR_PROPERTIES += \
    ro.camerax.extensions.enabled=true \

Validation

To test your implementation of the OEM vendor library during the development stage, use the example app at androidx-main/camera/integration-tests/extensionstestapp/ , which runs through various vendor extensions.

After you complete your implementation, use the Camera Extensions Validation Tool to run automated and manual tests to verify that the vendor library is implemented correctly.

Extended scene mode versus Camera Extensions

For the bokeh extension, in addition to exposing it using Camera Extensions, you can expose the extension using the extended scene mode, which is enabled through the CONTROL_EXTENDED_SCENE_MODE key. For more implementation details, see Camera Bokeh .

Extended scene mode has fewer restrictions compared to Camera Extensions for camera2 apps. For example, you can enable extended scene mode in a regular CameraCaptureSession instance that supports flexible stream combinations and capture request parameters. In contrast, camera extensions support only a fixed set of stream types and have limited support for capture request parameters.

A downside of extended scene mode is that you can only implement it in the camera HAL, which means that it must be verified to work across all orthogonal controls available to app developers.

We recommend exposing bokeh using both the extended scene mode and Camera Extensions because apps might prefer to use a particular API to enable bokeh. We recommend first using the extended scene mode because this is the most flexible way for apps to enable the bokeh extension. Then you can implement the camera extensions interface based on the extended scene mode. If implementing bokeh in the camera HAL is difficult, for example, because it requires a post processor running in the app layer to process images, we recommend implementing the bokeh extension using the Camera Extensions interface.

Frequently asked questions (FAQs)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation .

,

Device manufacturers can expose extensions such as bokeh, night mode, and HDR to third-party developers through the Camera Extensions interface provided by the OEM vendor library. Developers can use the Camera2 Extensions API and the CameraX Extensions API to access the extensions implemented in the OEM vendor library.

For a list of supported extensions, which is the same across Camera2 and CameraX, see CameraX Extensions API . If you want to add an extension, file a bug with the Issue Tracker .

This page describes how to implement and enable the OEM vendor library on devices.

Architecture

The following diagram describes the architecture of the Camera Extensions interface or extensions-interface :

Figure 1. Camera Extensions architecture diagram

As shown in the diagram, to support Camera Extensions, you need to implement the extensions-interface provided by the OEM vendor library. Your OEM vendor library enables two APIs: CameraX Extensions API and Camera2 Extensions API , which are used by CameraX and Camera2 apps, respectively, to access vendor extensions.

Implement the OEM vendor library

To implement the OEM vendor library, copy the camera-extensions-stub files into a system library project. These files define the Camera Extensions interface.

Note: You must keep the files under the androidx.camera.extensions.impl Java packages. For reference, see the sample project .

The camera-extensions-stub files are divided into the following categories:

Essential interface files (don't modify)

• PreviewExtenderImpl.java
• ImageCaptureExtenderImpl.java
• ExtenderStateListener.java
• ProcessorImpl.java
• PreviewImageProcessorImpl.java
• CaptureProcessorImpl.java
• CaptureStageImpl.java
• RequestUpdateProcessorImpl.java
• ProcessResultImpl.java
• advanced/AdvancedExtenderImpl.java
• advanced/Camera2OutputConfigImpl.java
• advanced/Camera2SessionConfigImpl.java
• advanced/ImageProcessorImpl.java
• advanced/ImageReaderOutputConfigImpl.java
• advanced/ImageReferenceImpl.java
• advanced/MultiResolutionImageReaderOutputConfigImpl.java
• advanced/OutputSurfaceImpl.java
• advanced/RequestProcessorImpl.java
• advanced/SessionProcessorImpl.java
• advanced/SurfaceOutputConfigImpl.java

Mandatory implementations (add your implementation)

• ExtensionVersionImpl.java
• InitializerImpl.java

Bokeh extender classes (implement it if Bokeh extension is supported)

• BokehImageCaptureExtenderImpl.java
• BokehPreviewExtenderImpl.java
• advanced/BokehAdvancedExtenderImpl.java

Night extender classes (implement it if Night extension is supported)

• NightImageCaptureExtenderImpl.java
• NightPreviewExtenderImpl.java
• advanced/NightAdvancedExtenderImpl.java

Auto extender classes (implement it if Auto extension is supported)

• AutoImageCaptureExtenderImpl.java
• AutoPreviewExtenderImpl.java
• advanced/AutoAdvancedExtenderImpl.java

HDR extender classes (implement it if HDR extension is supported)

• HdrImageCaptureExtenderImpl.java
• HdrPreviewExtenderImpl.java
• advanced/HdrAdvancedExtenderImpl.java

Face Retouch extender classes (implement it if Face Retouch extension is supported)

• BeautyImageCaptureExtenderImpl.java
• BeautyPreviewExtenderImpl.java
• advanced/BeautyAdvancedExtenderImpl.java

Utilities (optional, can be deleted)

• advanced/Camera2OutputConfigImplBuilder.java
• advanced/Camera2SessionConfigImplBuilder.java

You aren't required to provide an implementation for every extension. If you don't implement an extension, set isExtensionAvailable() to return false or remove the corresponding Extender classes. The Camera2 and CameraX Extensions APIs report to the app that the extension is unavailable.

Note: Because the OEM vendor library runs as part of a third-party app process, the vendor library must NOT require any special permissions (such as system permissions).

Let's walk through how the Camera2 and CameraX Extensions APIs interact with the vendor library to enable an extension. The following diagram illustrates the end-to-end flow using the Night extension as an example:

Figure 2. Night extension implementation

Note: For convenience, we refer to the Camera2 and CameraX Extensions APIs as Camera2/X .

1. Version verification:

   Camera2/X calls ExtensionVersionImpl.checkApiVersion() to ensure that the OEM-implemented extensions-interface version is compatible with Camera2/X supported versions.

2. Vendor library initialization:

   InitializerImpl has a method init() that initializes the vendor library. Camera2/X completes the initialization before accessing the Extender classes.

3. Instantiate Extender classes:

   Instantiates the Extender classes for the extension. There are two Extender types: Basic Extender and Advanced Extender. You must implement one Extender type for all Extensions. For more information, see Basic Extender versus Advanced Extender .

   Camera2/X instantiates and interacts with the Extender classes to retrieve information and enable the extension. For a given extension, Camera2/X can instantiate the Extender classes multiple times. As a result, don't do heavy-lifting initialization in the constructor or the init() call. Do the heavy lifting only when the camera session is about to start, such as when onInit() is called in Basic Extender or initSession() is called in Advanced Extender.

   For the Night extension, the following Extender classes are instantiated for the Basic Extender type:

   • NightImageCaptureExtenderImpl.java
   • NightPreviewExtenderImpl.java

   And for the Advanced Extender type:

   • NightAdvancedExtenderImpl.java

4. Check extension availability:

   Before enabling the extension, isExtensionAvailable() checks if the extension is available on the specified camera ID through the Extender instance.

5. Initialize the Extender with camera information:

   Camera2/X calls init() on the Extender instance and passes it the camera ID and CameraCharacteristics .

6. Query information:

   Invokes the Extender class to retrieve information such as supported resolutions, still capture estimated latency, and capture request keys from the Extender in preparation for enabling the extension.

7. Enable extension on the Extender:

   The Extender class provides all the interfaces needed to enable the class. It offers a mechanism to hook OEM implementation into the Camera2 pipeline such as injecting capture request parameters or enabling a post processor.

   For the Advanced Extender type, Camera2/X interacts with SessionProcessorImpl to enable the extension. Camera2/X retrieves the SessionProcessorImpl instance by calling createSessionProcessor() on the Extender.

The following sections describe the extension flow in greater detail.

Version verification

When loading the OEM vendor library from the device at runtime, Camera2/X verifies if the library is compatible with the extensions-interface version. The extensions-interface uses semantic versioning, or MAJOR.MINOR.PATCH, for example, 1.1.0 or 1.2.0. However, only the major and minor versions are used during the version verification.

To verify the version, Camera2/X calls ExtensionVersionImpl.checkApiVersion() with the supported extensions-interface version. Camera2/X then uses the version reported by the OEM library to determine if the extension can be enabled and what capabilities it should invoke.

Note: checkApiVersion() shouldn't return an extensions-interface version that's more recent than the Camera2/X supported version. For example, if Camera2/X supports 1.3.0 and the OEM vendor library supports 1.4.0, then it should return 1.3.0 instead of 1.4.0.

Major version compatibility

If the major versions of the extension-interface are different between Camera2/X and the vendor library, then it's considered incompatible and the extension is disabled.

Backward compatibility

As long as the major version is identical, Camera2/X ensures backward compatibility with OEM vendor libraries built with prior extensions-interface versions. For example, if Camera2/X supports extensions-interface 1.3.0, the OEM vendor libraries that implemented 1.0.0, 1.1.0, and 1.2.0 are still compatible. This also means that after you implement a specific version of the vendor library, Camera2/X makes sure the library is backward compatible with upcoming extension-interface versions.

Forward compatibility

Forward compatibility with vendor libraries of newer extensions-interface depends on you, the OEM. If you need some features to implement the extensions, you might want to enable the extensions starting from a certain version. In this case, you can return the supported extensions-interface version when the Camera2/X library version meets the requirements. If the Camera2/X versions aren't supported, you can return an incompatible version such as 99.0.0 to disable the extensions.

Vendor library initialization

After verifying the extensions-interface version implemented by the OEM library, Camera2/X starts the initialization process. The InitializerImpl.init() method signals to the OEM library that an app is trying to use extensions.

Camera2/X makes no other calls to the OEM library (aside from version checking) until the OEM vendor library calls OnExtensionsInitializedCallback.onSuccess() to notify the completion of initialization.

You must implement InitializerImpl as of extensions-interface 1.1.0. Camera2/X skips the library initialization step if the OEM vendor library implements extensions-interface 1.0.0.

Basic Extender versus Advanced Extender

There are two types of extensions-interface implementation: Basic Extender and Advanced Extender. Advanced Extender has been supported since extensions-interface 1.2.0.

Implement Basic Extender for extensions that process images in the camera HAL or use a post processor capable of processing YUV streams.

Implement Advanced Extender for extensions that need to customize the Camera2 stream configuration and send capture requests as needed.

See the following table for the comparison:

	Basic Extender	Advanced Extender
Stream configurations	Fixed Preview: PRIVATE or YUV_420_888 (if processor exists) Still capture: JPEG or YUV_420_888 (if processor exists)	Customizable by OEM.
Sending capture request	Only Camera2/X can send capture requests. You can set the parameters to these requests. When the processor is provided for image capture, Camera2/X can send multiple capture requests and send all the images and capture results to the processor.	A RequestProcessorImpl instance is provided to you to execute the camera2 capture request and get results and Image. Camera2/X invokes startRepeating and startCapture on SessionProcessorImpl to signal the OEM to start the repeating request for preview and start the still capture sequence respectively.
Hooks in the camera pipeline	onPresetSession provides session parameters. onEnableSession sends a single request right after CameraCaptureSession is configured. onDisableSession sends a single request before CameraCaptureSession is closed.	initSession initializes and returns a customized camera2 session configuration for creating the capture session. onCaptureSessionStart is invoked right after CameraCaptureSession is configured. onCaptureSessionEnd is invoked before CameraCaptureSession is closed.
Suitable for	Extensions implemented in the camera HAL or in a processor that processes YUV images.	Has Camera2-based implementations for the extensions. Needs customized stream configuration such as RAW stream. Needs Interactive capture sequence.
Supported API version	Camera2 Extensions: Android 13 or higher CameraX Extensions: camera-extensions 1.1.0 or higher	Camera2 Extensions: Android 12L or higher CameraX Extensions: camera-extensions 1.2.0-alpha03 or higher

Note: If ExtensionVersionImpl.isAdvancedExtenderImplemented() returns true , you must implement all extensions in Advanced Extender. If isAdvancedExtenderImplemented() returns false , you must implement all extensions in Basic Extender.

App flows

The following table shows three types of app flows and their corresponding Camera Extensions API calls. While Camera2/X provide these APIs, you must properly implement the vendor library to support these flows, which we describe in more detail in a later section.

	Camera2 extensions	CameraX extensions
Query extension availability	CameraExtensionCharacteristics . getSupportedExtensions	ExtensionsManager. isExtensionAvailable
Query information	CameraExtensionCharacteristics. getExtensionSupportedSizes CameraExtensionCharacteristics. getEstimatedCaptureLatencyRangeMillis CameraExtensionCharacteristics. getAvailableCaptureRequestKeys CameraExtensionCharacteristics. getAvailableCaptureResultKeys	ExtensionsManager. getEstimatedCaptureLatencyRange CameraX handles the rest of the information within the library.
Preview and still-capture with extension enabled	CameraDevice. createExtensionSession cameraExtensionsSession. setRepeatingRequest cameraExtensionsSession. capture	val cameraSelector = ExtensionsManager. getExtensionEnabledCameraSelector bindToLifecycle(lifecycleOwner, cameraSelector, preview, ...)

Basic Extender

The Basic Extender interface provides hooks into several places in the camera pipeline. Each extension type has corresponding Extender classes that OEMs need to implement.

The following table lists the Extender classes OEMS need to implement for each extension:

	Extender classes to implement
Night	NightPreviewExtenderImpl.java NightImageCaptureExtenderImpl.java
HDR	HdrPreviewExtenderImpl.java HdrImageCaptureExtenderImpl.java
Auto	AutoPreviewExtenderImpl.java AutoImageCaptureExtenderImpl.java
Bokeh	BokehPreviewExtenderImpl.java BokehImageCaptureExtenderImpl.java
Face retouch	BeautyPreviewExtenderImpl.java BeautyImageCaptureExtenderImpl.java

We use PreviewExtenderImpl and ImageCaptureExtenderImpl as placeholders in the following example. Replace these with the names of the actual files you're implementing.

Note: These Extender classes might be instantiated several times as needed inside Camera2/X.

Basic Extender has the following capabilities:

• Inject session parameters when configuring CameraCaptureSession ( onPresetSession ).
• Notify you of the capture session start and closing events and send a single request to notify the HAL with the returned parameters ( onEnableSession , onDisableSession ).
• Inject capture parameters for the request ( PreviewExtenderImpl.getCaptureStage , ImageCaptureExtenderImpl.getCaptureStages ).
• Add processors for preview and still capture that's capable of processing YUV_420_888 stream.

Note: You must implement both preview and still capture for the extensions that you want to enable. For example, if you want to enable the Night extension, then you must implement both NightPreviewExtenderImpl.java and NightCaptureExtenderImpl.java .

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows mentioned above.

Note: Camera2/X always initializes the vendor library ( InitializerImpl.init() ) before starting any of these app flows.

App flow 1: Check extension availability

Figure 3. App flow 1 on Basic Extender

In this flow, Camera2/X directly calls the isExtensionAvailable() method of both PreviewExtenderImpl and ImageCaptureExtenderImpl without calling init() . Both Extender classes must return true to enable the extensions.

This is often the first step for apps to check if the given extension type is supported for a given camera ID before enabling the extension. This is because some extensions are supported only on certain camera IDs.

App flow 2: Query information

Figure 4. App flow 2 on Basic Extender

After determining if the extension is available, apps should query the following information before enabling the extension.

• Still capture latency range: ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange returns the range of the capture latency for the app to evaluate if it's appropriate to enable the extension for the current scenario.

• Supported sizes for the preview and capture surface: ImageCaptureExtenderImpl.getSupportedResolutions and PreviewExtenderImpl.getSupportedResolutions return a list of image formats and the sizes that are supported for surface format and size.

• Supported request and result keys: Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

  • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys
  • ImageCaptureExtenderImpl.getAvailableCapturetResultKeys

Camera2/X always calls init() first on these Extender classes before querying for more information.

App flow 3: Preview/still capture with extension enabled (HAL implementation)

Figure 5. App flow 3 on Basic Extender

The above diagram illustrates the main flow of enabling preview and still capture with an extension without any processor. This means the camera HAL processes the extension.

In this flow, Camera2/X first calls init() then onInit , which notifies you that a camera session is about to start with the specified extensions. You can do heavy-lifting initialization in onInit() .

When configuring CameraCaptureSession , Camera2/X invokes onPresetSession to get the session parameters. After the capture session is configured successfully, Camera2/X invokes onEnableSession returning a CaptureStageImpl instance that contains the capture parameters. Camera2/X immediately sends a single request with these capture parameters to notify the HAL. Similarly, before the capture session is closed, Camera2/X invokes onDisableSession and then sends a single request with the returned capture parameters.

The repeating request triggered by Camera2/X contains the request parameters returned by PreviewExtenderImpl.getCaptureStage() . Furthermore, the still capture request contains the parameters returned by ImageCaptureExtenderImpl.getCaptureStages() .

Finally, Camera2/X invokes onDeInit() after the camera session has finished. You can release resources in onDeinit() .

Note: Camera2/X can have multiple camera sessions open and close between onInit() and onDeInit() , although this diagram shows only one camera session.

Preview processor

In addition to the camera HAL, you can also implement extensions in a processor.

Implement PreviewExtenderImpl.getProcessorType to specify the processor type as explained below:

• PROCESSOR_TYPE_NONE : No processor. Images are processed in the camera HAL.

• PROCESSOR_TYPE_REQUEST_UPDATE_ONLY : The processor type lets you update the repeating request with new capture request parameters based on the latest TotalCaptureResult .

  PreviewExtenderImpl.getProcessor must return a RequestUpdateProcessorImpl instance that processes the TotalCaptureResult instance and returns a CaptureStageImpl instance to update the repeating request. PreviewExtenderImpl.getCaptureStage() should also reflect the result of the processing and return the latest CaptureStageImpl .

• PROCESSOR_TYPE_IMAGE_PROCESSOR : This type allows you to implement a processor to process YUV_420_888 images and write the output to a PRIVATE surface.

  You need to implement and return a PreviewImageProcessorImpl instance in PreviewExtenderImpl.getProcessor . The processor is responsible for processing YUV_420_888 input images. It should write the output to the PRIVATE format of preview. Camera2/X uses a YUV_420_888 surface instead of PRIVATE to configure the CameraCaptureSession for preview.

  See following illustration for the flow:

Figure 6. Preview flow with PreviewImageProcessorImpl

The PreviewImageProcessorImpl interface extends ProcessImpl and has three important methods:

• onOutputSurface(Surface surface, int imageFormat) sets the output surface for the processor. For PreviewImageProcessorImpl , imageFormat is a pixel format such as PixelFormat.RGBA_8888 .

• onResolutionUpdate(Size size) sets the size of the input image.

• onImageFormatUpdate(int imageFormat) sets the image format of the input image. Currently, it can only be YUV_420_888 .

Image capture processor

For still capture, you can implement a processor by returning a CaptureProcessorImpl instance using ImageCaptureExtenderImpl.getCaptureProcessor . The processor is responsible to process a list of captured YUV_420_888 images and TotalCaptureResult instances and write the output to a YUV_420_888 surface.

You can safely assume that preview is enabled and running before sending the still capture request.

See the flow in the diagram below:

Figure 7. Still capture flow with CaptureProcessorImpl

1. Camera2/X uses a YUV_420_888 format surface for still capture to configure the capture session. Camera2/X prepares CaptureProcessorImpl by calling:

   • CaptureProcessorImpl.onImageFormatUpdate() with YUV_420_888 .
   • CaptureProcessorImpl.onResolutionUpdate() with the input image size.
   • CaptureProcessorImpl.onOutputSurface() with an output YUV_420_888 surface.

2. ImageCaptureExtenderImpl.getCaptureStages returns a list of CaptureStageImpl , where each element maps to a CaptureRequest instance with capture parameters that are sent by Camera2/X. For example, if it returns a list of three CaptureStageImpl instances, Camera2/X sends three capture requests with corresponding capture parameters using the captureBurst API.

   Note: getCaptureStages() is allowed to return more than one CaptureStageImpl instance only when getCaptureProcessor() returns an instance.

3. The received images and TotalCaptureResult instances are bundled together and sent to CaptureProcessorImpl for processing.

4. CaptureProcessorImpl writes the result Image ( YUV_420_888 format) to the output surface specified by the onOutputSurface() call. Camera2/X converts it into JPEG images if necessary.

Support capture request keys and results

In addition to camera preview and capture, apps can set zoom, flash parameters, or trigger a tap-to-focus. These parameters might not be compatible with your extension implementation.

The following methods have been added to extensions-interface 1.3.0 to allow you to expose the parameters that your implementation supports:

• ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys() returns the capture request keys supported by your implementation.
• ImageCaptureExtenderImpl.getAvailableCaptureResultKeys() returns the capture result keys that are contained in the capture result.

If the camera HAL processes the extension, Camera2/X retrieves the capture results in CameraCaptureSession.CaptureCallback . However, if the processor is implemented, then Camera2/X retrieves the capture results in ProcessResultImpl , which is passed to the process() method in PreviewImageProcessorImpl and CaptureProcessorImpl . You're responsible for reporting the capture result through ProcessResultImpl to Camera2/X.

See the definition of the CaptureProcessorImpl interface below as an example. In extensions-interface 1.3.0 or higher, the second process() call is invoked:

Interface CaptureProcessorImpl extends ProcessorImpl {
    // invoked when extensions-interface version < 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results);
    // invoked when extensions-interface version >= 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results,
            ProcessResultImpl resultCallback, Executor executor);
}

Note: The CaptureRequest#JPEG_QUALITY and CaptureRequest#JPEG_ORIENTATION keys are always supported regardless of whether they are added to the list.

For common camera operations like zoom, tap-to-focus, flash, and exposure compensation, we recommend supporting the following keys for both capture request and capture result:

• Zoom:
  • CaptureRequest#CONTROL_ZOOM_RATIO
  • CaptureRequest#SCALER_CROP_REGION
• Tap-to-focus:
  • CaptureRequest#CONTROL_AF_MODE
  • CaptureRequest#CONTROL_AF_TRIGGER
  • CaptureRequest#CONTROL_AF_REGIONS
  • CaptureRequest#CONTROL_AE_REGIONS
  • CaptureRequest#CONTROL_AWB_REGIONS
• Flash:
  • CaptureRequest#CONTROL_AE_MODE
  • CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
  • CaptureRequest#FLASH_MODE
• Exposure compensation:
  • CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION

For Basic Extenders that implement 1.2.0 or prior versions, the CameraX Extensions API explicitly supports all the above keys. For extensions-interface 1.3.0, both CameraX and Camera2 honor the returned list and support only the keys contained in it. For example, if you decide to return only CaptureRequest#CONTROL_ZOOM_RATIO and CaptureRequest#SCALER_CROP_REGION in the 1.3.0 implementation, then that means only zoom is supported for the app while tap-to-focus, flash, and exposure compensation aren't allowed.

Note: The Camera2 Extensions support setting only parameters JPEG_QUALITY and JPEG_ORIENTATION in Android 12 and Android 12L. Starting in Android 13, the Camera2 Extensions API supports other parameters such as zoom or flash as long as these keys are returned by the ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys in your OEM vendor library that implements extensions-interface 1.3.0 or higher.

Advanced Extender

Advanced Extender is a type of vendor implementation based on the Camera2 API. This Extender type was added in extensions-interface 1.2.0. Depending on the device manufacturer, extensions might be implemented in the app layer, which depends on the following factors:

• Custom stream configuration: Configure custom streams like RAW stream or have multiple streams for different physical camera IDs.

• Capability to send Camera2 requests: Support a complicated interaction logic that can send capture requests with parameters based on the results of previous requests.

Advanced Extender provides a wrapper, or an intermediate layer, so you can customize the stream configuration and send capture requests on demand.

Note: The Camera2 Extensions API has supported Advanced Extender since Android 12L. For the CameraX Extensions API, Advanced Extender has been supported since CameraX 1.2.0-alpha03.

Files to implement

To switch to the Advanced Extender implementation, the isAdvancedExtenderImplemented() method in ExtensionVersionImpl must return true . For each extension type, OEMs must implement the corresponding Extender classes. The Advanced Extender implementation files are in the advanced package.

	Extender classes to implement
Night	advanced/NightAdvancedExtenderImpl.java
HDR	advanced/HdrAdvancedExtenderImpl.java
Auto	advanced/AutoAdvancedExtenderImpl.java
Bokeh	advanced/BokehAdvancedExtenderImpl.java
Face Retouch	advanced/BeautyAdvancedExtenderImpl.java

We use AdvancedExtenderImpl as a placeholder in the following example. Replace it with the name of the Extender file for the extension you're implementing.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows.

Note: Camera2/X always initializes the vendor library ( InitializerImpl.init() ) before starting any of these app flows.

App flow 1: Check extensions availability

Figure 8. App flow 1 on Advanced Extender

First, the app checks if the given extension is supported.

Note: The app doesn't need to call AdvancedExtenderImpl.init() before calling isExtensionAvailable() .

App flow 2: Query information

Figure 9. App flow 2 on Advanced Extender

After calling AdvancedExtenderImpl.init() , the app can query the following the information on AdvancedExtenderImpl :

• Estimated still capture latency: AdvancedExtenderImpl.getEstimatedCaptureLatencyRange() returns the range of the capture latency for the app to evaluate if it is appropriate to enable the extension for the current scenario.

• Supported resolutions for preview and still capture:

  • AdvancedExtenderImpl.getSupportedPreviewOutputResolutions() returns a map of image format to the sizes list that are supported for preview surface format and size. OEMs must support at least the PRIVATE format.

  • AdvancedExtenderImpl.getSupportedCaptureOutputResolutions() returns the supported format and sizes for still capture surface. OEMs must support both JPEG and YUV_420_888 format output.

  • AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() returns the supported sizes for an extra YUV_420_888 stream for image analysis. If the image analysis YUV surface isn't supported, getSupportedYuvAnalysisResolutions() should return null or an empty list.

• Available capture request keys/results (added in extensions-interface 1.3.0): Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

  • AdvancedExtenderImpl.getAvailableCaptureRequestKeys
  • AdvancedExtenderImpl.getAvailableCaptureResultKeys

For more information, see Support capture request keys and results .

App flow 3: Preview/still capture with extension enabled

Figure 10. App flow 3 on Advanced Extender

The above diagram shows the main flow for starting preview and still capture for the Advanced Extender type. Let's walk through each step.

1. SessionProcessorImpl instance

   The core Advanced Extender implementation is in SessionProcessorImpl , which is responsible for providing customized session configuration and sending capture requests to initiate the preview and still capture request. AdvancedExtenderImpl.createSessionProcessor() is invoked to return the SessionProcessorImpl instance.

   Note: A camera session can use a SessionProcessorImpl instance multiple times. As a result, the implementation must be able to support multiple initSession() and deInitSession() calls for one SessionProcessorImpl instance.

2. initSession

   SessionProcessorImpl.initSession() initializes the session for the extension. This is where you allocate resources and return a session configuration for preparing a CameraCaptureSession .

   For the input parameters, Camera2/X specifies the output surface configurations for preview, still capture, and an optional YUV image analysis. This output surface configuration ( OutputSurfaceImpl ) contains the surface, size and image format that are retrieved by following methods in AdvancedExtenderImpl :

   • getSupportedPreviewOutputResolutions()
   • getSupportedCaptureOutputResolutions()
   • getSupportedYuvAnalysisResolutions()

   You must return a Camera2SessionConfigImpl instance, which consists of a list of Camera2OutputConfigImpl instances and the session parameters used for configuring CameraCaptureSession . You're responsible for outputting the correct camera images to the output surfaces passed in by Camera2/X. Here are some options to enable the output:

   • Processing in camera HAL: You can directly add the output surfaces to CameraCaptureSession with a SurfaceOutputConfigImpl implementation. This configures the supplied output surface to the camera pipeline and allows the camera HAL to process the image.

   • Processing intermediate ImageReader surface (RAW, YUV, etc): Add the intermediate ImageReader surfaces to the CameraCaptureSession with an ImageReaderOutputConfigImpl instance.

     You need to process the intermediate images and write the result image to the output surface.

   Note: Camera2/X creates and manages the ImageReader instances for you. You can set an ImageProcessorImpl instance to retrieve the image.
   • Use Camera2 surface sharing: Use surface sharing with another surface by adding any Camera2OutputConfigImpl instance to the getSurfaceSharingOutputConfigs() method of another Camera2OutputConfigImpl instance. The surface format and size must be identical.

   All Camera2OutputConfigImpl including SurfaceOutputConfigImpl and ImageReaderOutputConfigImpl must have a unique ID ( getId() ), which is used to specify the target surface and retrieve the image from ImageReaderOutputConfigImpl .

3. onCaptureSessionStart and RequestProcessorImpl

   When CameraCaptureSession starts and the Camera framework invokes onConfigured() , then Camera2/X invokes SessionProcessorImpl.onCaptureSessionStart() with the Camera2 request wrapper RequestProcessImpl . Camera2/X implements RequestProcessImpl , which enables you to execute the capture requests , and retrieve images if ImageReaderOutputConfigImpl is used.

   The RequestProcessImpl APIs are similar to the Camera2 CameraCaptureSession APIs in terms of executing requests. The differences are:

   • The target surface is specified by the ID of the Camera2OutputConfigImpl instance.
   • The capability of retrieving the image of the ImageReader .

   You can call RequestProcessorImpl.setImageProcessor() with a specified Camera2OutputConfigImpl ID to register an ImageProcessorImpl instance to receive images.

   The RequestProcessImpl instance becomes invalid after Camera2/X calls SessionProcessorImpl.onCaptureSessionEnd() .

4. Start the preview and take a picture

   In the Advanced Extender implementation, you can send capture requests through the RequestProcessorImpl interface. Camera2/X notifies you to start the repeating request for preview or the still capture sequence by calling SessionProcessorImpl#startRepeating and SessionProcessorImpl#startCapture respectively. You should send capture requests to satisfy these preview and still-capture requests.

   Camera2/X also sets the capture request parameters through SessionProcessorImpl#setParameters . You must set these request parameters (if parameters are supported) on both the repeating and single requests.

   You must support at least CaptureRequest.JPEG_ORIENTATION and CaptureRequest.JPEG_QUALITY . extensions-interface 1.3.0 supports request and result keys, which are exposed by the following methods:

   • AdvancedExtenderImpl.getAvailableCaptureRequestKeys()
   • AdvancedExtenderImpl.getAvailableCaptureResultKeys()

   When developers set the keys in the getAvailableCaptureRequestKeys list, you must enable the parameters and ensure the capture result contains the keys in the getAvailableCaptureResultKeys list.

5. startTrigger

   SessionProcessorImpl.startTrigger() is invoked to start the trigger such as CaptureRequest.CONTROL_AF_TRIGGER and CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER . You can disregard any capture request keys that weren't advertised in AdvancedExtenderImpl.getAvailableCaptureRequestKeys() .

   startTrigger() has been supported since extensions-interface 1.3.0. It enables apps to implement tap-to-focus and flash with extensions.

6. Clean up

   When finishing a capture session, SessionProcessorImpl.onCaptureSessionEnd() is invoked ahead of closing CameraCaptureSession . After the capture session has closed, deInitSession() performs the clean up.

Support preview, still capture, and image analysis

You should apply the extension for both the preview and still capture use cases. However, if the latency is too high to smoothly show the preview, you can apply the extension only for still capture.

For the Basic Extender type, regardless of enabling the extension for preview, you must implement both ImageCaptureExtenderImpl and PreviewExtenderImpl for a given extension. Often, an app also uses a YUV stream to analyze the image content such as finding QR codes or text. To better support this use case , you should support the stream combination of preview, still capture, and a YUV_420_888 stream for configuring CameraCaptureSession . This means that if you implement a processor, then you have to support the stream combination of three YUV_420_888 streams.

For Advanced Extender, Camera2/X passes three output surfaces to the SessionProcessorImpl.initSession() call. These output surfaces are for preview , still capture, and image analysis, respectively. You must ensure that preview and still capture output surfaces show the valid output. However, for the image analysis output surface, ensure it's working only when it's non-null. If your implementation can't support the image analysis stream, you can return an empty list in AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() . This ensures the image analysis output surface is always null in SessionProcessorImpl.initSession() .

Note: You aren't required to apply the extension on the image analysis stream. Furthermore, the Camera2 Extensions API only allows for configuring two surfaces: preview and still capture.

Support video capture

The current Camera Extension architecture supports only the preview and still capture use cases. We don't support enabling the extension on the MediaCodec or MediaRecorder surfaces for recording the video. However, it's possible for apps to record the preview output.

Supporting MediaCodec and MediaRecorder surfaces is under investigation.

Extensions interface version history

The following table shows the Camera Extension interface version history. You should always implement the vendor library with the latest version.

Version	Added features
1.0.0	Version verification ExtensionVersionImpl Basic Extender PreviewExtenderImpl ImageCaptureExtenderImpl Processor PreviewImageProcessorImpl CaptureProcessorImpl RequestUpdateProcessorImpl
1.1.0	Library initialization InitializerImpl Expose supported resolutions PreviewExtenderImpl.getSupportedResolutions ImageCaptureExtenderImpl.getSupportedResolutions
1.2.0	AdvancedExtender AdvancedExtenderImpl SessionProcessorImpl Get estimated capture latency ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange
1.3.0	Expose supported capture request keys/results keys ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys AdvancedExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys New process() call that takes ProcessResultImpl in PreviewImageProcessorImpl and CaptureProcessorImpl Support trigger type request AdvancedExtenderImpl.startTrigger

Reference implementation

The following reference OEM vendor library implementations are available in frameworks/ex .

• advancedSample : A basic implementation of Advanced Extender.

• sample : A basic implementation of Basic Extender.

• service_based_sample : An implementation that demonstrates how to host Camera Extensions in a Service . This implementation contains the following components:

  • oem_library : A Camera Extensions OEM library for Camera2 and CameraX Extensions APIs that implements Extensions-Interface . This acts as a passthrough that forwards calls from Extensions-Interface to the service. This library also provides AIDL files and wrapper classes to communicate with the service.

    Advanced Extender is enabled by default. To enable the Basic Extender, change ExtensionsVersionImpl#isAdvancedExtenderImplemented to return false .

  • extensions_service : A sample implementation of the Extensions Service. Add your implementation here. The interface to implement in the service is similar to the Extensions-Interface . For example, implementing the IAdvancedExtenderImpl.Stub performs the same operations as AdvancedExtenderImpl . ImageWrapper and TotalCaptureResultWrapper are required to make Image and TotalCaptureResult parcelable.

Set up the vendor library on a device

The OEM vendor library isn't built into an app; it's loaded from the device at runtime by Camera2/X. In CameraX, the <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file of the camera-extensions library, is a dependency of CameraX and must be loaded at runtime. In Camera2, the framework loads an extensions service that also declares that the <uses-library> loads the same androidx.camera.extensions.impl library at runtime.

This allows third-party apps using extensions to automatically load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device. Camera2/X handles this behavior automatically when an app tries to use a camera extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ ANY_FILENAME .xml . For example, /etc/permissions/camera_extensions.xml . The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

2. Use the example below to add the required information to the file.

   • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
   • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar ).

   <?xml version="1.0" encoding="utf-8"?>
   <permissions>
       <library name="androidx.camera.extensions.impl"
                file="OEM_IMPLEMENTED_JAR" />
   </permissions>
   

In Android 12 or higher, devices supporting CameraX extensions must have the ro.camerax.extensions.enabled property set to true , which allows for querying whether a device supports extensions. To do this, add the following line in the device make file:

PRODUCT_VENDOR_PROPERTIES += \
    ro.camerax.extensions.enabled=true \

Validation

To test your implementation of the OEM vendor library during the development stage, use the example app at androidx-main/camera/integration-tests/extensionstestapp/ , which runs through various vendor extensions.

After you complete your implementation, use the Camera Extensions Validation Tool to run automated and manual tests to verify that the vendor library is implemented correctly.

Extended scene mode versus Camera Extensions

For the bokeh extension, in addition to exposing it using Camera Extensions, you can expose the extension using the extended scene mode, which is enabled through the CONTROL_EXTENDED_SCENE_MODE key. For more implementation details, see Camera Bokeh .

Extended scene mode has fewer restrictions compared to Camera Extensions for camera2 apps. For example, you can enable extended scene mode in a regular CameraCaptureSession instance that supports flexible stream combinations and capture request parameters. In contrast, camera extensions support only a fixed set of stream types and have limited support for capture request parameters.

A downside of extended scene mode is that you can only implement it in the camera HAL, which means that it must be verified to work across all orthogonal controls available to app developers.

We recommend exposing bokeh using both the extended scene mode and Camera Extensions because apps might prefer to use a particular API to enable bokeh. We recommend first using the extended scene mode because this is the most flexible way for apps to enable the bokeh extension. Then you can implement the camera extensions interface based on the extended scene mode. If implementing bokeh in the camera HAL is difficult, for example, because it requires a post processor running in the app layer to process images, we recommend implementing the bokeh extension using the Camera Extensions interface.

Frequently asked questions (FAQs)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation .

,

Device manufacturers can expose extensions such as bokeh, night mode, and HDR to third-party developers through the Camera Extensions interface provided by the OEM vendor library. Developers can use the Camera2 Extensions API and the CameraX Extensions API to access the extensions implemented in the OEM vendor library.

For a list of supported extensions, which is the same across Camera2 and CameraX, see CameraX Extensions API . If you want to add an extension, file a bug with the Issue Tracker .

This page describes how to implement and enable the OEM vendor library on devices.

Architecture

The following diagram describes the architecture of the Camera Extensions interface or extensions-interface :

Figure 1. Camera Extensions architecture diagram

As shown in the diagram, to support Camera Extensions, you need to implement the extensions-interface provided by the OEM vendor library. Your OEM vendor library enables two APIs: CameraX Extensions API and Camera2 Extensions API , which are used by CameraX and Camera2 apps, respectively, to access vendor extensions.

Implement the OEM vendor library

To implement the OEM vendor library, copy the camera-extensions-stub files into a system library project. These files define the Camera Extensions interface.

Note: You must keep the files under the androidx.camera.extensions.impl Java packages. For reference, see the sample project .

The camera-extensions-stub files are divided into the following categories:

Essential interface files (don't modify)

• PreviewExtenderImpl.java
• ImageCaptureExtenderImpl.java
• ExtenderStateListener.java
• ProcessorImpl.java
• PreviewImageProcessorImpl.java
• CaptureProcessorImpl.java
• CaptureStageImpl.java
• RequestUpdateProcessorImpl.java
• ProcessResultImpl.java
• advanced/AdvancedExtenderImpl.java
• advanced/Camera2OutputConfigImpl.java
• advanced/Camera2SessionConfigImpl.java
• advanced/ImageProcessorImpl.java
• advanced/ImageReaderOutputConfigImpl.java
• advanced/ImageReferenceImpl.java
• advanced/MultiResolutionImageReaderOutputConfigImpl.java
• advanced/OutputSurfaceImpl.java
• advanced/RequestProcessorImpl.java
• advanced/SessionProcessorImpl.java
• advanced/SurfaceOutputConfigImpl.java

Mandatory implementations (add your implementation)

• ExtensionVersionImpl.java
• InitializerImpl.java

Bokeh extender classes (implement it if Bokeh extension is supported)

• BokehImageCaptureExtenderImpl.java
• BokehPreviewExtenderImpl.java
• advanced/BokehAdvancedExtenderImpl.java

Night extender classes (implement it if Night extension is supported)

• NightImageCaptureExtenderImpl.java
• NightPreviewExtenderImpl.java
• advanced/NightAdvancedExtenderImpl.java

Auto extender classes (implement it if Auto extension is supported)

• AutoImageCaptureExtenderImpl.java
• AutoPreviewExtenderImpl.java
• advanced/AutoAdvancedExtenderImpl.java

HDR extender classes (implement it if HDR extension is supported)

• HdrImageCaptureExtenderImpl.java
• HdrPreviewExtenderImpl.java
• advanced/HdrAdvancedExtenderImpl.java

Face Retouch extender classes (implement it if Face Retouch extension is supported)

• BeautyImageCaptureExtenderImpl.java
• BeautyPreviewExtenderImpl.java
• advanced/BeautyAdvancedExtenderImpl.java

Utilities (optional, can be deleted)

• advanced/Camera2OutputConfigImplBuilder.java
• advanced/Camera2SessionConfigImplBuilder.java

You aren't required to provide an implementation for every extension. If you don't implement an extension, set isExtensionAvailable() to return false or remove the corresponding Extender classes. The Camera2 and CameraX Extensions APIs report to the app that the extension is unavailable.

Note: Because the OEM vendor library runs as part of a third-party app process, the vendor library must NOT require any special permissions (such as system permissions).

Let's walk through how the Camera2 and CameraX Extensions APIs interact with the vendor library to enable an extension. The following diagram illustrates the end-to-end flow using the Night extension as an example:

Figure 2. Night extension implementation

Note: For convenience, we refer to the Camera2 and CameraX Extensions APIs as Camera2/X .

1. Version verification:

   Camera2/X calls ExtensionVersionImpl.checkApiVersion() to ensure that the OEM-implemented extensions-interface version is compatible with Camera2/X supported versions.

2. Vendor library initialization:

   InitializerImpl has a method init() that initializes the vendor library. Camera2/X completes the initialization before accessing the Extender classes.

3. Instantiate Extender classes:

   Instantiates the Extender classes for the extension. There are two Extender types: Basic Extender and Advanced Extender. You must implement one Extender type for all Extensions. For more information, see Basic Extender versus Advanced Extender .

   Camera2/X instantiates and interacts with the Extender classes to retrieve information and enable the extension. For a given extension, Camera2/X can instantiate the Extender classes multiple times. As a result, don't do heavy-lifting initialization in the constructor or the init() call. Do the heavy lifting only when the camera session is about to start, such as when onInit() is called in Basic Extender or initSession() is called in Advanced Extender.

   For the Night extension, the following Extender classes are instantiated for the Basic Extender type:

   • NightImageCaptureExtenderImpl.java
   • NightPreviewExtenderImpl.java

   And for the Advanced Extender type:

   • NightAdvancedExtenderImpl.java

4. Check extension availability:

   Before enabling the extension, isExtensionAvailable() checks if the extension is available on the specified camera ID through the Extender instance.

5. Initialize the Extender with camera information:

   Camera2/X calls init() on the Extender instance and passes it the camera ID and CameraCharacteristics .

6. Query information:

   Invokes the Extender class to retrieve information such as supported resolutions, still capture estimated latency, and capture request keys from the Extender in preparation for enabling the extension.

7. Enable extension on the Extender:

   The Extender class provides all the interfaces needed to enable the class. It offers a mechanism to hook OEM implementation into the Camera2 pipeline such as injecting capture request parameters or enabling a post processor.

   For the Advanced Extender type, Camera2/X interacts with SessionProcessorImpl to enable the extension. Camera2/X retrieves the SessionProcessorImpl instance by calling createSessionProcessor() on the Extender.

The following sections describe the extension flow in greater detail.

Version verification

When loading the OEM vendor library from the device at runtime, Camera2/X verifies if the library is compatible with the extensions-interface version. The extensions-interface uses semantic versioning, or MAJOR.MINOR.PATCH, for example, 1.1.0 or 1.2.0. However, only the major and minor versions are used during the version verification.

To verify the version, Camera2/X calls ExtensionVersionImpl.checkApiVersion() with the supported extensions-interface version. Camera2/X then uses the version reported by the OEM library to determine if the extension can be enabled and what capabilities it should invoke.

Note: checkApiVersion() shouldn't return an extensions-interface version that's more recent than the Camera2/X supported version. For example, if Camera2/X supports 1.3.0 and the OEM vendor library supports 1.4.0, then it should return 1.3.0 instead of 1.4.0.

Major version compatibility

If the major versions of the extension-interface are different between Camera2/X and the vendor library, then it's considered incompatible and the extension is disabled.

Backward compatibility

As long as the major version is identical, Camera2/X ensures backward compatibility with OEM vendor libraries built with prior extensions-interface versions. For example, if Camera2/X supports extensions-interface 1.3.0, the OEM vendor libraries that implemented 1.0.0, 1.1.0, and 1.2.0 are still compatible. This also means that after you implement a specific version of the vendor library, Camera2/X makes sure the library is backward compatible with upcoming extension-interface versions.

Forward compatibility

Forward compatibility with vendor libraries of newer extensions-interface depends on you, the OEM. If you need some features to implement the extensions, you might want to enable the extensions starting from a certain version. In this case, you can return the supported extensions-interface version when the Camera2/X library version meets the requirements. If the Camera2/X versions aren't supported, you can return an incompatible version such as 99.0.0 to disable the extensions.

Vendor library initialization

After verifying the extensions-interface version implemented by the OEM library, Camera2/X starts the initialization process. The InitializerImpl.init() method signals to the OEM library that an app is trying to use extensions.

Camera2/X makes no other calls to the OEM library (aside from version checking) until the OEM vendor library calls OnExtensionsInitializedCallback.onSuccess() to notify the completion of initialization.

You must implement InitializerImpl as of extensions-interface 1.1.0. Camera2/X skips the library initialization step if the OEM vendor library implements extensions-interface 1.0.0.

Basic Extender versus Advanced Extender

There are two types of extensions-interface implementation: Basic Extender and Advanced Extender. Advanced Extender has been supported since extensions-interface 1.2.0.

Implement Basic Extender for extensions that process images in the camera HAL or use a post processor capable of processing YUV streams.

Implement Advanced Extender for extensions that need to customize the Camera2 stream configuration and send capture requests as needed.

See the following table for the comparison:

	Basic Extender	Advanced Extender
Stream configurations	Fixed Preview: PRIVATE or YUV_420_888 (if processor exists) Still capture: JPEG or YUV_420_888 (if processor exists)	Customizable by OEM.
Sending capture request	Only Camera2/X can send capture requests. You can set the parameters to these requests. When the processor is provided for image capture, Camera2/X can send multiple capture requests and send all the images and capture results to the processor.	A RequestProcessorImpl instance is provided to you to execute the camera2 capture request and get results and Image. Camera2/X invokes startRepeating and startCapture on SessionProcessorImpl to signal the OEM to start the repeating request for preview and start the still capture sequence respectively.
Hooks in the camera pipeline	onPresetSession provides session parameters. onEnableSession sends a single request right after CameraCaptureSession is configured. onDisableSession sends a single request before CameraCaptureSession is closed.	initSession initializes and returns a customized camera2 session configuration for creating the capture session. onCaptureSessionStart is invoked right after CameraCaptureSession is configured. onCaptureSessionEnd is invoked before CameraCaptureSession is closed.
Suitable for	Extensions implemented in the camera HAL or in a processor that processes YUV images.	Has Camera2-based implementations for the extensions. Needs customized stream configuration such as RAW stream. Needs Interactive capture sequence.
Supported API version	Camera2 Extensions: Android 13 or higher CameraX Extensions: camera-extensions 1.1.0 or higher	Camera2 Extensions: Android 12L or higher CameraX Extensions: camera-extensions 1.2.0-alpha03 or higher

Note: If ExtensionVersionImpl.isAdvancedExtenderImplemented() returns true , you must implement all extensions in Advanced Extender. If isAdvancedExtenderImplemented() returns false , you must implement all extensions in Basic Extender.

App flows

The following table shows three types of app flows and their corresponding Camera Extensions API calls. While Camera2/X provide these APIs, you must properly implement the vendor library to support these flows, which we describe in more detail in a later section.

	Camera2 extensions	CameraX extensions
Query extension availability	CameraExtensionCharacteristics . getSupportedExtensions	ExtensionsManager. isExtensionAvailable
Query information	CameraExtensionCharacteristics. getExtensionSupportedSizes CameraExtensionCharacteristics. getEstimatedCaptureLatencyRangeMillis CameraExtensionCharacteristics. getAvailableCaptureRequestKeys CameraExtensionCharacteristics. getAvailableCaptureResultKeys	ExtensionsManager. getEstimatedCaptureLatencyRange CameraX handles the rest of the information within the library.
Preview and still-capture with extension enabled	CameraDevice. createExtensionSession cameraExtensionsSession. setRepeatingRequest cameraExtensionsSession. capture	val cameraSelector = ExtensionsManager. getExtensionEnabledCameraSelector bindToLifecycle(lifecycleOwner, cameraSelector, preview, ...)

Basic Extender

The Basic Extender interface provides hooks into several places in the camera pipeline. Each extension type has corresponding Extender classes that OEMs need to implement.

The following table lists the Extender classes OEMS need to implement for each extension:

	Extender classes to implement
Night	NightPreviewExtenderImpl.java NightImageCaptureExtenderImpl.java
HDR	HdrPreviewExtenderImpl.java HdrImageCaptureExtenderImpl.java
Auto	AutoPreviewExtenderImpl.java AutoImageCaptureExtenderImpl.java
Bokeh	BokehPreviewExtenderImpl.java BokehImageCaptureExtenderImpl.java
Face retouch	BeautyPreviewExtenderImpl.java BeautyImageCaptureExtenderImpl.java

We use PreviewExtenderImpl and ImageCaptureExtenderImpl as placeholders in the following example. Replace these with the names of the actual files you're implementing.

Note: These Extender classes might be instantiated several times as needed inside Camera2/X.

Basic Extender has the following capabilities:

• Inject session parameters when configuring CameraCaptureSession ( onPresetSession ).
• Notify you of the capture session start and closing events and send a single request to notify the HAL with the returned parameters ( onEnableSession , onDisableSession ).
• Inject capture parameters for the request ( PreviewExtenderImpl.getCaptureStage , ImageCaptureExtenderImpl.getCaptureStages ).
• Add processors for preview and still capture that's capable of processing YUV_420_888 stream.

Note: You must implement both preview and still capture for the extensions that you want to enable. For example, if you want to enable the Night extension, then you must implement both NightPreviewExtenderImpl.java and NightCaptureExtenderImpl.java .

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows mentioned above.

Note: Camera2/X always initializes the vendor library ( InitializerImpl.init() ) before starting any of these app flows.

App flow 1: Check extension availability

Figure 3. App flow 1 on Basic Extender

In this flow, Camera2/X directly calls the isExtensionAvailable() method of both PreviewExtenderImpl and ImageCaptureExtenderImpl without calling init() . Both Extender classes must return true to enable the extensions.

This is often the first step for apps to check if the given extension type is supported for a given camera ID before enabling the extension. This is because some extensions are supported only on certain camera IDs.

App flow 2: Query information

Figure 4. App flow 2 on Basic Extender

After determining if the extension is available, apps should query the following information before enabling the extension.

• Still capture latency range: ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange returns the range of the capture latency for the app to evaluate if it's appropriate to enable the extension for the current scenario.

• Supported sizes for the preview and capture surface: ImageCaptureExtenderImpl.getSupportedResolutions and PreviewExtenderImpl.getSupportedResolutions return a list of image formats and the sizes that are supported for surface format and size.

• Supported request and result keys: Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

  • ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys
  • ImageCaptureExtenderImpl.getAvailableCapturetResultKeys

Camera2/X always calls init() first on these Extender classes before querying for more information.

App flow 3: Preview/still capture with extension enabled (HAL implementation)

Figure 5. App flow 3 on Basic Extender

The above diagram illustrates the main flow of enabling preview and still capture with an extension without any processor. This means the camera HAL processes the extension.

In this flow, Camera2/X first calls init() then onInit , which notifies you that a camera session is about to start with the specified extensions. You can do heavy-lifting initialization in onInit() .

When configuring CameraCaptureSession , Camera2/X invokes onPresetSession to get the session parameters. After the capture session is configured successfully, Camera2/X invokes onEnableSession returning a CaptureStageImpl instance that contains the capture parameters. Camera2/X immediately sends a single request with these capture parameters to notify the HAL. Similarly, before the capture session is closed, Camera2/X invokes onDisableSession and then sends a single request with the returned capture parameters.

The repeating request triggered by Camera2/X contains the request parameters returned by PreviewExtenderImpl.getCaptureStage() . Furthermore, the still capture request contains the parameters returned by ImageCaptureExtenderImpl.getCaptureStages() .

Finally, Camera2/X invokes onDeInit() after the camera session has finished. You can release resources in onDeinit() .

Note: Camera2/X can have multiple camera sessions open and close between onInit() and onDeInit() , although this diagram shows only one camera session.

Preview processor

In addition to the camera HAL, you can also implement extensions in a processor.

Implement PreviewExtenderImpl.getProcessorType to specify the processor type as explained below:

• PROCESSOR_TYPE_NONE : No processor. Images are processed in the camera HAL.

• PROCESSOR_TYPE_REQUEST_UPDATE_ONLY : The processor type lets you update the repeating request with new capture request parameters based on the latest TotalCaptureResult .

  PreviewExtenderImpl.getProcessor must return a RequestUpdateProcessorImpl instance that processes the TotalCaptureResult instance and returns a CaptureStageImpl instance to update the repeating request. PreviewExtenderImpl.getCaptureStage() should also reflect the result of the processing and return the latest CaptureStageImpl .

• PROCESSOR_TYPE_IMAGE_PROCESSOR : This type allows you to implement a processor to process YUV_420_888 images and write the output to a PRIVATE surface.

  You need to implement and return a PreviewImageProcessorImpl instance in PreviewExtenderImpl.getProcessor . The processor is responsible for processing YUV_420_888 input images. It should write the output to the PRIVATE format of preview. Camera2/X uses a YUV_420_888 surface instead of PRIVATE to configure the CameraCaptureSession for preview.

  See following illustration for the flow:

Figure 6. Preview flow with PreviewImageProcessorImpl

The PreviewImageProcessorImpl interface extends ProcessImpl and has three important methods:

• onOutputSurface(Surface surface, int imageFormat) sets the output surface for the processor. For PreviewImageProcessorImpl , imageFormat is a pixel format such as PixelFormat.RGBA_8888 .

• onResolutionUpdate(Size size) sets the size of the input image.

• onImageFormatUpdate(int imageFormat) sets the image format of the input image. Currently, it can only be YUV_420_888 .

Image capture processor

For still capture, you can implement a processor by returning a CaptureProcessorImpl instance using ImageCaptureExtenderImpl.getCaptureProcessor . The processor is responsible to process a list of captured YUV_420_888 images and TotalCaptureResult instances and write the output to a YUV_420_888 surface.

You can safely assume that preview is enabled and running before sending the still capture request.

See the flow in the diagram below:

Figure 7. Still capture flow with CaptureProcessorImpl

1. Camera2/X uses a YUV_420_888 format surface for still capture to configure the capture session. Camera2/X prepares CaptureProcessorImpl by calling:

   • CaptureProcessorImpl.onImageFormatUpdate() with YUV_420_888 .
   • CaptureProcessorImpl.onResolutionUpdate() with the input image size.
   • CaptureProcessorImpl.onOutputSurface() with an output YUV_420_888 surface.

2. ImageCaptureExtenderImpl.getCaptureStages returns a list of CaptureStageImpl , where each element maps to a CaptureRequest instance with capture parameters that are sent by Camera2/X. For example, if it returns a list of three CaptureStageImpl instances, Camera2/X sends three capture requests with corresponding capture parameters using the captureBurst API.

   Note: getCaptureStages() is allowed to return more than one CaptureStageImpl instance only when getCaptureProcessor() returns an instance.

3. The received images and TotalCaptureResult instances are bundled together and sent to CaptureProcessorImpl for processing.

4. CaptureProcessorImpl writes the result Image ( YUV_420_888 format) to the output surface specified by the onOutputSurface() call. Camera2/X converts it into JPEG images if necessary.

Support capture request keys and results

In addition to camera preview and capture, apps can set zoom, flash parameters, or trigger a tap-to-focus. These parameters might not be compatible with your extension implementation.

The following methods have been added to extensions-interface 1.3.0 to allow you to expose the parameters that your implementation supports:

• ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys() returns the capture request keys supported by your implementation.
• ImageCaptureExtenderImpl.getAvailableCaptureResultKeys() returns the capture result keys that are contained in the capture result.

If the camera HAL processes the extension, Camera2/X retrieves the capture results in CameraCaptureSession.CaptureCallback . However, if the processor is implemented, then Camera2/X retrieves the capture results in ProcessResultImpl , which is passed to the process() method in PreviewImageProcessorImpl and CaptureProcessorImpl . You're responsible for reporting the capture result through ProcessResultImpl to Camera2/X.

See the definition of the CaptureProcessorImpl interface below as an example. In extensions-interface 1.3.0 or higher, the second process() call is invoked:

Interface CaptureProcessorImpl extends ProcessorImpl {
    // invoked when extensions-interface version < 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results);
    // invoked when extensions-interface version >= 1.3.0
    void process(Map<Integer, Pair<Image, TotalCaptureResult>> results,
            ProcessResultImpl resultCallback, Executor executor);
}

Note: The CaptureRequest#JPEG_QUALITY and CaptureRequest#JPEG_ORIENTATION keys are always supported regardless of whether they are added to the list.

For common camera operations like zoom, tap-to-focus, flash, and exposure compensation, we recommend supporting the following keys for both capture request and capture result:

• Zoom:
  • CaptureRequest#CONTROL_ZOOM_RATIO
  • CaptureRequest#SCALER_CROP_REGION
• Tap-to-focus:
  • CaptureRequest#CONTROL_AF_MODE
  • CaptureRequest#CONTROL_AF_TRIGGER
  • CaptureRequest#CONTROL_AF_REGIONS
  • CaptureRequest#CONTROL_AE_REGIONS
  • CaptureRequest#CONTROL_AWB_REGIONS
• Flash:
  • CaptureRequest#CONTROL_AE_MODE
  • CaptureRequest#CONTROL_AE_PRECAPTURE_TRIGGER
  • CaptureRequest#FLASH_MODE
• Exposure compensation:
  • CaptureRequest#CONTROL_AE_EXPOSURE_COMPENSATION

For Basic Extenders that implement 1.2.0 or prior versions, the CameraX Extensions API explicitly supports all the above keys. For extensions-interface 1.3.0, both CameraX and Camera2 honor the returned list and support only the keys contained in it. For example, if you decide to return only CaptureRequest#CONTROL_ZOOM_RATIO and CaptureRequest#SCALER_CROP_REGION in the 1.3.0 implementation, then that means only zoom is supported for the app while tap-to-focus, flash, and exposure compensation aren't allowed.

Note: The Camera2 Extensions support setting only parameters JPEG_QUALITY and JPEG_ORIENTATION in Android 12 and Android 12L. Starting in Android 13, the Camera2 Extensions API supports other parameters such as zoom or flash as long as these keys are returned by the ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys in your OEM vendor library that implements extensions-interface 1.3.0 or higher.

Advanced Extender

Advanced Extender is a type of vendor implementation based on the Camera2 API. This Extender type was added in extensions-interface 1.2.0. Depending on the device manufacturer, extensions might be implemented in the app layer, which depends on the following factors:

• Custom stream configuration: Configure custom streams like RAW stream or have multiple streams for different physical camera IDs.

• Capability to send Camera2 requests: Support a complicated interaction logic that can send capture requests with parameters based on the results of previous requests.

Advanced Extender provides a wrapper, or an intermediate layer, so you can customize the stream configuration and send capture requests on demand.

Note: The Camera2 Extensions API has supported Advanced Extender since Android 12L. For the CameraX Extensions API, Advanced Extender has been supported since CameraX 1.2.0-alpha03.

Files to implement

To switch to the Advanced Extender implementation, the isAdvancedExtenderImplemented() method in ExtensionVersionImpl must return true . For each extension type, OEMs must implement the corresponding Extender classes. The Advanced Extender implementation files are in the advanced package.

	Extender classes to implement
Night	advanced/NightAdvancedExtenderImpl.java
HDR	advanced/HdrAdvancedExtenderImpl.java
Auto	advanced/AutoAdvancedExtenderImpl.java
Bokeh	advanced/BokehAdvancedExtenderImpl.java
Face Retouch	advanced/BeautyAdvancedExtenderImpl.java

We use AdvancedExtenderImpl as a placeholder in the following example. Replace it with the name of the Extender file for the extension you're implementing.

Let's see how Camera2/X invokes the extensions-interface to achieve the three app flows.

Note: Camera2/X always initializes the vendor library ( InitializerImpl.init() ) before starting any of these app flows.

App flow 1: Check extensions availability

Figure 8. App flow 1 on Advanced Extender

First, the app checks if the given extension is supported.

Note: The app doesn't need to call AdvancedExtenderImpl.init() before calling isExtensionAvailable() .

App flow 2: Query information

Figure 9. App flow 2 on Advanced Extender

After calling AdvancedExtenderImpl.init() , the app can query the following the information on AdvancedExtenderImpl :

• Estimated still capture latency: AdvancedExtenderImpl.getEstimatedCaptureLatencyRange() returns the range of the capture latency for the app to evaluate if it is appropriate to enable the extension for the current scenario.

• Supported resolutions for preview and still capture:

  • AdvancedExtenderImpl.getSupportedPreviewOutputResolutions() returns a map of image format to the sizes list that are supported for preview surface format and size. OEMs must support at least the PRIVATE format.

  • AdvancedExtenderImpl.getSupportedCaptureOutputResolutions() returns the supported format and sizes for still capture surface. OEMs must support both JPEG and YUV_420_888 format output.

  • AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() returns the supported sizes for an extra YUV_420_888 stream for image analysis. If the image analysis YUV surface isn't supported, getSupportedYuvAnalysisResolutions() should return null or an empty list.

• Available capture request keys/results (added in extensions-interface 1.3.0): Camera2/X invokes the following methods to retrieve the supported capture request keys and result keys from your implementation:

  • AdvancedExtenderImpl.getAvailableCaptureRequestKeys
  • AdvancedExtenderImpl.getAvailableCaptureResultKeys

For more information, see Support capture request keys and results .

App flow 3: Preview/still capture with extension enabled

Figure 10. App flow 3 on Advanced Extender

The above diagram shows the main flow for starting preview and still capture for the Advanced Extender type. Let's walk through each step.

1. SessionProcessorImpl instance

   The core Advanced Extender implementation is in SessionProcessorImpl , which is responsible for providing customized session configuration and sending capture requests to initiate the preview and still capture request. AdvancedExtenderImpl.createSessionProcessor() is invoked to return the SessionProcessorImpl instance.

   Note: A camera session can use a SessionProcessorImpl instance multiple times. As a result, the implementation must be able to support multiple initSession() and deInitSession() calls for one SessionProcessorImpl instance.

2. initSession

   SessionProcessorImpl.initSession() initializes the session for the extension. This is where you allocate resources and return a session configuration for preparing a CameraCaptureSession .

   For the input parameters, Camera2/X specifies the output surface configurations for preview, still capture, and an optional YUV image analysis. This output surface configuration ( OutputSurfaceImpl ) contains the surface, size and image format that are retrieved by following methods in AdvancedExtenderImpl :

   • getSupportedPreviewOutputResolutions()
   • getSupportedCaptureOutputResolutions()
   • getSupportedYuvAnalysisResolutions()

   You must return a Camera2SessionConfigImpl instance, which consists of a list of Camera2OutputConfigImpl instances and the session parameters used for configuring CameraCaptureSession . You're responsible for outputting the correct camera images to the output surfaces passed in by Camera2/X. Here are some options to enable the output:

   • Processing in camera HAL: You can directly add the output surfaces to CameraCaptureSession with a SurfaceOutputConfigImpl implementation. This configures the supplied output surface to the camera pipeline and allows the camera HAL to process the image.

   • Processing intermediate ImageReader surface (RAW, YUV, etc): Add the intermediate ImageReader surfaces to the CameraCaptureSession with an ImageReaderOutputConfigImpl instance.

     You need to process the intermediate images and write the result image to the output surface.

   Note: Camera2/X creates and manages the ImageReader instances for you. You can set an ImageProcessorImpl instance to retrieve the image.
   • Use Camera2 surface sharing: Use surface sharing with another surface by adding any Camera2OutputConfigImpl instance to the getSurfaceSharingOutputConfigs() method of another Camera2OutputConfigImpl instance. The surface format and size must be identical.

   All Camera2OutputConfigImpl including SurfaceOutputConfigImpl and ImageReaderOutputConfigImpl must have a unique ID ( getId() ), which is used to specify the target surface and retrieve the image from ImageReaderOutputConfigImpl .

3. onCaptureSessionStart and RequestProcessorImpl

   When CameraCaptureSession starts and the Camera framework invokes onConfigured() , then Camera2/X invokes SessionProcessorImpl.onCaptureSessionStart() with the Camera2 request wrapper RequestProcessImpl . Camera2/X implements RequestProcessImpl , which enables you to execute the capture requests , and retrieve images if ImageReaderOutputConfigImpl is used.

   The RequestProcessImpl APIs are similar to the Camera2 CameraCaptureSession APIs in terms of executing requests. The differences are:

   • The target surface is specified by the ID of the Camera2OutputConfigImpl instance.
   • The capability of retrieving the image of the ImageReader .

   You can call RequestProcessorImpl.setImageProcessor() with a specified Camera2OutputConfigImpl ID to register an ImageProcessorImpl instance to receive images.

   The RequestProcessImpl instance becomes invalid after Camera2/X calls SessionProcessorImpl.onCaptureSessionEnd() .

4. Start the preview and take a picture

   In the Advanced Extender implementation, you can send capture requests through the RequestProcessorImpl interface. Camera2/X notifies you to start the repeating request for preview or the still capture sequence by calling SessionProcessorImpl#startRepeating and SessionProcessorImpl#startCapture respectively. You should send capture requests to satisfy these preview and still-capture requests.

   Camera2/X also sets the capture request parameters through SessionProcessorImpl#setParameters . You must set these request parameters (if parameters are supported) on both the repeating and single requests.

   You must support at least CaptureRequest.JPEG_ORIENTATION and CaptureRequest.JPEG_QUALITY . extensions-interface 1.3.0 supports request and result keys, which are exposed by the following methods:

   • AdvancedExtenderImpl.getAvailableCaptureRequestKeys()
   • AdvancedExtenderImpl.getAvailableCaptureResultKeys()

   When developers set the keys in the getAvailableCaptureRequestKeys list, you must enable the parameters and ensure the capture result contains the keys in the getAvailableCaptureResultKeys list.

5. startTrigger

   SessionProcessorImpl.startTrigger() is invoked to start the trigger such as CaptureRequest.CONTROL_AF_TRIGGER and CaptureRequest.CONTROL_AE_PRECAPTURE_TRIGGER . You can disregard any capture request keys that weren't advertised in AdvancedExtenderImpl.getAvailableCaptureRequestKeys() .

   startTrigger() has been supported since extensions-interface 1.3.0. It enables apps to implement tap-to-focus and flash with extensions.

6. Clean up

   When finishing a capture session, SessionProcessorImpl.onCaptureSessionEnd() is invoked ahead of closing CameraCaptureSession . After the capture session has closed, deInitSession() performs the clean up.

Support preview, still capture, and image analysis

You should apply the extension for both the preview and still capture use cases. However, if the latency is too high to smoothly show the preview, you can apply the extension only for still capture.

For the Basic Extender type, regardless of enabling the extension for preview, you must implement both ImageCaptureExtenderImpl and PreviewExtenderImpl for a given extension. Often, an app also uses a YUV stream to analyze the image content such as finding QR codes or text. To better support this use case , you should support the stream combination of preview, still capture, and a YUV_420_888 stream for configuring CameraCaptureSession . This means that if you implement a processor, then you have to support the stream combination of three YUV_420_888 streams.

For Advanced Extender, Camera2/X passes three output surfaces to the SessionProcessorImpl.initSession() call. These output surfaces are for preview , still capture, and image analysis, respectively. You must ensure that preview and still capture output surfaces show the valid output. However, for the image analysis output surface, ensure it's working only when it's non-null. If your implementation can't support the image analysis stream, you can return an empty list in AdvancedExtenderImpl.getSupportedYuvAnalysisResolutions() . This ensures the image analysis output surface is always null in SessionProcessorImpl.initSession() .

Note: You aren't required to apply the extension on the image analysis stream. Furthermore, the Camera2 Extensions API only allows for configuring two surfaces: preview and still capture.

Support video capture

The current Camera Extension architecture supports only the preview and still capture use cases. We don't support enabling the extension on the MediaCodec or MediaRecorder surfaces for recording the video. However, it's possible for apps to record the preview output.

Supporting MediaCodec and MediaRecorder surfaces is under investigation.

Extensions interface version history

The following table shows the Camera Extension interface version history. You should always implement the vendor library with the latest version.

Version	Added features
1.0.0	Version verification ExtensionVersionImpl Basic Extender PreviewExtenderImpl ImageCaptureExtenderImpl Processor PreviewImageProcessorImpl CaptureProcessorImpl RequestUpdateProcessorImpl
1.1.0	Library initialization InitializerImpl Expose supported resolutions PreviewExtenderImpl.getSupportedResolutions ImageCaptureExtenderImpl.getSupportedResolutions
1.2.0	AdvancedExtender AdvancedExtenderImpl SessionProcessorImpl Get estimated capture latency ImageCaptureExtenderImpl.getEstimatedCaptureLatencyRange
1.3.0	Expose supported capture request keys/results keys ImageCaptureExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys AdvancedExtenderImpl.getAvailableCaptureRequestKeys and getAvailableCaptureResultKeys New process() call that takes ProcessResultImpl in PreviewImageProcessorImpl and CaptureProcessorImpl Support trigger type request AdvancedExtenderImpl.startTrigger

Reference implementation

The following reference OEM vendor library implementations are available in frameworks/ex .

• advancedSample : A basic implementation of Advanced Extender.

• sample : A basic implementation of Basic Extender.

• service_based_sample : An implementation that demonstrates how to host Camera Extensions in a Service . This implementation contains the following components:

  • oem_library : A Camera Extensions OEM library for Camera2 and CameraX Extensions APIs that implements Extensions-Interface . This acts as a passthrough that forwards calls from Extensions-Interface to the service. This library also provides AIDL files and wrapper classes to communicate with the service.

    Advanced Extender is enabled by default. To enable the Basic Extender, change ExtensionsVersionImpl#isAdvancedExtenderImplemented to return false .

  • extensions_service : A sample implementation of the Extensions Service. Add your implementation here. The interface to implement in the service is similar to the Extensions-Interface . For example, implementing the IAdvancedExtenderImpl.Stub performs the same operations as AdvancedExtenderImpl . ImageWrapper and TotalCaptureResultWrapper are required to make Image and TotalCaptureResult parcelable.

Set up the vendor library on a device

The OEM vendor library isn't built into an app; it's loaded from the device at runtime by Camera2/X. In CameraX, the <uses-library> tag declares that the androidx.camera.extensions.impl library, which is defined in the AndroidManifest.xml file of the camera-extensions library, is a dependency of CameraX and must be loaded at runtime. In Camera2, the framework loads an extensions service that also declares that the <uses-library> loads the same androidx.camera.extensions.impl library at runtime.

This allows third-party apps using extensions to automatically load the OEM vendor library. The OEM library is marked as optional so apps can run on devices that don't have the library on the device. Camera2/X handles this behavior automatically when an app tries to use a camera extension as long as the device manufacturer places the OEM library on the device so that it can be discovered by the app.

To set up the OEM library on a device, do the following:

1. Add a permission file, which is required by the <uses-library> tag, using the following format: /etc/permissions/ ANY_FILENAME .xml . For example, /etc/permissions/camera_extensions.xml . The files in this directory provide a mapping of the library named in <uses-library> to the actual file path on the device.

2. Use the example below to add the required information to the file.

   • name must be androidx.camera.extensions.impl as that's the library that CameraX searches for.
   • file is the absolute path of the file that contains the extensions implementation (for example, /system/framework/androidx.camera.extensions.impl.jar ).

   <?xml version="1.0" encoding="utf-8"?>
   <permissions>
       <library name="androidx.camera.extensions.impl"
                file="OEM_IMPLEMENTED_JAR" />
   </permissions>
   

In Android 12 or higher, devices supporting CameraX extensions must have the ro.camerax.extensions.enabled property set to true , which allows for querying whether a device supports extensions. To do this, add the following line in the device make file:

PRODUCT_VENDOR_PROPERTIES += \
    ro.camerax.extensions.enabled=true \

Validation

To test your implementation of the OEM vendor library during the development stage, use the example app at androidx-main/camera/integration-tests/extensionstestapp/ , which runs through various vendor extensions.

After you complete your implementation, use the Camera Extensions Validation Tool to run automated and manual tests to verify that the vendor library is implemented correctly.

Extended scene mode versus Camera Extensions

For the bokeh extension, in addition to exposing it using Camera Extensions, you can expose the extension using the extended scene mode, which is enabled through the CONTROL_EXTENDED_SCENE_MODE key. For more implementation details, see Camera Bokeh .

Extended scene mode has fewer restrictions compared to Camera Extensions for camera2 apps. For example, you can enable extended scene mode in a regular CameraCaptureSession instance that supports flexible stream combinations and capture request parameters. In contrast, camera extensions support only a fixed set of stream types and have limited support for capture request parameters.

A downside of extended scene mode is that you can only implement it in the camera HAL, which means that it must be verified to work across all orthogonal controls available to app developers.

We recommend exposing bokeh using both the extended scene mode and Camera Extensions because apps might prefer to use a particular API to enable bokeh. We recommend first using the extended scene mode because this is the most flexible way for apps to enable the bokeh extension. Then you can implement the camera extensions interface based on the extended scene mode. If implementing bokeh in the camera HAL is difficult, for example, because it requires a post processor running in the app layer to process images, we recommend implementing the bokeh extension using the Camera Extensions interface.

Frequently asked questions (FAQs)

Are there any restrictions on API levels?

Yes. This depends on the Android API feature set that's required by the OEM vendor library implementation. For example, ExtenderStateListener.onPresetSession() uses the SessionConfiguration.setSessionParameters() call to set a baseline set of tags. This call is available only on API level 28 and higher. For details on specific interface methods, see the API reference documentation .