Monitor Flash Memory Usage

Watchdog monitors flash memory usage by tracking the total amount of disk I/O writes made by all applications and services using the per-UID disk I/O stats exposed by the Kernel at the location `/proc/uid_io/stats`. When an application or service exceeds the disk I/O overuse threshold, Watchdog takes actions on the application or service. The disk I/O overuse thresholds and the action to take on overuse is pre-defined in the disk I/O overuse configuration.

Overuse thresholds

  • The disk I/O overuse thresholds are enforced on a daily basis i.e., All writes made by an application/service are aggregated since the beginning of the current UTC calendar day and checked against the thresholds defined in the overuse configurations.
  • When a vehicle is started multiple times on a given day, the Watchdog module stores the disk I/O usage stats on flash memory and aggregates them since the beginning of the current UTC calendar day.

Overuse actions

When an application repeatedly exceeds the defined disk I/O overuse thresholds, Watchdog takes actions defined in the overuse configuration.

  • All vendor applications and services are considered critical for the overall system stability, so they are not terminated on disk I/O overuse. However, the overuse configuration can define a list of safe-to-terminate vendor applications and services.
  • All third-party applications are safe-to-terminate.

When an application or service is safe-to-terminate, Watchdog disables the application or service with the application component state PackageManager.COMPONENT_ENABLED_STATE_DISABLED_UNTIL_USED.

Overuse configuration

The overuse configuration contains the disk I/O overuse thresholds and actions. Default overuse configurations are defined in the system and vendor images, and shipped with the build. Vendors may optionally include the vendor configuration in the vendor image. When the vendor configuration is not provided, the system configuration is used for the vendor applications and services as well.

Watchdog exposes system APIs via CarWatchdogManager, that allows vendors applications or services to update the vendor configuration anytime.

Overuse configuration definition

Overuse configuration is split by the component type. For example, System, Vendor, and Third-Party. OEMs must update only the vendor component configuration.

Vendor configuration

Vendor configuration defines the disk I/O overuse thresholds and actions for all vendor applications and services, and all maps and media applications. The configuration contains the below configuration fields.

  • Vendor package prefixes. All packages installed in the vendor partition are considered as vendor packages. In addition to these packages, vendors may classify pre-installed packages as vendor packages by adding the package prefixes to the vendor package prefixes config. This config doesn't accept regular expressions.
  • Safe-to-terminate packages. Vendors may specify which vendor packages are safe to be terminated by adding the complete package names to the safe-to-terminate packages config.
  • Application category mappings. Vendors may map any package (including third-party packages) to one of the two supported application categories - Map and Media applications. This mapping is done to provide maps and media applications higher disk I/O overuse thresholds because these applications tend to download and write more data to disk than other application types.
  • Component level thresholds. Defines generic thresholds for all vendor packages (i.e., packages not covered by Package specific thresholds or Application category specific thresholds get these thresholds). Vendors must define non-zero component level thresholds when defining disk I/O overuse configuration.
  • Package specific thresholds. Vendors may define special thresholds for specific vendor packages. The mappings should contain the complete package names. The thresholds defined in this config takes precedence over thresholds defined in other configs for a given package.
  • Application category specific thresholds. Vendors may specify special thresholds for specific application categories. The application categories must be one of the supported categories - Maps and Media applications. The thresholds defined in this config are mapped to specific packages using the Application category mappings.
  • System-wide thresholds. Vendors must not specify this config.

Vendor package prefixes, Safe-to-terminate packages, Component level thresholds, and Package specific thresholds configs are updatable only by the vendor configuration for vendor applications and services. Application category specific thresholds config can be updated only by the vendor configuration for all maps and media applications.

The overuse thresholds contain the amount of bytes allowed to be written during

  • An application/service foreground mode vs background mode
  • and system garage mode.

This classification allows user facing foreground applications/services to write more data than background applications/services. In Garage mode, applications and services tend to download updates, so each needs a higher threshold than applications and services running in other modes.

System and third-party configurations

OEMs should not update the system and third-party configurations.

  • System configuration defines I/O overuse thresholds and actions for system applications and services.
    • This configuration may also update the Application category mappings. Thus, this config field is shared between System and Vendor configurations.
  • Third-party configuration defines thresholds for all third-party applications. All applications that are not pre-installed in the system are third-party applications.
    • All third-party applications receive the same thresholds (for example, no third-party application receives special thresholds) except maps and media applications, whose thresholds are defined by the vendor configuration.
    • The below disk I/O overuse thresholds are the default thresholds for the third-party applications. These thresholds are shipped with the system image.
      • 3GiB write in the application foreground mode.
      • 2GiB write in the application background mode.
      • 4GiB write in the system garage mode.
    • The aforementioned thresholds are base thresholds. These thresholds will be updated as we understand disk I/O usage better.

Overuse configuration XML format

Default vendor configuration may be placed (this is optional) at the location /vendor/etc/automotive/watchdog/resource_overuse_configuration.xml in the build image. When this configuration is not specified, the System defined configuration will be applied for vendor applications and services as well.

The XML file should contain only one tag for each config field. I/O overuse configuration must be defined in the XML file. All threshold values should be specified in the MiB unit.

A sample XML configuration is provided below:

<resourceOveruseConfiguration version="1.0">
      <componentType> VENDOR </componentType>

      <!-- List of safe to kill vendor packages. -->
      <safeToKillPackages>
            <package> com.vendor.package.A </package>
            <package> com.vendor.package.B </package>
      </safeToKillPackages>

      <!-- List of vendor package prefixes. -->
      <vendorPackagePrefixes>
            <packagePrefix> com.vendor.package </packagePrefix>
      </vendorPackagePrefixes>

      <!-- List of unique package names to app category mappings. -->
      <packagesToAppCategoryTypes>
            <packageAppCategory type="MEDIA"> com.vendor.package.A </packageAppCategory>
            <packageAppCategory type="MAPS"> com.google.package.B </packageAppCategory>
            <packageAppCategory type="MEDIA"> com.third.party.package.C </packageAppCategory>
      </packagesToAppCategoryTypes>

      <ioOveruseConfiguration>
        <!-- Thresholds in MiB for all vendor packages that don't have package specific thresholds. -->
            <componentLevelThresholds>
                  <state id="foreground_mode"> 1024 </state>
                  <state id="background_mode"> 512 </state>
                  <state id="garage_mode"> 3072 </state>
            </componentLevelThresholds>

            <packageSpecificThresholds>
                  <!-- IDs must be unique -->
                  <perStateThreshold id="com.vendor.package.C">
                    <state id="foreground_mode"> 400 </state>
                    <state id="background_mode"> 100 </state>
                    <state id="garage_mode"> 200 </state>
                  </perStateThreshold>

                  <perStateThreshold id="com.vendor.package.D">
                    <state id="foreground_mode"> 1024 </state>
                    <state id="background_mode"> 500 </state>
                    <state id="garage_mode"> 2048 </state>
                  </perStateThreshold>
            </packageSpecificThresholds>

            <!-- Application category specific thresholds. -->
            <appCategorySpecificThresholds>
                  <!-- One entry per supported application category -->
                  <perStateThreshold id="MEDIA">
                    <state id="foreground_mode"> 600 </state>
                    <state id="background_mode"> 700 </state>
                    <state id="garage_mode"> 1024 </state>
                  </perStateThreshold>

                  <perStateThreshold id="MAPS">
                    <state id="foreground_mode"> 800 </state>
                    <state id="background_mode"> 900 </state>
                    <state id="garage_mode"> 2048 </state>
                  </perStateThreshold>
            </appCategorySpecificThresholds>
      </ioOveruseConfiguration>
</resourceOveruseConfiguration>

Update overuse configuration through CarWatchdogManager system APIs

The above XML configuration can be provided only in the build image. If an OEM chooses to update the on-device configuration after a build is released, they can use the following APIs to make changes to the on-device configuration.

  • Grant the permission Car.PERMISSION_CONTROL_CAR_WATCHDOG_CONFIG to the caller.
  • Must use the existing configurations to update and set the new configurations. Use the API CarWatchdogManager.getResourceOveruseConfigurations to get the existing configurations. If existing configurations are not used, all configurations (including System and Third-party configurations) will be overwritten, which is not recommended.
  • Update the existing configurations with the delta changes and set the new configurations. Do not update the System and Third-party component configurations.
  • Use the API CarWatchdogManager.setResourceOveruseConfigurations to set the new configurations.
  • To get and set the disk I/O overuse configurations use the flag CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO.

A sample implementation that updates the resource overuse configurations is shown below.

void updateResourceOveruseConfigurations() {
    CarWatchdogManager manager =
        (CarWatchdogManager) car.getCarManager(Car.CAR_WATCHDOG_SERVICE);

    List<ResourceOveruseConfiguration> resourceOveruseConfigurations =
        manager.getResourceOveruseConfigurations(
            CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO);

    List<ResourceOveruseConfiguration> newResourceOveruseConfigurations =
            new List<>();
    ResourceOveruseConfiguration vendorConfiguration;
    for(ResourceOveruseConfiguration config : resourceOveruseConfigurations) {
        // Do not update the configurations of the system and third-party component types.
        if (config.getComponentType()
            != ResourceOveruseConfiguration.COMPONENT_TYPE_VENDOR) {
            newResourceOveruseConfigurations.add(config);
            continue;
        }
        vendorConfiguration = config;
    }

    if (vendorConfiguration == null) {
        ResourceOveruseConfiguration.Builder vendorConfigBuilder =
            new ResourceOveruseConfiguration.Builder();
        initializeConfig(vendorConfigBuilder);
        newResourceOveruseConfigurations.add(vendorConfigBuilder.build());
    } else {
        ResourceOveruseConfiguration newVendorConfig =
            updateConfig(vendorConfiguration);
        newResourceOveruseConfigurations.add(newVendorConfig);
    }
    int result = manager.setResourceOveruseConfigurations(
        newResourceOveruseConfigurations,

    if (result != CarWatchdogManager.RETURN_CODE_SUCCESS) {
        // Failed to set the resource overuse configurations.
    }
}

/** Sets the delta between the old configuration and the new configuration. */
ResourceOveruseConfiguration updateConfig(
    ResourceOveruseConfiguration oldConfiguration) {
    // Replace com.vendor.package.A with com.vendor.package.B in the safe-to-kill list.
    List<String> safeToKillPackages = oldConfiguration.getSafeToKillPackages();
    safeToKillPackages.remove("com.vendor.package.A");
    safeToKillPackages.add("com.vendor.package.B");

    ResourceOveruseConfiguration.Builder configBuilder =
        new ResourceOveruseConfiguration.Builder(
            oldConfiguration.getComponentType(),
            safeToKillPackages,
            oldConfiguration.getVendorPackagePrefixes(),
            oldConfiguration.getPackagesToAppCategoryTypes());

    configBuilder.addVendorPackagePrefixes("com.vendor.");
    configBuilder.addPackagesToAppCategoryTypes("com.vendor.package.B",
        ResourceOveruseConfiguration.APPLICATION_CATEGORY_TYPE_MAPS);

    IoOveruseConfiguration oldIoConfiguration = oldConfiguration.getIoOveruseConfiguration();
    IoOveruseConfiguration.Builder ioConfigBuilder =
        new IoOveruseConfiguration.Builder(
            oldIoConfiguration.getComponentLevelThresholds(),
            oldIoConfiguration.getPackageSpecificThresholds(),
            oldIoConfiguration.getAppCategorySpecificThresholds(),
            oldIoConfiguration.getSystemWideThresholds());

    // Define the amount of bytes based on the flash memory specification, expected lifetime,
    // and estimated average amount of bytes written by a package during different modes.
    ioConfigBuilder.addPackageSpecificThresholds("com.vendor.package.B",
        new PerStateBytes(/* foregroundModeBytes= */ 2 * 1024 * 1024 * 1024,
                          /* backgroundModeBytes= */ 500 * 1024 * 1024,
                          /* garageModeBytes= */ 3 * 1024 * 1024 * 1024));


    return configBuilder.setIoOveruseConfiguration(ioConfigBuilder.build()).build();
}

Applications monitoring their resource overuse

Vendor and third-party applications can listen for app specific resource overuse notifications from Watchdog or poll CarWatchdogManager for the app specific resource overuse statistics for up to the past 30 days.

Listen for resource overuse notifications

Applications can implement a resource overuse listener and register the listener with CarWatchdogManager to receive app specific notifications when they exceed 80% or 100% of their disk I/O overuse thresholds. Applications can use these notifications to:

  • Log the disk I/O overuse statistics for offline analysis. Application developers can use this logging to debug the disk I/O overuse issue.
  • Reduce the disk I/O writes until the overuse counters reset.

Java client

  1. Implement listener by inheriting CarWatchdogManager.ResourceOveruseListener: 
    class ResourceOveruseListenerImpl implements
      CarWatchdogManager.ResourceOveruseListener {
            @Override
            public void onOveruse(
                  @NonNull ResourceOveruseStats resourceOveruseStats) {
                  // 1. Log/Upload resource overuse metrics.
                  // 2. Reduce writes until the counters reset.

                  IoOveruseStats ioOveruseStats = resourceOveruseStats.getIoOveruseStats();
                  // Stats period - [ioOveruseStats.getStartTime(), ioOveruseStats.getStartTime()
                  //   + ioOveruseStats.getDurationInSeconds()]
                  // Total I/O overuses - ioOveruseStats.getTotalOveruses()
                  // Total bytes written - ioOveruseStats.getTotalBytesWritten()
                  // Remaining write bytes for the current UTC calendar day -
                  //    ioOveruseStats.getRemainingWriteBytes()
            }
      }
}
  2. Register the listener instance by calling CarWatchdogManager.addResourceOveruseListener 
    private void addResourceOveruseListener() {
      CarWatchdogManager manager =
            (CarWatchdogManager) car.getCarManager(Car.CAR_WATCHDOG_SERVICE);
      // Choose a proper executor to handle resource overuse notifications.
      Executor executor = mContext.getMainExecutor();
      manager.addResourceOveruseListener(
            executor, CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO,
            mListenerImpl);
}
  3. Unregister the listener instance when the application has finished listening to: 
    private void removeResourceOveruseListener() {
    CarWatchdogManager manager =
            (CarWatchdogManager) car.getCarManager(Car.CAR_WATCHDOG_SERVICE);
    mCarWatchdogManager.removeResourceOveruseListener(
          mListenerImpl);
}

Native client

  1. Include carwatchdog_aidl_interface-ndk_platform in the shared_libs dependency of the build rule.

    Android.bp

    cc_binary {
    name: "sample_native_client",
    srcs: [
        "src/*.cpp"
    ],
    shared_libs: [
        "carwatchdog_aidl_interface-ndk_platform",
        "libbinder_ndk",
    ],
    vendor: true,
}
  2. Add SELinux policy to allow the vendor service domain to use binder (binder_user macro) and add the vendor service domain to the carwatchdog client domain (carwatchdog_client_domain macro). See the code below for sample_client.te and file_contexts.

    sample_client.te

    type sample_client, domain;
type sample_client_exec, exec_type, file_type, vendor_file_type;

carwatchdog_client_domain(sample_client)

init_daemon_domain(sample_client)
binder_use(sample_client)

    file_contexts

    /vendor/bin/sample_native_client  u:object_r:sample_client_exec:s0
  3. Implement the resource overuse listener by inheriting BnResourceOveruseListener. Override BnResourceOveruseListener::onOveruse to handle resource overuse notifications.

    ResourceOveruseListenerImpl.h

    class ResourceOveruseListenerImpl : public BnResourceOveruseListener {
public:
    ndk::ScopedAStatus onOveruse(
        ResourceOveruseStats resourceOveruseStats) override;

private:
    void initialize();
    void terminate();

    std::shared_ptr<ICarWatchdog> mWatchdogServer;
    std::shared_ptr<IResourceOveruseListener> mListener;
}

    ResourceOveruseListenerImpl.cpp

    ndk::ScopedAStatus ResourceOveruseListenerImpl::onOveruse(
      ResourceOveruseStats resourceOveruseStats) {

      // 1. Log/Upload resource overuse metrics.
      // 2. Reduce writes until the counters reset.

      if (stats.getTag() != ResourceOveruseStats::ioOveruseStats) {
            // Received resourceOveruseStats doesn't contain I/O overuse stats.
      }

      const IoOveruseStats& ioOveruseStats = stats.get();
      // Stats period - [ioOveruseStats.startTime,
      //   ioOveruseStats.startTime + ioOveruseStats.durationInSeconds]
      // Total I/O overuses - ioOveruseStats.totalOveruses
      // Total bytes written - ioOveruseStats.writtenBytes
      // Remaining write bytes for the current UTC calendar day -
      //    ioOveruseStats.remainingWriteBytes

      return ndk::ScopedAStatus::ok();
}
  4. Start a binder thread pool and register the resource overuse listener with the watchdog server. Watchdog server is registered under the service name android.automotive.watchdog.ICarWatchdog/default.

    main.cpp

    int main(int argc, char** argv) {
    ABinderProcess_setThreadPoolMaxThreadCount(1);
    ABinderProcess_startThreadPool();
    std::shared_ptr<ResourceOveruseListenerImpl> listener =
        ndk::SharedRefBase::make<ResourceOveruseListenerImpl>();

    // The listener is added in initialize().
    listener->initialize();

    ... Run service ...

    // The listener is removed in terminate().
    listener->terminate();
}

    ResourceOveruseListenerImpl.cpp

    void ResourceOveruseListener::initialize() {
    ndk::SpAIBinder binder(AServiceManager_getService(
            "android.automotive.watchdog.ICarWatchdog/default"));
    std::shared_ptr<ICarWatchdog> server = ICarWatchdog::fromBinder(binder);
    mWatchdogServer = server;

    std::shared_ptr<IResourceOveruseListener> listener =
        IResourceOveruseListener::fromBinder(this->asBinder());
    mWatchdogServer->addResourceOveruseListener(
      std::vector<int>{ResourceType.IO}, listener);
    mListener = listener;
}

void ResourceOveruseListener::terminate() {
    mWatchdogServer->removeResourceOveruseListener(mListener);
}

Poll resource overuse statistics

Applications can poll CarWatchdogManager for the app-specific I/O overuse statistics ATS for the most recent 30 days.

Java client

Use CarWatchdogManager.getResourceOveruseStats to get the resource overuse stats. Pass the CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO flag to get the disk I/O overuse stats.

private void getResourceOveruseStats() {
      CarWatchdogManager manager =
            (CarWatchdogManager) car.getCarManager(Car.CAR_WATCHDOG_SERVICE);

      // Returns resource overuse stats with I/O overuse stats for the past
      // 7 days. Stats are available for up to the past 30 days.
      ResourceOveruseStats resourceOveruseStats =
            mCarWatchdogManager.getResourceOveruseStats(
                  CarWatchdogManager.FLAG_RESOURCE_OVERUSE_IO,
                  CarWatchdogManager.STATS_PERIOD_PAST_7_DAYS);

      IoOveruseStats ioOveruseStats = resourceOveruseStats.getIoOveruseStats();
      // Stats period - [ioOveruseStats.getStartTime(), ioOveruseStats.getStartTime()
      //   + ioOveruseStats.getDurationInSeconds()]
      // Total I/O overuses - ioOveruseStats.getTotalOveruses()
      // Total bytes written - ioOveruseStats.getTotalBytesWritten()
      // Remaining write bytes for the UTC calendar day -
      //    ioOveruseStats.getRemainingWriteBytes()
}

Native client

Use CarWatchdogServer.getResourceOveruseStats to get the resource overuse stats. Pass the ResourceType.IO enum to fetch disk I/O overuse stats.

void getResourceOveruseStats() {
      ndk::SpAIBinder binder(AServiceManager_getService(
            "android.automotive.watchdog.ICarWatchdog/default"));
      std::shared_ptr<ICarWatchdog> server = ICarWatchdog::fromBinder(binder);
      // Returns the stats only for the current UTC calendar day.
      const std::vector<ResourceOveruseStats> resourceOveruseStats;
      ndk::ScopedAStatus status = server.getResourceOveruseStats(
            std::vector<int>{ResourceType.IO}, &resourceOveruseStats);
      if (!status.isOk()) {
            // Failed to get the resource overuse stats.
            return;
      }

      for (const auto& stats : resourceOveruseStats) {
            if (stats.getTag() != ResourceOveruseStats::ioOveruseStats) {
                  continue;
            }
            const IoOveruseStats& ioOveruseStats = stats.get();
            // Stats period - [ioOveruseStats.startTime,
            //   ioOveruseStats.startTime + ioOveruseStats.durationInSeconds]
            // Total I/O overuses - ioOveruseStats.totalOveruses
            // Total bytes written - ioOveruseStats.writtenBytes
            // Remaining write bytes for the current UTC calendar day -
            //   ioOveruseStats.remainingWriteBytes
      }
}

Resource overuse UX

Prioritize app performance setting

Application settings page has the Prioritize app performance settings (see the image below), that allows the user to prioritize an application's performance over the system and long-term hardware performance. This setting is available only for applications that are safe to be terminated on resource overuse. Otherwise, this setting is grayed out. When this setting is toggled off (default setting) for an application, the application can be terminated on resource overuse. Otherwise, the application is not terminated on resource overuse.

When the user toggles on this setting, the following confirmation dialog describes the implications of toggling on the setting.

User notification

When an application or service repeatedly overuses disk I/O (for example, writes data to disk beyond the defined thresholds) within a certain period and is safe to be terminated on resource overuse, the user is notified after the vehicle enters the allow-driver-distraction state.

The first user notification (during a drive) is posted as a heads-up notification and the other notifications are posted on the notification center.

For example, when an app repeatedly overuses disk I/O, the user receives the following notification:

  • When the user clicks on the Prioritize app button, the application's settings page is launched, where the user can toggle on or off the Prioritize app performance setting.
  • When the user clicks on the Disable app button, the application is disabled until the user launches the app or enables it on the application's settings page.
  • For uninstallable applications, the Disable app button is replaced with the Uninstall app button. When the user clicks on the Uninstall app button, the application's Settings page is launched, from which the user can uninstall the app.

Recommendation for launcher implementation

When applications are disabled due to resource overuse, the applications disappear from the default launcher app because CarService updates the applications' enabled state as PackageManager.COMPONENT_ENABLED_STATE_DISABLED_UNTIL_USED. OEMs must update the builtin launcher implementation to display these applications as unusual, so users can use them if needed. See the following recommendations based on the build release.

Android SC V2 release