When the HWASan tool detects a memory bug, the process is terminated with abort()
, and
a report is printed to stderr and logcat. Like all native crashes on Android, HWASan errors can be
found under /data/tombstones
.
Compared to regular native crashes, HWASan carries extra information in the “Abort message” field near the top of the tombstone. See a sample heap-based crash below (for stack bugs, see the note below for the stack-specific sections).
Example report
*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** Build fingerprint: 'google/flame_hwasan/flame:Tiramisu/MASTER/7956676:userdebug/dev-keys' Revision: 'DVT1.0' ABI: 'arm64' Timestamp: 2019-04-24 01:13:22+0000 pid: 11154, tid: 11154, name: sensors@1.0-ser >>> /vendor/bin/hw/android.hardware.sensors@1.0-service <<< signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr -------- Abort message: '==9569==ERROR: HWAddressSanitizer: tag-mismatch on address 0x00433ae20045 at pc 0x00623ae2a9cc READ of size 1 at 0x00433ae20045 tags: 5b/83 (ptr/mem) in thread T0 #0 0x7240450c68 (/system/lib64/vndk-sp-R/libcutils.so+0x8c68) #1 0x723dffd490 (/vendor/lib64/sensors.ssc.so+0x34490) #2 0x723e0126e0 (/vendor/lib64/sensors.ssc.so+0x496e0) [...] [0x00433ae20040,0x00433ae20060) is a small unallocated heap chunk; size: 32 offset: 5 Cause: use-after-free 0x00433ae20045 is located 5 bytes inside of 10-byte region [0x00433ae20040,0x00433ae2004a) freed by thread T0 here: #0 0x72404d1b18 (/system/lib64/libclang_rt.hwasan-aarch64-android.so+0x10b18) #1 0x723af23040 (/vendor/lib64/libgralloccore.so+0x5040) #2 0x723af23fa4 (/vendor/lib64/libgralloccore.so+0x5fa4) [...] previously allocated here: #0 0x72404ce554 (/system/lib64/libclang_rt.hwasan-aarch64-android.so+0xd554) #1 0x7240115654 (/apex/com.android.runtime/lib64/bionic/libc.so+0x43654) #2 0x7240450ac8 (/system/lib64/vndk-sp-R/libcutils.so+0x8ac8) [...] hwasan_dev_note_heap_rb_distance: 1 1023 hwasan_dev_note_num_matching_addrs: 0 hwasan_dev_note_num_matching_addrs_4b: 0 Thread: T0 0x006a00002000 stack: [0x007fc1064000,0x007fc1864000) sz: 8388608 tls: [0x00737702ffc0,0x007377033000) Memory tags around the buggy address (one tag corresponds to 16 bytes): 0x006f33ae1f80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae1f90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae1fa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae1fb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae1fc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae1fd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae1fe0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae1ff0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 =>0x006f33ae2000: 08 00 08 00 [83] 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae2010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae2020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae2030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae2040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae2050: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae2060: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae2070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0x006f33ae2080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Tags for short granules around the buggy address (one tag corresponds to 16 bytes): 0x006f33ae1ff0: .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. =>0x006f33ae2000: 72 .. d0 .. [..] .. .. .. .. .. .. .. .. .. .. .. 0x006f33ae2010: .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. See https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html#short-granules for a description of short granule tags Registers where the failure occurred (pc 0x00623ae2a9cc): x0 0000007fc18623ec x1 5b0000433ae20045 x2 0000000000000013 x3 ffffffffffffffff x4 ffffffffffffffff x5 0000007fc1861da3 x6 6f7420676e696f47 x7 45522061206f6420 x8 0000000000000000 x9 0200006b00000000 x10 00000007fc18623f x11 5b0000433ae20040 x12 6f64206f7420676e x13 0a44414552206120 x14 0000000000000010 x15 ffffffffffffffff x16 000000737169ac94 x17 0000000000000007 x18 0000007377bd8000 x19 0000007fc1862498 x20 0200006b00000000 x21 0000007fc18624a8 x22 0000000000000001 x23 0000000000000000 x24 0000000000000000 x25 0000000000000000 x26 0000000000000000 x27 0000000000000000 x28 0000000000000000 x29 0000007fc1862410 x30 000000623ae2a9d0 sp 0000007fc18623d0 SUMMARY: HWAddressSanitizer: tag-mismatch (/system/lib64/vndk-sp-R/libcutils.so+0x8c68) [ … regular crash dump follows …]
This is very similar to an AddressSanitizer report. Unlike those, almost all HWASan bugs are “tag-mismatch”, i.e. a memory access where a pointer tag does not match the corresponding memory tag. This could be one of
- out of bounds access on stack or heap
- use after free on heap
- use after return on stack
Sections
An explanation of each of the sections of the HWASan report is below:
Access error
Contains information about the bad memory access, including:
- Access Type ("READ" vs. "WRITE")
- Access size (how many bytes were attempted to be accessed)
- Thread number of the access
- Pointer and memory tags (for advanced debugging)
Access stack trace
Stack trace of the bad memory access. See the Symbolization section to symbolize.
Cause
The potential cause for the bad access. If there are multiple candidates, they are listed in order of descending likelihood. Precedes the detailed info about the potential cause. HWASan can diagnose the following causes:
- use-after-free
- stack tag-mismatch: this can be stack use-after-return / use after-scope, or out of bounds
- heap-buffer-overflow
- global-overflow
Memory information
Describes what HWASan knows about the memory being accessed, and may differ based on the bug type.
Bug Type | Cause | Report Format |
---|---|---|
tag-mismatch | use-after-free |
<address> is located N bytes inside of M-byte region [<start>, <end>) freed by thread T0 here: |
heap-buffer-overflow | Note that this can also be an underflow.
<address> is located N bytes to the right of M-byte region [<start>, <end>) allocated here: |
|
stack tag-mismatch | Stack reports do not differentiate between overflow/underflow and use-after-return bugs. In addition, to find the stack allocation that is the source of the error, an offline symbolization step is required. See the Understanding stack reports section below. | |
invalid-free | use-after-free | This is a double free bug. If this happens on process shutdown, this can signify an
ODR violation.
<address> is located N bytes inside of M-byte region [<start>, <end>) freed by thread T0 here: |
cannot describe address | Either a wild free (free of memory that hadn't been allocated before), or a double free after the allocated memory was evicted from HWASan's free buffer. | |
0x… is HWAsan shadow memory. | This is definitely a wild free, as the app was attempting to free memory that is internal to HWASan. |
Deallocation stack trace
Stack trace of where the memory was deallocated. Only present for use-after-free or invalid-free bugs. See Symbolization section to symbolize.
Allocation stack trace
Stack trace of where the memory was allocated. See Symbolization section to symbolize.
Advanced debugging Information
The HWASan report also features some advanced debugging information, including (in order):
- The list of threads in the process
- The list of threads in the process
- The value of the memory tags near the faulting memory
- The dump of the registers at the point of memory access
Memory tag dump
The tag memory dump can be used to look for nearby memory allocations with the same tag as the pointer tag. These could point to an out of bounds access with a large offset. One tag corresponds to 16 bytes of memory; the pointer tag is the top 8 bits of the address. The tag memory dump can give hints, for example the following is a buffer overflow to the right:
tags: ad/5c (ptr/mem) [...] Memory tags around the buggy address (one tag corresponds to 16 bytes): 0x006f33ae1ff0: 0e 0e 0e 57 20 20 20 20 20 2e 5e 5e 5e 5e 5e b5 =>0x006f33ae2000: f6 f6 f6 f6 f6 4c ad ad ad ad ad ad [5c] 5c 5c 5c 0x006f33ae2010: 5c 04 2e 2e 2e 2e 2e 2f 66 66 66 66 66 80 6a 6a Tags for short granules around the buggy address (one tag corresponds to 16 bytes): 0x006f33ae1ff0: ab 52 eb .. .. .. .. .. .. .. .. .. .. .. .. .. =>0x006f33ae2000: .. .. .. .. .. .. .. .. .. .. .. .. [..] .. .. .. 0x006f33ae2010: .. 5c .. .. .. .. .. .. .. .. .. .. .. .. .. ..
If the size of an allocation is not a multiple of 16, the remainder of the size will be stored as the memory tag and the tag will be stored as a short granule tag. In the example above just after the bolded allocation tagged ad, we have a 5 × 16 + 4 = 84 byte allocation of tag 5c.
A zero memory tag (ex. tags: ad/00 (ptr/mem)
) usually indicates a
stack-use-after-return bug.
Register dump
The register dump in HWASan reports corresponds to the actual instruction that performed the invalid memory access. It is followed by another register dump from the regular Android signal handler - ignore the second one, it is taken when HWASan called abort() and is not relevant to the bug.
Symbolization
To get function names and line numbers in stack traces (and get variable names for use-after-scope bugs), an offline symbolization step is needed.
First-time setup: install llvm-symbolizer
To symbolize, your system must have llvm-symbolizer installed and accessible from $PATH. On Debian, you can
install it using sudo apt install llvm
.
Obtain symbol files
For symbolization, we require unstripped binaries containing symbols. Where these can be found depends on the type of build:
For local builds, the symbol files can be found in
out/target/product/<product>/symbols/
.
For AOSP builds (e.g. flashed from Flashstation), the
builds can be found on Android CI. In the "Artifacts" for the
build,
there will be a ${PRODUCT}-symbols-${BUILDID}.zip
file.
For internal builds from your organization, check your organization's documentation for help obtaining symbol files.
Symbolize
hwasan_symbolize –-symbols <DECOMPRESSED_DIR>/out/target/product/*/symbols < crash
Understand stack reports
For bugs that occur with stack variables, the HWASan report will contain details like this:
Cause: stack tag-mismatch Address 0x007d4d251e80 is located in stack of thread T64 Thread: T64 0x0074000b2000 stack: [0x007d4d14c000,0x007d4d255cb0) sz: 1088688 tls: [0x007d4d255fc0,0x007d4d259000) Previously allocated frames: record_addr:0x7df7300c98 record:0x51ef007df3f70fb0 (/apex/com.android.art/lib64/libart.so+0x570fb0) record_addr:0x7df7300c90 record:0x5200007df3cdab74 (/apex/com.android.art/lib64/libart.so+0x2dab74) [...]
To allow stack bugs to be understood, HWASan keeps track of the stack frames that happened in the past. Currently, HWASan doesn't transform this into human-understandable content in the bug report, and requires an additional symbolization step.
ODR violations
Some use-after-free bugs reported by HWASan can also indicate an One Definition Rule (ODR) violation. An ODR violation happens when the same variable is defined multiple times in the same program. This also means that the variable is destructed multiple times, which may lead to the use-after-free error.
After symbolization, ODR violations show a use-after-free with __cxa_finalize
,
on both the invalid access stack and the "freed here" stack. The "previously allocated
here" stack contains __dl__ZN6soinfo17call_constructorsEv
and should
point at the location in your program that defines the variable higher on the stack.
One reason why the ODR may be violated is when static libraries are used. If a static library that defines a C++ global is linked into multiple shared libraries or executables, multiple definitions of the same symbol may end up existing in the same address space, which will cause an ODR error.
Troubleshooting
HWAddressSanitizer can not describe address in more detail
Sometimes HWASan can run out of space for information about past memory allocations. In that case, the report will contain only one stack trace for the immediate memory access, followed by a note:
HWAddressSanitizer can not describe address in more detail.
In some cases this can be resolved by running the test multiple times. Another option is to increase HWASan
history size. This can be done globally in
build/soong/cc/sanitize.go
(look for
hwasanGlobalOptions
), or in your process environment (try
adb shell echo $HWASAN_OPTIONS
to see the current settings).
This can also happen if the accessed memory is not mapped, or allocated by a non-HWASan aware
allocator. In this case, the mem
tag listed in the crash header will generally be
00
. If you have access to the full tombstone, it might be helpful to consult the
memory maps dump to find out which mapping (if any) the address belongs to.
Nested bug in the same thread
This means there was a bug while generating the HWASan crash-report. This is usually due to a bug in the HWASan runtime, please file a bug and provide instructions on how to reproduce the issue if possible.