Android NE问题分析方法介绍

文章目录

- 简介
- Native内存布局
- NE问题常见类型
- 常用调试工具
- tombstone抓取流程
- 实例分析
- - tombstone文件内容
  - 举例

简介

NE，全称Native Exception，在Android中主要指在用户空间运行的native程序或者natvie库发生异常。NE问题通常带来程序奔溃现象，导致功能模块不稳定。

本文主要介绍有关NE的基本知识、NE问题出现后的基本分析方法、常见的NE问题和常用调试工具。

Native内存布局

这里主要介绍Native进程的虚拟地址空间，分32bit和64bit进程，cameraserver就是32bit的，而android.hardware.camera.provider@2.4-service_64是64bit的，主要是寻址范围不一样，32bit的进程寻址范围是4G，而64bit进程寻址范围是2的64次方，实际可用的是2的48次方，也就是256TB。

进程的内存空间按低地址到高地址通常分为代码段、数据段、BSS段、堆、栈和内核数据区这几个部分：

代码段：存放可执行的二进制指令，也就是代码占用的空间
数据段：存放初始化过的全局变量、静态变量
BSS段：Block Started by Symbol，存放程序中未初始化的全局变量和静态变量，默认初始化时全部为0
堆：存放进程运行中被动态分配的内存段，向上增长
栈：存放程序临时创建的局部变量，向下增长

(下文两张图来源于网络，出处忘记了…，见谅)

32位的虚拟地址空间
在这里插入图片描述

64位的虚拟地址空间
在这里插入图片描述

虚拟地址空间文件解读

通过adb shell cat /proc/$PID/maps或者adb shell pmap -x $PID可获取进程的虚拟地址空间映射文件，示例内容如：

00000070'ebb15000-00000070'ebc8ffff r--         0    17b000  /vendor/lib64/libmegface_denselmk.so (BuildId: 5bb83f57711ebbf34c416d3feb706e)
00000070'ebc90000-00000070'ebc94fff ---         0      5000
00000070'ebc95000-00000070'ec2e8fff r-x    180000    654000  /vendor/lib64/libmegface_denselmk.so (BuildId: 5bb83f57711ebbf34c416d3feb706e)
00000070'ec2e9000-00000070'ec2f4fff ---         0      c000
00000070'ec2f5000-00000070'ec328fff rw-    7e0000     34000  /vendor/lib64/libmegface_denselmk.so (BuildId: 5bb83f57711ebbf34c416d3feb706e)
00000070'ec329000-00000070'ec3aefff rw-         0     86000  [anon:.bss]00000070'fadd3000-00000070'fae4afff rw-         0     78000  anon_inode:dmabuf404400000071'7fa00000-00000071'801fffff rw-         0    800000  [anon:libc_malloc]0000007f'f1c61000-0000007f'f1d18fff rw-         0     b8000  [stack]

每一条记录对应一个VMA（虚拟内存区域）结构。

第一段 “r–” 则是这个lib 使用的只读变量段，第三段 “r-x” 则是只读并可执行的主体代码段，第五段 “rw-” 则是这个lib 使用的数据段，第六段则是bss段。

第8行是ion memory 段，ion buffer 的 vma name 标注成dmabuf，即已经mmap 的ion memory 可以从这个直接统计算出。

第10行是malloc通过jemalloc 所管控的空间, 常见的malloc leaks 都会可以看到这种libc_malloc段空间显著增长。

第12行是栈空间。

NE问题常见类型

下面介绍几种常见的Native Crash情况：

主动抛出异常，代码中调用了abort()，系统会给进程发送信号SIGABRT(6)，这种问题通常是代码执行到了异常分支，需要检查看看什么条件导致的，CamX代码使用的比较多。
进程被信号SIGKILL(9)杀死，比如通过adb shell kill -9 $PID或者某进程管理机制杀掉，APP进程可能进程会碰到这个问题，一般不会生成tombstone文件。
空指针，在进程的地址空间中，从0开始的第一个页面的权限被设置为不可读也不可写，当进程的指令试图访问该页面中的地址时(如读取空指针指向的内存)，处理器就会产生一个异常，然后Linux内核会给该进程发送一个段错误信号SIGSEGV(11)。
野指针，指向的是一个无效的地址，该地址如果是不可读不可写的，那么会马上Crash(内核给进程发送段错误信号SIGSEGV)，这时bug会很快被发现。

如果访问的地址为可写，而且通过野指针修改了该处的内存，那么很有可能会等一段时间(其它的代码使用了该处的内存后)才发生Crash。这时查看Crash时显示的调用栈，和野指针所在的代码部分，有可能基本上没有任何关联。是踩内存的一种。
数组越界/缓冲区溢出，情况与野指针类似，访问了无效的地址，踩了别人的内存区域。

常用调试工具

GNU C/C++工具，如addr2line、readelf、objdump、gdb等

prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/

Malloc Debug，检测内存踩踏、内存泄漏等问题
```
bionic/libc/malloc_debug/README.md
```
AddressSanitizer
AddressSanitizer (ASan) 是一种基于编译器的快速检测工具，用于检测原生代码中的内存错误。

ASan 可以检测以下问题：
- 堆栈和堆缓冲区上溢/下溢
- 释放之后的堆使用情况
- 超出范围的堆栈使用情况
- 重复释放/错误释放
ASan 可在 32 位和 64 位 ARM 以及 x86 和 x86-64 上运行。ASan 的 CPU 开销约为 2 倍，代码大小开销在一半到 2 倍之间，并且内存开销很大（具体取决于您的分配模式，但约为 2 倍）。

Android 10 和 AArch64 上的 AOSP master 分支支持硬件加速 ASan (HWASan)，这是一种 RAM 开销更小、检测到的错误范围更大的类似工具。除了 ASan 可以检测到的错误之外，HWASan 还可以检测返回之后的堆栈使用情况。

HWASan 具有类似的 CPU 和代码大小开销，但 RAM 开销要小得多 (15%)。HWASan 具有不确定性。只有 256 个可能的标记值，因此忽略任何错误的概率为 0.4%。ASan 对检测溢出规定了有限大小的红色区域，并对检测释放后使用情况规定了有限容量隔离区，而 HWAsan 没有这些规定，因此溢出大小或多久之前内存解除分配对 HWAsan 而言并不重要。这使得 HWASan 优于 ASan。您可以详细了解 HWAsan 的设计或 Android 上的 HWASan 的使用。

除了堆溢出外，ASan 还能检测堆栈/全局溢出，并能以最低的内存开销实现很高的速度。
HWAddressSanitizer 即HWASan

了解工作原理，可参考 ASAN和HWASAN原理解析

tombstone抓取流程

Native Exception发生时，CPU将产生中断触发异常处理流程。Kernel处理中断，统一成信号发送，应用进程注册和处理信号。

在Android中，所有so都需要通过linker(Android的链接器)加载，linker会默认注册信号处理函数，代码在bionic/linker/linker_main.cpp : debuggerd_init()，调用debuggerd_init()。debuggerd中注册接收的信号有9个: SIGABRT，SIGBUS，SIGFPE，SIGILL，SIGSEGV，SIGSTKFLT，SIGSYS，SIGTRAP，DEBUGGER_SIGNAL。

当NE发生时，debuggerd（代码在system/core/debuggerd）的默认信号处理函数被调用，它会打印"Fatal signal"异常信息，然后clone一直子进程，在其内执行/system/bin/crasher_dump64(32)，这个crasher_dump就会去抓取各种异常信息，如signal、寄存器、backtrace、maps等输出到log中，并通过socket通知tombstoned进程保存到/data/tombstones/tombstone_xx文件。

实例分析

tombstone文件内容

tombstone文件内容解读

举例

heap-buffer-overflow：

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'qti/ALPS/ALPS:10/QKQ1.200202.002/eng.xxxx.20200410.105316:userdebug/release-keys'
Revision: '0'
ABI: 'arm64'
Timestamp: 2020-04-14 15:50:43+0800
pid: 5822, tid: 6322, name: HwBinder:5822_3  >>> /vendor/bin/hw/android.hardware.camera.provider@2.4-service_64 <<<
uid: 1047
signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
Abort message: '=================================================================
==5822==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x0058e2a26f82 at pc 0x007e04e767a0 bp 0x007d2f68fe20 sp 0x007d2f68f5b0
READ of size 2048 at 0x0058e2a26f82 thread T307 (HwBinder:5822_3)#0 0x7e04e7679c  (/system/lib64/libclang_rt.asan-aarch64-android.so+0x7179c)#1 0x7d6a294200  (/vendor/lib64/libremosaic_wrapper.so+0x5200)#2 0x7d68a33bb4  (/vendor/lib64/hw/com.qti.chi.override.so+0x19fbb4)#3 0x7d68a76f80  (/vendor/lib64/hw/com.qti.chi.override.so+0x1e2f80)#4 0x7d689e5b18  (/vendor/lib64/hw/com.qti.chi.override.so+0x151b18)#5 0x7d689dbe28  (/vendor/lib64/hw/com.qti.chi.override.so+0x147e28)#6 0x7d7e0dda04  (/vendor/lib64/hw/camera.qcom.so+0x30ea04)#7 0x7d7e0d4578  (/vendor/lib64/hw/camera.qcom.so+0x305578)#8 0x7e00769a98  (/vendor/lib64/camera.device@3.4-impl.so+0x16a98)#9 0x7e007e6e3c  (/vendor/lib64/camera.device@3.5-impl.so+0x1ce3c)#10 0x7e04b7f234  (/system/lib64/vndk-29/android.hardware.camera.device@3.5.so+0x33234)#11 0x7e04b80164  (/system/lib64/vndk-29/android.hardware.camera.device@3.5.so+0x34164)#12 0x7e0421b6d8  (/system/lib64/vndk-sp-29/libhidlbase.so+0x986d8)#13 0x7e0421f078  (/system/lib64/vndk-sp-29/libhidlbase.so+0x9c078)#14 0x7e04220290  (/system/lib64/vndk-sp-29/libhidlbase.so+0x9d290)#15 0x7e0422f1d0  (/system/lib64/vndk-sp-29/libhidlbase.so+0xac1d0)#16 0x7e04beb5f0  (/system/lib64/vndk-sp-29/libutils.so+0x135f0)#17 0x7e047c3b60  (/apex/com.android.runtime/lib64/bionic/libc.so+0xe6b60)#18 0x7e04761b6c  (/apex/com.android.runtime/lib64/bionic/libc.so+0x84b6c)0x0058e2a26f82 is located 1 bytes to the right of 1793-byte region [0x0058e2a26880,0x0058e2a26f81)
allocated by thread T0 here:

heap-use-after-free：

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'XXXX/G5/G5:10/QKQ1.200216.002/eng.XXXX.20200723.094420:userdebug/release-keys'
Revision: '0'
ABI: 'arm64'
Timestamp: 2020-07-24 14:04:35+0800
pid: 26851, tid: 6803, name: provider@2.4-se  >>> /vendor/bin/hw/android.hardware.camera.provider@2.4-service_64 <<<
uid: 1047
signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
Abort message: '=================================================================
==26851==ERROR: AddressSanitizer: heap-use-after-free on address 0x0073896a5300 at pc 0x0075423717a0 bp 0x0074754e69e0 sp 0x0074754e6170
READ of size 777600 at 0x0073896a5300 thread T84843#0 0x754237179c  (/system/lib64/libclang_rt.asan-aarch64-android.so+0x7179c)#1 0x74adaec310  (/vendor/lib64/libanc_single_rt_bokeh.so+0x2eb310)#2 0x74ad896134  (/vendor/lib64/libanc_single_rt_bokeh.so+0x95134)#3 0x74ad8904cc  (/vendor/lib64/libanc_single_rt_bokeh.so+0x8f4cc)#4 0x74ad8901ec  (/vendor/lib64/libanc_single_rt_bokeh.so+0x8f1ec)#5 0x74ad876f88  (/vendor/lib64/libanc_single_rt_bokeh.so+0x75f88)#6 0x74ad874818  (/vendor/lib64/libanc_single_rt_bokeh.so+0x73818)#7 0x74ad874644  (/vendor/lib64/libanc_single_rt_bokeh.so+0x73644)#8 0x74ad8753ac  (/vendor/lib64/libanc_single_rt_bokeh.so+0x743ac)#9 0x7541fd8b60  (/apex/com.android.runtime/lib64/bionic/libc.so+0xe6b60)#10 0x7541f76b6c  (/apex/com.android.runtime/lib64/bionic/libc.so+0x84b6c)0x0073896a5300 is located 1555200 bytes inside of 2332800-byte region [0x007389529800,0x007389763080)
freed by thread T100 (HwBinder:26851_) here:    SUMMARY: AddressSanitizer: heap-use-after-free (/system/lib64/libclang_rt.asan-aarch64-android.so+0x7179c) 
Shadow bytes around the buggy address:0x001e712d4a10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd0x001e712d4a20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd0x001e712d4a30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd0x001e712d4a40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd0x001e712d4a50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x001e712d4a60:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd0x001e712d4a70: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd0x001e712d4a80: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd0x001e712d4a90: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd0x001e712d4aa0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd0x001e712d4ab0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):Addressable:           00Partially addressable: 01 02 03 04 05 06 07 Heap left redzone:       faFreed heap region:       fdStack left redzone:      f1Stack mid redzone:       f2Stack right redzone:     f3Stack after return:      f5Stack use after scope:   f8Global redzone:          f9Global init order:       f6Poisoned by user:        f7Container overflow:      fcArray cookie:            acIntra object redzone:    bbASan internal:           feLeft alloca redzone:     caRight alloca redzone:    cbShadow gap:              cc

空指针问题

定位代码行后，检查代码逻辑是否有问题，所诉指针是否可能空指针、指针是否可能被提前释放、代码前述条件异常？空指针问题很大一部分是这类情况。

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'GREE/G5/G5:10/QKQ1.200412.002/eng.buildf.20200729.014003:user/release-keys'
Revision: '0'
ABI: 'arm64'
Timestamp: 2020-08-03 09:42:21+0800
pid: 19367, tid: 19484, name: provider@2.4-se  >>> /vendor/bin/hw/android.hardware.camera.provider@2.4-service_64 <<<
uid: 1047
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x698
Cause: null pointer dereferencex0  0000000000000000  x1  0000000000000000  x2  0000000000003820  x3  495245535b0030ffx4  000000713301fc82  x5  000000711a1d170d  x6  0000000000000030  x7  0000000000000030x8  8000000000000000  x9  0000000000000000  x10 0000000000000001  x11 000000000000000ax12 8000000000000000  x13 0000000000000010  x14 0000000000000004  x15 ffffffffffffffffx16 000000713375ac48  x17 00000071b79f509c  x18 00000071122c4588  x19 000000711201a940x20 0000000000000000  x21 000000711201c888  x22 0000000000000001  x23 0000000000000001x24 0000000000000001  x25 0000000000000001  x26 0000000000000001  x27 0000000000000001x28 0000000000001f40  x29 000000711a1d39d0sp  000000711a1d12b0  lr  000000713352aa90  pc  000000713352c6d0backtrace:#00 pc 00000000005ab6d0  /vendor/lib64/hw/camera.qcom.so (CamX::CAECStatsProcessor::ExecuteProcessRequest(CamX::StatsProcessRequestData const*)+15528) (BuildId: a30cf952aef4cabc475e5545a5d6c1a7)#01 pc 00000000005e1cf4  /vendor/lib64/hw/camera.qcom.so (CamX::StatsProcessorManager::StatsProcessorManagerJobHandler(CamX::StatsProcessRequestData*)+340) (BuildId: a30cf952aef4cabc475e5545a5d6c1a7)#02 pc 00000000005dfa14  /vendor/lib64/hw/camera.qcom.so (CamX::StatsProcessingNode::ExecuteProcessRequest(CamX::ExecuteProcessRequestData*)+2108) (BuildId: a30cf952aef4cabc475e5545a5d6c1a7)#03 pc 0000000000721d60  /vendor/lib64/hw/camera.qcom.so (CamX::Node::ProcessRequest(CamX::NodeProcessRequestData*, unsigned long)+10088) (BuildId: a30cf952aef4cabc475e5545a5d6c1a7)#04 pc 00000000006c37a0  /vendor/lib64/hw/camera.qcom.so (CamX::DeferredRequestQueue::DeferredWorkerWrapper(void*)+368) (BuildId: a30cf952aef4cabc475e5545a5d6c1a7)#05 pc 000000000061ad40  /vendor/lib64/hw/camera.qcom.so (CamX::ThreadCore::WorkerThreadBody(void*)+1992) (BuildId: a30cf952aef4cabc475e5545a5d6c1a7)#06 pc 00000000000e6b60  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36) (BuildId: 778f9db29d872fa660c03bee8d69f746)#07 pc 0000000000084b6c  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 778f9db29d872fa660c03bee8d69f746)

FDLEAK问题

这类问题通常在Log中的表现是提示memory不足、文件打开失败，我们可以确认一下当时的memory信息、fd使用情况，一个进程的可用fd一般是32768个(在大概是Android 9这个版本之前是1024)，如果已经打开了这么多，新的文件就无法打开成功。

//main_log:
08-03 03:39:48.370 30827 19230 E CamX    : [ERROR][CSL    ] camxcslhwinternal.cpp:3214 CSLHwInternalDefaultIoctl2() Ioctl returned -1, failed for device /dev/video0 (Type:CSLHwRequestManager, FD:9, Index:-1)with error reason Invalid argument
08-03 03:39:48.370 30827 19230 E CamX    : [ERROR][CSL    ] camxcslhw.cpp:2381 CSLAllocHW() Allocating CSL Buffer failed for len = 8896
08-03 03:39:48.370 30827 19230 E CamX    : [ERROR][HAL    ] camxcslhw.cpp:2389 CSLAllocHW() open /proc/meminfo failed
08-03 03:39:48.370 30827 19230 E CamX    : [ERROR][MEMMGR ] camxcmdbuffermanager.cpp:417 InitializePool() Buffer Info 0x708ec458f0, CamxResult: 1
08-03 03:39:48.370 30827 19230 E CamX    : [ERROR][UTILS  ] camxcmdbuffermanager.cpp:435 InitializePool() Out of memory
08-03 03:39:48.370 30827 19230 E CamX    : [ERROR][CORE   ] camxnode.cpp:5216 CreateCmdBufferManager() Failed to Create command Buffer Manager(CBM_SATOfflinePreview_LRME0_PacketManager)08-03 03:39:48.370 30827 31068 E CHIZOOMTRANSLATOR: ReadCalibrationDataFromFile():124 ReadCalibrationDataFromFile:124: califile == null !
08-03 03:39:48.370 30827 31068 I CHIZOOMTRANSLATOR: ReadCalibrationDataFromFile():126 path=/mnt/vendor/persist/camera/dualcam_cali_sat.bin dataSize:419
08-03 03:39:48.370 30827 31068 E CHIZOOMTRANSLATOR: ReadCalibrationDataFromFile():129 ReadCalibrationDataFromFile:129: (OUT) failed !
08-03 03:39:48.370 30827 31068 E walog/ozm: calib data count wrong
08-03 03:39:48.375 30827 19231 E CamX    : [ERROR][CSL    ] camxcslhwinternal.cpp:3214 CSLHwInternalDefaultIoctl2() Ioctl returned -1, failed for device /dev/video0 (Type:CSLHwRequestManager, FD:9, Index:-1)with error reason Invalid argument
08-03 03:39:48.375 30827 19231 E CamX    : [ERROR][CSL    ] camxcslhw.cpp:2381 CSLAllocHW() Allocating CSL Buffer failed for len = 106752
08-03 03:39:48.375 30827 19231 E CamX    : [ERROR][HAL    ] camxcslhw.cpp:2389 CSLAllocHW() open /proc/meminfo failed
08-03 03:39:48.375 30827 19231 E CamX    : [ERROR][MEMMGR ] camxcmdbuffermanager.cpp:417 InitializePool() Buffer Info 0x70f378bc00, CamxResult: 1
08-03 03:39:48.375 30827 19231 E CamX    : [ERROR][UTILS  ] camxcmdbuffermanager.cpp:435 InitializePool() Out of memory
08-03 03:39:48.376 30827 19231 E CamX    : [ERROR][CORE   ] camxnode.cpp:5216 CreateCmdBufferManager() Failed to Create command Buffer //FDLEAK的特征，查看fd(来自tombstone_xx文件)
open files:fd 0: /dev/null (unowned)fd 1: /dev/null (unowned)fd 2: /dev/null (unowned)fd 3: socket:[17996876] (unowned)fd 4: /dev/vndbinder (unowned)fd 5: /dev/hwbinder (unowned)fd 6: /sys/kernel/debug/tracing/trace_marker (unowned)fd 7: /dev/ion (unowned)fd 8: pipe:[17975238] (unowned)fd 9: /dev/video0 (unowned)fd 10: pipe:[17975236] (unowned)fd 11: pipe:[17975236] (unowned)fd 12: /dev/v4l-subdev0 (unowned)fd 13: /dev/v4l-subdev1 (unowned)fd 14: /dev/v4l-subdev4 (unowned)fd 15: /dev/v4l-subdev5 (unowned)fd 16: /dev/v4l-subdev6 (unowned)fd 17: /dev/v4l-subdev7 (unowned)fd 18: /dev/v4l-subdev8 (unowned)fd 19: /dev/v4l-subdev9 (unowned)fd 20: /dev/v4l-subdev10 (unowned)fd 21: /dev/v4l-subdev11 (unowned)fd 22: /dev/v4l-subdev12 (unowned)fd 23: /dev/v4l-subdev13 (unowned)fd 24: /dev/v4l-subdev14 (unowned)fd 25: /dev/v4l-subdev15 (unowned)fd 26: /dev/v4l-subdev16 (unowned)fd 27: /dev/v4l-subdev17 (unowned)fd 28: /dev/v4l-subdev18 (unowned)fd 29: /dev/v4l-subdev19 (unowned)fd 30: /dev/v4l-subdev20 (unowned)fd 31: /dev/v4l-subdev21 (unowned)fd 32: /dev/v4l-subdev22 (unowned)fd 33: /dev/v4l-subdev23 (unowned)fd 34: /dev/v4l-subdev24 (unowned)fd 35: /dev/v4l-subdev25 (unowned)fd 36: /dev/v4l-subdev26 (unowned)fd 37: pipe:[17975238] (unowned)fd 38: /dev/video1 (unowned)fd 39: socket:[17996693] (unowned)fd 40: pipe:[17993457] (unowned)fd 41: pipe:[17993457] (unowned)fd 42: socket:[17993459] (unowned)fd 43: pipe:[17993460] (unowned)fd 44: pipe:[17993460] (unowned)fd 45: pipe:[17976879] (unowned)fd 46: pipe:[17976879] (unowned)fd 47: socket:[17976880] (unowned)fd 48: /dev/ashmem (unowned)fd 49: /dev/ashmem (unowned)fd 50: /dev/ion (unowned)fd 51: /dev/kgsl-3d0 (unowned)fd 52: /dev/ion (unowned)fd 53: anon_inode:sync_file (unowned)fd 54: anon_inode:dmabuf5272072 (unowned)fd 55: anon_inode:dmabuf5272073 (unowned)fd 56: anon_inode:dmabuf5272074 (unowned)fd 57: anon_inode:dmabuf5272075 (unowned)fd 58: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x7132c33570)fd 59: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e4299018)fd 60: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x7132c33018)fd 61: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x7132c330b0)fd 62: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e42991e0)fd 63: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e4299278)fd 64: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e4299440)fd 65: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e42994d8)fd 66: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x710ab540b0)fd 67: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x710ab54148)fd 68: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x710ab54310)fd 69: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x710ab543a8)fd 70: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x710ab54570)fd 71: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70caaa3c18)fd 72: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70caaa3de0)fd 73: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70caaa3e78)fd 74: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70caaa4040)fd 75: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70caaa40d8)fd 76: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e28f80b0)fd 77: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e28f8148)fd 78: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e28f8310)fd 79: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e28f83a8)fd 80: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e28f8570)fd 81: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e4299570)fd 82: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e28f9818)fd 83: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e28f99e0)fd 84: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e28f9a78)fd 85: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e28f9c40)fd 86: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e28f9cd8)fd 87: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70f52950b0)fd 88: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70f5295148)fd 89: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70f52951e0)fd 90: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70f5295278)fd 91: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e94dd418)fd 92: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e94dd4b0)fd 93: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e94dd678)fd 94: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70e94dd710)fd 95: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70e94dd8d8)// ... ...    fd 32761: anon_inode:dmabuf5272337 (unowned)fd 32762: anon_inode:dmabuf5272617 (unowned)fd 32763: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x708ddfe4d8)fd 32764: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x708ddfe570)fd 32765: /mnt/vendor/persist/camera/calib_result (owned by FILE* 0x70b7f78418)fd 32766: /mnt/vendor/persist/camera/verify_result (owned by FILE* 0x70b7f784b0)fd 32767: anon_inode:dmabuf5272320 (unowned)