Watchdog是什么
Watchdog是android framework中一个java类(在线源码), 也可以认为是一种系统检查和处理的机制。比如在Android系统中,SystemServer进程会对应用程序进行卡顿检测和处理(即ANR检测等),那么谁来检测和处理SystemServer进程的服务呢?如果AMS、PMS等核心服务出现了卡死谁来处理呢?答案是Watchdog,在Android中,设计了Watchdog机制来检测和处理系统核心服务是否能正常工作。
Watchdog怎么工作
Watchdog作为系统服务的检查者,在系统启动过程的较早阶段就已经启动了。
启动Watchdog
#SystemServer.java
private void startBootstrapServices() {// 当前方法在android系统启动后首先调用,主要作用是启动系统基础服务,而Watchdog是在启动基础服务之前就启动了final Watchdog watchdog = Watchdog.getInstance();watchdog.start();//.....}
#Watchdog.java
public class Watchdog extends Thread {static Watchdog sWatchdog;static final long DEFAULT_TIMEOUT = DB ? 10*1000 : 60*1000;final ArrayList<HandlerChecker> mHandlerCheckers = new ArrayList<>();final HandlerChecker mMonitorChecker;public static Watchdog getInstance() {if (sWatchdog == null) {//单例模式,说明Watchdog在SystemServer进程内只有一个实例 sWatchdog = new Watchdog();}return sWatchdog;}private Watchdog() {super("watchdog");//为每一个我们想检测的线程初始化HandlerChecker对象,该对象是Runnable的子类// 注意,此处并不会检测后台线程,因为后台线程可能会运行很长时间,无法保证及时性// 创建监视器检查者,这里加入的是前台线程的Handler, DEFAULT_TIMEOUT为60smMonitorChecker = new HandlerChecker(FgThread.getHandler(),"foreground thread", DEFAULT_TIMEOUT);//将监视器检查者加入队列 mHandlerCheckers.add(mMonitorChecker);// 添加主线程的检查者mHandlerCheckers.add(new HandlerChecker(new Handler(Looper.getMainLooper()),"main thread", DEFAULT_TIMEOUT));// 添加UI线程检查者mHandlerCheckers.add(new HandlerChecker(UiThread.getHandler(),"ui thread", DEFAULT_TIMEOUT));// 添加IO线程检查者mHandlerCheckers.add(new HandlerChecker(IoThread.getHandler(),"i/o thread", DEFAULT_TIMEOUT));// 添加Display线程检查者mHandlerCheckers.add(new HandlerChecker(DisplayThread.getHandler(),"display thread", DEFAULT_TIMEOUT));// 添加动画线程检查者mHandlerCheckers.add(new HandlerChecker(AnimationThread.getHandler(),"animation thread", DEFAULT_TIMEOUT));// 添加窗口动画线程检查者mHandlerCheckers.add(new HandlerChecker(SurfaceAnimationThread.getHandler(),"surface animation thread", DEFAULT_TIMEOUT));// 创建并添加Binder线程检查者addMonitor(new BinderThreadMonitor());//文件相关mOpenFdMonitor = OpenFdMonitor.create();}public void addMonitor(Monitor monitor) {synchronized (this) {mMonitorChecker.addMonitorLocked(monitor);}}
}
整理下Watchdog的创建源码,其重点如下:
- Watchdog继承自Thread,本质上是线程
- 在Watchdog是单例,该对象创建时添加了7个HandlerChecker
- 将BinderThreadMonitor加入了mMonitorChecker对象
那么HandlerChecker和Monitor到底是个什么东西呢 ?
#Watchdog.java
public final class HandlerChecker implements Runnable {private final Handler mHandler; //当前检查者工作使用的Handlerprivate final String mName; //当前检查者名字private final long mWaitMax; //最大等待时间private final ArrayList<Monitor> mMonitors = new ArrayList<Monitor>();private final ArrayList<Monitor> mMonitorQueue = new ArrayList<Monitor>();private boolean mCompleted; //当前检查是否已完成private Monitor mCurrentMonitor; //当前正在执行的监控器private long mStartTime; //检测的开始时间private int mPauseCount; //暂停次数HandlerChecker(Handler handler, String name, long waitMaxMillis) {mHandler = handler;mName = name;mWaitMax = waitMaxMillis;mCompleted = true;}void addMonitorLocked(Monitor monitor) {//这里先将监控器添加到mMonitorQueue集合而不是mMonitors集合,主要是为了安全,不想在mMonitors集合中的Monitor正在执行的时候更新集合,mMonitorQueue.add(monitor);}}public interface Monitor {void monitor();}
现在我们大致知道Watchdog、HandleChecker和Monitor分别是什么了,接下来看下watchdog.start(),watchdog是一个线程,所以直接到其run方法
#Watchdog.javapublic void run() {boolean waitedHalf = false;//无限循环while (true) {final List<HandlerChecker> blockedCheckers;final String subject;final boolean allowRestart;//是否允许重新启动系统int debuggerWasConnected = 0;//debug链接数量synchronized (this) {long timeout = CHECK_INTERVAL; //检测周期:debug下为5s,正式版本为30sfor (int i=0; i<mHandlerCheckers.size(); i++) {HandlerChecker hc = mHandlerCheckers.get(i);//调用对应HandlerChecker的scheduleCheckLocked方法,该方法会触发真正的检查,稍后分析hc.scheduleCheckLocked();}//...long start = SystemClock.uptimeMillis();//此处timeout > 0表明后续代码一定会在检测周期过后才会触发while (timeout > 0) {try {wait(timeout);} catch (InterruptedException e) {Log.wtf(TAG, e);}timeout = CHECK_INTERVAL - (SystemClock.uptimeMillis() - start);}boolean fdLimitTriggered = false;if (mOpenFdMonitor != null) {fdLimitTriggered = mOpenFdMonitor.monitor();}if (!fdLimitTriggered) {final int waitState = evaluateCheckerCompletionLocked();//评估HandlerChecker检查者的状态//处理检测结果,稍后分析}}
Watchdog的run方法大致做了如下事情:
- 遍历检查者集合,触发每一个HandlerChecker的scheduleCheckLocked来开启检查
- 等待一个检查周期后,开始获取检查结果
- 根据检查结果进行处理
接下来我们详细看下watchdog的run方法的三个流程:
1.发起检测
public void scheduleCheckLocked() {//mCompleted默认是trueif (mCompleted) {// 这里把mMonitorQueue集合中的数据添加搭配mMonitor中,因为此时HandlerChecker不在工作队列中,这样不会因为多线程操作触发快速失效机制mMonitors.addAll(mMonitorQueue);mMonitorQueue.clear();}if ((mMonitors.size() == 0 && mHandler.getLooper().getQueue().isPolling())|| (mPauseCount > 0)) {//没有需要监视的服务,或者当前MessageQueue正在poll时,不用检查mCompleted = true;return;}if (!mCompleted) {// 不用重复检查return;}mCompleted = false;mCurrentMonitor = null;mStartTime = SystemClock.uptimeMillis();mHandler.postAtFrontOfQueue(this);//将当前HandlerChecker添加到mHandler所属MessageQueue队列中头}
scheduleCheckLocked方法做了两件事:
- 将新添加的Monitor转移到mMonitor集合中
- 重置状态,并向持有的mHandler中post一个Runnable(HandlerChecker自身)。由消息机制可以理解其会在mHandler所属线程执行当前HandlerChecker的run方法,如下
public void run() {//遍历mMonitors集合中每一个元素final int size = mMonitors.size();for (int i = 0 ; i < size ; i++) {synchronized (Watchdog.this) {mCurrentMonitor = mMonitors.get(i);}//执行其monitor方法来检测,由于其是在HandlerCheck的run方法中调用的,所以是运行在Handler对应的线程中的。mCurrentMonitor.monitor();}synchronized (Watchdog.this) {mCompleted = true;mCurrentMonitor = null;}}
由源码分析可得
- HandlerChecker的run方法在Handler所属线程执行,其执行内容为遍历每个Monitor,执行monitor方法
- 所有Monitor检查完了之后,设置mCompleted为true,表示完成检查
2 获取检查结果
由Watchdog的run方法得知,检查结果主要是获取evaluateCheckerCompletionLocked方法的返回值,如下
#HandlerCheckerstatic final int COMPLETED = 0;static final int WAITING = 1;static final int WAITED_HALF = 2;static final int OVERDUE = 3;private int evaluateCheckerCompletionLocked() {int state = COMPLETED;for (int i=0; i<mHandlerCheckers.size(); i++) {HandlerChecker hc = mHandlerCheckers.get(i);//取得当前HandlerChecker集合中state最大值state = Math.max(state, hc.getCompletionStateLocked());}return state;}public int getCompletionStateLocked() {//在检测开始时mCompleted为false,检查结束时为trueif (mCompleted) {return COMPLETED;} else {long latency = SystemClock.uptimeMillis() - mStartTime;if (latency < mWaitMax/2) {//如果当前耗费时间小于最大等待时长的一半,返回WAITINGreturn WAITING;} else if (latency < mWaitMax) {//如果当前耗费时间大于等于最大等待时长的一半,且小于最大等待时长,返回WAITED_HALFreturn WAITED_HALF;}}return OVERDUE;//表示已经超时了,即阻塞或者挂起了}
由源码可知,检查结果一共有4种:
- COMPLETED:表示HandlerChecker检测完成,本次没有阻塞
- WAITING:表示本次检测还未完成,当前耗时小于最大耗时的一半
- WAITED_HALF:表示本次检测还未完成,当前耗时大于最大耗时的一半
- OVERDUE: 表示本次检测超时了
那么Watchdog针对上述4种检测结果,会如何处理呢?
3 处理检测结果
处理检测结果的代码在Watchdog的run方法中,如下:
#Watchdog.javapublic void run() {boolean waitedHalf = false;//无限循环while (true) {final List<HandlerChecker> blockedCheckers;final String subject;final boolean allowRestart;//是否允许重新启动系统int debuggerWasConnected = 0;//debug链接数量synchronized (this) {//... 检测阶段,前面分析过了if (!fdLimitTriggered) {final int waitState = evaluateCheckerCompletionLocked();//评估HandlerChecker检查者的状态if (waitState == COMPLETED) {// 全部完成,则认为一切工作正常,本次检查结束 重置waitedHalf为falsewaitedHalf = false;continue;//开始下一轮检测} else if (waitState == WAITING) {// 在等待状态,当前耗时小于最大限制的一半,continue;//再检查一次} else if (waitState == WAITED_HALF) {if (!waitedHalf) {//在等待状态,当前耗时大于最大限制的一半,但还未超时ArrayList<Integer> pids = new ArrayList<Integer>();pids.add(Process.myPid());//通过AMS保存当前进程的堆栈信息ActivityManagerService.dumpStackTraces(pids, null, null,getInterestingNativePids());waitedHalf = true;}continue;//在检查一次}// 走到这里表明一些检查器已经检查出问题了blockedCheckers = getBlockedCheckersLocked();//获取已阻塞的那些HandlerCheckersubject = describeCheckersLocked(blockedCheckers);//构建提示字符串} else {}allowRestart = mAllowRestart;}//代码走到这里,表示系统大概率已挂起了ArrayList<Integer> pids = new ArrayList<>();pids.add(Process.myPid());if (mPhonePid > 0) pids.add(mPhonePid);通过AMS保存再次当前进程的堆栈信息final File stack = ActivityManagerService.dumpStackTraces(pids, null, null, getInterestingNativePids());//等待5s,为了确认堆栈信息都被写入SystemClock.sleep(5000);// 触发内核以转储所有阻塞的线程,并将所有CPU上的回溯到内核日志doSysRq('w');doSysRq('l');// ...dropbox相关IActivityController controller;synchronized (this) {controller = mController;}if (controller != null) {//将阻塞状态报告给activity controller,try {Binder.setDumpDisabled("Service dumps disabled due to hung system process.");//返回值为1表示继续等待,-1表示杀死系统int res = controller.systemNotResponding(subject);if (res >= 0) {waitedHalf = false;continue; //设置ActivityController的某些情况下,可以让发生Watchdog时继续等待}} catch (RemoteException e) {}}//当debugger没有attach时,才杀死进程if (Debug.isDebuggerConnected()) {debuggerWasConnected = 2;}if (debuggerWasConnected >= 2) {Slog.w(TAG, "Debugger connected: Watchdog is *not* killing the system process");} else if (debuggerWasConnected > 0) {Slog.w(TAG, "Debugger was connected: Watchdog is *not* killing the system process");} else if (!allowRestart) {Slog.w(TAG, "Restart not allowed: Watchdog is *not* killing the system process");} else {Slog.w(TAG, "*** WATCHDOG KILLING SYSTEM PROCESS: " + subject);//遍历输出阻塞线程的栈信息for (int i=0; i<blockedCheckers.size(); i++) {Slog.w(TAG, blockedCheckers.get(i).getName() + " stack trace:");StackTraceElement[] stackTrace= blockedCheckers.get(i).getThread().getStackTrace();for (StackTraceElement element: stackTrace) {Slog.w(TAG, " at " + element);}}Slog.w(TAG, "*** GOODBYE!");//杀死进程system_serverProcess.killProcess(Process.myPid());System.exit(10);}waitedHalf = false;}}}private ArrayList<HandlerChecker> getBlockedCheckersLocked() {ArrayList<HandlerChecker> checkers = new ArrayList<HandlerChecker>();for (int i=0; i<mHandlerCheckers.size(); i++) {HandlerChecker hc = mHandlerCheckers.get(i);if (hc.isOverdueLocked()) {//统计超时那些checkercheckers.add(hc);}}return checkers;}
处理检测结果也是针对4种不同检测结果分开进行的:
- COMPLETED:继续下一轮检测
- WAITING:继续下一轮检测
- WAITED_HALF:dump堆栈,并继续下一轮检测
- OVERDUE: 超时,dump堆栈和内核线程,如果没有特例,则杀掉进程并重启
有细心的小伙伴
总结与反思
Watchdog是系统检测的最后保障,其设计采用了单线程+多个HanlderChecker的结构,流程上主要包含启动监控检测、添加监控、获取结果并处理。其巧妙的地方在于把检测过程的耗时分摊到了各Handler对应的工作线程,保证了Watchdog检测阶段的性能。该方案也可稍作变通,以解决其他问题,比如检测卡顿