前言

在之前笔者有介绍过《在Android设备上使用PaddleMobile实现图像分类》，使用的框架是百度开源的PaddleMobile。在本章中，笔者将会介绍使用小米的开源手机深度学习框架MACE来实现在Android手机实现图像分类。

MACE的GitHub地址：https://github.com/XiaoMi/mace

编译MACE库和模型

编译MACE库和模型有两种方式，一种是在Ubuntu本地上编译，另一种是使用docker编译。下面就介绍使用这两种编译方式。

使用Ubuntu编译

使用Ubuntu编译源码比较麻烦的是就要自己配置环境，所以下面我们就来配置一下环境。以下是官方给出的环境依赖：

所需依赖

Software	Installation command	Tested version
Python		2.7
Bazel	bazel installation guide	0.13.0
CMake	apt-get install cmake	>= 3.11.3
Jinja2	pip install -I jinja2==2.10	2.10
PyYaml	pip install -I pyyaml==3.12	3.12.0
sh	pip install -I sh==1.12.14	1.12.14
Numpy	pip install -I numpy==1.14.0	Required by model validation
six	pip install -I six==1.11.0	Required for Python 2 and 3 compatibility (TODO)

可选依赖

Software	Installation command	Remark
Android NDK	NDK installation guide	Required by Android build, r15b, r15c, r16b, r17b
ADB	apt-get install android-tools-adb	Required by Android run, >= 1.0.32
TensorFlow	pip install -I tensorflow==1.6.0	Required by TensorFlow model
Docker	docker installation guide	Required by docker mode for Caffe model
Scipy	pip install -I scipy==1.0.0	Required by model validation
FileLock	pip install -I filelock==3.0.0	Required by run on Android

安装依赖环境

• 安装Bazel

export BAZEL_VERSION=0.13.1
mkdir /bazel && \cd /bazel && \wget https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \chmod +x bazel-*.sh && \./bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \cd / && \rm -f /bazel/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh

• 安装Android NDK

# Download NDK r15c
cd /opt/ && \wget -q https://dl.google.com/android/repository/android-ndk-r15c-linux-x86_64.zip && \unzip -q android-ndk-r15c-linux-x86_64.zip && \rm -f android-ndk-r15c-linux-x86_64.zipexport ANDROID_NDK_VERSION=r15c
export ANDROID_NDK=/opt/android-ndk-${ANDROID_NDK_VERSION}
export ANDROID_NDK_HOME=${ANDROID_NDK}# add to PATH
export PATH=${PATH}:${ANDROID_NDK_HOME}

• 安装其他工具

apt-get install -y --no-install-recommends \cmake \android-tools-adb
pip install -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com setuptools
pip install -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com \"numpy>=1.14.0" \scipy \jinja2 \pyyaml \sh==1.12.14 \pycodestyle==2.4.0 \filelock

• 安装TensorFlow

pip install -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com tensorflow==1.6.0

编译库和模型

• 克隆MACE源码

git clone https://github.com/XiaoMi/mace.git

• 进入到官方的Android Demo上

cd mace/mace/examples/android/

• 修改当前目录下的build.sh，修成如下：

#!/usr/bin/env bashset -e -u -o pipefailpushd ../../../TARGET_ABI=armeabi-v7a
LIBRARY_DIR=mace/examples/android/macelibrary/src/main/cpp/
INCLUDE_DIR=$LIBRARY_DIR/include/mace/public/
LIBMACE_DIR=$LIBRARY_DIR/lib/$TARGET_ABI/rm -rf $LIBRARY_DIR/include/
mkdir -p $INCLUDE_DIRrm -rf $LIBRARY_DIR/lib/
mkdir -p $LIBMACE_DIRrm -rf $LIBRARY_DIR/model/python tools/converter.py convert --config=mace/examples/android/mobilenet.yml --target_abis=$TARGET_ABI
cp -rf builds/mobilenet/include/mace/public/*.h $INCLUDE_DIR
cp -rf builds/mobilenet/model $LIBRARY_DIRbazel build --config android --config optimization mace/libmace:libmace_static --define neon=true --define openmp=true --define opencl=true --cpu=$TARGET_ABI
cp -rf mace/public/*.h $INCLUDE_DIR
cp -rf bazel-genfiles/mace/libmace/libmace.a $LIBMACE_DIRpopd

• 修改模型的配置文件mobilenet.yml，修改成如下，这些属性的文件可以查看官方的文档，各个模型的配置可以参考Mobile Model Zoo下的各个模型，以下是以为MobileNet V2为例。

library_name: mobilenet
target_abis: [armeabi-v7a]
model_graph_format: code
model_data_format: code
models:mobilenet_v2:platform: tensorflowmodel_file_path: https://cnbj1.fds.api.xiaomi.com/mace/miai-models/mobilenet-v2/mobilenet-v2-1.0.pbmodel_sha256_checksum: 369f9a5f38f3c15b4311c1c84c032ce868da9f371b5f78c13d3ea3c537389bb4subgraphs:- input_tensors:- inputinput_shapes:- 1,224,224,3output_tensors:- MobilenetV2/Predictions/Reshape_1output_shapes:- 1,1001runtime: cpu+gpulimit_opencl_kernel_time: 0nnlib_graph_mode: 0obfuscate: 0winograd: 0

• 开始编译

./build.sh

• 编译完成之后，可以在mace/mace/examples/android/macelibrary/src/main/cpp/看到多了3个文件：

1. include是存放调用mace接口和模型配置的头文件
2. lib是存放编译好的mace库
3. model是存放模型的文件夹，比如我们编译的MobileNet V2模型

使用Docker编译

• 首先安装docker，命令如下：

apt-get install docker.io

• 拉取mace镜像：

docker pull registry.cn-hangzhou.aliyuncs.com/xiaomimace/mace-dev

• 获取MACE源码，并按照上一步修改mace/mace/examples/android/目录下的build.sh和mobilenet.yml这个两个文件。

git clone https://github.com/XiaoMi/mace.git

• 进入到MACE的根目录，执行以下命令：

docker run -it -v $PWD:/mace registry.cn-hangzhou.aliyuncs.com/xiaomimace/mace-dev

• 接着执行以下命令：

cd mace/mace/examples/android/
./build.sh

执行之后便可得到跟上一步获取的一样的文件。使用docker就简单很多，少了很多安装依赖环境的步骤。

开发Android项目

• 创建Android项目

在创建项目是要选择C++支持。

因为MACE最低支持版本是Android5.0，所以这里要选择Android5.0。

MACE使用的是C++11。

• 复制C++文件。删除cpp目录下自动生产的C++文件，并复制上一步编译得到的3个目录和本来就有的两C++文件到Android项目的cpp目录下。如下图：
  

• 修改CMakeLists.txt编译文件，修改如下，编译对应的是我们上一步复制的C++文件：

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html# Sets the minimum version of CMake required to build the native library.cmake_minimum_required(VERSION 3.4.1)# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.#set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/../app/libs/${ANDROID_ABI})include_directories(${CMAKE_SOURCE_DIR}/)
include_directories(${CMAKE_SOURCE_DIR}/src/main/cpp/include)
set(mace_lib ${CMAKE_SOURCE_DIR}/src/main/cpp/lib/armeabi-v7a/libmace.a)
set(mobilenet_lib ${CMAKE_SOURCE_DIR}/src/main/cpp/model/armeabi-v7a/mobilenet.a)
add_library (mace_lib STATIC IMPORTED)
set_target_properties(mace_lib PROPERTIES IMPORTED_LOCATION ${mace_lib})
add_library (mobilenet_lib STATIC IMPORTED)
set_target_properties(mobilenet_lib PROPERTIES IMPORTED_LOCATION ${mobilenet_lib})add_library( # Sets the name of the library.mace_mobile_jni# Sets the library as a shared library.SHARED# Provides a relative path to your source file(s).src/main/cpp/image_classify.cc )# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.find_library( # Sets the name of the path variable.log-lib# Specifies the name of the NDK library that# you want CMake to locate.log )# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.target_link_libraries( # Specifies the target library.mace_mobile_jnimace_libmobilenet_lib# Links the target library to the log library# included in the NDK.${log-lib} )

• 修改app目录下的build.gradle，修改如下：

把原来的

externalNativeBuild {cmake {cppFlags "-std=c++11"}}

修改成，因为我们只编译了armeabi-v7a支持：

externalNativeBuild {cmake {cppFlags "-std=c++11 -fopenmp"abiFilters "armeabi-v7a"}}

在android下加上：

    sourceSets {main {jniLibs.srcDirs = ["src/main/jniLibs"]jni.srcDirs = ['src/cpp']}}

• 修改Android项目使用的NDK版本，我们编译的时候是使用r15c，所以我们在Android项目上也要使用r15c，如下：
  

• 创建一个com.xiaomi.mace包，并复制官方demo中的java类JniMaceUtils.java到该包中，代码如下，这个就是使用mace的JNI接口：

package com.xiaomi.mace;public class JniMaceUtils {static {System.loadLibrary("mace_mobile_jni");}// 设置模型属性public static native int maceMobilenetSetAttrs(int ompNumThreads, int cpuAffinityPolicy, int gpuPerfHint, int gpuPriorityHint, String kernelPath);// 加载模型和选择使用GPU或CPUpublic static native int maceMobilenetCreateEngine(String model, String device);// 预测图片public static native float[] maceMobilenetClassify(float[] input);
}

• 在项目的包下创建一个InitData.java类，这个是配置mace的信息类，比如使用CPU还是GPU来预测，加载的是那个模型等等：

package com.example.myapplication;import android.os.Environment;import java.io.File;public class InitData {public static final String[] DEVICES = new String[]{"CPU", "GPU"};public static final String[] MODELS = new String[]{"mobilenet_v1", "mobilenet_v2"};private String model;private String device = "";private int ompNumThreads;private int cpuAffinityPolicy;private int gpuPerfHint;private int gpuPriorityHint;private String kernelPath = "";public InitData() {model = MODELS[1];ompNumThreads = 4;cpuAffinityPolicy = 0;gpuPerfHint = 3;gpuPriorityHint = 3;device = DEVICES[0];kernelPath = Environment.getExternalStorageDirectory().getAbsolutePath() + File.separator + "mace";File file = new File(kernelPath);if (!file.exists()) {file.mkdir();}}public String getModel() {return model;}public void setModel(String model) {this.model = model;}public String getDevice() {return device;}public void setDevice(String device) {this.device = device;}public int getOmpNumThreads() {return ompNumThreads;}public void setOmpNumThreads(int ompNumThreads) {this.ompNumThreads = ompNumThreads;}public int getCpuAffinityPolicy() {return cpuAffinityPolicy;}public void setCpuAffinityPolicy(int cpuAffinityPolicy) {this.cpuAffinityPolicy = cpuAffinityPolicy;}public int getGpuPerfHint() {return gpuPerfHint;}public void setGpuPerfHint(int gpuPerfHint) {this.gpuPerfHint = gpuPerfHint;}public int getGpuPriorityHint() {return gpuPriorityHint;}public void setGpuPriorityHint(int gpuPriorityHint) {this.gpuPriorityHint = gpuPriorityHint;}public String getKernelPath() {return kernelPath;}public void setKernelPath(String kernelPath) {this.kernelPath = kernelPath;}
}

• 同样是在项目的包下创建PhotoUtil.java类，这是一个工具类，包括启动相机获拍摄图片并返回该图片的绝对路径，还有一个是把图片转换成预测的数据，mace读取的预测数据是一个float数组。

package com.example.myapplication;import android.app.Activity;
import android.content.Context;
import android.content.Intent;
import android.database.Cursor;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.net.Uri;
import android.os.Build;
import android.provider.MediaStore;
import android.support.v4.content.FileProvider;import java.io.File;
import java.io.IOException;
import java.nio.FloatBuffer;public class PhotoUtil {// start camerapublic static Uri start_camera(Activity activity, int requestCode) {Uri imageUri;// save image in cache pathFile outputImage = new File(activity.getExternalCacheDir(), "out_image.jpg");try {if (outputImage.exists()) {outputImage.delete();}outputImage.createNewFile();} catch (IOException e) {e.printStackTrace();}if (Build.VERSION.SDK_INT >= 24) {// compatible with Android 7.0 or overimageUri = FileProvider.getUriForFile(activity,"com.example.myapplication", outputImage);} else {imageUri = Uri.fromFile(outputImage);}// set system camera ActionIntent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);// set save photo pathintent.putExtra(MediaStore.EXTRA_OUTPUT, imageUri);// set photo quality, min is 0, max is 1intent.putExtra(MediaStore.EXTRA_VIDEO_QUALITY, 0);activity.startActivityForResult(intent, requestCode);return imageUri;}// get picture in photopublic static void use_photo(Activity activity, int requestCode){Intent intent = new Intent(Intent.ACTION_PICK);intent.setType("image/*");activity.startActivityForResult(intent, requestCode);}// get photo from Uripublic static String get_path_from_URI(Context context, Uri uri) {String result;Cursor cursor = context.getContentResolver().query(uri, null, null, null, null);if (cursor == null) {result = uri.getPath();} else {cursor.moveToFirst();int idx = cursor.getColumnIndex(MediaStore.Images.ImageColumns.DATA);result = cursor.getString(idx);cursor.close();}return result;}// Compress the image to the size of the training imagepublic static float[] getScaledMatrix(Bitmap bitmap, int desWidth,int desHeight) {// create data bufferfloat[] floatValues = new float[desWidth * desHeight * 3];FloatBuffer floatBuffer = FloatBuffer.wrap(floatValues, 0, desWidth * desHeight * 3);floatBuffer.rewind();// get image pixelint[] pixels = new int[desWidth * desHeight];Bitmap bm = Bitmap.createScaledBitmap(bitmap, desWidth, desHeight, false);bm.getPixels(pixels, 0, bm  .getWidth(), 0, 0, desWidth, desHeight);// pixel to datafor (int clr : pixels) {floatBuffer.put((((clr >> 16) & 0xFF) - 128f) / 128f);floatBuffer.put((((clr >> 8) & 0xFF) - 128f) / 128f);floatBuffer.put(((clr & 0xFF) - 128f) / 128f);}if (bm.isRecycled()) {bm.recycle();}return floatBuffer.array();}// compress picturepublic static Bitmap getScaleBitmap(String filePath) {BitmapFactory.Options opt = new BitmapFactory.Options();opt.inJustDecodeBounds = true;BitmapFactory.decodeFile(filePath, opt);int bmpWidth = opt.outWidth;int bmpHeight = opt.outHeight;int maxSize = 500;// compress picture with inSampleSizeopt.inSampleSize = 1;while (true) {if (bmpWidth / opt.inSampleSize < maxSize || bmpHeight / opt.inSampleSize < maxSize) {break;}opt.inSampleSize *= 2;}opt.inJustDecodeBounds = false;return BitmapFactory.decodeFile(filePath, opt);}
}

• 修改MainActivity.java，修改如下，主要是有两个功能，第一个是打开相册选择图片进行预测，另一个是启动相机拍摄图片进行预测。在进入应用是就调用init_model()方法来设置mace的配置信息和加载模型，其中可以通过调用load_model(String model)该更换模型。通过调用predict_image(String image_path)方法预测图片并显示结果：

package com.example.myapplication;import android.Manifest;
import android.app.Activity;
import android.content.Intent;
import android.content.pm.PackageManager;
import android.content.res.AssetManager;
import android.graphics.Bitmap;
import android.net.Uri;
import android.os.Bundle;
import android.support.annotation.NonNull;
import android.support.annotation.Nullable;
import android.support.v4.app.ActivityCompat;
import android.support.v4.content.ContextCompat;
import android.support.v7.app.AppCompatActivity;
import android.text.method.ScrollingMovementMethod;
import android.util.Log;
import android.view.View;
import android.widget.Button;
import android.widget.ImageView;
import android.widget.TextView;
import android.widget.Toast;import com.bumptech.glide.Glide;
import com.bumptech.glide.load.engine.DiskCacheStrategy;
import com.bumptech.glide.request.RequestOptions;
import com.xiaomi.mace.JniMaceUtils;import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;public class MainActivity extends AppCompatActivity {private static final String TAG = MainActivity.class.getName();private static final int USE_PHOTO = 1001;private static final int START_CAMERA = 1002;private Uri camera_image_path;private ImageView show_image;private TextView result_text;private boolean load_result = false;private int[] ddims = {1, 3, 224, 224};private int model_index = 1;private InitData initData = new InitData();private List<String> resultLabel = new ArrayList<>();private static final String[] PADDLE_MODEL = {"mobilenet_v1","mobilenet_v2"};@Overrideprotected void onCreate(Bundle savedInstanceState) {super.onCreate(savedInstanceState);setContentView(R.layout.activity_main);init_view();init_model();readCacheLabelFromLocalFile();}// initialize viewprivate void init_view() {request_permissions();show_image = (ImageView) findViewById(R.id.show_image);result_text = (TextView) findViewById(R.id.result_text);result_text.setMovementMethod(ScrollingMovementMethod.getInstance());Button use_photo = (Button) findViewById(R.id.use_photo);Button start_photo = (Button) findViewById(R.id.start_camera);// use photo clickuse_photo.setOnClickListener(new View.OnClickListener() {@Overridepublic void onClick(View view) {if (!load_result) {Toast.makeText(MainActivity.this, "never load model", Toast.LENGTH_SHORT).show();return;}PhotoUtil.use_photo(MainActivity.this, USE_PHOTO);}});// start camera clickstart_photo.setOnClickListener(new View.OnClickListener() {@Overridepublic void onClick(View view) {if (!load_result) {Toast.makeText(MainActivity.this, "never load model", Toast.LENGTH_SHORT).show();return;}camera_image_path = PhotoUtil.start_camera(MainActivity.this, START_CAMERA);}});}// init mace environmentprivate void init_model() {int result = JniMaceUtils.maceMobilenetSetAttrs(initData.getOmpNumThreads(), initData.getCpuAffinityPolicy(),initData.getGpuPerfHint(), initData.getGpuPriorityHint(),initData.getKernelPath());Log.i(TAG, "maceMobilenetSetAttrs result = " + result);load_model(PADDLE_MODEL[model_index]);}// load infer modelprivate void load_model(String model) {// set will load model nameinitData.setModel(model);// load modelint result = JniMaceUtils.maceMobilenetCreateEngine(initData.getModel(), initData.getDevice());Log.i(TAG, "maceMobilenetCreateEngine result = " + result);// set load model resultload_result = result == 0;if (load_result) {Toast.makeText(MainActivity.this, model + " model load success", Toast.LENGTH_SHORT).show();Log.d(TAG, model + " model load success");} else {Toast.makeText(MainActivity.this, model + " model load fail", Toast.LENGTH_SHORT).show();Log.d(TAG, model + " model load fail");}}private void readCacheLabelFromLocalFile() {try {AssetManager assetManager = getApplicationContext().getAssets();BufferedReader reader = new BufferedReader(new InputStreamReader(assetManager.open("cacheLabel.txt")));String readLine = null;while ((readLine = reader.readLine()) != null) {resultLabel.add(readLine);}reader.close();} catch (Exception e) {Log.e("labelCache", "error " + e);}}@Overrideprotected void onActivityResult(int requestCode, int resultCode, @Nullable Intent data) {String image_path;RequestOptions options = new RequestOptions().skipMemoryCache(true).diskCacheStrategy(DiskCacheStrategy.NONE);if (resultCode == Activity.RESULT_OK) {switch (requestCode) {case USE_PHOTO:if (data == null) {Log.w(TAG, "user photo data is null");return;}Uri image_uri = data.getData();Glide.with(MainActivity.this).load(image_uri).apply(options).into(show_image);// get image path from uriimage_path = PhotoUtil.get_path_from_URI(MainActivity.this, image_uri);// predict imagepredict_image(image_path);break;case START_CAMERA:// show photoGlide.with(MainActivity.this).load(camera_image_path).apply(options).into(show_image);image_path = PhotoUtil.get_path_from_URI(MainActivity.this, camera_image_path);// predict imagepredict_image(image_path);break;}}}//  predict imageprivate void predict_image(String image_path) {// picture to float arrayBitmap bmp = PhotoUtil.getScaleBitmap(image_path);float[] inputData = PhotoUtil.getScaledMatrix(bmp, ddims[2], ddims[3]);try {// Data format conversion takes too long// Log.d("inputData", Arrays.toString(inputData));long start = System.currentTimeMillis();// get predict resultfloat[] result = JniMaceUtils.maceMobilenetClassify(inputData);long end = System.currentTimeMillis();Log.d(TAG, "origin predict result:" + Arrays.toString(result));long time = end - start;Log.d("result length", String.valueOf(result.length));// show predict result and timeint r = get_max_result(result);String show_text = "result：" + r + "\nname：" + resultLabel.get(r) + "\nprobability：" + result[r] + "\ntime：" + time + "ms";result_text.setText(show_text);} catch (Exception e) {e.printStackTrace();}}// get max probability labelprivate int get_max_result(float[] result) {float probability = result[0];int r = 0;for (int i = 0; i < result.length; i++) {if (probability < result[i]) {probability = result[i];r = i;}}return r;}// request permissionsprivate void request_permissions() {List<String> permissionList = new ArrayList<>();if (ContextCompat.checkSelfPermission(this, Manifest.permission.CAMERA) != PackageManager.PERMISSION_GRANTED) {permissionList.add(Manifest.permission.CAMERA);}if (ContextCompat.checkSelfPermission(this, Manifest.permission.WRITE_EXTERNAL_STORAGE) != PackageManager.PERMISSION_GRANTED) {permissionList.add(Manifest.permission.WRITE_EXTERNAL_STORAGE);}if (ContextCompat.checkSelfPermission(this, Manifest.permission.READ_EXTERNAL_STORAGE) != PackageManager.PERMISSION_GRANTED) {permissionList.add(Manifest.permission.READ_EXTERNAL_STORAGE);}// if list is not empty will request permissionsif (!permissionList.isEmpty()) {ActivityCompat.requestPermissions(this, permissionList.toArray(new String[permissionList.size()]), 1);}}@Overridepublic void onRequestPermissionsResult(int requestCode, @NonNull String[] permissions, @NonNull int[] grantResults) {super.onRequestPermissionsResult(requestCode, permissions, grantResults);switch (requestCode) {case 1:if (grantResults.length > 0) {for (int i = 0; i < grantResults.length; i++) {int grantResult = grantResults[i];if (grantResult == PackageManager.PERMISSION_DENIED) {String s = permissions[i];Toast.makeText(this, s + " permission was denied", Toast.LENGTH_SHORT).show();}}}break;}}
}

• 在main下创建一个asset目录并加入这个文件

• 最后别忘了在配置文件AndroidManifest.xml上加上权限

<uses-permission android:name="android.permission.CAMERA"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>

最后运行得到的结果如下图：

注意：该项目对Android7.0相机兼容不是很好。
源码下载： 上面已经是全部代码了，如果读者想更方便使用，可以直接下载该项目

参考资料

1. https://github.com/XiaoMi/mace
2. https://mace.readthedocs.io/en/latest/