[RFC] CoreML Runtime

Development pre-RFC
#1

In this RFC, we would like to propose adding a runtime to load and execute CoreML models from TVM.

Motivation

  • Currently, using CoreML is a defacto standard approach to run inference on iOS. This runtime is useful to obtain a baseline benchmark and compare it with TVM.

  • Using CoreML is the only way to leverage Apple’s Neural Engine on iOS devices. With this runtime, we could extract a subgraph and offload it to CoreML to achieve better performance in future.

How this runtime works

Here is a proposal of how to run CoreML inference via TVM RPC. In this workflow, we assume that a RPC proxy is running somewhere and iOS TVM RPC app is set up on the iOS device beforehand.

  1. Compile CoreML model

    Although there is an API to compile a CoreML model, we don’t use it. It is because the API works only on the device and it makes development hard. We have to use an iOS device (or simulator) even for testing. Instead, we compile a CoreML model with the xcode coremlc command. We can completely test CoreML runtime only on macOS with this approach.

  2. Send a compiled model to the device and launch RPC server

    We pack a compiled CoreML model (modelname.mlmodelc) into the iOS TVM RPC app bundle, and then, launch the RPC server and connect to the RPC proxy. These are automated with xcodebuild test.

  3. Create CoreML module and run inference

    We create a CoreML module with a packed func (tvm.coreml_runtime.create), and run CoreML inference on iOS via the RPC proxy like other RPC usecases.

Implementation

Implementation is available at https://github.com/apache/incubator-tvm/pull/5283.

cc @tqchen @ziheng @FrozenGene

#2

I think leveraging Apple’s Neural Engine is one good motivation (we could add one example how to leverage this). As we have TFLite’s runtime, I think add CoreML runtime is reasonable.

Does this have one version of XCore requires? My XCode is 10.1 but I can not find coremlc and only find coremlcompiler command.

#3

@FrozenGene Sorry, my description is ambiguous. You can compile a CoreML model with the following command:

$(xcode-select -p)/usr/bin/coremlc compile [model.mlmodel] [outputFolder]

or

xcrun coremlcompiler compile [model.mlmodel] [outputFolder]

I tried Xcode 11.4 and 10.3, and I could compile a model on both environment. I think Xcode 10.1 also works.

#4

XCode 10.1 works. (Post must be 20 characters)

1 Like
#5

Indeed, having a coreml runtime support would be quite interesting for our iOS backend