One of the challenges with using TVM as an ahead of time compiler is the lack of a simple interface to take a model and put it through the stack. The current solution of writing a python script has a few problems:
- It leads to a lot of code duplication as everyone writes their own bit of python to accomplish the same task in a subtly different way.
- It encourages writing special-cased solutions.
- To even begin writing these python scripts requires a reasonably comprehensive knowledge of the API.
The last point in particular makes TVM fairly inaccessible to anyone who is not interested in becoming a TVM developer. We have produced some python scripts to expose the core functionality of the TVM stack via a command line interface.
Compile takes a model in some format (eg. a .tflite file) and compiles it down to a tvm.Module which is saved to disk as a tar file (containing the graph/lib/params). Here we can set compiler flags and dump intermediates.
Run takes a compiled tvm.Module (as a tar file) and runs it over RPC. Input tensors can be filled with zeros/ones/random or alternatively a .npz file can be passed to initialise the inputs. The outputs can then also be saved as a .npz file. This generic approach lets us run many different types of network instead of just image classification. Additionally, ‘run’ can time the execution and produce a profiling dump via the debug runtime.
Tune performs auto-tuning for a given model. This is the most speculative command as there exist a few special cases in tuning that are still best suited to a python script. However, the intention is to provide a ‘good enough’ route for someone who doesn’t want to develop an intimate understanding of AutoTVM.
A typical workflow would likely consist of the following:
Perform some tuning
tvm tune resnet_v2_50.tflite -o history.log --host=220.127.116.11 --tracker-key=hikey960 --trials=1000 --target=llvm
Compile using the tuning history
tvm compile resnet_v2_50.tflite -o resnet_v2_50.tar --target=llvm --tuning-log=history.log
Run on a remote device
tvm run resnet_v2_50.tar --host=18.104.22.168 --tracker-key=hikey960
Or if you have an image converted into a .npz file
tvm run resnet_v2_50.tar --host=22.214.171.124 --tracker-key=hikey960 -i image.npz -o output.npz
And if you’re just interested in benchmarking
tvm run resnet_v2_50.tar --host=126.96.36.199 --tracker-key=hikey960 --time