Codegen and uTVM integration

[etom42]

Hi all, Two new features in v0.7 roadmap and their integration are highly interesting for us at CEVA. Namely end-to-end uTVM and External code generator (aka codegen). We are looking at compiling networks using TVM and generating code with our implementation of codegen. The generated code is compiled using proprietary compiler targeting CEVA’s proprietary hardware. Finally it needs to be executed on CEVA’s hardware using uTVM, as there is no OS running on the hardware.

Hence, the two features and their integration are highly interesting for us. I believe that the integration of codegen and uTVM are crucial for many edge devices that execute neural-networks and does not need/require OS.

The goal (beside getting it done) is to share via TVM github repo.

[tico]

Hi @etom42,

Great to hear that you are working on this. This is something that I am precisely also looking forward to use and to understand how they could be potentially interact.

First I have been trying to understand their particular use cases and I was also wondering if they can be somehow integrated.

To my understanding, on the one hand the use case of the external code generator feature (or Bring Your Own Codegen - BYOC) is to offload some operators to accelerators using libraries provided by the vendors, while on the other hand uTVM targets baremetal systems where there is no OS, and even no host core at all. However, as I said I was wondering if the BYOC feature can be integrated with uTVM in which there is a host core such as an ARM that offloads specific operators or the entire model on an programmable accelerator running uTVM.

cc @comaniac, @weberlo

[tqchen]

There are two types of integration here. If you are talking about a DSP with a cutomized C dialect backend. Then perhaps directly integrating into target/source would be a better idea.

In the other case where you want to directly compile from a graph, use the bring your own codegen and customized compiler. In both cases, you will likely need a customized runtime.

Note that uTVM itself is quite modular(as a runtime), so it should not be hard to reuse it and combine it with specific runtime module

[etom42]

Thank you @tqchen . We will compile directly from graph and use uTVM modules as needed. We will have more questions and ideas as we progress.