Hello,

Currently in TVM/NNVM automatic differentiation is implemented only at the NNVM level and it requires manually defining gradients for every operation. If there was automatic differentiation on the TVM level then the gradients could be derived automatically from the FTVMCompute function.

Some competitors already have this feature, e.g. PlaidML, and recently Halide.

We have implemented a proof-of-concept automatic differentiation for TVM. It performs reverse mode differentiation of a Tensor with respect to some other given Tensor. Currently our implementation is very slow because it lacks certain important optimizations consisting in transforming the domain of iteration to avoid summing over lots of zeros. We are considering to implement these optimizations as a separate IR pass.

The proof-of-concept implementation is on GitHub. The main source file is here. Some usage examples can be found in this test.

So, what do you think?