Currently in TVM/NNVM automatic differentiation is implemented only at the NNVM level and it requires manually defining gradients for every operation. If there was automatic differentiation on the TVM level then the gradients could be derived automatically from the FTVMCompute function.
We have implemented a proof-of-concept automatic differentiation for TVM. It performs reverse mode differentiation of a Tensor with respect to some other given Tensor. Currently our implementation is very slow because it lacks certain important optimizations consisting in transforming the domain of iteration to avoid summing over lots of zeros. We are considering to implement these optimizations as a separate IR pass.
So, what do you think?