Is there a way to deploy tvm module with python interface?

I’ve noticed in one TVM tutorial (https://docs.tvm.ai/deploy/cpp_deploy.html), we can export a compiled tvm.Module as shared library, and import the .so with cpp code.

Is there a way to have corresponding python API for such deployment?

I’ve made a simple test in which I:

  • renamed libtvm.so (as I don’t want to deploy that)
  • tried to import graph_runtime in python code
from tvm.contrib import graph_runtime

And there goes the undefined symbol error:

OSError: …/tvm/build/libtvm_runtime.so: undefined symbol: _ZTIN3tvm2ir9IRVisitorE

and the symbol is defined in libtvm.so

I’ve tried to change ctypes.CDLL mode to os.RTLD_LAZY, as well as defined TVM_USE_RUNTIME_LIB=1 but none of that works. And I stopped here.

Back to the questions, is there a way to deploy tvm module with python APIs? If so, a tutorial of that is very much appreciated. Thank you!

update

I tried the cpp flow just now (with libtvm.so moved back) and found similar undefined symbol errors, so I guess there might be something wrong with the cpp API as well.

$make lib/cpp_deploy_normal
g++ -std=c++11 -O2 -fPIC -I/home/yunjing.lh/Workspace/tvm/include -I/home/yunjing.lh/Workspace/tvm/3rdparty/dmlc-core/include -I/home/yunjing.lh/Workspace/tvm/3rdparty/dlpack/include -o lib/cpp_deploy_normal cpp_deploy.cc -ltvm_runtime -L/home/yunjing.lh/Workspace/tvm/build -ldl -lpthread
/home/yunjing.lh/Workspace/tvm/build/libtvm_runtime.so: undefined reference to tvm::IRPrinter::Print(tvm::NodeRef const&)' /home/yunjing.lh/Workspace/tvm/build/libtvm_runtime.so: undefined reference totvm::ir::IRVisitor::Visit_(tvm::ir::For const*)’

collect2: error: ld returned 1 exit status

Test Env:

tvm: master 16bed7e670b39e5c6416c3e13efdc69c91c7ac82
gcc: 8.3.1
llvm: 8.0.1
cmake: 3.5.2

This is interesting, libtvm_runtime.so should not have exposed the symbol of IRVisitor, and what you mentioned should work out of box.

Note that the original runtime CMake for runtime script exclude the anything related to the IR. Unfortunately I cannot confirm the case locally, but it would be great if you dig a bit.

Thank you for the great reminder, I’ve once added an ir_pass and accidentally put them in the RUNTIME_SRCS file group. After rolling back the CMakeLists.txt, the whole flow works like a charm.

BTW, I’ve also managed to deploy tvm module with python interface with libtvm_runtime.so only.
An example python script might be useful and is submitted with this PR.