How to deploy two different tvm compiled model in c++ statically?


#1

Hi @srkreddy1238, @tqchen is it possible to deploy and generate c++ API’s with two different tvm compiled models at the same time. Like I have two tvm compiled models for face detection and object detection can inference them same time?
Note: It should be statically not dynamically


#2

Check out

Likely you will want to use the system module. What you want is possible but with a bit of effort. Note that tvm’s graph runtime takes two inputs:

  • graph_json which is the graph json file
  • lib the tvm module containing all the functions needed by the graph.

To deploy two modules together, we somehow need to combine the generated code together to create a single module that contains functions needed by both modules. Then we can create to create two graph runtimes, one for each module.


#3

@tqchen thank you for your suggestion, i am able to deploy single tvm compiled model in both dynamic and static way. For deploying two models in one c++ code any more suggestions or samples above shared links gives idea about model deployment in c++. i need help on deploying two models.


#4

My comment above is for two models. You need to somehow generate a module that contains functions used in two models (in normal c code it could be as simple as linking everything together) and two versions of json


#5

Hi @tqchen Any samples to generate a module that contains functions used in two different models?


#6

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)


#7

@tqchen the symbol name of the model in the .o is tvm_runtime_create, tvm_runtime_run…, so if there are two models such as a.o and b.o, when we link a.o,b.o,runtime.o, main.o to the final executable, the symbol name will be the same and will cause a link error.