Change number of threads at runtime

#1

TVM runtime automatically determines number of threads to use based on number of Big cores. Also number of threads to use can be specified by env var TVM_NUM_THREADS.
But I’m not sure we can change number of threads while the program is running.

That is different from TFLite. TFLite Interpreter API allows to set number of threads at any time while the inference is running.
https://www.tensorflow.org/lite/api_docs/cc/class/tflite/interpreter#classtflite_1_1_interpreter_1ac6d9205e23cfe0a77ed2e69005bb7ec1

#2

Would this help?

BTW, what is a typical use case that requires one changes the number of threads on the fly instead of restarting the program?

1 Like
#3

We have Android app which uses TVM runtime
One of the control on the form allows to change number of threads.
It works fine for TFLite runtime but not for TVM Runtime
See Example:

1 Like
#4

I think @yidawang’s answer should work, because we have done similar things through that api.

#5

Not sure what API are you talking about.
What the app should do after user clicks + or -? Terminate itself, set env var and relaunch?

#6

config_threadpool.
When you click - +, you should reconfigure the thread nums via config_threadpool and then reload the .so again. That is to say, you doesn’t need relauch app / libtvm_runtime.so, but you should reconfigure the threads firstly and then reload deploy.so.