[solved]Auto-tuning a model for remote device using RPCtunner

Hi all,

While running auto tuning i am getting error.

DEBUG:autotvm:No: 1GFLOPS: 0.00/0.00	result: MeasureResult(costs=

(RuntimeError('Traceback (most recent call last):\n [bt] (8) /home/kalyan/libraries/tvm/build /libtvm.so(TVMSynchronize+0x51) [0x7f3a7341c13d]\n [bt] (7) /home/kalyan/libraries/tvm/build /libtvm.so(tvm::runtime::RPCDeviceAPI::StreamSync(DLContext, void*)+0x86) [0x7f3a73472388]\n [bt] (6) /home/kalyan/libraries/tvm/build/libtvm.so(tvm::runtime::RPCClientSession::StreamSync(DLContext, void*)+0x55) [0x7f3a7347b14d]\n [bt] (5) /home/kalyan/libraries/tvm/build /libtvm.so(tvm::runtime::TVMRetValue tvm::runtime::RPCEndpoint::SysCallRemote<DLContext&, void*&>(tvm::runtime::RPCCode, DLContext&, void*&)+0x72) [0x7f3a7347cf2e]\n [bt] (4) /home/kalyan /libraries/tvm/build/libtvm.so(tvm::runtime::TVMRetValue tvm::runtime::PackedFunc::operator()<int, DLContext&, void*&>(int&&, DLContext&, void*&) const+0x100) [0x7f3a7347e8f0]\n [bt] (3) /home/kalyan/libraries/tvm/build/libtvm.so(std::function<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const+0x5a) [0x7f3a72841bf6]\n ',),), error_no=4, all_cost=1.2250032424926758, timestamp=1588791000.1666398)[(‘tile_ic’, [-1, 8]), (‘tile_oc’, [-1, 1]), (‘tile_ow’, [-1, 1]), (‘tile_oh’, 1)],None,3

Can anyone help me in understandingc the issue??

The runtime error essems to happens in StreamSync, but the exact error message was not clear. The first thing to confirm might be to make sure that you can run the remote function and call remote_ctx.sync() without the RPC(to see if that is an rpc problem).

Note that we had a recent RPC protocol update that means you need also need to update your tvm runtime on the remote

@tqchen tvm on both devices are latest and i am still getting the issue

ok, please follow the steps I mentioned above and see if you can get more info

Hi,

@tqchen actually i don’t understand where and what to call.Could you please help me.

thanks

Try to take a look at the example here https://tvm.apache.org/docs/tutorials/cross_compilation_and_rpc.html#sphx-glr-tutorials-cross-compilation-and-rpc-py

@tqchen , Thanks for your reply, the example is working fine and also I have verified resnet50 with same target platform (Jetson Nano in CPU mode ) now i am trying run same example resnet50 along with Auto-tvm and where I see the above crash. Somehow I didn’t get how to use remote_ctx.sync() without RPC. meaning local machine ctx.sync() ? Also see my earlier solved issue (Error while loading params to target in RPC session)

thanks,

kalyan

You can do remote.gpu(0).sync()

hi @tqchen ,

I have tried with remote.cpu(0).sync() i am getting same error as before

File “/home/kalyan/libraries/tvm/python/tvm/_ffi/runtime_ctypes.py”, line 255, in sync check_call(_LIB.TVMSynchronize(self.device_type, self.device_id, None))

File “/home/kalyan/libraries/tvm/python/tvm/_ffi/base.py”, line 329, in check_call raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last): [bt] (8) /home/kalyan/libraries/tvm/build/libtvm.so(TVMSynchronize+0x51) [0x7fb5979aae0d] [bt] (7) /home/kalyan/libraries/tvm/build/libtvm.so(tvm::runtime::RPCDeviceAPI::StreamSync(DLContext, void*)+0x86) [0x7fb597a00a28] [bt] (6) /home/kalyan/libraries/tvm/build/libtvm.so(tvm::runtime::RPCClientSession::StreamSync(DLContext, void*)+0x55) [0x7fb597a097c9] [bt] (5) /home/kalyan/libraries/tvm/build/libtvm.so(tvm::runtime::TVMRetValue tvm::runtime::RPCEndpoint::SysCallRemote<DLContext&, void*&>(tvm::runtime::RPCCode, DLContext&, void*&)+0x72) [0x7fb597a0b5ba] [bt] (4) /home/kalyan/libraries/tvm/build/libtvm.so(tvm::runtime::TVMRetValue tvm::runtime::PackedFunc::operator()<int, DLContext&, void*&>(int&&, DLContext&, void*&) const+0x100) [0x7fb597a0cf7c] [bt] (3) /home/kalyan/libraries/tvm/build/libtvm.so(std::function<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const+0x5a) [0x7fb596de86ca] [bt] (2) /home/kalyan/libraries/tvm/build/libtvm.so(+0x2630a6d) [0x7fb597a04a6d] [bt] (1) /home/kalyan/libraries/tvm/build/libtvm.so(+0x262d6c3) [0x7fb597a016c3] [bt] (0) /home/kalyan/libraries/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4a) [0x7fb596d87ad4] File “/home/kalyan/libraries/tvm/src/runtime/rpc/rpc_endpoint.cc”, line 656 TVMError: Check failed: code == RPCCode: :kReturn: code=1

OK, I get a reproduce, will look into it. Thanks for reporting

https://github.com/apache/incubator-tvm/pull/5538 should fix the problem. Thanks for the help

1 Like

thank you for your support @tqchen. Problem solved with that fix.

Actually i am confused with this line line. remote_ctx.sync() without RPC. I mean how we will get remote ctx without RPC.