TVM tuning interruption errors

Troubleshooting
#1

Hi TVM community,

I am trying to use TVM auto-tuning to tune a model in Mali-GPU. I am following this tutorial to do so: https://docs.tvm.ai/tutorials/autotvm/tune_nnvm_mobile_gpu.html

I set it up RPC tracker and used a desktop server a host. However, the tuning always interupted for some reason. My host server gave me an error when the tuning was interrupted: Error in RPC Tracker: Connection reset by peer. My host server is wired connected with Internet, and my mali-board is also wired connected with internet. Has anyone experienced this before?

I attached two screenshots below: 1) the error on my host server, 2) the tuning process interruption.

Thanks!

image
image.jpg949×659 178 KB

Crashed in auto-tunning - mobile GPU (Mali)
#2

I have solved this problem. The issue was due to the timeout parameter was not passed into check_remote function in here: https://github.com/apache/incubator-tvm/blob/master/python/tvm/autotvm/measure/measure_methods.py

I think it is very important for big networks running on Mali.


    def set_task(self, task):
        self.task = task

        if check_remote(task.target, self.key, self.host, self.port, timeout=self.timeout):
            logger.info("Get devices for measurement successfully!")
1 Like
#3

@Xin, thanks for catching. A fix PR would be definitely helpful!

#4

Sounds good. Will do it then.