Auto-tune faster-rcnn from gluonCV


Recently I am auto-tuning faster-rcnn from GluonCV in auto-tvm
My setting is:

  • RTX2080Ti
  • Centos 7.5.1804
  • Cuda 10 with cudnn 7.4

However,during the tuning process,error happens as follows:

what(): [23:13:20] /home/liuxin/3rdparty/source_code/tvm-1/src/runtime/cuda/ CUDAError: cuModuleUnload(module_[i]) failed with error: CUDA_ERROR_ILLEGAL_ADDRESS

Stack trace: [bt] (0) /home/liuxin/3rdparty/source_code/tvm-1/build/ [0x7fa22d678952]
[bt] (1) /home/liuxin/3rdparty/source_code/tvm-1/build/ [0x7fa22ddaaee8]
[bt] (2) /home/liuxin/3rdparty/source_code/tvm-1/build/ [0x7fa22dd4d954]
[bt] (3) /home/liuxin/3rdparty/source_code/tvm-1/build/ [0x7fa22dd59062]
[bt] (4) /home/liuxin/3rdparty/source_code/tvm-1/build/ [0x7fa22dd809e0]
[bt] (5) /home/liuxin/3rdparty/source_code/tvm-1/build/ [0x7fa22dd80cc0]
[bt] (6) /home/liuxin/3rdparty/source_code/tvm-1/build/ [0x7fa22dd85df8]
[bt] (7) /home/liuxin/3rdparty/source_code/tvm-1/build/ [0x7fa22dd82ba7]
[bt] (8) /home/liuxin/3rdparty/source_code/tvm-1/build/ [0x7fa22dd86b5d]

I have checked the error code line but I don’t know how the error happens.Anyone knows this ?
In addition,it happens in the process.To reproduce the error,I have to wait a long time , how to avoid this happening?


src/schedule/ not in feed graph consumer = extern(nms, 0x56190de09ac0)

Besides tuning faster-rcnn,when tuning ssd and yolo,the above message also pops


Maybe faster-rcnn and ssd-mobilenet, etc. have not supported by auto-tuning.


No,they have supported.You search Gluon in the community and you will find that they have supported detection models in GluonCV.


I means that TVM’s auto-tuning doesn’t support above models.


Yeah, maybe many bugs exist in auto-tuning these detection models.