[IR builder] cudaErrorCudartUnloading

Laurawly · April 17, 2019, 10:26pm

I used IR builder to write get_valid_counts, and in topi unit test, there’s no problem running individual test. But when I put multiple individual tests, it would give me this error:
CUDA: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: an illegal memory access was encountered

I think there’s something to do with codegen caching for IR builder.
Error message attached here:

gist.github.com

https://gist.github.com/Laurawly/2bd2521fc3060df1cf14f461413ae55f

gistfile1.txt

Traceback (most recent call last):
  File "topi/tests/python/test_topi_vision.py", line 437, in <module>
    test_get_valid_counts()
  File "topi/tests/python/test_topi_vision.py", line 84, in test_get_valid_counts
    verify_get_valid_counts((1, 15360, 5), 0)
  File "topi/tests/python/test_topi_vision.py", line 74, in verify_get_valid_counts
    check_device(device)
  File "topi/tests/python/test_topi_vision.py", line 70, in check_device
    tvm.testing.assert_allclose(tvm_out1.asnumpy(), np_out1, rtol=1e-3)
  File "/home/ubuntu/workplace/tvm/python/tvm/_ffi/ndarray.py", line 283, in asnumpy

This file has been truncated. show original

Generated IR looks like this:

gist.github.com

https://gist.github.com/Laurawly/776483e7bd61bb4b9ca05f6d8c1c766e

gistfile1.txt

// attr [get_valid_counts_phase_one.v0] storage_scope = "global"
allocate get_valid_counts_phase_one.v0[int32 * 15360]
// attr [get_valid_counts_phase_one.v1] storage_scope = "global"
allocate get_valid_counts_phase_one.v1[int32 * 15360]
// attr [get_valid_counts_phase_two.v1] storage_scope = "global"
allocate get_valid_counts_phase_two.v1[int32 * 496]
// attr [get_valid_counts_phase_two.v0] storage_scope = "global"
allocate get_valid_counts_phase_two.v0[int32 * 15360]
// attr [get_valid_counts_phase_three] storage_scope = "global"
allocate get_valid_counts_phase_three[int32 * 496]

This file has been truncated. show original

@tqchen

tqchen · April 18, 2019, 5:39pm

please check if it is related to wrong rewrite of the code https://github.com/dmlc/tvm/blob/master/python/tvm/build_module.py#L390