[VTA] Recursion error

I’m following this guide https://tvm.apache.org/docs/vta/install.html to install VTA, but I get I get a recursion error (RecursionError: maximum recursion depth exceeded in comparison) while executing this script in hardware (not in simulation):

python <tvm root>/vta/tests/python/integration/test_benchmark_topi_conv2d.py

Here is the error in detail (host side):

Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=64, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
CPU CONV2D TEST PASSED: Time cost = 0.0158294 sec/op, 14.6064 GOPS
Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=128, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00839859 sec/op, 13.7649 GOPS
Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=128, hkernel=1, wkernel=1, hpad=0, wpad=0, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00140844 sec/op, 9.12003 GOPS
Conv2DWorkload(batch=1, height=28, width=28, in_filter=128, out_filter=128, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
CPU CONV2D TEST PASSED: Time cost = 0.0166927 sec/op, 13.8511 GOPS
Conv2DWorkload(batch=1, height=28, width=28, in_filter=128, out_filter=256, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00881398 sec/op, 13.1162 GOPS
Conv2DWorkload(batch=1, height=28, width=28, in_filter=128, out_filter=256, hkernel=1, wkernel=1, hpad=0, wpad=0, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00126221 sec/op, 10.1766 GOPS
Conv2DWorkload(batch=1, height=14, width=14, in_filter=256, out_filter=256, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
CPU CONV2D TEST PASSED: Time cost = 0.0138982 sec/op, 16.6361 GOPS
Conv2DWorkload(batch=1, height=14, width=14, in_filter=256, out_filter=512, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.011544 sec/op, 10.0143 GOPS
Conv2DWorkload(batch=1, height=14, width=14, in_filter=256, out_filter=512, hkernel=1, wkernel=1, hpad=0, wpad=0, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00127813 sec/op, 10.0499 GOPS
Conv2DWorkload(batch=1, height=7, width=7, in_filter=512, out_filter=512, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
CPU CONV2D TEST PASSED: Time cost = 0.0168659 sec/op, 13.7088 GOPS
Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=64, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
Traceback (most recent call last):
  File "../../outils/tvm/vta/tests/python/integration/test_benchmark_topi_conv2d.py", line 257, in <module>
    test_conv2d(device="vta")
  File "../../outils/tvm/vta/tests/python/integration/test_benchmark_topi_conv2d.py", line 253, in test_conv2d
    vta.testing.run(_run)
  File "/home/julien/outils/tvm/vta/python/vta/testing/util.py", line 75, in run
    run_func(env, remote)
  File "../../outils/tvm/vta/tests/python/integration/test_benchmark_topi_conv2d.py", line 252, in _run
    run_conv2d(env, remote, wl, target)
  File "../../outils/tvm/vta/tests/python/integration/test_benchmark_topi_conv2d.py", line 183, in run_conv2d
    data_arr = tvm.nd.array(data_np, ctx)
  File "/home/julien/outils/tvm/python/tvm/runtime/ndarray.py", line 503, in array
    return empty(arr.shape, arr.dtype, ctx).copyfrom(arr)
  File "/home/julien/outils/tvm/python/tvm/runtime/ndarray.py", line 271, in empty
    check_call(_LIB.TVMArrayAlloc(
  File "/home/julien/outils/tvm/python/tvm/_ffi/base.py", line 334, in check_call
    raise get_last_ffi_error()

[...] (error too long)

  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 64, in ext_dev_callback
    return tvm.get_global_func("device_api.ext_dev")()
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 64, in ext_dev_callback
    return tvm.get_global_func("device_api.ext_dev")()
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 64, in ext_dev_callback
    return tvm.get_global_func("device_api.ext_dev")()
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 63, in ext_dev_callback
    load_vta_dll()
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 53, in load_vta_dll
    logging.info("Loading VTA library: %s", dll_path)
  File "/usr/lib/python3.6/logging/__init__.py", line 1900, in info
    root.info(msg, *args, **kwargs)
  File "/usr/lib/python3.6/logging/__init__.py", line 1306, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.6/logging/__init__.py", line 1441, in _log
    exc_info, func, extra, sinfo)
  File "/usr/lib/python3.6/logging/__init__.py", line 1411, in makeRecord
    sinfo)
  File "/usr/lib/python3.6/logging/__init__.py", line 277, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.Mapping)
  File "/home/julien/outils/tvm/src/runtime/rpc/rpc_endpoint.cc", line 370
RPCError: Error caught from RPC call:
RecursionError: maximum recursion depth exceeded in comparison

The detailed error (RPC server side):

INFO:RPCServer:bind to 0.0.0.0:9091
INFO:RPCServer:connection from ('192.168.3.102', 48578)
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
Process Process-1:1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xilinx/tvm/python/tvm/rpc/server.py", line 85, in _serve_loop
    _ffi_api.ServerLoop(sockfd)
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
AttributeError: Traceback (most recent call last):
  [bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(+0xb4d54) [0x7fade35d54]
  [bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0xac) [0x7fade347b4]
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::ServerLoop()+0x1c8) [0x7fade15bb8]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 84, in server_shutdown
    runtime_dll[0].VTARuntimeShutdown()
  File "/usr/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so: undefined symbol: VTARuntimeShutdown
INFO:RPCServer:connection from ('192.168.3.102', 48584)
INFO:root:Program FPGA with 1x16_i8w8a32_15_15_18_17.bit 
INFO:root:Skip reconfig_runtime due to same config.
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmpk08kcwj4/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
Process Process-1:2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xilinx/tvm/python/tvm/rpc/server.py", line 85, in _serve_loop
    _ffi_api.ServerLoop(sockfd)
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
AttributeError: Traceback (most recent call last):
  [bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(+0xb4d54) [0x7fade35d54]
  [bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0xac) [0x7fade347b4]
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::ServerLoop()+0x1c8) [0x7fade15bb8]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 84, in server_shutdown
    runtime_dll[0].VTARuntimeShutdown()
  File "/usr/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so: undefined symbol: VTARuntimeShutdown

I know that we can solve the problem by adding these lines of code, but here it do not work here. Certainly because this code is not offloaded to FPGA board:

import sys
sys.setrecursionlimit(10**6)

Thanks in advance for your help :slight_smile:

Same here… any help?

I tried to change the recursion limit to 1,000,000 (instead of 10,000) in the following files (both host and target):

$TVM_HOME/python/tvm/relay/__init__.py
$TVM_HOME/python/tvm/relay/_parser.py

But still the same error. Any idea?

Any update on this issue?

somehow, I could run without this error. Then this popped out of nowhere again. In just a few seconds. This is what happened.

prev : I could run tutorials without any problems.

  1. I tried to change some parameters (in vta_config.json file) with the configured bitstream file.
  2. Same issue.
  3. I re-changed to the default configuration, and get the same issue with the ones that I could run successfully.

There must be some kind of bug here.

This way, I could make it run again without problem.

  1. re-install from source
  2. make the config file with ‘pynq config’
  • still have same runtime shutdown issue
  1. change the target to ‘ultra96’ and re make

BTW I am using the ZCU104 board so my target is ultra96. But as soon as I try another configuration bit file or some other attempts using different configs, I go back to the same issue, and returning the setting to the values that used to work, does not work. Then I have to re-install again in that order above.

Haven’t found the solution yet.

I encountered a similar situation. I had everything running flawlessly in my PYNQ boards but when I tried modifying the parameters from the configuration file the remote execution on the FPGA side started failing. I tried re-installing and using the default configuration but It hasn’t worked since then. Haven’t found any solution to the problem either.

I also encountered recurrsion error. In my case I just ran vta tutorial - Get Started with VTA - with changing freq and pipeline setting. Although I rebuild both bitstream file in host and cmake in pynq with changed pkg_config.py setting, I cannot run jupytercode with recurrsion error.

If you build vta on device, you’ll find a “libvta.so” file. But as you change small bit on the host side and run any code, 'VTARuntimeShutdown" disappears. I don’t know how this works, but this is the reason of this failure.

For example, if you change LOG_BATCH to 1 (vta/config/vta_config.json) and run any code on pynq, the .so file changes and occurs error. Anyone have any advice?

Best solution for me rn, just make clean on device side and rebuild runtime, vta but this is too big job to do every time I run different code which has different parameters. (Or copy the first default .so file and paste it every time it’s changed, but didn’t try this.)

In my case I haven’t been able to successfully ran any operation on the devices since I changed the configuration parameters. I tried rebuilding and reinstalling everything but I still meet the recursion error.