Error when run pynq-z1 test

#1

Hello, there is a error when I run the tests test_benchmark_topi_conv2d.py on FPGA pynq-Z1.
My setting is completely follow the VTA FPGA pynq-z1 setup.
However, when I run the test test_benchmark_topi_conv2d.py from my host, I get the following errors:

key=resnet-cfg[1]
Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=64, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
----- CONV2D CPU End-to-End Test-------

Traceback (most recent call last):
File “test_benchmark_topi_conv2d.py”, line 268, in
test_cpu_conv2d()
File “test_benchmark_topi_conv2d.py”, line 134, in test_cpu_conv2d
vta.testing.run(_run)
File “/home/fdsoc/Documents/tvm/vta/python/vta/testing/util.py”, line 45, in run
run_func(env, remote)
File “test_benchmark_topi_conv2d.py”, line 129, in _run
run_cpu_conv2d(env, remote, key, batch_size, wl)
File “test_benchmark_topi_conv2d.py”, line 103, in run_cpu_conv2d
conv_normal(False)
File “test_benchmark_topi_conv2d.py”, line 99, in conv_normal
cost = verify(s, True)
File “test_benchmark_topi_conv2d.py”, line 72, in verify
f = remote.load_module(“conv2d.o”)
File “/home/fdsoc/Documents/tvm/python/tvm/rpc/client.py”, line 119, in load_module
return base._LoadRemoteModule(self._sess, path)
File “/home/fdsoc/Documents/tvm/python/tvm/_ffi/_ctypes/function.py”, line 185, in call
ctypes.byref(ret_val), ctypes.byref(ret_tcode)))
File “/home/fdsoc/Documents/tvm/python/tvm/_ffi/base.py”, line 68, in check_call
raise TVMError(py_str(_LIB.TVMGetLastError()))
tvm._ffi.base.TVMError: Except caught from RPC call: TVMCall CFunc Error:
Traceback (most recent call last):
File “/home/xilinx/tvm/python/tvm/_ffi/_ctypes/function.py”, line 55, in cfun
rv = local_pyfunc(*pyargs)
File “/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py”, line 44, in load_module
return _load_module(file_name)
File “/home/xilinx/tvm/python/tvm/_ffi/_ctypes/function.py”, line 185, in call
ctypes.byref(ret_val), ctypes.byref(ret_tcode)))
File “/home/xilinx/tvm/python/tvm/_ffi/base.py”, line 68, in check_call
raise TVMError(py_str(_LIB.TVMGetLastError()))
tvm._ffi.base.TVMError: TVMCall CFunc Error:
Traceback (most recent call last):
File “/home/xilinx/tvm/python/tvm/_ffi/_ctypes/function.py”, line 55, in cfun
rv = local_pyfunc(*pyargs)
File “/home/xilinx/tvm/python/tvm/rpc/server.py”, line 50, in load_module
m = _load_module(path)
File “/home/xilinx/tvm/python/tvm/module.py”, line 216, in load
_cc.create_shared(path + “.so”, path)
File “/home/xilinx/tvm/python/tvm/contrib/cc.py”, line 33, in create_shared
_linux_shared(output, objects, options, cc)
File “/home/xilinx/tvm/python/tvm/contrib/cc.py”, line 58, in _linux_shared
raise RuntimeError(msg)
RuntimeError: Compilation error:
/usr/bin/arm-linux-gnueabihf-ld: /tmp/tmpz1_us39g/conv2d.o: relocation R_ARM_MOVW_ABS_NC against __TVMAPISetLastError' can not be used when making a shared object; recompile with -fPIC /usr/bin/arm-linux-gnueabihf-ld: /tmp/tmpz1_us39g/conv2d.o(.text+0x79c): unresolvable R_ARM_CALL relocation against symbol__aeabi_idiv@@GCC_3.5’
/usr/bin/arm-linux-gnueabihf-ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status

I can not figure it out! I need help!

#2

what version of the pynq image are you using? v2.3? and what version of LLVM are you using to cross-compiler binaries?

#3

Thanks. I have solved the problem. The problem is that I use the LLVM 7.0.0 version, but TVM now seems do not support 7.0.0 version. So I replace it with the LLVM 4.0.0 version. Then, the TVM VTA in pynq-Z1 work well.

1 Like
#4

Glad to hear that your issue was resolved.

#5

On the other hand, we might need to notify the community which versions of LLVM are we currently supporting, otherwise, many users might spend much useless time in figuring out that. Or, we might need to update the llvm support in the runtime, in order to be compatible with the latest version.

#6

good point regarding the version number - we’ll update the docs accordingly

1 Like
#7

We should always keep up with the mainline, https://github.com/dmlc/tvm/issues/2005 is still an open issue that we need help with

2 Likes
#8

Excuse me! I also found a problem when I run the test_program_rpc.py after I generated custom bitstream. My setting is also completely follow the VTA FPGA pynq-z1 setup. I modified the HW_FREQ=142 and HW_CLK_TARGET=6 in vta_config.json .But when I run the script , I get the following errors:
Traceback (most recent call last):
File “test_program_rpc.py”, line 27, in
program_rpc_bitstream()
File “test_program_rpc.py”, line 18, in program_rpc_bitstream
program_fpga(remote, path)
File “/home/qd/tvm/vta/python/vta/rpc_client.py”, line 38, in program_fpga
fprogram = remote.get_function(“tvm.contrib.vta.init”)
File “/home/qd/tvm/python/tvm/rpc/client.py”, line 41, in get_function
return self._sess.get_function(name)
File “/home/qd/tvm/python/tvm/_ffi/function.py”, line 103, in get_function
ctypes.byref(ret_handle)))
File “/home/qd/tvm/python/tvm/_ffi/base.py”, line 72, in check_call
raise TVMError(py_str(_LIB.TVMGetLastError()))
tvm._ffi.base.TVMError: [16:18:23] /home/qd/tvm/src/runtime/rpc/rpc_session.cc:857: Check failed: code == RPCCode::kReturn code=4

Stack trace returned 10 entries:
[bt] (0) /home/qd/tvm/build/libtvm.so(dmlc::StackTrace[abi:cxx11](unsigned long)+0x1fd) [0x7fbf6a1382ed]
[bt] (1) /home/qd/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x2d) [0x7fbf6a138f3d]
[bt] (2) /home/qd/tvm/build/libtvm.so(+0x99164c) [0x7fbf6a7f264c]
[bt] (3) /home/qd/tvm/build/libtvm.so(tvm::runtime::RPCModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::shared_ptrtvm::runtime::ModuleNode const&)+0x8c2) [0x7fbf6a7ff942]
[bt] (4) /home/qd/tvm/build/libtvm.so(TVMModGetFunction+0x79) [0x7fbf6a7d1d49]
[bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fbf76bbfe40]
[bt] (6) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7fbf76bbf8ab]
[bt] (7) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7fbf76dcf3df]
[bt] (8) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7fbf76dd3d82]
[bt] (9) python(PyEval_EvalFrameEx+0x578d) [0x4c166d]

Can you give me some instruction?

#9

Excuse me!I also find a problem like you.Do you have sloved it?Can you give me some suggestion?

thanks！

#10

Did you pre-compile your model? which script are you running here?

#11

I use zcu104 board.I run test_program_rpc.py,but i get a problem like this:
xiaoliyang@xiaoliyang-virtual-machine:~/tvm$ python3 vta/tests/python/pynq/test_program_rpc.py
Traceback (most recent call last):
File “vta/tests/python/pynq/test_program_rpc.py”, line 44, in
program_rpc_bitstream()
File “vta/tests/python/pynq/test_program_rpc.py”, line 35, in program_rpc_bitstream
program_fpga(remote, path)
File “/home/xiaoliyang/tvm/vta/python/vta/rpc_client.py”, line 56, in program_fpga
fprogram(os.path.basename(bitstream))
File “/home/xiaoliyang/tvm/python/tvm/_ffi/_ctypes/function.py”, line 209, in call
raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (8) /home/xilinx/tvm/build/libtvm_runtime.so(+0x950fc) [0x7f9dc850fc]
[bt] (7) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0xc0) [0x7f9dc84190]
[bt] (6) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::ServerLoop()+0x104) [0x7f9dc7a89c]
[bt] (5) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::HandleUntilReturnEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0x1b8) [0x7f9dc7a570]
[bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleNextEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0x584) [0x7f9dc80944]
[bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleRecvPackedSeqArg()+0x14c) [0x7f9dc7fd6c]
[bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::SwitchToState(tvm::runtime::RPCSession::EventHandler::State)+0x320) [0x7f9dc7ec88]
[bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandlePackedCall()+0x300) [0x7f9dc78748]
[bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x280e8) [0x7f9dc180e8]
File “/home/xilinx/tvm/python/tvm/_ffi/_ctypes/function.py”, line 72, in cfun
rv = local_pyfunc(*pyargs)
File “/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py”, line 74, in program_fpga
program_bitstream.bitstream_program(env.TARGET, path)
File “/home/xilinx/tvm/vta/python/vta/program_bitstream.py”, line 52, in bitstream_program
pynq_bitstream_program(bitstream)
File “/home/xilinx/tvm/vta/python/vta/program_bitstream.py”, line 48, in pynq_bitstream_program
bitstream.download()
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1945, in download
fd.write(self.binfile_name)
TVMError: Except caught from RPC call: OSError: [Errno 22] Invalid argument
During handling of the above exception, another exception occurred:

OSError: [Errno 22] Invalid argument

Is there a problem with my RPC deployment? RPC like this:
xiaoliyang@xiaoliyang-virtual-machine:~$ ssh xilinx@192.168.2.99
xilinx@192.168.2.99’s password:
Welcome to PYNQ Linux, based on Ubuntu 18.04 (GNU/Linux 4.14.0-xilinx-v2018.3 aarch64)

Last login: Fri Jul 19 02:25:46 2019 from 192.168.2.1
xilinx@pynq:~$ cd tvm
xilinx@pynq:~/tvm$ sudo ./apps/pynq_rpc/start_rpc_server.sh
[sudo] password for xilinx:
INFO:RPCServer:bind to 0.0.0.0:9091
INFO:RPCServer:connection from (‘192.168.2.1’, 62460)
INFO:RPCServer:Finish serving (‘192.168.2.1’, 62460)

Thank you very much for spending time to help me.

#12

I see - ZCU104 is not supported yet, but we can release support (I’m working on an Ultra96 release). One issue with the Zynq Ultrascale+ MPSoC is that they don’t have coherent memory support out of the box; I’ll try to get in tough with the Xilinx Pynq engineers to get a patch so we can support ZCU104. I’ll keep you in the loop.

#13

Here’s the PR on Ultra96 support; ZCU104 will follow. https://github.com/dmlc/tvm/pull/3496

Is TVM VTA support pynq ZCU104 now?
#14

Thank you very much.Thank you for taking the time to help me .

#15

Hello, I’m running on pynq-z1. And when I execute ‘python3 test_program_rpc.py’, I get an error.

Traceback (most recent call last):
File “test_program_rpc.py”, line 44, in
reconfig_rpc_runtime()
File “test_program_rpc.py”, line 41, in reconfig_rpc_runtime
reconfig_runtime(remote)
File “/home/rd/Documents/Code/DLSYS/tvm/vta/python/vta/rpc_client.py”, line 33, in reconfig_runtime
freconfig(env.pkg_config().cfg_json)
File “/home/rd/.local/lib/python3.6/site-packages/tvm-0.6.dev0-py3.6-linux-x86_64.egg/tvm/_ffi/_ctypes/function.py”, line 210, in call
raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (8) /home/xilinx/tvm/build/libtvm_runtime.so(+0x6d04a) [0xb598704a]
[bt] (7) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0x91) [0xb598666e]
[bt] (6) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::ServerLoop()+0xa1) [0xb59800aa]
[bt] (5) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::HandleUntilReturnEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0x103) [0xb597fed0]
[bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleNextEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0x203) [0xb5983c08]
[bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleRecvPackedSeqArg()+0x23f) [0xb5983718]
[bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::SwitchToState(tvm::runtime::RPCSession::EventHandler::State)+0x1eb) [0xb5982a4c]
[bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandlePackedCall()+0x417) [0xb597ea00]
[bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x20348) [0xb593a348]
File “/home/xilinx/tvm/python/tvm/_ffi/_ctypes/function.py”, line 72, in cfun
rv = local_pyfunc(*pyargs)
File “/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py”, line 97, in reconfig_runtime
pkg = PkgConfig(cfg, proj_root)
File “/home/xilinx/tvm/vta/python/vta/pkg_config.py”, line 55, in init
cfg[“LOG_BLOCK_IN”] = cfg[“LOG_BLOCK”]
TVMError: Except caught from RPC call: KeyError: ‘LOG_BLOCK’

Please help me. Thank you very much!

#16

can you share the vta_config.json that you are using?

#17

{
“TARGET” : “pynq”,
“HW_FREQ” : 100,
“HW_CLK_TARGET” : 8,
“HW_VER” : “0.0.0”,
“LOG_INP_WIDTH” : 3,
“LOG_WGT_WIDTH” : 3,
“LOG_ACC_WIDTH” : 5,
“LOG_OUT_WIDTH” : 3,
“LOG_BATCH” : 0,
“LOG_BLOCK_IN” : 4,
“LOG_BLOCK_OUT” : 4,
“LOG_UOP_BUFF_SIZE” : 15,
“LOG_INP_BUFF_SIZE” : 15,
“LOG_WGT_BUFF_SIZE” : 18,
“LOG_ACC_BUFF_SIZE” : 17
}

#18

Oh I think that the issue might arise from a TVM mismatch between your host and your FPGA. Can you make sure they are both up to date and have been rebuilt? It’s possible you updated your TVM on your host but didn’t on your FPGA, and consequently the RPC is out of sync.

#19

Thank you. I’ll check later.

#20

Hi, my project team has been executing ‘python3 test_program_rpc.py’, but continues to get the following error with it.

Traceback (most recent call last):

File “vta/tests/python/pynq/test_program_rpc.py”, line 44, in program_rpc_bitstream()

File “vta/tests/python/pynq/test_program_rpc.py”, line 35, in program_rpc_bitstream program_fpga(remote, path)

File “/home/jmarshall/tvm/vta/python/vta/rpc_client.py”, line 57, in program_fpga fprogram = remote.get_function(“tvm.contrib.vta.init”)

File “/home/jmarshall/tvm/python/tvm/rpc/client.py”, line 57, in get_function return self._sess.get_function(name)

File “/home/jmarshall/tvm/python/tvm/runtime/module.py”, line 85, in get_function ctypes.byref(ret_handle)))

File “/home/jmarshall/tvm/python/tvm/_ffi/base.py”, line 330, in check_call raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last): [bt] (4) /home/jmarshall/tvm/build/libtvm.so(TVMModGetFunction+0x81) [0x7f30591fdde1] [bt] (3) /home/jmarshall/tvm/build/libtvm.so(tvm::runtime::ModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool)+0x49) [0x7f3059218629] [bt] (2) /home/jmarshall/tvm/build/libtvm.so(tvm::runtime::RPCModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, tvm::runtime::ObjectPtrtvm::runtime::Object const&)+0x5eb) [0x7f305926526b] [bt] (1) /home/jmarshall/tvm/build/libtvm.so(+0xcb2fec) [0x7f305926afec] [bt] (0) /home/jmarshall/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x43) [0x7f3058948ef3] File “/home/jmarshall/tvm/src/runtime/rpc/rpc_session.cc”, line 903 TVMError: Check failed: code == RPCCode: :kReturn: code=4

Process Process-1:1: Traceback (most recent call last): File “/usr/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap self.run() File “/usr/lib/python3.6/multiprocessing/process.py”, line 93, in run self.target(self._args, **self._kwargs) File “/home/xilinx/tvmForBoard/python/tvm/rpc/server.py”, line 85, in _serve_loop _ffi_api.ServerLoop(sockfd) File “/home/xilinx/tvmForBoard/python/tvm/_ffi/_ctypes/packed_func.py”, line 225, in call raise get_last_ffi_error() tvm._ffi.base.TVMError: Traceback (most recent call last): [bt] (8) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::ServerLoop()+0x97) [0xb599e764] [bt] (7) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::HandleUntilReturnEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)+0x10b) [0xb599e3b0] [bt] (6) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)+0x7f) [0xb59a3c08] [bt] (5) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleProcessPacket(std::function<void (tvm::runtime::TVMArgs)>)+0xd7) [0xb59a3b58] [bt] (4) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleSyscall(tvm::runtime::RPCCode)+0x115) [0xb599e1ca] [bt] (3) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(void tvm::runtime::RPCEndpoint::EventHandler::SysCallHandler<void ()(tvm::runtime::RPCSession*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>(void ()(tvm::runtime::RPCSession, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*))+0x31) [0xb59a3586] [bt] (2) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(void tvm::runtime::RPCReference::RecvPackedSeqtvm::runtime::RPCEndpoint::EventHandler(TVMValue**, int**, int*, tvm::runtime::RPCEndpoint::EventHandler*)+0x97) [0xb59a28f8] [bt] (1) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::Read(void*, unsigned int)+0xff) [0xb59a07a0] [bt] (0) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x49) [0xb5956c02] File “/home/xilinx/tvmForBoard/src/runtime/rpc/rpc_endpoint.cc”, line 579 TVMError: Check failed: size <= pending_request_bytes (4 vs. 1) :

Does anyone know how to help? Thanks!