Error when run pynq-z1 test

#8

Excuse me! I also found a problem when I run the test_program_rpc.py after I generated custom bitstream. My setting is also completely follow the VTA FPGA pynq-z1 setup. I modified the HW_FREQ=142 and HW_CLK_TARGET=6 in vta_config.json .But when I run the script , I get the following errors:
Traceback (most recent call last):
File “test_program_rpc.py”, line 27, in
program_rpc_bitstream()
File “test_program_rpc.py”, line 18, in program_rpc_bitstream
program_fpga(remote, path)
File “/home/qd/tvm/vta/python/vta/rpc_client.py”, line 38, in program_fpga
fprogram = remote.get_function(“tvm.contrib.vta.init”)
File “/home/qd/tvm/python/tvm/rpc/client.py”, line 41, in get_function
return self._sess.get_function(name)
File “/home/qd/tvm/python/tvm/_ffi/function.py”, line 103, in get_function
ctypes.byref(ret_handle)))
File “/home/qd/tvm/python/tvm/_ffi/base.py”, line 72, in check_call
raise TVMError(py_str(_LIB.TVMGetLastError()))
tvm._ffi.base.TVMError: [16:18:23] /home/qd/tvm/src/runtime/rpc/rpc_session.cc:857: Check failed: code == RPCCode::kReturn code=4

Stack trace returned 10 entries:
[bt] (0) /home/qd/tvm/build/libtvm.so(dmlc::StackTrace[abi:cxx11](unsigned long)+0x1fd) [0x7fbf6a1382ed]
[bt] (1) /home/qd/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x2d) [0x7fbf6a138f3d]
[bt] (2) /home/qd/tvm/build/libtvm.so(+0x99164c) [0x7fbf6a7f264c]
[bt] (3) /home/qd/tvm/build/libtvm.so(tvm::runtime::RPCModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::shared_ptrtvm::runtime::ModuleNode const&)+0x8c2) [0x7fbf6a7ff942]
[bt] (4) /home/qd/tvm/build/libtvm.so(TVMModGetFunction+0x79) [0x7fbf6a7d1d49]
[bt] (5) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7fbf76bbfe40]
[bt] (6) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7fbf76bbf8ab]
[bt] (7) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7fbf76dcf3df]
[bt] (8) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7fbf76dd3d82]
[bt] (9) python(PyEval_EvalFrameEx+0x578d) [0x4c166d]

Can you give me some instruction?

#9

Excuse me!I also find a problem like you.Do you have sloved it?Can you give me some suggestion?

thanks！

#10

Did you pre-compile your model? which script are you running here?

#11

I use zcu104 board.I run test_program_rpc.py,but i get a problem like this:
xiaoliyang@xiaoliyang-virtual-machine:~/tvm$ python3 vta/tests/python/pynq/test_program_rpc.py
Traceback (most recent call last):
File “vta/tests/python/pynq/test_program_rpc.py”, line 44, in
program_rpc_bitstream()
File “vta/tests/python/pynq/test_program_rpc.py”, line 35, in program_rpc_bitstream
program_fpga(remote, path)
File “/home/xiaoliyang/tvm/vta/python/vta/rpc_client.py”, line 56, in program_fpga
fprogram(os.path.basename(bitstream))
File “/home/xiaoliyang/tvm/python/tvm/_ffi/_ctypes/function.py”, line 209, in call
raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (8) /home/xilinx/tvm/build/libtvm_runtime.so(+0x950fc) [0x7f9dc850fc]
[bt] (7) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0xc0) [0x7f9dc84190]
[bt] (6) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::ServerLoop()+0x104) [0x7f9dc7a89c]
[bt] (5) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::HandleUntilReturnEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0x1b8) [0x7f9dc7a570]
[bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleNextEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0x584) [0x7f9dc80944]
[bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleRecvPackedSeqArg()+0x14c) [0x7f9dc7fd6c]
[bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::SwitchToState(tvm::runtime::RPCSession::EventHandler::State)+0x320) [0x7f9dc7ec88]
[bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandlePackedCall()+0x300) [0x7f9dc78748]
[bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x280e8) [0x7f9dc180e8]
File “/home/xilinx/tvm/python/tvm/_ffi/_ctypes/function.py”, line 72, in cfun
rv = local_pyfunc(*pyargs)
File “/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py”, line 74, in program_fpga
program_bitstream.bitstream_program(env.TARGET, path)
File “/home/xilinx/tvm/vta/python/vta/program_bitstream.py”, line 52, in bitstream_program
pynq_bitstream_program(bitstream)
File “/home/xilinx/tvm/vta/python/vta/program_bitstream.py”, line 48, in pynq_bitstream_program
bitstream.download()
File “/usr/local/lib/python3.6/dist-packages/pynq/pl.py”, line 1945, in download
fd.write(self.binfile_name)
TVMError: Except caught from RPC call: OSError: [Errno 22] Invalid argument
During handling of the above exception, another exception occurred:

OSError: [Errno 22] Invalid argument

Is there a problem with my RPC deployment? RPC like this:
xiaoliyang@xiaoliyang-virtual-machine:~$ ssh xilinx@192.168.2.99
xilinx@192.168.2.99’s password:
Welcome to PYNQ Linux, based on Ubuntu 18.04 (GNU/Linux 4.14.0-xilinx-v2018.3 aarch64)

Last login: Fri Jul 19 02:25:46 2019 from 192.168.2.1
xilinx@pynq:~$ cd tvm
xilinx@pynq:~/tvm$ sudo ./apps/pynq_rpc/start_rpc_server.sh
[sudo] password for xilinx:
INFO:RPCServer:bind to 0.0.0.0:9091
INFO:RPCServer:connection from (‘192.168.2.1’, 62460)
INFO:RPCServer:Finish serving (‘192.168.2.1’, 62460)

Thank you very much for spending time to help me.

#12

I see - ZCU104 is not supported yet, but we can release support (I’m working on an Ultra96 release). One issue with the Zynq Ultrascale+ MPSoC is that they don’t have coherent memory support out of the box; I’ll try to get in tough with the Xilinx Pynq engineers to get a patch so we can support ZCU104. I’ll keep you in the loop.

#13

Here’s the PR on Ultra96 support; ZCU104 will follow. https://github.com/dmlc/tvm/pull/3496

Is TVM VTA support pynq ZCU104 now?
#14

Thank you very much.Thank you for taking the time to help me .

#15

Hello, I’m running on pynq-z1. And when I execute ‘python3 test_program_rpc.py’, I get an error.

Traceback (most recent call last):
File “test_program_rpc.py”, line 44, in
reconfig_rpc_runtime()
File “test_program_rpc.py”, line 41, in reconfig_rpc_runtime
reconfig_runtime(remote)
File “/home/rd/Documents/Code/DLSYS/tvm/vta/python/vta/rpc_client.py”, line 33, in reconfig_runtime
freconfig(env.pkg_config().cfg_json)
File “/home/rd/.local/lib/python3.6/site-packages/tvm-0.6.dev0-py3.6-linux-x86_64.egg/tvm/_ffi/_ctypes/function.py”, line 210, in call
raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (8) /home/xilinx/tvm/build/libtvm_runtime.so(+0x6d04a) [0xb598704a]
[bt] (7) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0x91) [0xb598666e]
[bt] (6) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::ServerLoop()+0xa1) [0xb59800aa]
[bt] (5) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::HandleUntilReturnEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0x103) [0xb597fed0]
[bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleNextEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0x203) [0xb5983c08]
[bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleRecvPackedSeqArg()+0x23f) [0xb5983718]
[bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::SwitchToState(tvm::runtime::RPCSession::EventHandler::State)+0x1eb) [0xb5982a4c]
[bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandlePackedCall()+0x417) [0xb597ea00]
[bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x20348) [0xb593a348]
File “/home/xilinx/tvm/python/tvm/_ffi/_ctypes/function.py”, line 72, in cfun
rv = local_pyfunc(*pyargs)
File “/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py”, line 97, in reconfig_runtime
pkg = PkgConfig(cfg, proj_root)
File “/home/xilinx/tvm/vta/python/vta/pkg_config.py”, line 55, in init
cfg[“LOG_BLOCK_IN”] = cfg[“LOG_BLOCK”]
TVMError: Except caught from RPC call: KeyError: ‘LOG_BLOCK’

Please help me. Thank you very much!

#16

can you share the vta_config.json that you are using?

#17

{
“TARGET” : “pynq”,
“HW_FREQ” : 100,
“HW_CLK_TARGET” : 8,
“HW_VER” : “0.0.0”,
“LOG_INP_WIDTH” : 3,
“LOG_WGT_WIDTH” : 3,
“LOG_ACC_WIDTH” : 5,
“LOG_OUT_WIDTH” : 3,
“LOG_BATCH” : 0,
“LOG_BLOCK_IN” : 4,
“LOG_BLOCK_OUT” : 4,
“LOG_UOP_BUFF_SIZE” : 15,
“LOG_INP_BUFF_SIZE” : 15,
“LOG_WGT_BUFF_SIZE” : 18,
“LOG_ACC_BUFF_SIZE” : 17
}

#18

Oh I think that the issue might arise from a TVM mismatch between your host and your FPGA. Can you make sure they are both up to date and have been rebuilt? It’s possible you updated your TVM on your host but didn’t on your FPGA, and consequently the RPC is out of sync.

#19

Thank you. I’ll check later.

#20

Hi, my project team has been executing ‘python3 test_program_rpc.py’, but continues to get the following error with it.

Traceback (most recent call last):

File “vta/tests/python/pynq/test_program_rpc.py”, line 44, in program_rpc_bitstream()

File “vta/tests/python/pynq/test_program_rpc.py”, line 35, in program_rpc_bitstream program_fpga(remote, path)

File “/home/jmarshall/tvm/vta/python/vta/rpc_client.py”, line 57, in program_fpga fprogram = remote.get_function(“tvm.contrib.vta.init”)

File “/home/jmarshall/tvm/python/tvm/rpc/client.py”, line 57, in get_function return self._sess.get_function(name)

File “/home/jmarshall/tvm/python/tvm/runtime/module.py”, line 85, in get_function ctypes.byref(ret_handle)))

File “/home/jmarshall/tvm/python/tvm/_ffi/base.py”, line 330, in check_call raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last): [bt] (4) /home/jmarshall/tvm/build/libtvm.so(TVMModGetFunction+0x81) [0x7f30591fdde1] [bt] (3) /home/jmarshall/tvm/build/libtvm.so(tvm::runtime::ModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, bool)+0x49) [0x7f3059218629] [bt] (2) /home/jmarshall/tvm/build/libtvm.so(tvm::runtime::RPCModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, tvm::runtime::ObjectPtrtvm::runtime::Object const&)+0x5eb) [0x7f305926526b] [bt] (1) /home/jmarshall/tvm/build/libtvm.so(+0xcb2fec) [0x7f305926afec] [bt] (0) /home/jmarshall/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x43) [0x7f3058948ef3] File “/home/jmarshall/tvm/src/runtime/rpc/rpc_session.cc”, line 903 TVMError: Check failed: code == RPCCode: :kReturn: code=4

Process Process-1:1: Traceback (most recent call last): File “/usr/lib/python3.6/multiprocessing/process.py”, line 258, in _bootstrap self.run() File “/usr/lib/python3.6/multiprocessing/process.py”, line 93, in run self.target(self._args, **self._kwargs) File “/home/xilinx/tvmForBoard/python/tvm/rpc/server.py”, line 85, in _serve_loop _ffi_api.ServerLoop(sockfd) File “/home/xilinx/tvmForBoard/python/tvm/_ffi/_ctypes/packed_func.py”, line 225, in call raise get_last_ffi_error() tvm._ffi.base.TVMError: Traceback (most recent call last): [bt] (8) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::ServerLoop()+0x97) [0xb599e764] [bt] (7) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::HandleUntilReturnEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)+0x10b) [0xb599e3b0] [bt] (6) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)+0x7f) [0xb59a3c08] [bt] (5) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleProcessPacket(std::function<void (tvm::runtime::TVMArgs)>)+0xd7) [0xb59a3b58] [bt] (4) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleSyscall(tvm::runtime::RPCCode)+0x115) [0xb599e1ca] [bt] (3) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(void tvm::runtime::RPCEndpoint::EventHandler::SysCallHandler<void ()(tvm::runtime::RPCSession*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>(void ()(tvm::runtime::RPCSession, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*))+0x31) [0xb59a3586] [bt] (2) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(void tvm::runtime::RPCReference::RecvPackedSeqtvm::runtime::RPCEndpoint::EventHandler(TVMValue**, int**, int*, tvm::runtime::RPCEndpoint::EventHandler*)+0x97) [0xb59a28f8] [bt] (1) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::Read(void*, unsigned int)+0xff) [0xb59a07a0] [bt] (0) /home/xilinx/tvmForBoard/build/libtvm_runtime.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x49) [0xb5956c02] File “/home/xilinx/tvmForBoard/src/runtime/rpc/rpc_endpoint.cc”, line 579 TVMError: Check failed: size <= pending_request_bytes (4 vs. 1) :

Does anyone know how to help? Thanks!

#21

Hi @hjiang , is this the same RPC problem you referred to?

#22

I’ve encountered several RPC issues while running test_benchmark_topi_conv2d.py and also some of the tutorials like deploy_classification.py. This errors report VTARuntimeShutdown symbol not found. Further information about the issues were discussed in the following posts: VTA build on PYNQ fails at VTA.cmake:97 and VTA build and execution error with new 3rdparty vta-hw. The building problem was solved in a recent commit but RPC still fails.

#23

@liangfu, I experience similar issue, a temporary work ground is to roll back TVM to April old build for example 6b840fa9672124bebccff5322a59ab1f159e74b8, and manually patch this VTA build patch https://github.com/apache/incubator-tvm/commit/5d75992d8849dac60f1a01e47073f2ca3191999f.

#24

There were some bugs that were introduced from the recent refactor. Upon this PR being merged the classification example should be working again.

#25

Hi I was experiencing a same problem with yours, and I’ve solved it this way. VTARuntimeShutdown is placed on tvm/vta/runtime/runtime.cc file. It originally seems like this

void VTARuntimeShutdown() { vta::CommandQueue::Shutdown(); }

and I added ‘return’ keyword to this method

void VTARuntimeShutdown() { return vta::CommandQueue::Shutdown(); }

and build again with

cmake …

make runtime vta -j2

as the tutorial shows.

#27

Hi, It seems that I have a similar problem. Can you help me to find the error? Thannnks!

k@ubuntu:~/tvm$ python /home/k/tvm/vta/tests/python/pynq/test_program_rpc.py Traceback (most recent call last): File “/home/k/tvm/vta/tests/python/pynq/test_program_rpc.py”, line 47, in program_rpc_bitstream() File “/home/k/tvm/vta/tests/python/pynq/test_program_rpc.py”, line 36, in program_rpc_bitstream remote = rpc.connect(host, port) File “/home/k/tvm/python/tvm/rpc/client.py”, line 501, in connect sess = _ffi_api.Connect(url, port, key, *session_constructor_args) File “/home/k/tvm/python/tvm/_ffi/_ctypes/packed_func.py”, line 237, in call raise get_last_ffi_error() tvm._ffi.base.TVMError: Traceback (most recent call last): [bt] (4) /home/k/tvm/build/libtvm.so(TVMFuncCall+0x61) [0x7fbd51566081] [bt] (3) /home/k/tvm/build/libtvm.so(+0x1451a27) [0x7fbd515bca27] [bt] (2) /home/k/tvm/build/libtvm.so(tvm::runtime::RPCClientConnect(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, tvm::runtime::TVMArgs)+0xdd) [0x7fbd515bc40d] [bt] (1) /home/k/tvm/build/libtvm.so(tvm::runtime::RPCConnect(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, tvm::runtime::TVMArgs)+0x2b50) [0x7fbd515bb720] [bt] (0) /home/k/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x61) [0x7fbd506f90f1] File “/home/k/tvm/src/runtime/rpc/…/…/support/socket.h”, line 140 TVMError:

An internal invariant was violated during the execution of TVM. Please read TVM’s error reporting guidelines. More details can be found here: https://discuss.tvm.ai/t/error-reporting/7793.

Check failed: sig == 0 && res != nullptr == false: cannot obtain address of pynq

#28

image
image1856×450 98.9 KB