[Solved][Hybrid Script] Operator compiled well with tvm but failed with nnvm/relay

kevinthesun · January 14, 2019, 6:02am

This operator works well while building with tom, but failed with nnvm and relay.

nnvm test https://github.com/kevinthesun/tvm/blob/RefactorSSDOperator/nnvm/tests/python/compiler/test_top_level4.py#L544 will return error:

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 315, in 'calling callback function'
  File "/home/ubuntu/tvm/python/tvm/_ffi/_ctypes/function.py", line 63, in cfun
    raise ValueError("PackedFunction can only support one return value")
ValueError: PackedFunction can only support one return value
Traceback (most recent call last):
  File "test_top_level4.py", line 791, in <module>
    test_multibox_transform_loc()
  File "test_top_level4.py", line 569, in test_multibox_transform_loc
    "anchors": (1, num_anchors, 4)})
  File "/home/ubuntu/tvm/nnvm/python/nnvm/compiler/build_module.py", line 305, in build
    graph = graph.apply("GraphCompile")
  File "/home/ubuntu/tvm/nnvm/python/nnvm/graph.py", line 234, in apply
    check_call(_LIB.NNGraphApplyPasses(self.handle, npass, cpass, ctypes.byref(ghandle)))
  File "/home/ubuntu/tvm/nnvm/python/nnvm/_base.py", line 75, in check_call
    raise NNVMError(py_str(_LIB.NNGetLastError()))
nnvm._base.NNVMError: TVMCall CFunc Error:

relay test https://github.com/kevinthesun/tvm/blob/RefactorSSDOperator/tests/python/relay/test_op_level5.py#L191 returns error:

Exception ValueError: ValueError('PackedFunction can only support one return value',) in 'core.tvm_callback' ignored
Traceback (most recent call last):
  File "test_op_level5.py", line 317, in <module>
    test_multibox_transform_loc()
  File "test_op_level5.py", line 309, in test_multibox_transform_loc
    test_default_value()
  File "test_op_level5.py", line 273, in test_default_value
    op_res1 = intrp1.evaluate(func)(np_cls_prob, np_loc_preds,
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/backend/interpreter.py", line 178, in evaluate
    return self._make_executor(expr)
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/build_module.py", line 404, in _make_executor
    graph_json, mod, params = build(func, target=self.target)
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/build_module.py", line 274, in build
    graph_json, lowered_funcs, params = graph_gen.codegen(func)
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/backend/graph_runtime_codegen.py", line 394, in codegen
    self.heads = self.visit(func.body)
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/expr_functor.py", line 27, in visit
    res = self.visit_call(expr)
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/backend/graph_runtime_codegen.py", line 270, in visit_call
    res = self.visit(arg)
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/expr_functor.py", line 39, in visit
    res = self.visit_tuple_getitem(expr)
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/backend/graph_runtime_codegen.py", line 184, in visit_tuple_getitem
    vtuple = self.visit(op.tuple_value)
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/expr_functor.py", line 27, in visit
    res = self.visit_call(expr)
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/backend/graph_runtime_codegen.py", line 260, in visit_call
    self.target[call_dev_type])
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/backend/compile_engine.py", line 86, in lower
    raise RuntimeError(msg)
RuntimeError: Traceback (most recent call last):
  File "/Users/wayao/anaconda/lib/python2.7/site-packages/tvm-0.5.dev0-py2.7-macosx-10.6-x86_64.egg/tvm/relay/backend/compile_engine.py", line 78, in lower
    return _backend._CompileEngineLower(self, key)
  File "tvm/_ffi/_cython/function.pxi", line 286, in core.FunctionBase.__call__ (tvm/_ffi/_cython/core.cpp:6689)
  File "tvm/_ffi/_cython/function.pxi", line 221, in core.FuncCall (tvm/_ffi/_cython/core.cpp:5851)
  File "tvm/_ffi/_cython/function.pxi", line 210, in core.FuncCall3 (tvm/_ffi/_cython/core.cpp:5761)
  File "tvm/_ffi/_cython/base.pxi", line 143, in core.CALL (tvm/_ffi/_cython/core.cpp:1670)
TVMError: [01:34:38] /Users/wayao/Documents/tvm/src/relay/backend/compile_engine.cc:189: Check failed: tuple_type->fields.size() == outputs.size() (2 vs. 0) 

Stack trace returned 10 entries:
[bt] (0) 0   libtvm.dylib                        0x0000000109e4631b dmlc::StackTrace(unsigned long) + 299
[bt] (1) 1   libtvm.dylib                        0x0000000109e460c4 dmlc::LogMessageFatal::~LogMessageFatal() + 52
[bt] (2) 2   libtvm.dylib                        0x000000010a18e9e7 tvm::relay::ScheduleGetter::VisitExpr_(tvm::relay::CallNode const*) + 3991
[bt] (3) 3   libtvm.dylib                        0x000000010a191e80 std::__1::__function::__func<tvm::relay::ExprFunctor<tvm::relay::Value (tvm::relay::Expr const&)>::InitVTable()::'lambda4'(tvm::NodeRef const&, tvm::relay::ExprFunctor<tvm::relay::Value (tvm::relay::Expr const&)>*), std::__1::allocator<tvm::relay::ExprFunctor<tvm::relay::Value (tvm::relay::Expr const&)>::InitVTable()::'lambda4'(tvm::NodeRef const&, tvm::relay::ExprFunctor<tvm::relay::Value (tvm::relay::Expr const&)>*)>, tvm::relay::Value (tvm::NodeRef const&, tvm::relay::ExprFunctor<tvm::relay::Value (tvm::relay::Expr const&)>*)>::operator()(tvm::NodeRef const&, tvm::relay::ExprFunctor<tvm::relay::Value (tvm::relay::Expr const&)>*&&) + 32
[bt] (4) 4   libtvm.dylib                        0x000000010a18fca5 tvm::IRFunctor<tvm::Array<tvm::Tensor, void> (tvm::NodeRef const&, tvm::relay::ExprFunctor<tvm::Array<tvm::Tensor, void> (tvm::relay::Expr const&)>*)>::operator()(tvm::NodeRef const&, tvm::relay::ExprFunctor<tvm::Array<tvm::Tensor, void> (tvm::relay::Expr const&)>*) const + 149
[bt] (5) 5   libtvm.dylib                        0x000000010a18f777 tvm::relay::ExprFunctor<tvm::Array<tvm::Tensor, void> (tvm::relay::Expr const&)>::VisitExpr(tvm::relay::Expr const&) + 119
[bt] (6) 6   libtvm.dylib                        0x000000010a18cc7c tvm::relay::ScheduleGetter::VisitExpr(tvm::relay::Expr const&) + 300
[bt] (7) 7   libtvm.dylib                        0x000000010a18bcf0 tvm::relay::ScheduleGetter::Create(tvm::relay::Function const&) + 2352
[bt] (8) 8   libtvm.dylib                        0x000000010a18a3bd tvm::relay::CompileEngineImpl::CreateSchedule(tvm::relay::Function const&, tvm::Target const&) + 77
[bt] (9) 9   libtvm.dylib                        0x000000010a1895a5 tvm::relay::CompileEngineImpl::LowerInternal(tvm::relay::CCacheKey const&) + 677


Error during compile func
--------------------------
fn (%p0: Tensor[(1, 3, 3), float32],
    %p1: Tensor[(1, 12), float32],
    %p2: Tensor[(1, 3, 4), float32])
    -> Tuple[Tensor[(1, 3, 6), float32], Tensor[(1,), int32]] {
  %0 = vision.multibox_transform_loc(%p0, %p1, %p2) # ty=Tuple[Tensor[(1, 3, 6), float32], Tensor[(1,), int32]]
  %0
}
--------------------------

It’s worth notice that all tests under topi is fine, and while I rolled back to pervious ir_builder implementation implementation, nnvm/relay tests passed. Is it possible to be related to allocated buffer? So far this is the only op using allocated buffer.

@were

were · January 13, 2019, 7:04pm

Can you rebase my latest hybrid script and commit your latest change to this branch so that I can replicate your results easily?

were · January 13, 2019, 7:52pm

Yet, I cannot replicate your errors.

kevinthesun · January 14, 2019, 3:35am

Updated more detailed error msg while running in Ubuntu.

were · January 14, 2019, 5:10am

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 234, in 'calling callback function'
  File "/home/weng/tvm-dev/yao/python/tvm/_ffi/_ctypes/function.py", line 63, in cfun
    raise ValueError("PackedFunction can only support one return value")
ValueError: PackedFunction can only support one return value
Traceback (most recent call last):
  File "nnvm/tests/python/compiler/test_top_level4.py", line 791, in <module>
    test_multibox_transform_loc()
  File "nnvm/tests/python/compiler/test_top_level4.py", line 569, in test_multibox_transform_loc
    "anchors": (1, num_anchors, 4)})
  File "/home/weng/tvm-dev/yao/nnvm/python/nnvm/compiler/build_module.py", line 305, in build
    graph = graph.apply("GraphCompile")
  File "/home/weng/tvm-dev/yao/nnvm/python/nnvm/graph.py", line 234, in apply
    check_call(_LIB.NNGraphApplyPasses(self.handle, npass, cpass, ctypes.byref(ghandle)))
  File "/home/weng/tvm-dev/yao/nnvm/python/nnvm/_base.py", line 75, in check_call
    raise NNVMError(py_str(_LIB.NNGetLastError()))
nnvm._base.NNVMError: TVMCall CFunc Error:

This is what I get, slightly different from yours.

kevinthesun · January 14, 2019, 6:05am

This is unrelated to hybrid script. The problem is that python topi operator must return a list instead of tuple, the format like

return a, b

is not allowed.