Annotate Target not working

IssacXid · March 11, 2024, 7:56pm

I’m trying to establish the BYOC flow for a customized processor using the C source codegen method. The processor, we call it IMP, supports only single operations and I’ve specified which all relay operators that is to be offloaded to the accelerator in python/tvm/relay/op/contrib/imp.py and registered the codegen and implemented it as per article. Also I’ve added support for the processor in src/runtime as it is done for DNNL to define the APIs for the processor. But, when trying to print the output of the code snippet like: mod = relay.transform.AnnotateTarget(“imp”)(mod) print(mod)
it’s output is giving out to be:

def @main(%w: Tensor[(2, 2), int32] /* ty=Tensor[(2, 2), int32] */, %x: Tensor[(2, 2), int32] /*     ty=Tensor[(2, 2), int32] */, %y: Tensor[(2, 2), int32] /* ty=Tensor[(2, 2), int32] */) -> Tensor[(2, 2), int32] {
      %0 = annotation.compiler_begin(%w, compiler="default") /* ty=Tensor[(2, 2), int32] */;
      %1 = annotation.compiler_begin(%x, compiler="default") /* ty=Tensor[(2, 2), int32] */;
      %2 = add(%0, %1) /* ty=Tensor[(2, 2), int32] */;
      %3 = annotation.compiler_end(%2, compiler="default") /* ty=Tensor[(2, 2), int32] */;
      %4 = annotation.compiler_begin(%3, compiler="default") /* ty=Tensor[(2, 2), int32] */;
      %5 = annotation.compiler_begin(%y, compiler="default") /* ty=Tensor[(2, 2), int32] */;
      %6 = multiply(%4, %5) /* ty=Tensor[(2, 2), int32] */;
      annotation.compiler_end(%6, compiler="default") /* ty=Tensor[(2, 2), int32] */
    }

whereas, in case of relay.transform.AnnotateTarget("dnnl")(mod), it’s output has compiler="dnnl" in it. I’m trying to validate whether the offloading will work or not by checking the annotations. Can anyone help me in figuring out why is this difference in the output of AnnotateTarget present? Thanks in advance for any help.

IssacXid · March 20, 2024, 6:57am

Can you please help? I’ve tried with the BYOC integration PRs also, but couldn’t get much hint on what might be missing. I’ve double-checked the documentation and followed the steps meticulously, but I’m still encountering the Annotate Target issues. @masahi @comaniac @zhiics

cgerum · March 22, 2024, 2:34pm

@IssacXid your sounds, like you forgot to register your target patterns. If you followed the other targets it is most likely sufficient to import your .py module in the init.py of of relay/op/contrib

Otherwise could you add a link to your code, so that we can have a look.

IssacXid · March 27, 2024, 1:37pm

Hello @cgerum, Sorry for the late reply. It would be really helpful if you could look at the changes I’ve made for BYOC integration of IMP. Here’s the link to the byoc branch of the forked repo. In order to build tvm with IMP_BYOC, setting the USE_IMP flag to ON, a libImp.so and a few header files are required. However, I cannot provide the driver files as it is proprietary. I’d used the single operator annotation rules in imp.py and added the .py module in init.py.

I’m using this file for generating the cross-compiled subgraphs: import numpy as np

import tvm

from tvm import te

import tvm.relay as relay

from tvm.contrib import cc, utils, graph_executor

from tvm.contrib.download import download_testdata

import os

target = "llvm -mtriple=aarch64-linux-gnu"

target_host = "llvm -mtriple=aarch64-linux-gnu"

dev = tvm.cpu(0)

def export_tvm_module(lib,module_name):

    if(not os.path.exists("compiled")):

        os.makedirs("compiled")

    kwargs = {}

    kwargs["options"] = ["-O2", "-std=c++14", "-I" + "/home/dut3kor/tvm/src/runtime/contrib", "--sysroot="+"<sysroot for the cross-toolchain>"]

    dylib_path = os.path.join("compiled", module_name + "_compiled.so")

    poky_gcc='<cross toolchain gcc>'

    lib.export_library(dylib_path, fcompile=cc.cross_compiler(poky_gcc), **kwargs) # Cross compile toolchain for risc-v

    print("dumping graph...")

    with open("compiled" + "/"  + module_name + "_graph.json", "w") as f:

        f.write(lib.graph_json)

    print("dumping params...")

    with open("compiled"+ "/" + module_name + "_param.params", "wb") as f:

        f.write(tvm.runtime.save_param_dict(lib.params))

num_r=2

num_c=2

a_np = np.random.choice(10, num_r*num_c).astype("int32").reshape((num_r, num_c))

b_np = np.random.choice(10, num_r*num_c).astype("int32").reshape((num_r, num_c))

c_np = np.random.choice(10, num_r*num_c).astype("int32").reshape((num_r, num_c))

d_np = np.add(a_np, b_np)

e_np = np.multiply(c_np, d_np)

w = relay.var("w", shape=a_np.shape, dtype="int32")

x = relay.var("x", shape=b_np.shape, dtype="int32")

y = relay.var("y", shape=c_np.shape, dtype="int32")

z = relay.add(w, x)

z = relay.multiply(z, y)

func = relay.Function([w, x, y], z)

mod = tvm.IRModule.from_expr(func)

mod = relay.transform.AnnotateTarget("imp")(mod)

print(mod)

mod = relay.transform.InferType()(mod)

print(mod)

mod = relay.transform.PartitionGraph()(mod)

with tvm.transform.PassContext(opt_level=3, disabled_pass=["AlterOpLayout"]):

    lib = relay.build(mod, target=target, params=None)

print(lib.ir_mod)

export_tvm_module(lib,"op_subgraph_relay_aarch64")

It is throwing this error:

Can you check the changes and see if anything else is missing? Thank you for your time.

IssacXid · March 28, 2024, 10:27am

Hello @cgerum, By importing the .py file in init.py file, the AnnotateTarget function gave compiler="imp" as output after resolving the above error by using the registration like this:

    @tvm.ir.register_op_attr("add", "target.imp")
    def _imp_add_wrapper(expr):
      return True

instead of the way given in DNNL byoc doc like:

    @tvm.ir.register_op_attr("add", "target.imp")
    def _imp_add_wrapper(args, attrs):
      return True

in the .py file. But now I’m getting this error:

Traceback (most recent call last):
  File "crossapplication.py", line 56, in <module>
    lib = relay.build(mod, target=target, params=None)
  File "/home/dut3kor/tvm/python/tvm/relay/build_module.py", line 364, in build
    graph_json, runtime_mod, params = bld_mod.build(
  File "/home/dut3kor/tvm/python/tvm/relay/build_module.py", line 161, in build
    self._build(
  File "/home/dut3kor/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in __call__
    raise_last_ffi_error()
  File "/home/dut3kor/tvm/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
    raise py_err
tvm._ffi.base.TVMError: Traceback (most recent call last):
  17: tvm::relay::backend::RelayBuildModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#3}::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
  16: tvm::relay::backend::RelayBuildModule::BuildRelay(tvm::IRModule, tvm::runtime::String const&)
  15: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::relay::backend::GraphExecutorCodegenModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#2}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
  14: tvm::relay::backend::GraphExecutorCodegen::Codegen(tvm::IRModule, tvm::relay::Function, tvm::runtime::String)
  13: tvm::transform::Pass::operator()(tvm::IRModule) const
  12: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  11: tvm::transform::SequentialNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  10: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  9: tvm::transform::ModulePassNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  8: _ZN3tvm7runtime13PackedFuncObj9ExtractorINS0_16PackedFuncSubObjIZNS0_15TypedPackedFuncIFNS_8IRModuleES5_NS_9transform11PassContextEEE17AssignTypedLambdaIZNS_5relay3tec7LowerTEENS0_6StringENS_17CompilationConfigESt8functionIFvNS_8BaseFuncEEEEUlS5_S7_E_EEvT_EUlRKNS0_7TVMArgsEPNS0_11TVMRetValueEE_EEE4CallEPKS1_SL_SP_
  7: tvm::relay::tec::LowerTE(tvm::IRModule const&, tvm::runtime::String const&, std::function<void (tvm::BaseFunc)>, tvm::CompilationConfig)
  6: tvm::relay::tec::TECompilerImpl::LowerExternalFunctions()
  5: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::TypedPackedFunc<tvm::runtime::Module (tvm::runtime::ObjectRef const&)>::AssignTypedLambda<tvm::runtime::Module (*)(tvm::runtime::ObjectRef const&)>(tvm::runtime::Module (*)(tvm::runtime::ObjectRef const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
  4: tvm::relay::contrib::IMPCompiler(tvm::runtime::ObjectRef const&)
  3: tvm::relay::contrib::IMPModuleCodegen::CreateCSourceModule(tvm::runtime::ObjectRef const&)
  2: _ZN3tvm7runtime13PackedFuncObj
  1: tvm::runtime::TypedPackedFunc<tvm::runtime::Module (tvm::runtime::String, tvm::runtime::String, tvm::runtime::Array<tvm::runtime::String, void>, tvm::runtime::Array<tvm::runtime::String, void>)>::AssignTypedLambda<tvm::codegen::{lambda(tvm::runtime::String, tvm::runtime::String, tvm::runtime::Array<tvm::runtime::String, void>, tvm::runtime::Array<tvm::runtime::String, void>)#1}>(tvm::codegen::{lambda(tvm::runtime::String, tvm::runtime::String, tvm::runtime::Array<tvm::runtime::String, void>, tvm::runtime::Array<tvm::runtime::String, void>)#1}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs const, tvm::runtime::TVMRetValue) const
  0: tvm::runtime::TVMMovableArgValueWithContext_::operator tvm::runtime::Array<tvm::runtime::String, void><tvm::runtime::Array<tvm::runtime::String, void> >() const
  18: tvm::relay::backend::RelayBuildModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#3}::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
  17: tvm::relay::backend::RelayBuildModule::BuildRelay(tvm::IRModule, tvm::runtime::String const&)
  16: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::relay::backend::GraphExecutorCodegenModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#2}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
  15: tvm::relay::backend::GraphExecutorCodegen::Codegen(tvm::IRModule, tvm::relay::Function, tvm::runtime::String)
  14: tvm::transform::Pass::operator()(tvm::IRModule) const
  13: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  12: tvm::transform::SequentialNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  11: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  10: tvm::transform::ModulePassNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  9: _ZN3tvm7runtime13PackedFuncObj9ExtractorINS0_16PackedFuncSubObjIZNS0_15TypedPackedFuncIFNS_8IRModuleES5_NS_9transform11PassContextEEE17AssignTypedLambdaIZNS_5relay3tec7LowerTEENS0_6StringENS_17CompilationConfigESt8functionIFvNS_8BaseFuncEEEEUlS5_S7_E_EEvT_EUlRKNS0_7TVMArgsEPNS0_11TVMRetValueEE_EEE4CallEPKS1_SL_SP_
  8: tvm::relay::tec::LowerTE(tvm::IRModule const&, tvm::runtime::String const&, std::function<void (tvm::BaseFunc)>, tvm::CompilationConfig)
  7: tvm::relay::tec::TECompilerImpl::LowerExternalFunctions()
  6: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::TypedPackedFunc<tvm::runtime::Module (tvm::runtime::ObjectRef const&)>::AssignTypedLambda<tvm::runtime::Module (*)(tvm::runtime::ObjectRef const&)>(tvm::runtime::Module (*)(tvm::runtime::ObjectRef const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
  5: tvm::relay::contrib::IMPCompiler(tvm::runtime::ObjectRef const&)
  4: tvm::relay::contrib::IMPModuleCodegen::CreateCSourceModule(tvm::runtime::ObjectRef const&)
  3: _ZN3tvm7runtime13PackedFuncObj
  2: tvm::runtime::TypedPackedFunc<tvm::runtime::Module (tvm::runtime::String, tvm::runtime::String, tvm::runtime::Array<tvm::runtime::String, void>, tvm::runtime::Array<tvm::runtime::String, void>)>::AssignTypedLambda<tvm::codegen::{lambda(tvm::runtime::String, tvm::runtime::String, tvm::runtime::Array<tvm::runtime::String, void>, tvm::runtime::Array<tvm::runtime::String, void>)#1}>(tvm::codegen::{lambda(tvm::runtime::String, tvm::runtime::String, tvm::runtime::Array<tvm::runtime::String, void>, tvm::runtime::Array<tvm::runtime::String, void>)#1}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs const, tvm::runtime::TVMRetValue) const
  1: tvm::runtime::TVMMovableArgValueWithContext_::operator tvm::runtime::Array<tvm::runtime::String, void><tvm::runtime::Array<tvm::runtime::String, void> >() const
  0: tvm::runtime::Array<tvm::runtime::String, void> tvm::runtime::TVMPODValue_::AsObjectRef<tvm::runtime::Array<tvm::runtime::String, void> >() const
  File "/home/dut3kor/tvm/include/tvm/runtime/packed_func.h", line 785
TVMError: In function runtime.CSourceModuleCreate(0: runtime.String, 1: runtime.String, 2: Array<runtime.String>, 3: Array<runtime.String>) -> runtime.Module: error while converting argument 2: [15:02:14] /home/dut3kor/tvm/include/tvm/runtime/packed_func.h:2056: InternalError: Check failed: (!checked_type.defined()) is false: Expected Array[runtime.String], but got runtime.String

Can you give some suggestions on this? Thanks in advance.

cgerum · March 28, 2024, 11:18am

Hi @IssacXid,

The error probably originates from this line: tvm/src/relay/backend/contrib/imp/codegen.cc at 32805429ebcb3fd1b9a7144330a0e0b49d50f231 · IssacXid/tvm · GitHub .

sym should be an Array>String> here.

See for example here: tvm/src/relay/backend/contrib/dnnl/codegen.cc at 83e7e9b2eb8dbeeb16dcfdbaf3336caa81071877 · apache/tvm · GitHub

Kind Regards, Christoph

IssacXid · April 2, 2024, 4:19am

Thanks @cgerum, it worked the way you suggested. Now, relay.build() is working.