How to build a module with one op using tvm

puddingfjz · April 6, 2020, 11:00am

I am trying to use the templates which are implemented by tvm to tune single operators. I use the code mentioned in this discussion. The code is:

import os

import numpy as np

import tvm
from tvm import te
from tvm import autotvm
from tvm import relay
import tvm.relay.testing
from tvm.autotvm.tuner import XGBTuner, GATuner, RandomTuner, GridSearchTuner
from tvm.contrib.util import tempdir
import tvm.contrib.graph_runtime as runtime

target = tvm.target.cuda()

N, H, W, CO, CI, KH, KW, strides, padding = 1, 7, 7, 512, 512, 3, 3, (1, 1), (1, 1)
data = te.placeholder((N, CI, H, W), name='data')
kernel = te.placeholder((CO, CI, KH, KW), name='kernel')
dilation=1
dtype = "float32"
kernel_shape = (CO, CI, KH, KW)

ctx = tvm.gpu()

out = relay.nn.conv2d(data, kernel, strides=strides, padding=padding, dilation=dilation, channels = CO, kernel_size = (KH, KW), data_layout='NCHW',out_layout='NCHW', out_dtype=dtype)

mod = relay.Module.from_expr(out)

kernel_weights = tvm.nd.array(np.ones(kernel_shape, dtype=dtype), ctx)

dict_params = {'kernel': kernel_weights}

task = autotvm.task.extract_from_program(mod["main"], target=target, params=dict_params, ops=(relay.op.nn.conv2d,))

The output is:

Traceback (most recent call last):

  File "tune_single_op_builtin_tmp.py", line 25, in <module>
    out = relay.nn.conv2d(data, kernel, strides=strides, padding=padding, dilation=dilation, channels = CO, kernel_size = (KH, KW), data_layout='NCHW',out_layout='NCHW', out_dtype=dtype)

  File "/usr/tvm/python/tvm/relay/op/nn/nn.py", line 209, in conv2d
    kernel_layout, out_layout, out_dtype)

  File "/usr/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 213, in __call__
    raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (3) /usr/tvm/build/libtvm.so(TVMFuncCall+0x65) [0x7fe8d57fab85]
  [bt] (2) /usr/tvm/build/libtvm.so(+0x7f1898) [0x7fe8d53d6898]
  [bt] (1) /usr/tvm/build/libtvm.so(tvm::RelayExpr tvm::runtime::TVMPODValue_::AsObjectRef<tvm::RelayExpr>() const+0x14b) [0x7fe8d4fd7d5b]
  [bt] (0) /usr/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x43) [0x7fe8d4f75d63]
  File "/usr/tvm/include/tvm/runtime/packed_func.h", line 1456
TVMError: Check failed: ObjectTypeChecker<TObjectRef>: :Check(ptr): Expect relay.Expr but get Tensor

Can anyone help on this?

puddingfjz · April 6, 2020, 5:31pm

I changed my code as follows:

import os

import numpy as np

import tvm
from tvm import te
from tvm import autotvm
from tvm import relay
import tvm.relay.testing
from tvm.autotvm.tuner import XGBTuner, GATuner, RandomTuner, GridSearchTuner
from tvm.contrib.util import tempdir
import tvm.contrib.graph_runtime as runtime

target = tvm.target.cuda()

N, H, W, CO, CI, KH, KW, strides, padding = 1, 7, 7, 512, 512, 3, 3, (1, 1), (1, 1)
data = relay.var('data', shape=(N, CI, H, W))
#data = te.placeholder((N, CI, H, W), name='data')
#kernel = relay.const(np.random.random((CO, CI, KH, KW)).astype("float32"))
kernel = relay.var('kernel', shape=(CO, CI, KH, KW))
#kernel = te.placeholder((CO, CI, KH, KW), name='kernel')
dilation=(1,1)
dtype = "float32"
kernel_shape = (CO, CI, KH, KW)

ctx = tvm.gpu()

out = relay.op.nn.nn.conv2d(data, kernel, strides=strides, padding=padding, dilation=dilation, channels = CO, kernel_size = (KH, KW), data_layout='NCHW', out_dtype=dtype)

mod = tvm.IRModule.from_expr(out)

kernel_weights = tvm.nd.array(np.ones(kernel_shape, dtype=dtype), ctx)

dict_params = {'kernel': kernel_weights}

task = autotvm.task.extract_from_program(mod["main"], target=target, params=dict_params, ops=(relay.op.nn.nn.conv2d,))
print(len(task))

The output is 0. It seems that no task is extracted. Could anyone give some advice on this situation? Thanks in advance.

OriAlpha · April 6, 2020, 10:58pm

your code seems to be fine but change the ops to

ops=(relay.op.get(“nn.conv2d”),)

then it will work fine and remember the ops has to be in the same format specified in TVM else you will get an empty array as the return value

for more info, you were stuck at for

task_name, args in env.get_tasks():

in replay_integration.py debug internally you get more info

puddingfjz · April 7, 2020, 4:54am

Thank you for your reply. The code runs well now. But I still have some questions:

How can I know which name to be the parameter of the function here:

How to determine the params here:

Which variables are supposed to be the params? How can I know the name of a variable to use as the key in the dictionary (is the name the same as what I define in kernel = relay.var('kernel', shape=(CO, CI, KH, KW)))? If I make the kernel be a constant instead of a variable (using relay.const), then can I use an empty dictionary as the params for autotvm.task.extract_from_program()?

OriAlpha · April 7, 2020, 8:32am

For these

How can I know which name to be the parameter of the function here:

you can find functions for ops in replay.op.nn.nn.py in TVM build files

You cant use the empty dictionary as param, param should have all the conv terms in it for tuning these can contain a param

puddingfjz · April 7, 2020, 8:57am

Thank you for your reply.

For the question about

I find that the function in the file is as follows:

github.com

apache/incubator-tvm/blob/dbd805c171ad8f66b78bbaeb3eaa4bd8e562de6a/python/tvm/relay/op/nn/nn.py#L118


        strides = (strides, )
    if isinstance(dilation, int):
        dilation = (dilation, )
    if isinstance(padding, int):
        padding = (padding, padding)
    return _make.conv1d(data, weight, strides, padding, dilation,
                        groups, channels, kernel_size, data_layout,
                        kernel_layout, out_layout, out_dtype)




def conv2d(data,
           weight,
           strides=(1, 1),
           padding=(0, 0),
           dilation=(1, 1),
           groups=1,
           channels=None,
           kernel_size=None,
           data_layout="NCHW",
           kernel_layout="OIHW",
           out_layout="",

So can I write the function as relay.op.get(“conv2d”) instead of relay.op.get(“nn.conv2d”)?

You privide an example of a param,

but I still do not quite understand it, could you explain more?

You also mentioned that

But in my previous code, I only put kernel in the params, not including other parameters for conv2d like data, strides, … Could you explain more about this?

Thank you very much！

======================================================

I try this code below:

target = tvm.target.cuda()

N, H, W, CO, CI, KH, KW, strides, padding = 1, 7, 7, 512, 512, 3, 3, (1, 1), (1, 1)
data = relay.var('data', shape=(N, CI, H, W))
#data = te.placeholder((N, CI, H, W), name='data')
#kernel = relay.const(np.random.random((CO, CI, KH, KW)).astype("float32"))
kernel = relay.var('kernel', shape=(CO, CI, KH, KW))
#kernel = te.placeholder((CO, CI, KH, KW), name='kernel')
dilation=(1,1)
dtype = "float32"
kernel_shape = (CO, CI, KH, KW)

ctx = tvm.gpu()

out = relay.op.nn.nn.conv2d(data, kernel, strides=strides, padding=padding, dilation=dilation, channels = CO, kernel_size = (KH, KW), data_layout='NCHW', out_dtype=dtype)

mod = tvm.IRModule.from_expr(out)

kernel_weights = tvm.nd.array(np.ones(kernel_shape, dtype=dtype), ctx)

dict_params = {'kernel': kernel_weights}
dict_params = dict()

task = autotvm.task.extract_from_program(mod["main"], target=target, params=dict_params, ops=(relay.op.get('nn.conv2d'),))
print(task[0].config_space)
print(task[1].config_space)
env = autotvm.task.topi_integration.TaskExtractEnv.get()
print(env.get_tasks())

It seems that using an empty dictionary as params can still extract tasks successfully:

ConfigSpace (len=844800, space_map=
   0 tile_f: Split(policy=factors, product=512, num_outputs=4) len=220
   1 tile_y: Split(policy=factors, product=7, num_outputs=4) len=4
   2 tile_x: Split(policy=factors, product=7, num_outputs=4) len=4
   3 tile_rc: Split(policy=factors, product=512, num_outputs=2) len=10
   4 tile_ry: Split(policy=factors, product=3, num_outputs=2) len=2
   5 tile_rx: Split(policy=factors, product=3, num_outputs=2) len=2
   6 auto_unroll_max_step: OtherOption([0, 512, 1500]) len=3
   7 unroll_explicit: OtherOption([0, 1]) len=2
)
ConfigSpace (len=462000, space_map=
   0 tile_b: Split(policy=factors, product=16, num_outputs=4) len=1
   1 tile_y: Split(policy=factors, product=512, num_outputs=4) len=220
   2 tile_x: Split(policy=factors, product=16, num_outputs=4) len=35
   3 tile_rc: Split(policy=factors, product=512, num_outputs=2) len=10
   4 auto_unroll_max_step: OtherOption([0, 128, 1500]) len=3
   5 unroll_explicit: OtherOption([0, 1]) len=2
)
[('conv2d_nchw.cuda', (('TENSOR', (1, 512, 7, 7), 'float32'), ('TENSOR', (512, 512, 3, 3), 'float32'), (1, 1), (1, 1, 1, 1), (1, 1), 'float32')), ('conv2d_nchw_winograd.cuda', (('TENSOR', (1, 512, 7, 7), 'float32'), ('TENSOR', (512, 512, 3, 3), 'float32'), (1, 1), (1, 1, 1, 1), (1, 1), 'float32'))]

OriAlpha · April 7, 2020, 1:31pm

ops=(relay.op.get(‘nn.conv2d’),)…This works you can use this.
for param it has elements of type, target info (ctx), shape these are the most important

in your case, it can be as followed

About your code

here the params are passed in the variable mods so you are able to get results

puddingfjz · April 7, 2020, 1:46pm

Thank you for your reply.

If I want to build a module with one log_softmax operator, then I can use ops=(relay.op.get(‘nn.log_softmax’),)? (Is my guess right? The name is just nn + operator_name?)

For the params, according to my understanding, for the conv2d operator, all parameters it needs are data, weight, strides, padding, dilation, groups, channels, kernel_size, data_layout, kernel_layout, out_layout, out_dtype, but only data and kernel are set to be variables by me. So I only need to pass data and kernel as the params to extract tasks.

If I pass kernel, then it means that the kernel value during the module inference will not change. If I pass an empty dictionary to the function, then it means that both kernel and data are treated as variables during the module inference.

Is my understanding right? Sorry for bothering you so many times.

OriAlpha · April 7, 2020, 1:58pm

Correct and you can modify for ur need.
ya these are the values need parameters for function data, weight, strides, padding, dilation, groups, channels, kernel_size, data_layout, kernel_layout, out_layout, out_dtype,
exactly, you have to mention else it takes kernel and data

No need to apologize, this helps to understand more by questions

puddingfjz · April 7, 2020, 2:06pm

Got it. Thank you again.