I am trying to build tvm relay for the following model BTS
while converting i am getting the following error
/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/relay/frontend/pytorch.py in from_pytorch(script_module, input_shapes, custom_convert_map)
2254
2255 ret = convert_operators(_get_operator_nodes(graph.nodes()),
-> 2256 outputs, ret_name, convert_map, prelude)
2257
2258 mod["main"] = tvm.relay.Function(_analysis.free_vars(ret[0]), ret[0])
~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/relay/frontend/pytorch.py in convert_operators(operators, outputs, ret_names, convert_map, prelude)
2168 else:
2169 relay_op = convert_map[operator]
-> 2170 relay_out = relay_op(inputs, _get_input_types(op_node))
2171
2172 if isinstance(relay_out, tuple):
~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/relay/frontend/pytorch.py in _impl(inputs, input_types)
324 print("data>>>",data)
325 print("reps>>>",reps)
--> 326 return _op.transform.tile(data, reps=reps)
327 return _impl
328
~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/relay/op/transform.py in tile(data, reps)
446 """
447
--> 448 return _make.tile(data, reps)
449
450
~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/_ffi/_ctypes/packed_func.py in __call__(self, *args)
208 """
209 temp_args = []
--> 210 values, tcodes, num_args = _make_tvm_args(args, temp_args)
211 ret_val = TVMValue()
212 ret_tcode = ctypes.c_int()
~/Documents/Projects/Kyocera_depth_estimation/optimization/tvm/python/tvm/_ffi/_ctypes/packed_func.py in _make_tvm_args(args, temp_args)
174 temp_args.append(arg)
175 else:
--> 176 raise TypeError("Don't know how to handle type %s" % type(arg))
177 return values, type_codes, num_args
178
**TypeError: Don't know how to handle type <class 'torch.Tensor'>**
I have already raised an issue which is as follows
As suggested in the above forum Currently tvm-pytorch frontend doesnt support taking each value of reps as another tensor or function. It is expecting as an simple constant and in this case we are getting a multiplication operator,
i tried changing the lines in local_planar_guidance method in bts.py from
u = self.u.repeat(plane_eq.size(0), plane_eq.size(2) * int(self.upratio), plane_eq.size(3))
v = self.v.repeat(plane_eq.size(0), plane_eq.size(2), plane_eq.size(3) * int(self.upratio))
to
u = self.u.repeat(plane_eq.size(0), int(plane_eq.size(2) * int(self.upratio)), plane_eq.size(3))
v = self.v.repeat(plane_eq.size(0), plane_eq.size(2), int(plane_eq.size(3) * int(self.upratio)))
but still the error remains same and additionally i am getting the following warning
/home/gopinathr/Documents/Projects/Kyocera_depth_estimation/optimization/bts_tvm/bts_inference_code/bts.py:151: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
u = self.u.repeat(plane_eq.size(0), int(plane_eq.size(2) * int(self.upratio)), plane_eq.size(3))
/home/gopinathr/Documents/Projects/Kyocera_depth_estimation/optimization/bts_tvm/bts_inference_code/bts.py:156: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
v = self.v.repeat(plane_eq.size(0), plane_eq.size(2), int(plane_eq.size(3) * int(self.upratio)))
I found that this is a fundamental limitation of tracing: You can’t trace arbitrary Python objects, including numbers, and they will be constant to the traced functions. So the typical workaround is to trace a wrapper for repeat
so i added a wrapper for repeat as follows
@torch.jit.script
def repeat_u(x, plane_eq, upratio:int):
return x.repeat(plane_eq.size(0), int(plane_eq.size(2)) * upratio, plane_eq.size(3))
@torch.jit.script
def repeat_v(y, plane_eq, upratio:int):
return y.repeat(plane_eq.size(0), plane_eq.size(2), int(plane_eq.size(3)) * upratio)
and changed the linesin local_planar_guidance module from
u = self.u.repeat(plane_eq.size(0), plane_eq.size(2) * int(self.upratio), plane_eq.size(3))
v = self.v.repeat(plane_eq.size(0), plane_eq.size(2), plane_eq.size(3) * int(self.upratio))
to
u = repeat_u(self.u, plane_eq, int(self.upratio))
v = repeat_v(self.v, plane_eq, int(self.upratio))
this resolved the warning while tracing but still i am getting TypeError: Don’t know how to handle type <class ‘torch.Tensor’>
before usinga torch.jit.script wrapper for tracing following are the values of input and input_types for repeat are
input>>> [tensor([[[0., 1., 2., 3., 4., 5., 6., 7.]]]), Var(1739, ty=TensorType([1, 1216, 44], float32))] input_types>>> [‘float’, ‘ListType’] data>>> tensor([[[0., 1., 2., 3., 4., 5., 6., 7.]]]) reps>>> (1, 1216, 44)
values of input and input_types for repeat when i use torch.jit.script wrapper are
input>>> [tensor([[[0., 1., 2., 3., 4., 5., 6., 7.]]]), [1, CallNode(Op(multiply), [Constant(152), Constant(8)], (nullptr), []), 44]] input_types>>> [‘float’, ‘ListType’] data>>> tensor([[[0., 1., 2., 3., 4., 5., 6., 7.]]]) reps>>> [1, CallNode(Op(multiply), [Constant(152), Constant(8)], (nullptr), []), 44]
If anyoneknow what is the reason and how to resolve this it would be really helpful