[Relay][ONNX][Frontend] Implement Resize Operation

uenoku · December 16, 2019, 4:23am

Hi all,

I’m trying to create a converter for ONNX Resize these days. As far as I see relay/frontend/onnx.py, a conveter for Resize is not implemented now.

But I’m having difficulty because ONNX Resize is generalized to N dim and has recursion.

I guess I need to simulate this function in relay.

github.com

onnx/onnx/blob/79bd5042ada4d9fa6004f351bce0704a566522ea/onnx/backend/test/case/node/resize.py#L161




    if exclude_outside:
        for i, idx in enumerate(idxes):
            if idx < 0 or idx >= input_width:
                coeffs[i] = 0
        coeffs /= sum(coeffs)


    return np.dot(coeffs, points).item()




def interpolate_nd_with_x(data,                      # type: np.ndarray
                          n,                         # type: int
                          scale_factors,             # type: List[float]
                          x,                         # type: List[float]
                          get_coeffs,                # type: Callable[[float], np.ndarray]
                          roi=None,                  # type: np.ndarray
                          **kwargs                   # type: Any
                          ):                         # type: (...) -> np.ndarray
    if n == 1:
        return interpolate_1d_with_x(data, scale_factors[0], x[0], get_coeffs, roi=roi,
                                     **kwargs)

def interpolate_nd_with_x(data,                      # type: np.ndarray
                          n,                         # type: int
                          scale_factors,             # type: List[float]
                          x,                         # type: List[float]
                          get_coeffs,                # type: Callable[[float], np.ndarray]
                          roi=None,                  # type: np.ndarray
                          **kwargs                   # type: Any
                          ):                         # type: (...) -> np.ndarray
    if n == 1:
        return interpolate_1d_with_x(data, scale_factors[0], x[0], get_coeffs, roi=roi,
                                     **kwargs)
    return interpolate_1d_with_x(
        [interpolate_nd_with_x(data[i], n - 1, scale_factors[1:], x[1:], get_coeffs,
                               roi=None if roi is None else np.concatenate(
                                   [roi[1:n], roi[n + 1:]]),
                               **kwargs)
         for i in range(data.shape[0])], scale_factors[0], x[0], get_coeffs,
        roi=None if roi is None else [roi[0], roi[n]], **kwargs)

How can I implement a converter for such kinds of recursive functions?

Thanks in advance.

masahi · December 16, 2019, 5:10am

Since we only support 4D inputs for resize op, you don’t have to implement a generic ND Resize op converter.

I have a very basic converter working that ignores all weird options in ONNX spec. Maybe you can start from here.

class Resize(OnnxOpConverter):
    @classmethod
    def _impl_v11(cls, inputs, attr, params):
        mode = attr.get('mode')
        if mode == b'nearest':
            method = "nearest_neighbor"
        elif mode == b'linear':
            method = "bilinear"
        else:
            raise tvm.error.OpAttributeInvalid(
                'Value {} in attribute "mode" of operator Resize is not valid.'.format(mode))
        scale = infer_value_simulated(inputs[2], params).asnumpy()
        size = infer_value_simulated(inputs[3], params).asnumpy()
        in_size = np.array(infer_shape(inputs[0]))
        if len(scale) != 0:
            assert len(size) == 0
            size = in_size * scale
        else:
            assert len(size) != 0
        return _op.image.resize(inputs[0], (size[-2], size[-1]), "NCHW", method, align_corners=True)

uenoku · December 16, 2019, 6:33am

Thank you! It is really helpful. I’ll try it.

jonso · December 16, 2019, 6:31pm

@masahi, any chance you can commit this to the public repo? You can add a comment over it that it does not support all of the weird ONNX cases.

masahi · December 16, 2019, 9:59pm

@jonso ok, I’ll see if I can make sense of the coordinate_transformation_mode option in the onnx spec. Among the possible options, it seems TVM resize op supports “align_corners” and “asymmetric”, but PyTorch actually uses “pytorch_half_pixel” when align_corners = False. So for the PyTorch use case we need to have “half_pixel” option too (“pytorch_half_pixel” being practically the same as “half_pixel”).

Rather than passing “align_corners” boolean flag, to better support ONNX models we should have a more general “coordinate_transformation_mode” or similar in our resize op. As a possible value for the “coordinate_transformation_mode” option, we can support

asymmetric (for nearest)
align_corners
half_pixel (for PyTorch and possibly other use cases)

What do you think? @jonso @jwfromm Let me know if you need to support other options. “tf_crop_and_resize” looks complicated and requires other arguments to be passed in, so out of scope for now.

jwfromm · December 16, 2019, 10:33pm

@masahi, I’m definitely in favor of a more general transformation argument and think the three you’ve listed should cover just about everything. One thing to note is that there is a PR open right now that adds crop_and_resize #4417 and includes a refactor of some of the existing resize functions.

masahi · December 20, 2019, 3:35am

@uenoku I have a PR at https://github.com/apache/incubator-tvm/pull/4536 which should handle most use cases of converting ONNX resize op. But it won’t be merged soon because of CI issues. You can use that one in the mean time.

uenoku · December 20, 2019, 5:20am

Thank you. The current implementation is working well for my use case:)