Autotvm - A fallback configuration is used, which may bring great performance regression

Rms45 · March 10, 2020, 2:51pm

Hi,

I am trying to run an online demo of the gesture recognition model on the NVIDIA Jetson TX2 and the model uses TVM to autotune for the underlying ARM CPU.

I get the following warning multiple times: … WARNING:autotvm:Cannot find config for target=llvm -target=aarch64-linux-gnu, workload=(‘conv2d’, (1, 3, 224, 224, ‘float32’), (32, 3, 3, 3, ‘float32’), (2, 2), (1, 1), (1, 1), ‘NCHW’, ‘float32’). A fallback configuration is used, which may bring great performance regression. …

I also noticed that the performance is degraded as the warning suggests.

Some info about setup: The model uses MobileNetV2 as backbone, and I am trying to use only the CPU to run the model. I tried modifying the -target argument after finding the target from running “gcc -v” on the device. Also i do not use RPC from the host. I am running the demo on the device.

Can someone tell me how to fix this?

comaniac · March 10, 2020, 4:30pm

Did you tune the model by yourself before building it? And how did you build the model?

Rms45 · March 13, 2020, 10:16am

The online demo uses a pretrained torch model. I modified the target and ran the demo. The demo seems to download a torch model and convert to onnx and then to tvm module using the specified target. Hence i expect that model should already be tuned to be run on TX2. Or am i wrong somewhere?

Apart from this, for target = ‘cuda’, there seems to only one warning of this kind ,

WARNING:autotvm:Cannot find config for target=cuda, workload=(‘dense’, (1, 1280, ‘float32’), (27, 1280, ‘float32’), 0, ‘float32’). A fallback configuration is used, which may bring great performance regression.

This is the code which converts torch to tvm using relay:

github.com

mit-han-lab/temporal-shift-module/blob/832a758f0c1e4a835cb0a47d957eff776d35dd91/online_demo/main.py#L21-L38


def torch2tvm_module(torch_module: torch.nn.Module, torch_inputs: Tuple[torch.Tensor, ...], target):
    torch_module.eval()
    input_names = []
    input_shapes = {}
    with torch.no_grad():
        for index, torch_input in enumerate(torch_inputs):
            name = "i" + str(index)
            input_names.append(name)
            input_shapes[name] = torch_input.shape
        buffer = io.BytesIO()
        torch.onnx.export(torch_module, torch_inputs, buffer, input_names=input_names, output_names=["o" + str(i) for i in range(len(torch_inputs))])
        outs = torch_module(*torch_inputs)
        buffer.seek(0, 0)
        onnx_model = onnx.load_model(buffer)
        relay_module, params = tvm.relay.frontend.from_onnx(onnx_model, shape=input_shapes)
    with tvm.relay.build_config(opt_level=3):
        graph, tvm_module, params = tvm.relay.build(relay_module, target, params=params)
    return graph, tvm_module, params

comaniac · March 13, 2020, 4:20pm

So it’s not doing auto-tuning. If you didn’t tune the model by yourself, TVM will try to use pre-tuned logs, but if a workload in your model doesn’t appear in the pre-tuned logs, you will see the WARNING.

You could follow this tutorial to tune your model on ARM CPU or TX2 after converting it to Relay: https://docs.tvm.ai/tutorials/autotvm/tune_relay_arm.html#sphx-glr-tutorials-autotvm-tune-relay-arm-py

Rms45 · March 16, 2020, 11:00am

Is there any way to see these pre-tuned logs?

Managed to find the link to the log file being downloaded by TVM during compilation.

comaniac · March 16, 2020, 4:24pm

https://github.com/uwsampl/tvm-distro/tree/master/tophub