Performance regression with relay (opt_level=3)

[gasgallo]

I’m having some problems when deploying this model on x86 architecture using relay.
When setting opt_level=3 performance get worse by almost 3 times compared to using opt_level=1.
Any clue why does this happen?

I’ve also tuned convolution on x86 using autotvm, but inference is still quite slow. Is there any other way I can further speed up the inference?

[kevinthesun]

Recently there are several similar issues on x86. Did you try debugger runtime to see the performance difference between opt_level=1/3?

[tico]

Hi @kevinthesun,

I have been also working on deploying model on x86 and getting bad performance results. For example, the situation in explained in the following post:

[Quantization][AutoTVM] Performance degradation due to transpose ops when NCHW layout is used

Hi, I would like to deploy a tensorflow model in x86 and also use quantization. First, after doing some preliminary benchmarking I noticed that the NHWC provides better performance than NCHW. However, for quantization NCHW is required, but this results in a performance degradation. Moreover, I noticed that due to the use of the NCHW layout, lots of transpose ops are added and this might be the reason of the performance degradation. When I tune with AutoTVM, the performance of the model gets be…

You mentioned that recently there are issues on x86. Could you please elaborate on the current situation for x86?

Thanks

[kevinthesun]

quantization is a different topic. @janimesh

[kevinthesun]

Can you share your tuning_option? That might be the reason why autotuning can’t get great performance.