TVM graph quantization is giving poor accuracy

Questions
#1

Hi All,

I tried to quantize the inception model using tvm quantization.
Non-Quantized model accuracy on imagenet validation dataset is 76.1%
Quantized model accuracy on imagenet validation dataset is 15.7%

Below script modification we have done to enable the quantization.

shape_dict = {‘DecodeJpeg/contents’: (299, 299, 3)}
dtype_dict = {‘DecodeJpeg/contents’: ‘uint8’}
mod, params = relay.frontend.from_tensorflow(graph_def,
layout=layout,
shape=shape_dict)
mod = relay.quantize.quantize(mod[‘main’], params)
with relay.build_config(opt_level=3):
graph, lib, params = relay.build(mod,
target=target,
target_host=target_host,
params=params)
m = graph_runtime.create(graph, lib, ctx)
m.set_input(‘DecodeJpeg/contents’, tvm.nd.array(x.astype(dtype)))
m.set_input(**params)
m.run()

Someone please correct me if anything wrong in the script?
Or the current TVM quantization is giving poor accuracy?

2 Likes
#2

I think @vinx13 @ziheng could help this.

#3

What is the structure of your model，I have similar test results. My test results show that all the results are not good except Inception_v3

#4

@ydy I have used inceptionV1 for my test and found the above results.
I am trying to run inceptionV3, but getting some issues related to labels mapping while measuring accuracy. I working on this part.
@ziheng @vinx13 Is this a known issue that the accuracy will be poor? and working is going on towards improving the accuracy?

#5

The recent change may hurt the accuracy, see [Quantization] Problems with recent refactoring changes in the quantization pass