Pre-quantized model ingestion

alopez_13 · January 31, 2020, 4:44pm

Hello,

I was trying a pre-quantized model in TFLite when I got an “NotImplementedError: Tensor type 9 is currently not supported” So I looked at the tflite.py ingestion code and found explicit checks that disable INT8 tensors. That was OK on old versions of TFLite, but according to the TFLite spec [https://www.tensorflow.org/lite/performance/quantization_spec] they have switched to symmetric quantization so values should be encoded as INT8.

I started adding code to support the INT8 data type and removing the assertions on the type checking for UINT8 only, but I wonder if there is a more disciplined way of doing this since I don’t want to break the ingestion process if there are assumptions on data types.

Also, is TVM going to support ingestion of pre-quantized models created with newer versions of TFLite? I know there is a quantizer in TVM, I have used it already, but there is still the case for ingesting pre-quantized models created in more recent versions of TFLite.

Thanks!

janimesh · February 3, 2020, 10:26pm

Thanks @alopez_13

Yes, we are invested in pre-quantized model support. Older TFLite version had uint8 inputs, and thus the checks that you see.

What you are doing is the right way to go. There is an e2e test for the older version of TFLite, so you wont break any current ingestion process. Obviously, we will do a PR review, so any other major issues will be addressed as well

Thanks for working on this. Let me know if you need any help.

alopez_13 · February 5, 2020, 7:30pm

Right now I need to take care of some other loose ends, but I expect to work on this ingestion path in the near future. I’ll let you know if I get stuck or if I can generate code to ingest TFLite files with INT8 inputs.

ramana-arm · February 8, 2020, 1:39am

Our CI currently only tests tflite 1.13.1 and thus keeping this up to date with newer models remains a challenge. I have off and on tried to move up to tflite 1.15 and tensorflow 1.15 but that’s proved harder than I thought.

So before we start ingesting new models , the community needs to decide whether we are going to have 2 different testing modes for tf1.15 and tf2.x and what versions of the frameworks are we going to test in our CI to ensure test coverage for newer operators as they land.

There are also some tests in the testsuite which are version specific as certain data types of certain operators are only available in newer versions of tflite IIRC.

Ramana