Use block data format for whole model


#1

I’d like to ask if there is any plan or any simple way to enable block (NCHWc) data format for whole model instead of plain (NCHW)?

After some playing with TVM it seems that the preferred data type is plain. Significant exceptions are for Intel CPU and Intel Graphics scheduler which are capable of converting from plain to block and back, but it’s not optimal as overall approach.

It appears that Intel HW prefers block data format and it would be profitable for performance to remove those conversions all together. The idea was to add on the model reading stage decision if model should be converted to block or plain, but it requires a lot of other changes down the compilation pipeline.

Could anyone comment here what’s the plan or solution?


#2

Check this tutorial out which tunes networks to take advantage of AVX512, and thus imposes NCHWc constraint on the data layout: https://docs.tvm.ai/tutorials/autotvm/tune_nnvm_x86.html


#3

Ok, if I understood this example correctly here is a way to take advantage of NCHWc in case of model already being in this format. What I’m looking for is more universal approach, allowing to convert whole model to NCHWc and not each operation.


#4

We do support layout conversion passes, what you talked about is related to AlterOpLayout pass. see related discussions [RFC] Functionality of AlterOpLayout and Possible Refactoring