KeyError: 'tile_ic' when tuning MobileNet for mobile GPU


#1

Hello,
I’m trying to auto-tune MobileNet (TensorFlow front-end) for mobile GPUs (Adreno 630 GPU in Google Pixel 3 and Adreno 430 GPU in Snapdragon 810 board).

It seems the tuning itself works well.
However, when compiling the model with the tuned records, I get the following error:

Tuning…
[Task 1/20] Current/Best: 21.40/ 43.65 GFLOPS | Progress: (216/1000) | 304.67 s Done.
[Task 2/20] Current/Best: 3.97/ 3.97 GFLOPS | Progress: (1/1000) | 1.08 s Done.
[Task 3/20] Current/Best: 20.59/ 35.72 GFLOPS | Progress: (644/1000) | 1326.11 s Done.
[Task 4/20] Current/Best: 2.23/ 2.23 GFLOPS | Progress: (1/1000) | 0.86 s Done.
[Task 5/20] Current/Best: 12.50/ 48.18 GFLOPS | Progress: (472/1000) | 1130.66 s Done.
[Task 6/20] Current/Best: 3.43/ 3.43 GFLOPS | Progress: (1/1000) | 6.41 s Done.
[Task 7/20] Current/Best: 30.32/ 46.65 GFLOPS | Progress: (520/1000) | 1282.59 s Done.
[Task 8/20] Current/Best: 1.15/ 1.15 GFLOPS | Progress: (1/1000) | 6.31 s Done.
[Task 9/20] Current/Best: 30.55/ 55.75 GFLOPS | Progress: (480/1000) | 983.67 s Done.
[Task 10/20] Current/Best: 2.60/ 2.60 GFLOPS | Progress: (1/1000) | 1.63 s Done.
[Task 11/20] Current/Best: 17.64/ 52.84 GFLOPS | Progress: (612/1000) | 1327.27 s Done.
[Task 12/20] Current/Best: 0.97/ 0.97 GFLOPS | Progress: (1/1000) | 1.29 s Done.
[Task 13/20] Current/Best: 13.91/ 61.94 GFLOPS | Progress: (476/1000) | 887.94 s Done.
[Task 14/20] Current/Best: 1.39/ 1.39 GFLOPS | Progress: (1/1000) | 1.09 s Done.
[Task 15/20] Current/Best: 18.77/ 56.70 GFLOPS | Progress: (492/1000) | 1035.34 s Done.
[Task 16/20] Current/Best: 0.78/ 0.78 GFLOPS | Progress: (1/1000) | 0.94 s Done.
[Task 17/20] Current/Best: 6.87/ 52.32 GFLOPS | Progress: (440/1000) | 970.66 s Done.
[Task 18/20] Current/Best: 1.10/ 1.10 GFLOPS | Progress: (1/1000) | 1.05 s Done.
[Task 19/20] Current/Best: 18.85/ 50.63 GFLOPS | Progress: (484/1000) | 1132.70 s Done.
[Task 20/20] Current/Best: 0.92/ 3.53 GFLOPS | Progress: (88/1000) | 191.53 s Done.
Compile…
Traceback (most recent call last):

File “tf_mobilenet_v1_android_gpu_tune.py”, line 330, in
tune_and_evaluate(tuning_option)

File “tf_mobilenet_v1_android_gpu_tune.py”, line 287, in tune_and_evaluate
mod, target=target, params=params, target_host=target_host)

File “/workspace/python/tvm/relay/build_module.py”, line 207, in build
graph_json, mod, params = bld_mod.build(func, target, target_host, params)

File “/workspace/python/tvm/relay/build_module.py”, line 108, in build
self._build(func, target, target_host)

File “tvm/_ffi/_cython/./function.pxi”, line 310, in tvm._ffi._cy3.core.FunctionBase.call

File “tvm/_ffi/_cython/./function.pxi”, line 245, in tvm._ffi._cy3.core.FuncCall

File “tvm/_ffi/_cython/./function.pxi”, line 234, in tvm._ffi._cy3.core.FuncCall3

File “tvm/_ffi/_cython/./base.pxi”, line 171, in tvm._ffi._cy3.core.CALL

tvm.ffi.base.TVMError: Traceback (most recent call last):
[bt] (8) /workspace/build/libtvm.so(tvm::relay::ForwardRewriter::VisitExpr
(tvm::relay::CallNode const*)+0x65a) [0x7f5af2035ffa]
[bt] (7) /workspace/build/libtvm.so(tvm::relay::ExprMutator::VisitExpr(tvm::relay::Expr const&)+0x9e) [0x7f5af21922de]
[bt] (6) /workspace/build/libtvm.so(tvm::relay::ExprFunctor<tvm::relay::Expr (tvm::relay::Expr const&)>::VisitExpr(tvm::relay::Expr const&)+0x161) [0x7f5af200bb61]
[bt] (5) /workspace/build/libtvm.so(std::_Function_handler<tvm::relay::Expr (tvm::NodeRef const&, tvm::relay::ExprFunctor<tvm::relay::Expr (tvm::relay::Expr const&)>), tvm::relay::ExprFunctor<tvm::relay::Expr (tvm::relay::Expr const&)>::InitVTable()::{lambda(tvm::NodeRef const&, tvm::relay::ExprFunctor<tvm::relay::Expr (tvm::relay::Expr const&)>)#6}>::_M_invoke(std::Any_data const&, tvm::NodeRef const&, tvm::relay::ExprFunctor<tvm::relay::Expr (tvm::relay::Expr const&)>*&&)+0x34) [0x7f5af1fff7b4]
[bt] (4) /workspace/build/libtvm.so(tvm::relay::ForwardRewriter::VisitExpr
(tvm::relay::CallNode const*)+0x7e6) [0x7f5af2036186]
[bt] (3) /workspace/build/libtvm.so(std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), void tvm::runtime::TypedPackedFunc<tvm::relay::Expr (tvm::relay::Call const&, tvm::Array<tvm::relay::Expr, void> const&, tvm::NodeRef const&)>::AssignTypedLambda<tvm::relay::Expr ()(tvm::relay::Call const&, tvm::Array<tvm::relay::Expr, void> const&, tvm::NodeRef const&)>(tvm::relay::Expr ()(tvm::relay::Call const&, tvm::Array<tvm::relay::Expr, void> const&, tvm::NodeRef const&))::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)+0xad) [0x7f5af202ee5d]
[bt] (2) /workspace/build/libtvm.so(tvm::relay::alter_op_layout::AlterOpLayoutRewrite(tvm::relay::Call const&, tvm::Array<tvm::relay::Expr, void> const&, tvm::NodeRef const&)+0x13cd) [0x7f5af202b4bd]
[bt] (1) /workspace/build/libtvm.so(tvm::relay::alter_op_layout::CallAlter(tvm::relay::Call const&, std::vector<tvm::relay::Expr, std::allocatortvm::relay::Expr > const&)+0x735) [0x7f5af2028da5]
[bt] (0) /workspace/build/libtvm.so(+0xcc10fb) [0x7f5af23020fb]
File “tvm/_ffi/_cython/./function.pxi”, line 56, in tvm._ffi._cy3.core.tvm_callback
File “/workspace/python/tvm/relay/op/nn/_nn.py”, line 207, in alter_op_layout_conv2d
return topi.nn.conv2d_alter_layout(attrs, inputs, tinfos, op)
File “</usr/local/lib/python3.6/dist-packages/decorator.py:decorator-gen-29>”, line 2, in conv2d_alter_layout
File “/workspace/python/tvm/target.py”, line 372, in dispatch_func
return dispatch_dict[k](*args, **kwargs)
File “/workspace/topi/python/topi/x86/conv2d_alter_op.py”, line 98, in _alter_conv2d_layout
ic_bn, oc_bn = cfg[“tile_ic”].size[-1], cfg[“tile_oc”].size[-1]
File “/workspace/python/tvm/autotvm/task/space.py”, line 805, in getitem
return self._entity_map[name]
KeyError: ‘tile_ic’

Does anyone have an idea how to solve this error?

Thanks!


#2

A similar issue was reported here.


#3

Yes, I’ve already seen that,
but the solution was for the x86 CPU.


#4

In general, there are two possibilities can cause this error.

  1. No available config in the history file. For example, the tuning space is just 1 and unfortunately that only one config cannot be built for some reasons. In this case, since you already loaded the history, the builder will not fallback to the default one and resulted in the error.

  2. The AutoTVM graph tuning messes up the config and layout (provided by @kevinthesun).

If you didn’t enable graph tuning, then the first point might be the killer.