Keras import: dimensions don't match in 3 layer example


#1

Here’s the minimal example of non-working code. I fixed one of the problems and it happens to be a bug in relay/frontend/keras.py . But I can’t find how to fix the second problem.

I need help to say me where to look for the bug, because probably it is in C++ code. I use a version from gihub (66235d1c3 commit)

import sys
import time
import math
import itertools

import numpy as np

import keras
from keras.layers import Input, Layer, Conv2D, UpSampling2D
from keras.models import Model

X = Input(shape=(1, 16384, 1), name="X")
x = X

x = UpSampling2D(size=2)(x)
x = (Conv2D(filters=32,
            kernel_size=(1, 54),
            activation=None, padding='same',
            strides=(1, 1)))(x)
x = (Conv2D(filters=1,
            kernel_size=(1, 54),
            activation=None, padding='same',
            strides=(1, 1)))(x)
model = keras.Model(inputs = X, outputs = x, name='generator')

#keras works fine

data = np.zeros(16384)
data = data.reshape((1, 1, len(data), 1))
#make sure that the keras model works itself
P = model.predict(data)

import tvm
import tvm.relay as relay

#tried different shapes - no success
shape_dict = {'X': [48, 1, 16384, 1]}
mod, params = relay.frontend.from_keras(model, shape_dict)

Output:

In `main`: 
v0.0.3
fn (%X: Tensor[(48, 1, 16384, 1), float32], %v_param_1: Tensor[(32, 1, 1, 54), float32], %v_param_2: Tensor[(32), float32], %v_param_3: Tensor[(1, 32, 1, 54), float32], %v_param_4: Tensor[(1), float32]) {
  %0 = nn.upsampling(%X, scale=2, layout="NHWC");
  %1 = nn.pad(%0, pad_width=[[0, 0], [0, 0], [0, 0], [26, 27]]);
  %2 = nn.conv2d(%1, %v_param_1, channels=32, kernel_size=[1, 54]) in particular dimension 1 conflicts 2 does not match 1; unable to unify: `Tensor[(32, 2, 1, 54), float32]` and `Tensor[(32, 1, 1, 54), float32]`; ;
  %3 = nn.bias_add(%2, %v_param_2);
  %4 = nn.pad(%3, pad_width=[[0, 0], [0, 0], [0, 0], [26, 27]]);
  %5 = nn.conv2d(%4, %v_param_3, channels=1, kernel_size=[1, 54]) in particular dimension 0 conflicts 32 does not match 1dimension 1 conflicts 0 does not match 32; unable to unify: `Tensor[(32, 0, 1, 54), float32]` and `Tensor[(1, 32, 1, 54), float32]`; ;
  nn.bias_add(%5, %v_param_4)
}

If I change _convert_upsample in relay/frontend/keras.py code to this, the first error is gone:

def _convert_upsample(inexpr, keras_layer, _):
    _check_data_format(keras_layer)
    upsample_type = type(keras_layer).__name__
    params = {'layout': 'NHWC'}
    params = {}#!!!!!!!
In `main`: 
v0.0.3
fn (%X: Tensor[(48, 1, 16384, 1), float32], %v_param_1: Tensor[(32, 1, 1, 55), float32], %v_param_2: Tensor[(32), float32], %v_param_3: Tensor[(1, 32, 1, 55), float32], %v_param_4: Tensor[(1), float32]) {
  %0 = nn.upsampling(%X, scale=2);
  %1 = nn.conv2d(%0, %v_param_1, padding=[0, 27], channels=32, kernel_size=[1, 55]);
  %2 = nn.bias_add(%1, %v_param_2);
  %3 = nn.conv2d(%2, %v_param_3, padding=[0, 27], channels=1, kernel_size=[1, 55]) in particular dimension 0 conflicts 32 does not match 1dimension 1 conflicts 0 does not match 32; unable to unify: `Tensor[(32, 0, 1, 55), float32]` and `Tensor[(1, 32, 1, 55), float32]`; ;
  nn.bias_add(%3, %v_param_4)
}

But I can’t get rid of that strange Tensor[(32, 0, 1, 55), float32]. I don’t get why it sets one of the dimensions to zero. To me it seems like a mistake in C++ code, as I traced the python code to the C++ call and I don’t see any mention of that weird 0-dimension shape. What files in the C++ code I should check to see where this shape appears?


#2

You actually have run into two separate bugs. The first was an issue in the keras frontend for upsampling that you correctly identified. The second issue was due to your convolution having 1 channel. This made relay think that it was a group convolution and infer the wrong weight shape. I’ve issued a PR to fix both of these problems and have confirmed they allow your example to work without any errors. https://github.com/dmlc/tvm/pull/3937


#3

Thanks for the help!