How does TVM support batch>1, while import a tensorflow pb which is frozen as batch=1

Questions
#1

Hi, I have a tensorflow pb frozen as batch=1. And I import the pb to TVM, now I want to run multi-batches, e.g. 16, how should I do? The following is my code, it works fine with batch=1. But if I change x = np.ones((1,248,248,3)) to
x = np.ones((16,248,248,3)), I will get error poped in from_tensorflow stage.

import tvm
import tensorflow as tf
from tvm import relay
import numpy as np
import time

Tensorflow utility functions

import tvm.relay.testing.tf as tf_testing
import logging
logging.basicConfig(filename=‘myapp.log’, level=logging.INFO)
model_path = ‘/home/zheenwang/fp32_model/new_add_shape_true/9rxf0co6_162k.pb’
with tf.gfile.GFile(model_path, “rb”) as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())
graph = tf.import_graph_def(graph_def, name=’’)
#tf.train.write_graph(graph_def, ‘/home/zheenwang/fp32_model/new_add_shape_true’, ‘fp32.pbtxt’, as_text=True)
graph_def = tf_testing.ProcessGraphDefParam(graph_def)
# Add shapes to the graph.
with tf.Session() as sess:
graph_def = tf_testing.AddShapesToGraphDef(sess, ‘fc/fba_unit/BiasAdd’)

#graph = tf.get_default_graph()
#input_images = g.get_tensor_by_name(‘XXX/image_tensor:0’)
#output_features = g.get_tensor_by_name(‘XXX/scores:0’)
x = np.ones((1,248,248,3))
shape_dict = {‘graph_input’: x.shape}
layout = “NCHW”
#import pdb
#pdb.set_trace()
mod, params = tvm.relay.frontend.from_tensorflow(graph_def,layout=layout,shape=shape_dict)
print(“Tensorflow protobuf imported to relay frontend.”)
target = ‘cuda’
target_host = ‘llvm’
layout = “NCHW”
ctx = tvm.gpu(0)
#target = ‘llvm’
#target_host = ‘llvm’
#layout = None
#ctx = tvm.cpu(0)
with relay.build_config(opt_level=3):
graph, lib, params = relay.build(mod,
target=target,
target_host=target_host,
params=params)
print(“relay built.”)
#print(graph)
from tvm.contrib import graph_runtime
dtype = ‘float32’
m = graph_runtime.create(graph, lib, ctx)

set inputs

m.set_input(‘graph_input’, tvm.nd.array(x.astype(dtype)))
m.set_input(**params)

execute

start = time.time()

m.run()

end = time.time()
print(end - start)

get outputs

#tvm_output = m.get_output(0, tvm.nd.empty(((1, 1008)), ‘float32’))

Traceback (most recent call last):

File “from_tensorflow.py”, line 29, in
mod, params = tvm.relay.frontend.from_tensorflow(graph_def,layout=layout,shape=shape_dict)

File “/home/zheenwang/tvm/python/tvm/relay/frontend/tensorflow.py”, line 2520, in from_tensorflow
mod, params = g.from_tensorflow(graph, layout, shape, outputs)

File “/home/zheenwang/tvm/python/tvm/relay/frontend/tensorflow.py”, line 2216, in from_tensorflow
self._mod[“main”] = func

File “/home/zheenwang/tvm/python/tvm/relay/module.py”, line 85, in setitem
return self._add(var, val)

File “/home/zheenwang/tvm/python/tvm/relay/module.py”, line 94, in _add
_module.Module_Add(self, var, val, update)

File “/home/zheenwang/tvm/python/tvm/_ffi/_ctypes/function.py”, line 207, in call
raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (7) /home/zheenwang/tvm/build/libtvm.so(TVMFuncCall+0x61) [0x7f9f15eb5911]
[bt] (6) /home/zheenwang/tvm/build/libtvm.so(+0xa63f94) [0x7f9f15dd3f94]
[bt] (5) /home/zheenwang/tvm/build/libtvm.so(+0xa636e6) [0x7f9f15dd36e6]
[bt] (4) /home/zheenwang/tvm/build/libtvm.so(tvm::relay::ModuleNode::Add(tvm::relay::GlobalVar const&, tvm::relay::Function const&, bool)+0x28c) [0x7f9f15dd0edc]
[bt] (3) /home/zheenwang/tvm/build/libtvm.so(tvm::relay::InferType(tvm::relay::Function const&, tvm::relay::Module const&, tvm::relay::GlobalVar const&)+0x1d7) [0x7f9f15cf7257]
[bt] (2) /home/zheenwang/tvm/build/libtvm.so(tvm::relay::TypeInferencer::Infer(tvm::relay::Expr)+0x86) [0x7f9f15cf6ad6]
[bt] (1) /home/zheenwang/tvm/build/libtvm.so(tvm::relay::ErrorReporter::RenderErrors(tvm::relay::Module const&, bool)+0x230c) [0x7f9f15db119c]
[bt] (0) /home/zheenwang/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x32) [0x7f9f156f5052]
File “/home/zheenwang/tvm/src/relay/ir/error.cc”, line 133
TVMError: ^[[1m
Error(s) have occurred. The program has been annotated with them:

%2338 = multiply(%2334, %2337)^[[31m Incompatible broadcast type TensorType([(int64)1, (int64)143360], float32) and TensorType([8960], float32); ^[[39m;

#2

It requires dynamic batch support which is ongoing.

cc @kevinthesun

#3

I think my problem is not real “dynamic” batch in TVM, my problem is that I want to build TVM with any batch dim based on a single TF pb file. My pb is as follows, where the graph input has a fixed batch size of 1. In anotehr thread, I find that @FrozenGene implies that if the batch dim in pb is -1, then you can feed any batch dim with shape_dict when calling from_tensorflow() in TVM. Is that right? So how to make pb generated batch dim= -1 when freeze in tensorflow?

node {
name: “graph_input”
op: “Placeholder”
attr {
key: “_output_shapes”
value {
list {
shape {
dim {
size: 1
}
dim {
size: 248
}
dim {
size: 248
}
dim {
size: 3
}
}
}
}
}
attr {
key: “dtype”
value {
type: DT_FLOAT
}
}
attr {
key: “shape”
value {
shape {
dim {
size: 1
}
dim {
size: 248
}
dim {
size: 248
}
dim {
size: 3
}
}
}
}
}

#4

I have unblocked myself by re-freeze the pb by setting the first dim of the input shape to ‘None’