Introduce new frontend for Caffe

Caffe Frontend

Background & Motivation

Caffe is a deep learning framework made with expression, speed, and modularity in mind. Because of its simplicity, good scalability, fast and other characteristics, it is favored by many people. According to RISELab who makes statistics of papers collected in arxiv.org, Caffe is ranked in the top four in the deep learning framework, which shows to some extent that Caffe’s user base is still large, please refer to blog. In addition, according to our company’s research on the market, the demand for Caffe in the production environment is still strong, and many models based on Caffe need to be deployed. For example, existing deployment frameworks, such as MNN, NCNN, MACE, etc., directly support the deployment of Caffe.

TVM only supports caffe2 at present, and the difference between Caffe and caffe2 is quite large. At present, there are two ways to deploy Caffe model in TVM: one is to convert Caffe model to Tensorflow or Pytorch model, the other is to convert Caffe model to onnx and then to relay IR. The two methods are essentially the same. They are all indirectly converted to relay IR through the third-party transformation. However, the problem is that some ops will fail in the process of model transformation, and even the result of transfer out may be different.

Based on the above situation, we decided to open our Caffe frontend codes, hoping to enrich the use scenarios of TVM.

Implementation Approach

The whole process of Caffe front end importing model is divided into:

  1. Read Model:The model graph and related parameters are read through the protobuffer API of Caffe;
  2. Rebuild Graph:Traverse the graph, then replace the top of the in-place layer with the name of the layer, and update all related layers at the same time;
  3. Model Conversion:Read the parameters of each layer and convert them into corresponding TVM OP and parameters;
  4. Layer Fusion:fuse batchnorm and scale layers;
  5. Convert to Relay IR:It mainly includes its module, params and the real name of the output layer。

Finally, we can import the Caffe model as follows:

from google.protobuf import text_format
from tvm.relay.frontend import caffe_pb2 as pb

init_net = pb.NetParameter()
predict_net = pb.NetParameter()

# load model
with open(proto_file, 'r') as f:
    text_format.Merge(f.read(), predict_net)
# load blob
with open(blob_file, 'rb') as f:
    init_net.ParseFromString(f.read())

shape_dict = {'data': [1,3,224,224]}
dtype_dict = {'data': 'float32'}

mod, params, model_outputs = relay.frontend.from_caffe(init_net, predict_net, shape_dict, dtype_dict)

Work Done

All of the things that we have done are listed as following:

1. List of supported Ops

  • BatchNorm
  • Concat
  • Convolution
  • Crop
  • Deconvolution
  • Dropout
  • Eltwise
  • Flatten
  • InnerProduct
  • Input
  • LRN
  • Normalize
  • Permute
  • Pooling
  • PReLU
  • PriorBox
  • proposal
  • Python
  • ReLU
  • Reshape
  • Resize
  • ROIPooling
  • Scale
  • Sigmoid
  • Slice
  • Softmax
  • TanH
  • Upsample

2. List of supported complete models

  • Alexnet
  • Resnet50
  • Mobilenetv1
  • Mobilenetv2
  • Inceptionv1
  • Inceptionv3
  • Inceptionv4
  • Vgg16
  • Squeezenetv1
  • SSDMobilenetv1
  • SSDMobilenetv2
  • YOLOv3
  • ENet

3. Caffe frontend test cases

4. Caffe frontend tutorial

TODO

  • [ ] Support more ops and more complete models.

According to the above implementation scheme, based on the front-end framework we built, you can add any new op, you only need to: firstly, add a method in the operatorconverter class, which needs to include your extraction of the layer parameters and the logic of conversion to TVM OP, secondly, register the method to convert_ map.

@tqchen @FrozenGene

2 Likes

Good Job! One quick question, many folks use caffe often define its own proto (though this is rare in other frameworks). Do we consider this situation?

Yes, we will provide a .proto file or caffe_pb2.py directly. Now, we’re just going to ask if I can provide caffe_pb2.py directly, which folder should we put it in?

I think put caffe_pb2.py in the frontend folder is fine. We have already some other stuff like mxnet_qnn_op_utils.py / tflite_flexbuffers.py and so on.

which branch does this work on?Thanks!

It has not been merged, please ref to this pr: https://github.com/apache/incubator-tvm/pull/6206

@zhupijuan_lkl When I run one of the tests in caffe test_forward.py, it reports an error from protobuf. I know it is not TVM’s issue, it seems the model is too large(alexnet.caffemodel is 233M in my side ) really appreciate if you can give me some hint. I am curious why it is OK in CI, I used the same script docker/install/ubuntu_install_caffe.sh to install the caffe

tvm/tests/python/frontend/caffe/test_forward.py:816: in _test_alexnet
    _test_network(data_process, proto_file, blob_file)
tvm/tests/python/frontend/caffe/test_forward.py:261: in _test_network
    tvm_out = _run_tvm(data, proto_file, blob_file)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

data = array([[[[  54.060997 ,  112.061    ,  -82.939    , ...,   37.060997 ,
           -69.939    ,  -78.939    ],
        ... [  54.32     ,   55.32     ,  126.32     , ...,  -32.68     ,
           -59.68     ,   -8.68     ]]]], dtype=float32)
proto_file = '/root/.tvm_test_data/model/alexnet.prototxt', blob_file = '/root/.tvm_test_data/model/alexnet.caffemodel'

    def _run_tvm(data, proto_file, blob_file):
        """ Run caffe model by TVM according to .caffemodel and .prototxt"""
        init_net = pb.NetParameter()
        predict_net = pb.NetParameter()
    
        # load model
        with open(proto_file, "r") as f:
            text_format.Merge(f.read(), predict_net)
        # load blob
        with open(blob_file, "rb") as f:
>           init_net.ParseFromString(f.read())
E           google.protobuf.message.DecodeError: Error parsing message

tvm/tests/python/frontend/caffe/test_forward.py:187: DecodeError
---------------------------------------------------------------------------------------------- Captured stdout call -----------------------------------------------------------------------------------------------
File /root/.tvm_test_data/model/alexnet.prototxt exists, skip.
File /root/.tvm_test_data/model/alexnet.caffemodel exists, skip.
---------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------
[libprotobuf ERROR google/protobuf/io/coded_stream.cc:207] A protocol message was rejected because it was too big (more than 67108864 bytes).  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.

There are two mothods to solve this problem.:

  1. follow the suggestion as error log: increase memory limits at CodedInputStream::SetTotalBytesLimit() and re-compile protobuf
  2. upgrade protobuf into >= 3.2

Hope to help you

1 Like

Hi all. We’re working on adding MobileNet-SSD specific layers (PriorBox, DetectionOutput and Permute) support to Caffe frontend.

Is anyone working on a similar work? If no, we’re going to contribute the source code to upstream after finishing our internal implementation.

Thanks.

Hello! We’re currently trying to convert our ssd-caffe network to tmv, but missing that layers (PriorBox, DetectionOutput and Permute) too, do you have any progress with them (may i have a look)? Thank you!

Hi @Alvern

We’ve already implemented Caffe-SSD support in our internal codebase. And we’re also upstreaming the codebase to TVM.

Here’s some related PRs. We’lll create additional PRs soon. Stay tuned!

1 Like

Hi there,

I’ve implemented PriorBox operator in TVM. Everything works fine.

I would appreciate it if anyone could give me some comment or feedback.