[External Codegen] Ensure FuseOps ignores partitioned subgraphs

#1

I’ve been using the graph partitioning infrastructure and have noticed that the FuseOps pass doesn’t seem to ignore subgraphs that have been designated as external. This causes the build to fail as, in my case, the partitioned subgraphs still contain unlowered qnn ops. I note the subgraphs are designated as ‘Primitive’ functions, which I assume means they should be left alone by any further lowering passes.

I can’t get away with disabling the pass entirely as it’s a required pass for TVM to work. So do we want FuseOps to still run on partitioned subgraphs or should this be considered a bug?

#2

Please see this topic where already has discussions for it: [External codegen] How to prevent batch norm from being decomposed?

#3

We should discuss here since that post contains some off topic stuff.

I think all optimization should be disabled by default, and enable passes on opt in basis. Otherwise it defeats the purpose of pat matching.

#4

This was my initial thinking - just turn off all the passes. However, FuseOps in particular is hard-wired to always run. We also will frequently have the case where there’s only a partial offloading to an external codegen and so we need to keep the optimisation passes for the subgraphs that are still passing through TVM.

The simplest idea I can think of is to run the codegen for the external functions before we call Optimize and do TVM codegen. That way if the subgraphs are mangled by the optimization passes it doesn’t matter because we’ve already generated the runtime modules. This would have an added benefit by making possible the introduction of a fallback path if external compilation fails, although it’s unclear if that would be useful.

#5

@comaniac @masahi I’ve opened a PR to attempt to address this: 4864. It’s a bit of a straw man to open up discussion but essentially alters the default behaviour of visitor patterns to skip over functions marked as ‘Primitive’.

#6

I replied in the PR. Probably this is a better place for discussion