Using Model Dynamism in Frontends

As the Relay VM becomes closer to being the primary way to run models in TVM, we get a huge benefit of being able to execute models with dynamic shape. A lot of operators, particularly in the TensorFlow and ONNX frontends, utilize infer_value_simulated to get input values. I’d like to start a discussion about how we can move away from these calls and better support dynamism.

@jwfromm @haichen do you have any thoughts?

I agree using infer_value_simulated looks unreliable. Could you investigate a bit on this and check why we need to use infer_value_simulated to get input values?

For example, I see range op in Tensorflow needs to call infer_value_simulated to get limit value. Currently relay.arange already support taking an expr as start/stop/step. Probably we can fix this one already?

I agree that we want to move towards full dynamic shape support in our frontends, with Onnx being a natural fit due to its “dynamic everything” philosophy. If it weren’t for autotuning, we could probably do this today as Relay VM and relay.Any have pretty much all the functionality we need. However, my understanding is that autotuning for dynamic shape isn’t fully fleshed out yet. Once that’s figured out I think we can start adding much more dynamism to our importers.