As the Relay VM becomes closer to being the primary way to run models in TVM, we get a huge benefit of being able to execute models with dynamic shape. A lot of operators, particularly in the TensorFlow and ONNX frontends, utilize infer_value_simulated
to get input values. I’d like to start a discussion about how we can move away from these calls and better support dynamism.
I agree using infer_value_simulated
looks unreliable. Could you investigate a bit on this and check why we need to use infer_value_simulated
to get input values?
For example, I see range
op in Tensorflow needs to call infer_value_simulated
to get limit
value. Currently relay.arange
already support taking an expr as start/stop/step. Probably we can fix this one already?
I agree that we want to move towards full dynamic shape support in our frontends, with Onnx being a natural fit due to its “dynamic everything” philosophy. If it weren’t for autotuning, we could probably do this today as Relay VM and relay.Any
have pretty much all the functionality we need. However, my understanding is that autotuning for dynamic shape isn’t fully fleshed out yet. Once that’s figured out I think we can start adding much more dynamism to our importers.