As the VTA infrastructure is taking shape, we need to slide into the industry ecosystem, and be able to demonstrate viability. MLperf appears to be a reasonable path to take. I would like to start collaborating with folks that are working on VTA implementations to create an MLPerf capability for testing and measuring the performance of these VTA implementations.
Getting MLperf coverage on VTA and it’s variants should definitely be a target. I’ll be pushing coverage for more models once Relay support is upstreamed (almost there!). Let me know if you wish to get involved.
yes, please let me know where I can help
Hi, is there any VTA MLperf results available by now?
Progress has been slow on adding more workloads to VTA to MLPerf due to major updates on the Relay compiler (e.g. bring your own codegen: Bring Your Own Codegen to TVM). Once that lands and better support for graph partitioning and re-writing is in place, we should be able to much more easily tackle the problem of compiling more models to VTA. At the moment, the hard-wired compiler flow for VTA is brittle and error prone. Once it’ll migrate to the new infrastructure, it should be much easier for end users to extend VTA coverage to new models etc.