Improve testing times using pytest-xdist or pytest-parallel

broune · November 20, 2019, 9:24pm

Definitely agree that we need a consistent limit on how many cores are occupied, which is going to depend on e.g. how many cores the machine has and how many executors we have. The higher levels of parallelization already present for build in some cases may also explain variances in load. It looks like Docker should make it easy to control this: https://docs.docker.com/config/containers/resource_constraints/

Though limiting it in docker doesn’t solve the problem of consistently parallelizing to the same amount in all places, where going above or below that is unfortunate. We don’t have a consistent control of this today:

*** build 1-way parallel (i.e. not parallel)

github.com

apache/incubator-tvm/blob/master/tests/scripts/task_python_integration.sh#L35


export TVM_NUM_THREADS=2


rm -rf python/tvm/*.pyc python/tvm/*/*.pyc python/tvm/*/*/*.pyc


# Test TVM
make cython3


# Test extern package
cd apps/extension
rm -rf lib
make
cd ../..


python3 -m pytest -v apps/extension/tests


TVM_FFI=ctypes python3 -m pytest -v tests/python/integration
TVM_FFI=ctypes python3 -m pytest -v tests/python/contrib


TVM_FFI=ctypes python3 -m pytest -v tests/python/relay


# Do not enable OpenGL

*** build 2-way parallel

github.com

apache/incubator-tvm/blob/master/Jenkinsfile#L206


         echo set\\(USE_GRAPH_RUNTIME_DEBUG ON\\) >> config.cmake
         echo set\\(USE_VM_PROFILER ON\\) >> config.cmake
         echo set\\(USE_LLVM llvm-config-8\\) >> config.cmake
         echo set\\(USE_NNPACK ON\\) >> config.cmake
         echo set\\(NNPACK_PATH /NNPACK/build/\\) >> config.cmake
         echo set\\(USE_ANTLR ON\\) >> config.cmake
         echo set\\(CMAKE_CXX_COMPILER g++\\) >> config.cmake
         echo set\\(CMAKE_CXX_FLAGS -Werror\\) >> config.cmake
         echo set\\(HIDE_PRIVATE_SYMBOLS ON\\) >> config.cmake
         """
      make(ci_cpu, 'build', '-j2')
      pack_lib('cpu', tvm_lib)
      timeout(time: max_time, unit: 'MINUTES') {
        sh "${docker_run} ${ci_cpu} ./tests/scripts/task_python_unittest.sh"
        sh "${docker_run} ${ci_cpu} ./tests/scripts/task_python_integration.sh"
        sh "${docker_run} ${ci_cpu} ./tests/scripts/task_python_vta.sh"
        sh "${docker_run} ${ci_cpu} ./tests/scripts/task_golang.sh"
      }
    }
  }
},

*** build 4-way parallel

github.com

apache/incubator-tvm/blob/ce90f0d0eec8fc02e98a63968a796f1598351001/tests/scripts/task_web_build.sh#L20


#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
cp /emsdk-portable/.emscripten ~/.emscripten
source /emsdk-portable/emsdk_env.sh
make -j4

*** build 8-way parallel

github.com

apache/incubator-tvm/blob/master/tests/scripts/task_cpp_unittest.sh#L27


# under the License.


set -e
set -u


export LD_LIBRARY_PATH="lib:${LD_LIBRARY_PATH:-}"


# Remove existing testcases
rm -f build/*_test


make cpptest -j8
for test in build/*_test; do
    ./$test
done

A consistent place where each thing gets run, and/or a consistent set of environment variables, should help here.

ramana-arm · November 21, 2019, 9:31am

The last couple of days have been a bit busy for me and I’ve not been able to pay attention to this thread at all.

Having read this thread again and given that it’s going into what feels with a wider set of issues with the test harness.

Running the tests as part of the upstream PR CI pipeline. There is the upstream CI pipeline and that needs to be considered. It appears to me that the current test infrastructure is set up to make that efficient.
Running the tests as a developer on the machine I’m working on is a use case from where I originally started this conversation. If possible as a developer, I would like to run all the CI CPU tests using a command line option and get all the results for all the tests in a single run. Also it would be nice if the test run gave us clear results for all the test failures and not to be in a situation of fixing one test group at a time. Preferably this interface to running the tests is the same as what the CI does but with an easier set of command line options.
Have a standard way of specifying the environment and the target architecture to execute and test on. Also differentiating between architecture specific test runs and non-architecture specific test runs.
Through the scripts and environment, it would be nice to be able to specify a single test to execute in a standardized manner. Not having to run every single test when doing this would be nice.
From all the conversations pytest-parallel is something that may not cut it because of thread safety for a large number of our tests.
Using pytest-xdist or other ways of parallelism is hard because we have varied ways of running our tests today with the examples that @broune has shown in the previous post.

Regards, Ramana