[CI][Docker] How to match Docker version tags in source control with the ones in Docker Hub?

leandron · December 17, 2019, 3:58pm

It seems some of the versions numbers of the TVM Docker images being used for the public CI, for example the usual “tvm > master” Jenkins, are not matching with the version tag of the Docker files in Git.

Taking as an example a recent build, it reports it is using the latest versions of the Docker images from tvmai DockerHub, which looks correct. This is a summary of versions being used by Jenkins:

tvmai/ci-lint v0.51
tvmai/ci-cpu v0.54
tvmai/ci-i386 v0.53
tvmai/ci-gpu v0.56

When comparing these versions above with the versions in the source, some versions are not matching. This is a summary of the versions present in the source:

tvmai/ci-lint <none>
tvmai/ci-cpu v0.41
tvmai/ci-i386 v0.50
tvmai/ci-gpu v0.54

I understand how tags work on Docker images, but based on the information above, I have the following questions:

how do we generate the official tvmai/* images that are uploaded on DockerHub?
by building images based on what is in the incubator-tmv/docker/* right now, do I get the “same” images that we have in CI ? (note: I am aware that probably will have a small set of differences due to Ubuntu software updates or pip packages pointing to latest version, but that is probably a minor problem).

If that is just a version number typo in the files, let me know and I can send a PR to update/fix them. I just want to understand how the broader Docker image process works here.

hcho3 · December 17, 2019, 9:24pm

Proposal: let CI build Docker image on the fly. This need not be expensive if we cache previously built Docker images to a Docker repository such as ECR. So only changes in Dockerfile will trigger re-build of the corresponding Docker image.

See example at dmlc/xgboost:

github.com

dmlc/xgboost/blob/63ffd2f686e74686eae0e026188e4d4a2c104550/tests/ci_build/ci_build.sh#L141-L164


if [[ -n "${DOCKER_CACHE_ECR_ID}" && -n "${DOCKER_CACHE_ECR_REGION}" ]]
then
    # Format Docker repo URL
    DOCKER_CACHE_REPO="${DOCKER_CACHE_ECR_ID}.dkr.ecr.${DOCKER_CACHE_ECR_REGION}.amazonaws.com"
    echo "Using AWS ECR; repo URL = ${DOCKER_CACHE_REPO}"
    # Login for Docker registry
    echo "\$(python3 -m awscli ecr get-login --no-include-email --region ${DOCKER_CACHE_ECR_REGION} --registry-ids ${DOCKER_CACHE_ECR_ID})"
    $(python3 -m awscli ecr get-login --no-include-email --region ${DOCKER_CACHE_ECR_REGION} --registry-ids ${DOCKER_CACHE_ECR_ID})
    # Pull pre-build container from Docker build cache,
    # if one exists for the particular branch or pull request
    echo "docker pull ${DOCKER_CACHE_REPO}/${DOCKER_IMG_NAME}:${BRANCH_NAME}"
    if docker pull "${DOCKER_CACHE_REPO}/${DOCKER_IMG_NAME}:${BRANCH_NAME}"
    then
      CACHE_FROM_CMD="--cache-from ${DOCKER_CACHE_REPO}/${DOCKER_IMG_NAME}:${BRANCH_NAME}"
    else
      # If the build cache is empty of the particular branch or pull request,
      # use the build cache associated with the master branch
      echo "docker pull ${DOCKER_CACHE_REPO}/${DOCKER_IMG_NAME}:master"
      docker pull "${DOCKER_CACHE_REPO}/${DOCKER_IMG_NAME}:master" || true
      CACHE_FROM_CMD="--cache-from ${DOCKER_CACHE_REPO}/${DOCKER_IMG_NAME}:master"

This file has been truncated. show original

tqchen · December 17, 2019, 10:10pm

See https://docs.tvm.ai/contribute/pull_request.html#ci-environment for the general docker update process. I think in your case the version number is a typo

We stick with a binary image mainly to avoid differences caused by upstream changes and keep the docker stable. In most cases, the docker images used by Jenkins are generated by the latest Dockerfile in the branch.

leandron · January 6, 2020, 1:41pm

Follow-up: I’ve submitted this PR to amend the versions mentioned in the Dockerfiles.