Update all ci- containers to reflect main

[areusch]

This is a tracking issue for the process of updating the TVM ci- containers to reflect the following PRs:

• Use new SBT Debian Repo before bintray is shutdown #7926
• [microTVM] Refactor zephyr installation + Update Zephyr RVM doc #7915

It will also unblock PRs:

• [ETHOSN] Removed support for 20.08 version of the driver stack. #7858
• Bumped Ubuntu version to 18.04 for ci_gpu #7970 (it's possible we could just include this PR in this update)

Steps:

• Pick the latest revision of main and use that specific git hash for following steps. Document it here.
• Build docker containers, tagging them as e.g. <username>/ci-<container>:v0.<ver> (we may not need to build all of these if Bumped Ubuntu version to 18.04 for ci_gpu #7970 is not included here)
  • ci-arm
  • ci-cpu
  • ci-gpu
  • ci-i386
  • ci-lint
  • ci-qemu
  • ci-wasm
• Push docker containers to Docker Hub
• Create a draft PR modifying Jenkinsfile to point all containers at <username>/ci-<container>:v0.<ver>
• Force-push the PR to ci-docker-staging branch
• Jenkins will notice the push and start a build here
• Debug the build and repeat these steps until the build passes
• Push the valid containers to tlcpack/ci-<container>:v0.<ver>
• Update the PR to point Jenkinsfile to the new containers.
• Merge the PR.

Let's use this tracking issue to record challenges we face in updating the drivers.

@jroesch can you please note down where you were with this?

cc @tqchen @u99127 @d-smirnov @leandron @tristan-arm

[jroesch]

So last time I tried to update the images but ran into issues where tests no longer passed under updated GPU and CPU images (see docker pull tlcpack/ci-gpu:v0.73-t3).

I had chased down the image building, there are a few important things.

First disable caching in order to make sure you pull the freshest apt as if you have built before you can have a stale layer which pulled an apt-registry which is no longer valid.

Second there were some updates to the drivers which caused breakage, specifically this test no longer works because it incorrectly believes that CUDA is on (even though there is no GPU):

https://github.com/apache/tvm/blob/main/tests/cpp/build_module_test.cc#L84

These needs to be patched to actually check for the GPU's existence.

There were also changes intended to fix Rust CI which should included in master today allowing us to turn Rust CI back on.

[areusch]

@tkonolige and i will attempt to update to 18.04 (e.g. include #7970) with these updates.

[tkonolige]

#8031 contains the updated docker images

[areusch]

@tkonolige is kindly rebuilding the containers. @tkonolige , can you document the main hash used following the steps above?

[areusch]

Testing: https://ci.tlcpack.ai/job/tvm/job/ci-docker-staging/87/

[areusch]

Built from: 55471632d97bb9d0d4535bd977ca4fb7e7cdcc28

[areusch]

retesting: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/88/pipeline

[areusch]

we accidentally tested the wrong thing; so re-re-testing here: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/89/pipeline/6

[areusch]

https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/90/pipeline

[areusch]

seems like the last run was unexpectedly unable to access to the gpu in "Frontend : GPU" phase. retrying with printing nvidia-smi before running those tests: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/91/pipeline/

[areusch]

apologies for the light updates here. we determined that the "Frontend : GPU" tests get into a state where either the GPU hardware is inaccessible after a while or TVM's existence check is wrong. Since we didn't change the CUDA version used here--we just updated to 18.04--the theory is that there is some interoperability problem between CUDA running in the containers (at 10.0) and the CUDA driver loaded on the docker host side (either 10.2 or 11.0, depending which CI node you hit).

@tkonolige and I have spent the last couple days running on a test TVM CI cluster using the same AMI (which has only CUDA 11.0). With CUDA 10.0 (ci-gpu) and 11.0 (host), we ran into another similar-looking bug during the GPU unit tests:

[ RUN      ] BuildModule.Heterogeneous
[22:11:58] /workspace/src/target/opt/build_cuda_on.cc:89: Warning: cannot detect compute capability from your device, fall back to compute_30.
unknown file: Failure
C++ exception with description "[22:11:58] /workspace/src/runtime/cuda/cuda_device_api.cc:117: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (e == cudaSuccess || e == cudaErrorCudartUnloading) is false: CUDA: no CUDA-capable device is detected

We then upgraded ci-gpu to use CUDA 11.0, and this test seemed to pass all the way to the end of the GPU integration tests, modulo a tolerance issue:

tests/python/contrib/test_cublas.py::test_batch_matmul FAILED
// ...
>       np.testing.assert_allclose(actual, desired, rtol=rtol, atol=atol, verbose=True)
E       AssertionError: 
E       Not equal to tolerance rtol=1e-05, atol=1e-07
E       
E       Mismatched elements: 2875175 / 3866624 (74.4%)
E       Max absolute difference: 0.00541687
E       Max relative difference: 0.00015383
E        x: array([[[29.647408, 31.88966 , 33.90233 , ..., 34.673954, 32.908764,
E                31.219051],
E               [30.993076, 30.78019 , 33.67124 , ..., 36.1395  , 29.176218,...
E        y: array([[[29.646427, 31.889557, 33.900528, ..., 34.673126, 32.90791 ,
E                31.21726 ],
E               [30.991737, 30.780437, 33.67001 , ..., 36.139397, 29.174744,...

we'll try and push this CUDA 11.0 ci-gpu container through the test CI cluster to see how far we can get. feel free to comment if there are concerns updating to CUDA 11.0.

[areusch]

update: #8130 is an alternate solution to the issues presented in #8108, which doesn't sacrifice accuracy. we have tested this in an instance of the TVM CI using CUDA 11 on the host and docker container side, and all tests pass. we'll now attempt to merge #8130.

following that, we'll disable all CI nodes running CUDA 10 and re-run ci-docker-staging against main using our CUDA 11 containers. based on our experiments, we think this will pass, and we can then promote those containers to tlcpack and declare victory.

[areusch]

#8130 is merged, testing the containers again. hoping we see green this time: https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/ci-docker-staging/92/pipeline

[areusch]

It looks like our most recent run passed enough to merge the containers. the failure here is actually a separate CI problem triggered by #8023. will submit another PR to fix the issue triggered there, but we should be able to proceed here.

[d-smirnov]

@Areush Not sure but could these failures be related to the container update?
https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-8151/2
https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/PR-8149/2

[areusch]

@d-smirnov should be resolved by #8160

[d-smirnov]

Indeed, resolved. Thank you! -- Dmitriy Smirnov

…

On Tue, 1 Jun 2021 at 18:35, Andrew Reusch ***@***.***> wrote: @d-smirnov <https://github.com/d-smirnov> should be resolved by #8160 <#8160> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7995 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGADO3676QIY5GVC67ENWKTTQUK7ZANCNFSM44H4RGXA> .