[Dev Containers] v0.354.0 new image is not being tagged properly

[terrynfredrickson-yahoo]

• VSCode Version: 1.88.1
• Local OS Version: Windows 10 & MacOS 14.4.1
• Remote OS Version: Rhel8.9
• Remote Extension/Connection Type: WSL & server

Issue: When starting up a container with an image that hasn't been used before, VSCode will sit at the following phase:

STEP 1/9: FROM vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a
? Please select an image: 
  ▸ registry.access.redhat.com/vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a:latest
    registry.redhat.io/vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a:latest
    docker.io/library/vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a:latest

This prevents vscode from continuing into the container loading process.

• Logs:

- [255 ms] Dev Containers 0.354.0 over Remote - SSH 0.110.1 in VS Code 1.88.1 (e170252f762678dec6ca2cc69aba1570769a5d39).
[254 ms] Initializing configuration support...
[255 ms] Internal initialization of dev container support package...
[1462 ms] Start: Resolving Remote
[1495 ms] Host OS: linux 4.18.0-477.13.1.el8_8.x86_64
[1495 ms] Host CWD: /home/<USER>/<PROJECT_ID>
[1495 ms] Start: Run: /bin/sh 
[1546 ms] Start: Run in host: echo ~
[1613 ms] /home/<USER>
[1614 ms] 
[1615 ms] Start: Run in host: id -un
[1681 ms] <USER>
[1682 ms] 
[1682 ms] Start: Run in host:  (command -v getent >/dev/null 2>&1 && getent passwd '<USER>' || grep -E '^<USER>|^[^:]*:[^:]*:<USER>:' /etc/passwd || true)
[1750 ms] userEnvProbe: loginInteractiveShell (default)
[1751 ms] userEnvProbe: not found in cache
[1751 ms] userEnvProbe shell: /bin/bash
[1846 ms] userEnvProbe PATHs:
Probe:     '/home/<USER>/.local/bin:/home/<USER>/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/var/lib/snapd/snap/bin'
Container: None
[1921 ms] Setting up container for folder or workspace: /home/<USER>/<PROJECT_ID>
[1922 ms] Host authority: ssh-remote+7b22686f73744e616d65223a2231302e35332e3134392e313239222c2275736572223a2265646566656e64227d
[1923 ms] Start: Run: wsl -l -v
[2116 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c echo ~
[2394 ms] Start: Run: wsl -d Ubuntu -e /bin/sh -c cd '/home/<USER>' && /bin/sh
[2399 ms] Start: Run in host: id -un
[2569 ms] <USER>
[2569 ms] 
[2569 ms] Start: Run in host:  (command -v getent >/dev/null 2>&1 && getent passwd '<USER>' || grep -E '^<USER>|^[^:]*:[^:]*:<USER>:' /etc/passwd || true)
[2573 ms] Start: Run in host: echo ~
[2573 ms] /home/<USER>
[2574 ms] 
[2574 ms] Start: Run in host: test -x '/home/<USER>/.vscode-remote-containers/bin/e170252f762678dec6ca2cc69aba1570769a5d39/node'
[2575 ms] 
[2575 ms] 
[2575 ms] Start: Run in host: test -f '/home/<USER>/.vscode-remote-containers/dist/vscode-remote-containers-server-0.354.0.js'
[2576 ms] 
[2576 ms] 
[2578 ms] userEnvProbe: loginInteractiveShell (default)
[2578 ms] userEnvProbe: not found in cache
[2578 ms] userEnvProbe shell: /bin/bash
[2667 ms] userEnvProbe PATHs:
Probe:     '/usr/lib/wsl-ssh-yinit:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/Oath/OpenSSH:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files (x86)/Sennheiser/SenncomSDK/:/mnt/c/Program Files/Docker/Docker/resources/bin:/mnt/c/Users/<USER>/AppData/Local/Programs/Python/Python311/Scripts/:/mnt/c/Users/<USER>/AppData/Local/Programs/Python/Python311/:/mnt/c/Users/<USER>/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/<USER>/AppData/Local/GitHubDesktop/bin:/mnt/c/Users/<USER>/AppData/Local/Programs/Git/cmd:/mnt/c/Users/<USER>/AppData/Local/JetBrains/Toolbox/scripts:/mnt/c/Users/<USER>/AppData/Local/Programs/Microsoft VS Code/bin:/snap/bin'
Container: None
[2669 ms] Start: Check Docker is running
[2669 ms] Start: Run: docker version
[2758 ms] Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.

[2758 ms] Client:       Podman Engine
Version:      4.6.1
API Version:  4.6.1
Go Version:   go1.20.12
Built:        Tue Feb  6 13:04:01 2024
OS/Arch:      linux/amd64
[2759 ms] Start: Run: docker volume ls -q
[2848 ms] Start: Run: docker ps -q -a --filter label=vsch.local.folder=/home/<USER>/<PROJECT_ID> --filter label=vsch.quality=stable
[2947 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/home/<USER>/<PROJECT_ID> --filter label=devcontainer.config_file=/home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json
[3047 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/home/<USER>/<PROJECT_ID>
[3141 ms] Start: Run: docker inspect --type container ce81cc684e47
[3278 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/home/<USER>/<PROJECT_ID>
[3366 ms] Start: Run: docker inspect --type container ce81cc684e47
[3502 ms] Start: Run: docker exec -i -u root ce81cc684e476a83752b4f338e49ea214f7ff5f202c64fb099a814d9fb1ea4cc /bin/sh -c echo "Container already running. Keep-alive process started." ; export VSCODE_REMOTE_CONTAINERS_SESSION=719b1ec7-181d-4a76-b3b0-f8f1468fed7a1713366701733 ; /bin/sh
[3502 ms] Running Dev Containers CLI:   read-configuration --workspace-folder /home/<USER>/<PROJECT_ID> --id-label devcontainer.local_folder=/home/<USER>/<PROJECT_ID> --id-label devcontainer.config_file=/home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json --log-level debug --log-format json --config /home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json --mount-workspace-git-root
[3577 ms] Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.

[3641 ms] Start: Run: /home/<USER>/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node /home/<USER>/.vscode-remote-containers/dist/dev-containers-cli-0.354.0/dist/spec-node/devContainersSpecCLI.js read-configuration --workspace-folder /home/<USER>/<PROJECT_ID> --id-label devcontainer.local_folder=/home/<USER>/<PROJECT_ID> --id-label devcontainer.config_file=/home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json --log-level debug --log-format json --config /home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json --mount-workspace-git-root
[3664 ms] Container already running. Keep-alive process started.
[3936 ms] @devcontainers/cli 0.58.0. Node.js v18.18.2. linux 4.18.0-477.13.1.el8_8.x86_64 x64.
[3936 ms] Start: Run: git rev-parse --show-cdup
[3947 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/home/<USER>/<PROJECT_ID> --filter label=devcontainer.config_file=/home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json
[4017 ms] Running Dev Containers CLI:   up --container-session-data-folder /tmp/devcontainers-719b1ec7-181d-4a76-b3b0-f8f1468fed7a1713366701733 --workspace-folder /home/<USER>/<PROJECT_ID> --workspace-mount-consistency cached --id-label devcontainer.local_folder=/home/<USER>/<PROJECT_ID> --id-label devcontainer.config_file=/home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json --log-level debug --log-format json --config /home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --terminal-columns 230 --terminal-rows 40
[4041 ms] Start: Run: /home/<USER>/.vscode-server/cli/servers/Stable-e170252f762678dec6ca2cc69aba1570769a5d39/server/node /home/<USER>/.vscode-remote-containers/dist/dev-containers-cli-0.354.0/dist/spec-node/devContainersSpecCLI.js up --container-session-data-folder /tmp/devcontainers-719b1ec7-181d-4a76-b3b0-f8f1468fed7a1713366701733 --workspace-folder /home/<USER>/<PROJECT_ID> --workspace-mount-consistency cached --id-label devcontainer.local_folder=/home/<USER>/<PROJECT_ID> --id-label devcontainer.config_file=/home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json --log-level debug --log-format json --config /home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json --default-user-env-probe loginInteractiveShell --mount type=volume,source=vscode,target=/vscode,external=true --skip-post-create --update-remote-user-uid-default on --mount-workspace-git-root --terminal-columns 230 --terminal-rows 40
[4297 ms] @devcontainers/cli 0.58.0. Node.js v18.18.2. linux 4.18.0-477.13.1.el8_8.x86_64 x64.
[4297 ms] Start: Run: docker buildx version
[4371 ms] buildah 1.31.3
[4371 ms] Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.

[4371 ms] Start: Resolving Remote
[4375 ms] Start: Run: git rev-parse --show-cdup
[4476 ms] Start: Run: docker ps -q -a --filter label=devcontainer.local_folder=/home/<USER>/<PROJECT_ID> --filter label=devcontainer.config_file=/home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote/devcontainer.json
[4522 ms] Start: Run: docker inspect --type image <INTERNAL_DOCKER_URL>
[4576 ms] Start: Run: docker buildx build --load --build-arg BUILDKIT_INLINE_CACHE=1 -f /tmp/devcontainercli-<USER>/container-features/0.58.0-1713366706881/Dockerfile-with-features -t vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a --target dev_containers_target_stage --build-arg _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label /home/<USER>/<PROJECT_ID>/.devcontainer/rhel8-<ORG>1.0-remote
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
[1/2] STEP 1/1: FROM <INTERNAL_DOCKER_URL> AS dev_container_auto_added_stage_label
--> 3b57cab8d48c
[2/2] STEP 1/2: FROM 3b57cab8d48c24efebc04191b5edc1e532b4b1857bf62d480c7d073512eb46c0 AS dev_containers_target_stage
[2/2] STEP 2/2: LABEL devcontainer.metadata="{\"postCreateCommand\":\"mkdir -p .vscode/ && cp .devcontainer/unified/<ORG>1.0/c_cpp_properties.json .vscode/ && cp .devcontainer/unified/<ORG>1.0/settings.json .vscode/ && cp .devcontainer/unified/.clang-tidy . && cp .devcontainer/unified/.clang-format .\",\"postStartCommand\":\"source /opt/rh/gcc-toolset-10/enable -s && .devcontainer/setup.sh && sudo yum install rpmdevtools patchelf -y && ./setMakefile.sh 10 && ./bootstrap.sh && .devcontainer/init_lua_testing.sh\",\"customizations\":{\"vscode\":{\"extensions\":[\"cschlosser.doxdocgen\",\"eamodio.gitlens\",\"GitHub.copilot\",\"jeff-hykin.better-cpp-syntax\",\"josetr.cmake-language-support-vscode\",\"lextudio.restructuredtext\",\"Meezilla.json\",\"mhutchie.git-graph\",\"ms-vscode.cpptools\",\"ms-python.python\",\"ms-vscode.cpptools-themes\",\"ms-vscode.cpptools-extension-pack\",\"ms-vscode.makefile-tools\",\"mutantdino.resourcemonitor\",\"streetsidesoftware.code-spell-checker\",\"twxs.cmake\",\"xaver.clang-format\"]}}}"
--> Using cache ac19325f35a6532e26342ba107a8b58dafd388c3469f39a44de6e27fdfb279d0
[2/2] COMMIT vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a
--> ac19325f35a6
[Warning] one or more build args were not consumed: [BUILDKIT_INLINE_CACHE]
Successfully tagged localhost/vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a:latest
ac19325f35a6532e26342ba107a8b58dafd388c3469f39a44de6e27fdfb279d0
[4902 ms] Start: Run: docker inspect --type image vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a
[5033 ms] Start: Run: docker build -f /tmp/devcontainercli-<USER>/updateUID.Dockerfile-0.58.0 -t vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a-uid --platform linux/amd64 --build-arg BASE_IMAGE=vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a --build-arg REMOTE_USER=vscode --build-arg NEW_UID=136356 --build-arg NEW_GID=100 --build-arg IMAGE_USER=vscode /tmp/devcontainercli-<USER>/empty-folder
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
STEP 1/9: FROM vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a
? Please select an image: 
  ▸ registry.access.redhat.com/vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a:latest
    registry.redhat.io/vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a:latest
    docker.io/library/vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a:latest

[bash]~/% podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
localhost/vsc-<PROJECT_ID>-bc8222b6a8aff1778c01e7d7fbb03b2f20fb236dfb89dc6549856e61f1d9b97a latest 0b3b912629fd 21 seconds ago 1.65 GB
<INTERNAL_DOCKER_URL> latest 3b57cab8d48c 19 hours ago 1.65 GB

Steps to Reproduce:
Currently the only way we've been able to reproduce (these are internal docker urls) is

1. Using the devcontainer V0.354.0, people trying to load up new images are being stuck on this step.
2. Rolling back to 0.348.0 seems to fix this issue.

Validated that it wasn't related to the host machine / podman that we are using for docker emulation, I went and spun up a new host to set up, and was able to connect just fine (I am currently using 0.315.1) and I did not encounter this problem.

3 different engineers have encountered this problem starting about 2 weeks ago which lines up with the release of 0.354.0.

Does this issue occur when you try this locally?: Unknown / didn't try
Does this issue occur when you try this locally and all extensions are disabled?: Unknown / didn't try

[chrmarti]

Tracking in #9748.