Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Using Dockerfile to build devcontainer with Podman causes VSCode to hang #9748

Closed
awesomekyle opened this issue Apr 4, 2024 · 16 comments · Fixed by devcontainers/cli#804
Closed
Assignees
Labels
bug Issue identified by VS Code Team member as probable bug containers Issue in vscode-remote containers podman Dev Container using Podman upstream Issue identified as 'upstream' component related (exists outside of VS Code Remote)

Comments

@awesomekyle
Copy link

awesomekyle commented Apr 4, 2024

When creating a devcontainer from a Dockerfile using Podman, VSCode hangs because Podman doesn't pull from localhost/ when using the --platform build arg. Because the terminal Dev Container terminal window can't be interacted with at all, this means VSCode hangs forever and the only way to stop it is to kill VSCode.

The Dockerfile builds correctly ("Successfully tagged localhost/vsc-podman....") and then VSCode does some additional setup using the /tmp/devcontainercli-kyle/updateUID.Dockerfile-0.58.0 file, which has the following first two lines:

ARG BASE_IMAGE
FROM $BASE_IMAGE
...

BASE_IMAGE=vsc-podman-devcontainer-41....53dbb is being set in the podman build command. The issue is that podman build prompts for the registry as shown in the image below:

Screenshot from 2024-04-04 19-26-22

.devcontainer/devcontainer.json

If I use the image option, the container is created correctly. It only fails when BASE_IMAGE is a local image.

{
  "name": "Alpine",

  // "image": "mcr.microsoft.com/devcontainers/base:alpine-3.19",
  "build": { "dockerfile": "Dockerfile" },

  "workspaceMount": "",
  "runArgs": [
    "--userns=keep-id",
    "--volume=${localWorkspaceFolder}:/workspaces/${localWorkspaceFolderBasename}:z"
  ],
  "containerUser": "vscode"
}

.devcontainer/Dockerfile

FROM mcr.microsoft.com/devcontainers/base:alpine-3.19

Workaround:

  • If /etc/containers/registries.conf only has one unqualified-search-registries, podman won't prompt for a selection so the container creation will work correctly.
  • After it fails, manually run the command that hung, choosing any prompted registry (it doesn't matter, Podman pulls locally regardless). Edit: After further testing this isn't true. VSCode tries to rebuild it regardless.
  • Push the base container to a registry instead of using the build option. Edit: Using features also causes new containers to be built, so this workaround only works if you don't use devcontainer features.
@vscodenpa vscodenpa added the containers Issue in vscode-remote containers label Apr 5, 2024
@chrmarti
Copy link
Contributor

chrmarti commented Apr 5, 2024

Is this a bug in Podman? You mention --platform, does it not ask for the registry without --platform?

@chrmarti chrmarti added the info-needed Issue requires more information from poster label Apr 5, 2024
@awesomekyle
Copy link
Author

I'm not sure if this is a bug, I haven't yet filed a bug with Podman. But that's correct, if --platform is not passed in, Podman doesn't prompt for anything. This could be intentional behavior in Podman. If you don't specify a platform, it will automatically use localhost because it's the correct platform by definition.

Testing a bit more locally this also means that features can't be used either because they also build a new container dynamically. The following devcontainer.json will cause a hang when running with Podman on Fedora with "dev.containers.dockerPath": "podman",.

{
  "name": "Alpine",

  "image": "mcr.microsoft.com/devcontainers/base:alpine-3.19",
  // "build": { "dockerfile": "Dockerfile" },

  "workspaceMount": "",
  "runArgs": [
    "--userns=keep-id",
    "--volume=${localWorkspaceFolder}:/workspaces/${localWorkspaceFolderBasename}:z"
  ],
  "containerUser": "vscode",
  "features": {
    "ghcr.io/wxw-matt/devcontainer-features/command_runner:0": {}
  }
}

After testing more I updated my original post; manually running the failed Podman command doesn't work, VSCode continues trying to build the container.

@awesomekyle
Copy link
Author

In case any body else happens to run into this, you can use the CONTAINERS_REGISTRIES_CONF environment variable to override the registries file:

~/.local/bin/podman-devcontainers:

#!/bin/bash
CONTAINERS_REGISTRIES_CONF="$HOME/.config/devcontainers-registries.conf" podman "$@"

~/.config/devcontainers-registries.conf:

unqualified-search-registries = ["docker.io"]

And then use "dev.containers.dockerPath": "/home/username/.local/bin/podman-devcontainers", in VSCode's settings. This will then ensure Podman only has one registry to pull from and doesn't hang.

@chrmarti, at the very least there should be a way to cancel the container creation process. If the process fails due to an error, VSCode prompts the user what to do. In this case there's no option other than killing VSCode entirely.

@lgonzalezsa
Copy link

I was having same issue described here and reversed to v0.348.0 of the extension and now works fine with podman
Issue with v0.354.0:
image

@chrmarti chrmarti added podman Dev Container using Podman and removed info-needed Issue requires more information from poster labels Apr 8, 2024
@chrmarti
Copy link
Contributor

chrmarti commented Apr 8, 2024

I think devcontainers/cli#746 might have changed it, so we now always pass --platform, also when the platform matches the local hardware.

@chrmarti chrmarti added upstream Issue identified as 'upstream' component related (exists outside of VS Code Remote) bug Issue identified by VS Code Team member as probable bug labels Apr 8, 2024
@pjlammertyn
Copy link

pjlammertyn commented Apr 11, 2024

@awesomekyle correct, restricting the number of registries is a workaround

mkdir $HOME/.config/containers
cat <<EOF > $HOME/.config/containers/registries.conf
unqualified-search-registries = ['docker.io']
EOF

@soiamsoNG
Copy link

@chrmarti
encounter same issue and after some search get below

podman tag behavior is not as docker
podman will alwasy add localhost/ before tag of image without specify registry part
even you are not hosting a local registry server.

but BASE_IMAGE miss localhost/ before it.

@chrmarti
Copy link
Contributor

Not sure what the correct fix is here. It's unfortunate that a change on our side broke things, but I also think adding --platform shouldn't suddenly require user input. I'm leaning towards considering this a Podman issue.

@pgourinchas
Copy link

Is it possible to prefix the BASE_IMAGE argument with localhost/ when the docker path setting contains podman ?
image

@chrmarti
Copy link
Contributor

That appears to work. We already use docker/podman -v to check if it is Podman and can reuse that.

Adding the suggested fix, could someone file an issue with Podman to check if the behavior is expected? It seems the addition of --platform triggered this, correct?

@chrmarti
Copy link
Contributor

This fix is available with Dev Containers 0.360.0-pre-release. Let us know if this works, thanks!

@lgonzalezsa
Copy link

Testing Dev Containers 0.360.0-pre-release when the local image exist and works.
But if I do a rebuild without cache, I do have an image in our own docker hub (hub.docker.mycorp.net) but seems the extension now adds localhost/ in front of that one as well, causing certainly the connection refused because I do not host that repository on my local machine.

Let me know if you need other logs from my side of if I should post on other GitHub issue.

Thanks!

[2024-04-19T14:07:43.813Z] STEP 1/9: FROM localhost/hub.docker.mycorp.net/my_repo/base:latest
[2024-04-19T14:07:43.816Z] Trying to pull localhost/hub.docker.mycorp.net/my_repo/base:latest...
[2024-04-19T14:07:43.818Z] WARN[0000] Failed, retrying in 2s ... (1/3). Error: initializing source docker://localhost/hub.docker.mycorp.net/my_repo/base:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 
[2024-04-19T14:07:45.820Z] WARN[0002] Failed, retrying in 2s ... (2/3). Error: initializing source docker://localhost/hub.docker.mycorp.net/my_repo/base:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 
[2024-04-19T14:07:47.822Z] WARN[0004] Failed, retrying in 2s ... (3/3). Error: initializing source docker://localhost/hub.docker.mycorp.net/my_repo/base:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused 
[2024-04-19T14:07:49.825Z] Error: creating build container: initializing source docker://localhost/hub.docker.mycorp.net/my_repo/base:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused

@chrmarti
Copy link
Contributor

Thanks @lgonzalezsa!

I need to add a check if there is already a registry name since we go through the same code path with that. PR is out: devcontainers/cli#810

@chrmarti chrmarti reopened this Apr 19, 2024
@damaestro
Copy link

Using the pre-release channel resolved this issue for me.

@chrmarti
Copy link
Contributor

@lgonzalezsa The updated fix is available with Dev Containers 0.361.0-pre-release. Please give that a try, thanks!

@lgonzalezsa
Copy link

Awesome! works for me!

@microsoft microsoft locked and limited conversation to collaborators Jun 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Issue identified by VS Code Team member as probable bug containers Issue in vscode-remote containers podman Dev Container using Podman upstream Issue identified as 'upstream' component related (exists outside of VS Code Remote)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants