Closed
Conversation
It seems like gVisor fixed a bug where a file's capabilities weren't being properly applied when creating the process: google/gvisor@586c38d I haven't fully finished investigating yet, but I think it's the above commit that caused the tests to start failing once it got released and used in CI.
Contributor
Author
|
In case it's helpful to anyone picking this up, I wrote a lima config file to help quickly spin up a VM that is ready to test this: default.yaml
# Based on https://github.com/lima-vm/lima/blob/9d31f2a7ee7c6a699298c66d80f26ad2d39fae76/examples/docker.yaml
# A template to use Docker instead of containerd & nerdctl
# $ limactl start ./docker.yaml
# $ limactl shell docker docker run -it -v $HOME:$HOME --rm alpine
# To run `docker` on the host (assumes docker-cli is installed):
# $ export DOCKER_HOST=$(limactl list docker --format 'unix://{{.Dir}}/sock/docker.sock')
# $ docker ...
# This template requires Lima v0.8.0 or later
images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-amd64.img"
arch: "x86_64"
digest: "sha256:054db2d88c454bb0ad8dfd8883955e3946b57d2b0bf0d023f3ade3c93cdd14e5"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20231026/ubuntu-22.04-server-cloudimg-arm64.img"
arch: "aarch64"
digest: "sha256:eafa7742ce5ff109222ea313d31ea366d587b4e89b900b11d8285ae775dfe8c3"
cpus: 8
memory: 16GiB
mounts:
- location: "~"
writable: true
- location: "/tmp/lima"
writable: true
# containerd is managed by Docker, not by Lima, so the values are set to false here.
containerd:
system: false
user: false
provision:
- mode: system
script: |
#!/bin/bash
apt-get install -y jq make
- mode: system
# This script defines the host.docker.internal hostname when hostResolver is disabled.
# It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
# Names defined in /etc/hosts inside the VM are not resolved inside containers when
# using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
script: |
#!/bin/sh
sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
# Install docker packages
- mode: system
script: |
#!/bin/bash
set -eux -o pipefail
command -v docker >/dev/null 2>&1 && exit 0
export DEBIAN_FRONTEND=noninteractive
curl -fsSL https://get.docker.com | sh
# NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
systemctl disable --now docker
apt-get install -y uidmap dbus-user-session
# Setup rootless docker
- mode: user
script: |
#!/bin/bash
set -eux -o pipefail
systemctl --user start dbus
dockerd-rootless-setuptool.sh install
docker context use rootless
# Install gVisor (runsc)
- mode: user
script: |
mkdir -p "$HOME/bin"
(
set -e
ARCH="$(uname -m)"
URL="https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}"
wget --quiet "${URL}/runsc" "${URL}/runsc.sha512" \
"${URL}/containerd-shim-runsc-v1" "${URL}/containerd-shim-runsc-v1.sha512"
sha512sum -c runsc.sha512 \
-c containerd-shim-runsc-v1.sha512
rm -f -- *.sha512
chmod a+rx runsc containerd-shim-runsc-v1
mv runsc containerd-shim-runsc-v1 "$HOME/bin"
)
mkdir -p ~/.config/docker/
tee ~/.config/docker/daemon.json <<EOF
{
"runtimes": {
"runsc": {
"path": "$HOME/bin/runsc",
"runtimeArgs": [
"--host-uds=create",
"--ignore-cgroups"
]
}
}
}
EOF
systemctl --user restart docker
# Install go
- mode: user
script: |
set -euo pipefail
VERSION="$(curl --silent https://go.dev/dl/?mode=json | jq -r '.[0].version')"
wget "https://dl.google.com/go/${VERSION}.linux-arm64.tar.gz"
mkdir -p "$HOME/bin"
rm -rf "$HOME/bin/go" && tar -C "$HOME/bin" -xzf "${VERSION}.linux-arm64.tar.gz"
echo '[ "${PATH#*$HOME/bin/go/bin}" == "$PATH" ] && export PATH="$PATH:$HOME/bin/go/bin"' >> "${HOME}/.bashrc"
probes:
- script: |
#!/bin/bash
set -eux -o pipefail
if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
echo >&2 "docker is not installed yet"
exit 1
fi
if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
echo >&2 "rootlesskit (used by rootless docker) is not running"
exit 1
fi
hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
# hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
# resolve inside containers, and not just inside the VM itself.
hosts:
host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/user/{{.UID}}/docker.sock"
hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
cd to the Vault repo on your host, run `lima` to enter the VM, and then:
make dev # Optional: Pull all the dependencies to populate the VM's Go module cache etc.
go test -v -count=1 -run="^TestExternalPluginInContainer_MountAndUnmount/rootless_runsc$" github.com/hashicorp/vault/vault |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It seems like gVisor fixed a bug where a file's capabilities weren't being properly applied when creating the process: google/gvisor@586c38d
I haven't fully finished investigating yet, but I think it's the above commit that caused the tests to start failing once it got released and used in CI.