Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docker-compose: Passing gpu with driver: cdi is not supported #19338

Closed
raldone01 opened this issue Jul 24, 2023 · 38 comments · May be fixed by ublue-os/bluefin-docs#93
Closed

docker-compose: Passing gpu with driver: cdi is not supported #19338

raldone01 opened this issue Jul 24, 2023 · 38 comments · May be fixed by ublue-os/bluefin-docs#93
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@raldone01
Copy link

raldone01 commented Jul 24, 2023

Issue Description

May be related to #19330.
See also NVIDIA/nvidia-container-toolkit#126.

CC @elezar

CDI Support in Docker will be an experimental feature in the Docker 25 release.

It will support the following docker-compose.yml file:

version: '3.8'
services:
cdi_test: # this is not working
    image: ubuntu:20.04
    deploy:
      resources:
        reservations:
          devices:
           - driver: cdi
             device_ids:
                - nvidia.com/gpu=all
                - nvidia.com/gds=all
                - nvidia.com/mofed=all
    command:
      - bash
      - -c
      - |
        nvidia-smi -L

Steps to reproduce the issue

Steps to reproduce:

Prerequisites:

  • Have the NVIDIA Container Toolkit (e.g. v1.13.5) installed
  • Have a v25 Docker daemon configured in experimental mode
  • Have generated the CDI specification for NVIDIA GPUs by running sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml --device-name-strategy=uuid
  • Make sure you are using the podman-docker-emulation-daemon

Observe the following docker-compose.yml:

version: '3.8'
services:
  nvidia_test: # this is not working
    image: ubuntu:20.04
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu, utility, compute]
    command:
      - bash
      - -c
      - |
        nvidia-smi -L
  runtime_test: # this is working
    image: ubuntu:20.04
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
    command:
      - bash
      - -c
      - |
        nvidia-smi -L
  cdi_test: # this is not working
    image: ubuntu:20.04
    deploy:
      resources:
        reservations:
          devices:
           - driver: cdi
             device_ids:
                - nvidia.com/gpu=all
                - nvidia.com/gds=all
                - nvidia.com/mofed=all
    command:
      - bash
      - -c
      - |
        nvidia-smi -L

Launch via docker-compose up.

Describe the results you received

Only the runtime container works.

Describe the results you expected

All three containers should show the same output.

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.7-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: f633919178f6c8ee4fb41b848a056ec33f8d707d'
  cpuUtilization:
    idlePercent: 98.59
    systemPercent: 0.38
    userPercent: 1.02
  cpus: 72
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  hostname: argon
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.4.4-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 54521659392
  memTotal: 135022321664
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.8.5-1
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 42947567616
  swapTotal: 42947567616
  uptime: 21h 35m 45.00s (Approximately 0.88 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/main/.config/containers/storage.conf
  containerStore:
    number: 3
    paused: 0
    running: 0
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/main/.local/share/containers/storage
  graphRootAllocated: 857601998848
  graphRootUsed: 271414001664
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/main/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685139594
  BuiltTime: Sat May 27 00:19:54 2023
  GitCommit: 9eef30051c83f62816a1772a743e5f1271b196d7-dirty
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

No response

@raldone01 raldone01 added the kind/bug Categorizes issue or PR as related to a bug. label Jul 24, 2023
@nicoya
Copy link

nicoya commented Aug 11, 2023

Is this expected to work when using podman-compose? None of the three containers in the above docker-compose.yml seem to work for me, however I can get nVidia support in a container via a plain podman commandline:

$ podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L
GPU 0: NVIDIA GeForce RTX 2070 (UUID: GPU-336c5a8a-839f-69a8-08cf-e3fdcd01e019)

@raldone01
Copy link
Author

raldone01 commented Aug 11, 2023

Interesting.
for me the runtime_test is working.

Thing is I had the understanding that podman-docker should be a fully compatible emulation so I consider this an issue.

Since nvidia.com/gpu cdi driver is supported the docker gpu api should be mapped to the podman cdi api in podman-docker.

@nicoya
Copy link

nicoya commented Aug 13, 2023

Got it working with this:

version: '3.8'
services:
  nvidia_test:
    image: ubuntu
    devices:
      - nvidia.com/gpu=all
    command:
      - bash
      - -c
      - |
        nvidia-smi -L

@itzsimpl
Copy link

For me, none of these docker-compose.yml approaches works, however I can get nVidia support in a container via a plain podman commandline.

@raldone01
Copy link
Author

nvidia_test:
image: ubuntu
devices:
- nvidia.com/gpu=all
command:
- bash
- -c
- |
nvidia-smi -L

When I try this I get the following: Error response from daemon: container create: stat nvidia.com/gpu=all: no such file or directory

What version of podman are you running?
Do you have any special configuration active?

@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@elezar
Copy link
Contributor

elezar commented Sep 13, 2023

@raldone01 if you could point me at the code for podman-compose, I could try to see what additions are required to add CDI support there. I would assume that this would need similar extensions to those made to the --device command line argument. Your error message indicates that the entries in devices: are being interpreted as device nodes.

Note that Docker does not yet support CDI devices and these will only be available in Docker 25. In addition, the mechanism to request these device in the compose YAML do not match your expectations.

@raldone01
Copy link
Author

raldone01 commented Sep 15, 2023

if you could point me at the code for podman-compose, I could try to see what additions are required to add CDI support there.

What do you want me to point you at? The code for the podman-compose implementation and its argument parser? I myself am not familiar with the code base. I could clone the repo and do some searches for you but I am not sure what code you are referring to. Please clarify.

Note that Docker does not yet support CDI devices and these will only be available in Docker 25. In addition, the mechanism to request these device in the compose YAML do not match your expectations.

Can you show me the correct yaml format?
If possible please link me to documentation outlining the correct planned yaml format.
I tried to search for it and found lots of conflicting information.

Note that Docker does not yet support CDI devices and these will only be available in Docker 25.

I know and mentioned it in the issue description.

@arielnmz
Copy link

arielnmz commented Oct 2, 2023

Can confirm too that none of the services presented in the OP work for me @raldone01 did you configure the container runtime using the docker instructions? The NVIDIA CTK guide says that CDI is recommended for podman

@raldone01
Copy link
Author

I configured the runtime using sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml --device-name-strategy=uuid. Afterwards podman containers with sudo podman run -ti --rm --device=nvidia.com/gpu=all ubuntu:18.04 nvidia-smi -L worked. I am pretty sure this uses cdi.

I will look into the link you sent on Friday.

@arielnmz
Copy link

arielnmz commented Oct 3, 2023

@raldone01 okay but that is using podman only, right? normal podman commands seem to work fine with CDI, but how did you manage to make docker-compose work using CDI?

@raldone01
Copy link
Author

raldone01 commented Oct 3, 2023

Yes this is using podman commands directly. This issue is exactly about cdi not working with podman-compose. Maybe this wasn't clear. I was not able to get cdi to work with podman-compose.

Cdi works fine with podman directly.
Nvidia Runtime (legacy) works with podman-compose but will be or is deprecated.

@raldone01
Copy link
Author

If anyone has a working podman/docker-compose yml with cdi please share it. As far as I know it is not supported.

@arielnmz
Copy link

arielnmz commented Oct 3, 2023

@raldone01 sorry I got confused by my own post. I am curious how you got the runtime_test part working. I do have the runtime installed and configured (https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-docker) but I just get nvidia-smi command not found

@arielnmz
Copy link

arielnmz commented Oct 4, 2023

Aight, just to follow up on this issue, I added the nvidia runtime to the podman (containers) configuration and set it as the default:

$cat  /usr/share/containers/containers.conf
runtime = "nvidia"
...
[engine.runtimes]
...
nvidia = [
  "/usr/bin/nvidia-container-runtime",
]

So this works:

$ podman --log-level DEBUG system service -t0
...
Using OCI runtime "/usr/bin/nvidia-container-runtime"
...

This works, too:

$ podman run --rm  --security-opt=label=disable nvidia/cuda:12.2.0-runtime-ubuntu22.04 nvidia-smi -L
...
GPU 0: NVIDIA RTX A5000 (UUID: GPU-82e4260a-3604-4c94-4abf-3ecccd9b28f8)

But tried running the runtime_test service from the OP and still get:

$ docker-compose up runtime_test
...
/opt/nvidia/nvidia_entrypoint.sh: line 67: exec: nvidia-smi: not found

podman info reports using the nvidia runtime:

...
  ociRuntime:
    name: nvidia
    package: nvidia-container-toolkit-base-1.14.2-1.x86_64
    path: /usr/bin/nvidia-container-runtime
    version: |-
      NVIDIA Container Runtime version 1.14.2
      commit: 807c87e057e13fbd559369b8fd722cc7a6f4e5bb
      spec: 1.1.0-rc.2

      runc version 1.1.8
      commit: v1.1.8-0-g82f18fe
      spec: 1.0.2-dev
      go: go1.19.11
      libseccomp: 2.5.3
...

@raldone01
Copy link
Author

raldone01 commented Oct 4, 2023

As far as I can tell all options in /usr/share/containers/containers.conf are commented out and I can't remember changing it.
I will try to get a reproduce script working using distrobox or something like that.
My default ociRuntime in podman info is crun.
I just verified docker is not even installed on the machine.
I am using Docker Compose version 2.20.3.

Does sudo docker run --rm -e NVIDIA_VISIBLE_DEVICES=all ubuntu nvidia-smi -L work for you?

❯ sudo docker run --rm -e NVIDIA_VISIBLE_DEVICES=all ubuntu nvidia-smi -L
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
GPU 0: NVIDIA T600 (UUID: GPU-STUFF)
GPU 1: NVIDIA GeForce GTX 1080 (UUID: GPU-STUFF)

Without env:

❯ sudo docker run --rm ubuntu nvidia-smi -L
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

The command you provided does not work on my machine as I use crun as the default runtime for podman.

❯ sudo podman run --rm  --security-opt=label=disable ubuntu nvidia-smi -L
Error: crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

Try to do the following (pulled from my command history):

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml --device-name-strategy=uuid
sudo nvidia-ctk runtime configure --runtime=podman
sudo nano /etc/nvidia-container-runtime/config.toml
no-cgroups = false
runtimes = [
  "docker-runc",
  "runc",
  "crun"
]

@arielnmz
Copy link

arielnmz commented Oct 4, 2023

I think the culprit might be that I'm not using compose v2. I was able to get this working, however, by using podman-compose and passing run args to the call:

podman-compose --podman-run-args '--device nvidia.com/gpu=all --security-opt=label=disable' run --rm runtime_test

And this works too:

podman-compose --podman-run-args '--device nvidia.com/gpu=all --security-opt=label=disable' up runtime_test

This works like a charm:

$ podman-compose run --rm runtime_test

version: "3.8"

services:
  runtime_test: # this is working
    image: nvidia/cuda:12.2.0-runtime-ubuntu22.04
    devices:
     - nvidia.com/gpu=all
    security_opt:
      - "label=disable"
    command: nvidia-smi -L

@rhatdan
Copy link
Member

rhatdan commented Oct 5, 2023

Can we close this issue now, since there is a workaround, or is there still something Podman needs to fix?

@arielnmz
Copy link

arielnmz commented Oct 5, 2023

@rhatdan I think the original issue remains: device reservations is broken using podman-docker and it works using normal docker

@rhatdan
Copy link
Member

rhatdan commented Oct 7, 2023

Does it work with rootful podman? Is this more a case of rootless versus rootful?

@raldone01
Copy link
Author

raldone01 commented Oct 7, 2023

I got a bit of new information. I'm currently reworking the issue. I will provide a reproducible Distrobox script to better test the issue.
I have not tried it with rootless yet.
The issue still exists.
Should I update the issue itself or add a new comment.

Edit:
You can track my progress here: https://github.com/raldone01/podman-docker-gpu-issue.git
Distrobox was not the right tool for the job. The reproduce script will be using lxd and qemu.

@rhatdan
Copy link
Member

rhatdan commented Oct 10, 2023

Add a new comment.

@mrnerdhair
Copy link

I am experiencing the same issue here, using the standalone docker-compose 2.23.0. I can do sudo podman run -ti --rm --device=nvidia.com/gpu=all ubuntu:18.04 nvidia-smi -L successfully, but none of the above flavors of compose file work.

@jesulo
Copy link

jesulo commented Jan 10, 2024

Hi, is the some option for enable gpu using podman-compose.yml? regards

@ageisen2000
Copy link

I am still experiencing this issue.

@gautaz
Copy link

gautaz commented Mar 8, 2024

Hello,

Here is the current behavior I am experiencing:

podman version
Client:       Podman Engine
Version:      4.9.3
API Version:  4.9.3
Go Version:   go1.21.7
Built:        Tue Jan  1 01:00:00 1980
OS/Arch:      linux/amd64podman compose version
>>>> Executing external compose provider "/home/del/.nix-profile/bin/docker-compose". Please refer to the documentation for details. <<<<

Docker Compose version 2.24.6podman run --rm --device nvidia.com/gpu=all -it docker.io/library/ubuntu nvidia-smi
Fri Mar  8 15:28:56 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14              Driver Version: 550.54.14      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A500 Laptop GPU     Off |   00000000:01:00.0 Off |                  N/A |
| N/A   37C    P8              4W /   30W |     232MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
+-----------------------------------------------------------------------------------------+podman compose run testgpu
>>>> Executing external compose provider "/home/del/.nix-profile/bin/docker-compose". Please refer to the documentation for details. <<<<

unable to upgrade to tcp, received 409
Error: executing /home/del/.nix-profile/bin/docker-compose run testgpu: exit status 1podman run --rm -it docker.io/library/ubuntu nvidia-smi
Error: crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

Here is the compose file used:

services:
  testgpu:
    image: docker.io/library/ubuntu
    command: nvidia-smi
    deploy:
      resources:
        reservations:
          devices:
            - driver: cdi
              device_ids:
                - nvidia.com/gpu=all

Is this related to this issue or is it another issue ?

@elezar
Copy link
Contributor

elezar commented Mar 12, 2024

@gautaz what docker compose does is map the devices section in the compose file to HostConfig modifications (specifically DeviceRequest entries) that are interpreted by the Docker daemon. I am not sure how this interacts with Podman.

If someone could point me at the execution path from running podman-compose -- and how Podman is invoked -- we can identify where support for the cdi driver needs to be added.

@laurensV
Copy link

Maybe related to my problem: I am trying to use the Docker API to create a container on a remote podman server with GPU support. With docker this can be done with device requests, but this comment #21180 (comment) suggests that podman is not handling data via device_requests.. How would I use the docker api to enable gpu support in podman?

@rhatdan
Copy link
Member

rhatdan commented Mar 12, 2024

Have you tried with podman 5.0, some fixes whent in to handle GPUs from remote API.

@gautaz
Copy link

gautaz commented Mar 13, 2024

@elezar @laurensV The way docker-compose/podman-compose are supposed to communicate with podman is in fact also unclear to me. As @elezar pointed out, docker-compose communicates with docker through the daemon API but, in my particular case, I am using podman in its original rootless design, which means, AFAIU, no daemon, no API access through a socket.

@rhatdan I guess your question was for @laurensV.
In the meantime, I have also tried podman 5 but I am hitting other difficulties that I need to fully understand. These difficulties are related to the fact that since version 5, podman uses pasta for its network layer and not slirp4netns anymore. The fact that the Linux distribution I am using (NixOS) does not support pasta adds another layer of difficulties (this seems to be a work in progress though).

@qdm12
Copy link

qdm12 commented Apr 2, 2024

Any news on this? Podman 5.x.x on Ubuntu is for now unusable / too-unstable as @gautaz mentioned.

So back to using Podman 4.9.3, I did a few experiments using a docker-compose.yml, all with the common body:

services:
  issue-19338:
    image: ubuntu
    entrypoint: /bin/bash -c "ls /dev/ | grep nvidia"

where ls /dev/ | grep nvidia shows nvidia devices if the passthrough is successful.
CDI definitions are generated using sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml.

The tries and results are in the following table:

Compose program YML setting NVIDIA devices passthrough?
podman-compose devices block Success
podman-compose deploy block with nvidia driver Fail
podman-compose deploy block with cdi driver Fail
compose devices block Fail: container won't start Error response from daemon: container create: stat nvidia.com/gpu=0: no such file or directory
compose devices block + privileged: true Fail
compose deploy block with nvidia driver Fail
compose deploy block with cdi driver Fail

For more details:

  • the deploy block is:

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia # nvidia or cdi
              device_ids: ['nvidia.com/gpu=0']
              capabilities: [gpu]
  • the devices block is:

    devices:
      - nvidia.com/gpu=0

Conclusions:

  • CDI devices are well supported with the devices block from the podman run API, but not through the Podman socket.
  • deploy block does not work at all, maybe it's just fully missing from Podman.

@gautaz
Copy link

gautaz commented Apr 24, 2024

Hello,

A little follow-up since podman 5 has reached unstable:

podman version
Client:       Podman Engine
Version:      5.0.2
API Version:  5.0.2
Go Version:   go1.22.2
Built:        Tue Jan  1 01:00:00 1980
OS/Arch:      linux/amd64docker compose version
>>>> Executing external compose provider "/home/del/.nix-profile/bin/docker-compose". Please refer to the documentation for details. <<<<

Docker Compose version 2.26.1ls /dev | grep nvidia
nvidia0
nvidia-caps
nvidiactl
nvidia-modeset
nvidia-uvm
nvidia-uvm-toolspodman run --rm --device nvidia.com/gpu=all ubuntu ls /dev | grep nvidia
nvidia-modeset
nvidia-uvm
nvidia-uvm-tools
nvidia0
nvidiactlcat compose.yaml
services:
  testgpu:
    image: ubuntu
    command: sh -c "ls /dev | grep nvidia"
    deploy:
      resources:
        reservations:
          devices:
            - driver: cdi
              device_ids:
                - nvidia.com/gpu=alldocker compose up
>>>> Executing external compose provider "/home/del/.nix-profile/bin/docker-compose". Please refer to the documentation for details. <<<<

[+] Running 1/0
 ✔ Container test-testgpu-1  Created                                                                                                 0.0s
Attaching to testgpu-1
testgpu-1 exited with code 1

So still no access to the GPU with compose/podman through CDI.

@loopback-kr
Copy link

This test was run inside a Slurm cluster with podman support.
There will be differences from an on-premises server, but hopefully this will help.

hyunseoki@Master:~/projects/calc-pi$ srun --pty --gpus 2  bash
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

hyunseoki@gpu2:~/projects/calc-pi$ docker version
Client:       Podman Engine
Version:      4.6.2
API Version:  4.6.2
Go Version:   go1.18.1
Built:        Thu Jan  1 09:00:00 1970
OS/Arch:      linux/amd64
hyunseoki@gpu2:~/projects/calc-pi$ docker-compose version
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.6.2
podman-compose version 1.0.6
podman --version 
podman version 4.6.2
exit code: 0
hyunseoki@gpu2:~/projects/calc-pi$ nvidia-smi -L
GPU 0: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-55585a22-a409-88ad-4376-c3b613036a7d)
GPU 1: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-b18e6aa7-83f0-0ae4-2bb1-0f3050e0e160)
hyunseoki@gpu2:~/projects/calc-pi$ docker run -it --rm --device nvidia.com/gpu=all python nvidia-smi -L
GPU 0: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-55585a22-a409-88ad-4376-c3b613036a7d)
GPU 1: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-b18e6aa7-83f0-0ae4-2bb1-0f3050e0e160)
hyunseoki@gpu2:~/projects/calc-pi$ cat /etc/containers/pre-exec-hooks/000_nvidia_cdi.sh 
#!/bin/bash
if [ -e /usr/bin/nvidia-ctk ]; then
  if [ -e /var/run/cdi/nvidia.yaml ]; then
    sleep $(( (RANDOM % 6) + 2 ))
    nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml > /dev/null 2>&1
  else
    nvidia-ctk cdi generate --output=/var/run/cdi/nvidia.yaml > /dev/null 2>&1
  fi
fi
exit 0
hyunseoki@gpu2:~/projects/calc-pi$ cat docker-compose.yml 
services:
  vscode:
    image: docker.io/python
    # build:
      # dockerfile: vscode.Dockerfile
    tty: true
    stdin_open: true
    shm_size: 1TB
    ports:
      - $FORWARD_PORT:8888
    devices:
      - nvidia.com/gpu=all
    volumes:
      - $PWD:/workspace
    environment:
      - SLURMD_NODENAME=$SLURMD_NODENAME
      - PASSWORD=1
      - SUBDOMAIN=$USER
      - FORWARD_PORT=$FORWARD_PORT
    command: nvidia-smi -L
hyunseoki@gpu2:~/projects/calc-pi$ docker-compose up
podman-compose version: 1.0.6
['podman', '--version', '']
using podman version: 4.6.2
** excluding:  set()
['podman', 'ps', '--filter', 'label=io.podman.compose.project=calc-pi', '-a', '--format', '{{ index .Labels "io.podman.compose.config-hash"}}']
recreating: ...
** excluding:  set()
podman stop -t 10 calc-pi_vscode_1
calc-pi_vscode_1
exit code: 0
podman rm calc-pi_vscode_1
calc-pi_vscode_1
exit code: 0
recreating: done


['podman', 'network', 'exists', 'calc-pi_default']
podman create --name=calc-pi_vscode_1 --label io.podman.compose.config-hash=a7dda14fa6087ec8f2e0557b2bae6ed958ed5468c68034ee6c61d734c7b3e456 --label io.podman.compose.project=calc-pi --label io.podman.compose.version=1.0.6 --label [email protected] --label com.docker.compose.project=calc-pi --label com.docker.compose.project.working_dir=/home/hyunseoki/projects/calc-pi --label com.docker.compose.project.config_files=docker-compose.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=vscode --device nvidia.com/gpu=all -e SLURMD_NODENAME=gpu2 -e PASSWORD=1 -e SUBDOMAIN=hyunseoki -e FORWARD_PORT=20580 -v /home/hyunseoki/projects/calc-pi:/workspace --net calc-pi_default --network-alias vscode -p 20580:8888 --shm-size 1TB -i --tty docker.io/python nvidia-smi -L
7ab542be0eb79b26afd526a70955c7ae34b08d74c878b816ac73813d6bdf35d0
exit code: 0
podman start -a calc-pi_vscode_1
[vscode] | GPU 0: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-55585a22-a409-88ad-4376-c3b613036a7d)
[vscode] | GPU 1: NVIDIA GeForce GTX 1080 Ti (UUID: GPU-b18e6aa7-83f0-0ae4-2bb1-0f3050e0e160)
exit code: 0

@sreevatsank1999
Copy link

What is the TL;DR ?

is it that we cannot use docker-compose as the external compose provider for podman? (given that we need gpu support)

is there a chance that this will be fixed in the near future?

@rhatdan
Copy link
Member

rhatdan commented Aug 1, 2024

Podman does not implement the client side of compose on the server side, so it requires either docker-compose or podman-compose (a separate repository) to process the compose syntax and generate podman containers.

@sreevatsank1999
Copy link

sreevatsank1999 commented Aug 2, 2024

Podman does not implement the client side of compose on the server side, so it requires either docker-compose or podman-compose (a separate repository) to process the compose syntax and generate podman containers.

Snooping on the socket connection /run/user/3938/podman/podman.sock using socat, I find that docker-compose makes the following API calls to the podman system service API server, and I think it might be possible to handle this correctly at the podman API server.

docker-compose + deploy block makes the following API call:

POST /v1.41/containers/create?name=bec3219e5b3c_test-issue-19338-1 HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Content-Length: 2561\r
Content-Type: application/json\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-187626433197f294-01\r
\r
{
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "nvidia-smi"
            ],
            "Image": "ubuntu",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "test",
                "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                "com.docker.compose.service": "issue-19338",
                "com.docker.compose.version": "2.29.1"
            },
            "HostConfig": {
                "Binds": null,
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "",
                    "Config": null
                },
                "NetworkMode": "test_default",
                "PortBindings": {},
                "RestartPolicy": {
                    "Name": "",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "ConsoleSize": [
                    0,
                    0
                ],
                "CapAdd": null,
                "CapDrop": null,
                "CgroupnsMode": "",
                "Dns": null,
                "DnsOptions": null,
                "DnsSearch": null,
                "ExtraHosts": [],
                "GroupAdd": null,
                "IpcMode": "",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 0,
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": null,
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": null,
                "DeviceCgroupRules": null,
                "DeviceRequests": [
                    {
                        "Driver": "",
                        "Count": 1,
                        "DeviceIDs": null,
                        "Capabilities": [
                            [
                                "gpu"
                            ]
                        ],
                        "Options": null
                    }
                ],
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": null,
                "OomKillDisable": false,
                "PidsLimit": null,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": null,
                "ReadonlyPaths": null
            },
            "NetworkingConfig": {
                "EndpointsConfig": {
                    "test_default": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": [
                            "test-issue-19338-1",
                            "issue-19338"
                        ],
                        "MacAddress": "",
                        "DriverOpts": null,
                        "NetworkID": "",
                        "EndpointID": "",
                        "Gateway": "",
                        "IPAddress": "",
                        "IPPrefixLen": 0,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "DNSNames": null
                    }
                }
            }
        }

docker-compose + device block API call:

POST /v1.41/containers/create?name=bec3219e5b3c_test-issue-19338-1 HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Content-Length: 2577\r
Content-Type: application/json\r
Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-18aadb2ff5c0b9ec-01\r
\r
{
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "nvidia-smi"
            ],
            "Image": "ubuntu",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "4a70b70543a273e8ee7f45bf50783bcd3aece1f5ae5bbfba74f61d41f5bcc59b",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "test",
                "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                "com.docker.compose.service": "issue-19338",
                "com.docker.compose.version": "2.29.1"
            },
            "HostConfig": {
                "Binds": null,
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "",
                    "Config": null
                },
                "NetworkMode": "test_default",
                "PortBindings": {},
                "RestartPolicy": {
                    "Name": "",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "ConsoleSize": [
                    0,
                    0
                ],
                "CapAdd": null,
                "CapDrop": null,
                "CgroupnsMode": "",
                "Dns": null,
                "DnsOptions": null,
                "DnsSearch": null,
                "ExtraHosts": [],
                "GroupAdd": null,
                "IpcMode": "",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 0,
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": null,
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": [
                    {
                        "PathOnHost": "nvidia.com/gpu=0",
                        "PathInContainer": "nvidia.com/gpu=0",
                        "CgroupPermissions": "rwm"
                    }
                ],
                "DeviceCgroupRules": null,
                "DeviceRequests": null,
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": null,
                "OomKillDisable": false,
                "PidsLimit": null,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": null,
                "ReadonlyPaths": null
            },
            "NetworkingConfig": {
                "EndpointsConfig": {
                    "test_default": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": [
                            "test-issue-19338-1",
                            "issue-19338"
                        ],
                        "MacAddress": "",
                        "DriverOpts": null,
                        "NetworkID": "",
                        "EndpointID": "",
                        "Gateway": "",
                        "IPAddress": "",
                        "IPPrefixLen": 0,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "DNSNames": null
                    }
                }
            }
        }

Full request/response trace captures:

`docker-compose` + deploy block capture
> 2024/07/31 18: 56: 53.457915  length=101 from=0 to=100
HEAD /_ping HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: Docker-Client/unknown-version (linux)\r
\r
< 2024/07/31 18: 56: 53.458332  length=318 from=0 to=317
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Builder-Version: \r
Buildkit-Version: \r
Cache-Control: no-cache\r
Docker-Experimental: true\r
Libpod-Api-Version: 5.1.2\r
Libpod-Buildah-Version: 1.36.0\r
Ostype: linux\r
Pragma: no-cache\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0000fc548\r
Date: Thu,
01 Aug 2024 01: 56: 53 GMT\r
\r
> 2024/07/31 18: 56: 53.459053  length=153 from=101 to=253
GET /v1.41/info HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-04b12442401aad2a-01\r
\r
> 2024/07/31 18: 56: 54.467236  length=297 from=0 to=296
GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-2dc839f2dc7ce974-01\r
\r
< 2024/07/31 18: 56: 54.473446  length=1696 from=0 to=1695
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
01 Aug 2024 01: 56: 54 GMT\r
Content-Length: 1480\r
\r
[
    {
        "Id": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
        "Names": [
            "/test-issue-19338-1"
        ],
        "Image": "docker.io/library/ubuntu:latest",
        "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
        "Command": "nvidia-smi",
        "Created": 1722475833,
        "Ports": [],
        "Labels": {
            "com.docker.compose.config-hash": "0e4992f78ef02e6d2b4d2586b96e15393e23549641a57cbbcd3a5d9b45061d29",
            "com.docker.compose.container-number": "1",
            "com.docker.compose.depends_on": "",
            "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
            "com.docker.compose.oneoff": "False",
            "com.docker.compose.project": "test",
            "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
            "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
            "com.docker.compose.service": "issue-19338",
            "com.docker.compose.version": "2.29.1",
            "org.opencontainers.image.ref.name": "ubuntu",
            "org.opencontainers.image.version": "24.04"
        },
        "State": "created",
        "Status": "Created",
        "NetworkSettings": {
            "Networks": {
                "test_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "test-issue-19338-1",
                        "issue-19338",
                        "bec3219e5b3c"
                    ],
                    "MacAddress": "",
                    "NetworkID": "test_default",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DriverOpts": null,
                    "DNSNames": null
                }
            }
        },
        "Mounts": [],
        "Name": "",
        "Config": null,
        "NetworkingConfig": null,
        "Platform": null,
        "DefaultReadOnlyNonRecursive": false
    }
]
> 2024/07/31 18: 56: 54.476142  length=167 from=297 to=463
GET /v1.41/images/ubuntu/json HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-5bf8faacf8205821-01\r
\r
< 2024/07/31 18: 56: 54.482639  length=1976 from=1696 to=3671
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
01 Aug 2024 01: 56: 54 GMT\r
Content-Length: 1760\r
\r
{
    "Id": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
    "RepoTags": [
        "docker.io/library/ubuntu:latest"
    ],
    "RepoDigests": [
        "docker.io/library/ubuntu@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30",
        "docker.io/library/ubuntu@sha256:c920ba4cfca05503764b785c16b76d43c83a6df8d1ab107e7e6610000d94315c"
    ],
    "Parent": "",
    "Comment": "",
    "Created": "2024-06-07T12:00:09.099611108Z",
    "ContainerConfig": {
        "Hostname": "35a88802559",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": null,
        "Image": "",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": null
    },
    "DockerVersion": "24.0.5",
    "Author": "",
    "Config": {
        "Hostname": "",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Cmd": [
            "/bin/bash"
        ],
        "Image": "",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": {
            "org.opencontainers.image.ref.name": "ubuntu",
            "org.opencontainers.image.version": "24.04"
        }
    },
    "Architecture": "amd64",
    "Os": "linux",
    "OsVersion": "24.0.5",
    "Size": 80563381,
    "VirtualSize": 80563381,
    "GraphDriver": {
        "Data": {
            "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff",
            "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/work"
        },
        "Name": "overlay"
    },
    "RootFS": {
        "Type": "layers",
        "Layers": [
            "sha256:a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea"
        ]
    },
    "Metadata": {
        "LastTagTime": "0001-01-01T00:00:00Z"
    },
    "Container": ""
}
> 2024/07/31 18: 56: 54.486734  length=170 from=464 to=633
GET /v1.41/networks/test_default HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-9efda98ca27b3e24-01\r
\r
< 2024/07/31 18: 56: 54.491938  length=785 from=3672 to=4456
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
01 Aug 2024 01: 56: 54 GMT\r
Content-Length: 570\r
\r
{
    "Name": "test_default",
    "Id": "72257744b1a3a1880aa5397dbc01e790027fe57a942c08c1355d7c2c888725ca",
    "Created": "2024-07-31T18:30:32.237867878-07:00",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
        "Driver": "default",
        "Options": {
            "driver": "host-local"
        },
        "Config": [
            {
                "Subnet": "10.89.1.0/24",
                "Gateway": "10.89.1.1"
            }
        ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
        "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {},
    "Options": {},
    "Labels": {
        "com.docker.compose.network": "default",
        "com.docker.compose.project": "test",
        "com.docker.compose.version": "2.29.1"
    }
}
> 2024/07/31 18: 56: 54.494158  length=156 from=634 to=789
GET /v1.41/version HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-388d483afddbd5fe-01\r
\r
< 2024/07/31 18: 56: 54.543292  length=2712 from=318 to=3029
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0000fc548\r
Date: Thu,
01 Aug 2024 01: 56: 54 GMT\r
Transfer-Encoding: chunked\r
\r
800\r
{
    "ID": "5a8824e0-b67f-4973-9335-777669ab056e",
    "Containers": 10,
    "ContainersRunning": 0,
    "ContainersPaused": 0,
    "ContainersStopped": 8,
    "Images": 7,
    "Driver": "overlay",
    "DriverStatus": [
        [
            "Backing Filesystem",
            "btrfs"
        ],
        [
            "Supports d_type",
            "true"
        ],
        [
            "Native Overlay Diff",
            "true"
        ],
        [
            "Using metacopy",
            "false"
        ],
        [
            "Supports shifting",
            "false"
        ],
        [
            "Supports volatile",
            "true"
        ]
    ],
    "Plugins": {
        "Volume": [
            "local"
        ],
        "Network": [
            "bridge",
            "macvlan",
            "ipvlan"
        ],
        "Authorization": null,
        "Log": [
            "k8s-file",
            "none",
            "passthrough",
            "journald"
        ]
    },
    "MemoryLimit": true,
    "SwapLimit": false,
    "CpuCfsPeriod": false,
    "CpuCfsQuota": false,
    "CPUShares": false,
    "CPUSet": false,
    "PidsLimit": true,
    "IPv4Forwarding": true,
    "BridgeNfIptables": true,
    "BridgeNfIp6tables": true,
    "Debug": false,
    "NFd": 15,
    "OomKillDisable": false,
    "NGoroutines": 15,
    "SystemTime": "2024-07-31T18:56:54.543021208-07:00",
    "LoggingDriver": "",
    "CgroupDriver": "systemd",
    "NEventsListener": 0,
    "KernelVersion": "6.5.0-44-generic",
    "OperatingSystem": "ubuntu",
    "OSVersion": "22.04",
    "OSType": "linux",
    "Architecture": "amd64",
    "IndexServerAddress": "",
    "RegistryConfig": {
        "AllowNondistributableArtifactsCIDRs": [],
        "AllowNondistributableArtifactsHostnames": [],
        "InsecureRegistryCIDRs": [],
        "IndexConfigs": {},
        "Mirrors": []
    },
    "NCPU": 64,
    "MemTotal": 472804831232,
    "GenericResources": null,
    "DockerRootDir": "/home/groberts/.local/share/containers/storage",
    "HttpProxy": "",
    "HttpsProxy": "",
    "NoProxy": "",
    "Name": "wcsng-36",
    "Labels": null,
    "ExperimentalBuild": true,
    "ServerVersion": "5.1.2",
    "Runtimes": {
        "crun": {
            "path": "/usr/bin/crun"
        },
        "crun-vm": {
            "path": "/usr/bin/crun-vm"
        },
        "crun-wasm": {
            "path": "/usr/bin/crun-wasm"
        },
        "kata": {
            "path": "/usr/bin/kata-runtime"
        },
        "krun": {
            "path": "/usr/bin/krun"
        },
        "ocijail": {
            "path": "/usr/local/bin/ocijail"
        },
        "runc": {
            "path": "/usr/bin/runc"
        },
        "runj": {
            "path": "/usr/local/bin/runj"
        },
        "runsc": {
            "path": "/usr/bin/runsc"
        },
        "youki": {
            "path": "/usr/local/bin/youki"
        }
    },
    "DefaultRuntime": "crun",
    "Swarm": {
        "NodeID": "",
        "NodeAddr": "",
        "LocalNodeState": "inactive",
        "ControlAvailable": false,
        "Error": "",
        "RemoteManagers": null
    },
    "LiveRestoreEnabled": false,
    "Isolation": "",
    "InitBinary": "",
    "ContainerdCommit": {
        "ID": "","Expec\r1a7\r
ted":""},"RuncCommit":{"ID":"","Expected":""},"InitCommit":{"ID":"","Expected":""},"SecurityOptions":["name=apparmor","name=seccomp,profile=default","name=rootless"],"ProductLicense":"Apache-2.0","CDISpecDirs":null,"Warnings":[],"BuildahVersion":"1.36.0","CPURealtimePeriod":false,"CPURealtimeRuntime":false,"CgroupVersion":"2","Rootless":true,"SwapFree":0,"SwapTotal":0,"Uptime":"1h 39m 40.00s (Approximately 0.04 days)"}
\r
0\r
\r
2024/07/31 18: 56: 54 socat[
            80523
        ] E write(6,
        0x638cd71cc000,
        2712): Broken pipe
< 2024/07/31 18: 56: 55.550681  length=1108 from=4457 to=5564
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
        01 Aug 2024 01: 56: 55 GMT\r
Content-Length: 893\r
\r
{
            "Platform": {
                "Name": "linux/amd64/ubuntu-22.04"
            },
            "Components": [
                {
                    "Name": "Podman Engine",
                    "Version": "5.1.2",
                    "Details": {
                        "APIVersion": "5.1.2",
                        "Arch": "amd64",
                        "BuildTime": "1969-12-31T16:00:00-08:00",
                        "Experimental": "false",
                        "GitCommit": "",
                        "GoVersion": "go1.22.5",
                        "KernelVersion": "6.5.0-44-generic",
                        "MinAPIVersion": "4.0.0",
                        "Os": "linux"
                    }
                },
                {
                    "Name": "Conmon",
                    "Version": "conmon version 2.1.2, commit: ",
                    "Details": {
                        "Package": "conmon_100:2.1.2~0_amd64"
                    }
                },
                {
                    "Name": "OCI Runtime (crun)",
                    "Version": "crun version 1.15\\ncommit: e6eacaf4034e84185fd8780ac9262bbf57082278\\nrundir: /run/user/3141/crun\\nspec: 1.0.0\\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL",
                    "Details": {
                        "Package": "crun_100:1.15-1_amd64"
                    }
                }
            ],
            "Version": "5.1.2",
            "ApiVersion": "1.41",
            "MinAPIVersion": "1.24",
            "GitCommit": "",
            "GoVersion": "go1.22.5",
            "Os": "linux",
            "Arch": "amd64",
            "KernelVersion": "6.5.0-44-generic",
            "BuildTime": "1969-12-31T16:00:00-08:00"
        }
> 2024/07/31 18: 56: 55.552874  length=167 from=790 to=956
GET /v1.41/images/ubuntu/json HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-b9516f77fec95279-01\r
\r
< 2024/07/31 18: 56: 55.555169  length=1976 from=5565 to=7540
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
        01 Aug 2024 01: 56: 55 GMT\r
Content-Length: 1760\r
\r
{
            "Id": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
            "RepoTags": [
                "docker.io/library/ubuntu:latest"
            ],
            "RepoDigests": [
                "docker.io/library/ubuntu@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30",
                "docker.io/library/ubuntu@sha256:c920ba4cfca05503764b785c16b76d43c83a6df8d1ab107e7e6610000d94315c"
            ],
            "Parent": "",
            "Comment": "",
            "Created": "2024-06-07T12:00:09.099611108Z",
            "ContainerConfig": {
                "Hostname": "35a88802559",
                "Domainname": "",
                "User": "",
                "AttachStdin": false,
                "AttachStdout": false,
                "AttachStderr": false,
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": null,
                "Cmd": null,
                "Image": "",
                "Volumes": null,
                "WorkingDir": "",
                "Entrypoint": null,
                "OnBuild": null,
                "Labels": null
            },
            "DockerVersion": "24.0.5",
            "Author": "",
            "Config": {
                "Hostname": "",
                "Domainname": "",
                "User": "",
                "AttachStdin": false,
                "AttachStdout": false,
                "AttachStderr": false,
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                ],
                "Cmd": [
                    "/bin/bash"
                ],
                "Image": "",
                "Volumes": null,
                "WorkingDir": "",
                "Entrypoint": null,
                "OnBuild": null,
                "Labels": {
                    "org.opencontainers.image.ref.name": "ubuntu",
                    "org.opencontainers.image.version": "24.04"
                }
            },
            "Architecture": "amd64",
            "Os": "linux",
            "OsVersion": "24.0.5",
            "Size": 80563381,
            "VirtualSize": 80563381,
            "GraphDriver": {
                "Data": {
                    "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff",
                    "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/work"
                },
                "Name": "overlay"
            },
            "RootFS": {
                "Type": "layers",
                "Layers": [
                    "sha256:a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea"
                ]
            },
            "Metadata": {
                "LastTagTime": "0001-01-01T00:00:00Z"
            },
            "Container": ""
        }
> 2024/07/31 18: 56: 55.560339  length=2819 from=957 to=3775
POST /v1.41/containers/create?name=bec3219e5b3c_test-issue-19338-1 HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Content-Length: 2561\r
Content-Type: application/json\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-187626433197f294-01\r
\r
{
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "nvidia-smi"
            ],
            "Image": "ubuntu",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "test",
                "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                "com.docker.compose.service": "issue-19338",
                "com.docker.compose.version": "2.29.1"
            },
            "HostConfig": {
                "Binds": null,
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "",
                    "Config": null
                },
                "NetworkMode": "test_default",
                "PortBindings": {},
                "RestartPolicy": {
                    "Name": "",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "ConsoleSize": [
                    0,
                    0
                ],
                "CapAdd": null,
                "CapDrop": null,
                "CgroupnsMode": "",
                "Dns": null,
                "DnsOptions": null,
                "DnsSearch": null,
                "ExtraHosts": [],
                "GroupAdd": null,
                "IpcMode": "",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 0,
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": null,
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": null,
                "DeviceCgroupRules": null,
                "DeviceRequests": [
                    {
                        "Driver": "",
                        "Count": 1,
                        "DeviceIDs": null,
                        "Capabilities": [
                            [
                                "gpu"
                            ]
                        ],
                        "Options": null
                    }
                ],
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": null,
                "OomKillDisable": false,
                "PidsLimit": null,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": null,
                "ReadonlyPaths": null
            },
            "NetworkingConfig": {
                "EndpointsConfig": {
                    "test_default": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": [
                            "test-issue-19338-1",
                            "issue-19338"
                        ],
                        "MacAddress": "",
                        "DriverOpts": null,
                        "NetworkID": "",
                        "EndpointID": "",
                        "Gateway": "",
                        "IPAddress": "",
                        "IPPrefixLen": 0,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "DNSNames": null
                    }
                }
            }
        }
< 2024/07/31 18: 56: 55.588605  length=307 from=7541 to=7847
HTTP/1.1 201 Created\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
        01 Aug 2024 01: 56: 55 GMT\r
Content-Length: 88\r
\r
{
            "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd",
            "Warnings": []
        }
> 2024/07/31 18: 56: 55.589584  length=229 from=3776 to=4004
GET /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/json HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-03167f6a8e160f24-01\r
\r
< 2024/07/31 18: 56: 55.592408  length=4585 from=7848 to=12432
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
        01 Aug 2024 01: 56: 55 GMT\r
Transfer-Encoding: chunked\r
\r
84b\r
{
            "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd",
            "Created": "2024-08-01T01:56:55.574628083Z",
            "Path": "nvidia-smi",
            "Args": [
                "nvidia-smi"
            ],
            "State": {
                "Status": "created",
                "Running": false,
                "Paused": false,
                "Restarting": false,
                "OOMKilled": false,
                "Dead": false,
                "Pid": 0,
                "ExitCode": 0,
                "Error": "",
                "StartedAt": "0001-01-01T00:00:00Z",
                "FinishedAt": "0001-01-01T00:00:00Z"
            },
            "Image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
            "ResolvConfPath": "",
            "HostnamePath": "",
            "HostsPath": "",
            "LogPath": "",
            "Name": "/bec3219e5b3c_test-issue-19338-1",
            "RestartCount": 0,
            "Driver": "overlay",
            "Platform": "linux",
            "MountLabel": "",
            "ProcessLabel": "",
            "AppArmorProfile": "",
            "ExecIDs": [],
            "HostConfig": {
                "Binds": [],
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "journald",
                    "Config": null
                },
                "NetworkMode": "bridge",
                "PortBindings": {},
                "RestartPolicy": {
                    "Name": "no",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "ConsoleSize": [
                    0,
                    0
                ],
                "CapAdd": [],
                "CapDrop": [],
                "CgroupnsMode": "",
                "Dns": [],
                "DnsOptions": [],
                "DnsSearch": [],
                "ExtraHosts": [],
                "GroupAdd": [],
                "IpcMode": "shareable",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "private",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": [],
                "UTSMode": "private",
                "UsernsMode": "",
                "ShmSize": 65536000,
                "Runtime": "oci",
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "user.slice",
                "BlkioWeight": 0,
                "BlkioWeightDevice": null,
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": [],
                "DeviceCgroupRules": null,
                "DeviceRequests": null,
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": 0,
                "OomKillDisable": false,
                "PidsLimit": 2048,
                "Ulimits": [
                    {
                        "Name": "RLIMIT_NPROC",
                        "Hard": 4194304,
                        "Soft": 4194304
                    }
                ],
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": null,
                "ReadonlyPaths": null
            },
            "GraphDriver": {
                "Data": {
                    "LowerDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff"\r
800\r
,
                    "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/3d08674f5d19b671c4faad6b4f9ad29d77a6e47196e190cb119aa38fdbafe659/diff",
                    "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/3d08674f5d19b671c4faad6b4f9ad29d77a6e47196e190cb119aa38fdbafe659/work"
                },
                "Name": "overlay"
            },
            "SizeRootFs": 0,
            "Mounts": [],
            "Config": {
                "Hostname": "f4bbe1f14929",
                "Domainname": "",
                "User": "",
                "AttachStdin": false,
                "AttachStdout": false,
                "AttachStderr": false,
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "container=podman"
                ],
                "Cmd": [
                    "nvidia-smi"
                ],
                "Image": "docker.io/library/ubuntu:latest",
                "Volumes": null,
                "WorkingDir": "/",
                "Entrypoint": [],
                "OnBuild": null,
                "Labels": {
                    "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d",
                    "com.docker.compose.container-number": "1",
                    "com.docker.compose.depends_on": "",
                    "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                    "com.docker.compose.oneoff": "False",
                    "com.docker.compose.project": "test",
                    "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                    "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                    "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                    "com.docker.compose.service": "issue-19338",
                    "com.docker.compose.version": "2.29.1",
                    "org.opencontainers.image.ref.name": "ubuntu",
                    "org.opencontainers.image.version": "24.04"
                },
                "StopSignal": "15",
                "StopTimeout": 10
            },
            "NetworkSettings": {
                "Bridge": "",
                "SandboxID": "",
                "SandboxKey": "",
                "Ports": {},
                "HairpinMode": false,
                "LinkLocalIPv6Address": "",
                "LinkLocalIPv6PrefixLen": 0,
                "SecondaryIPAddresses": null,
                "SecondaryIPv6Addresses": null,
                "EndpointID": "",
                "Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAddress": "",
                "IPPrefixLen": 0,
                "IPv6Gateway": "",
                "MacAddress": "",
                "Networks": {
                    "test_default": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": [
                            "test-issue-19338-1",
                            "issue-19338",
                            "f4bbe1f14929"
                        ],
                        "MacAddress": "",
                        "NetworkID":"test_defa\r
a7\r
ult","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DriverOpts":null,"DNSNames":null}}}}
\r
0\r
\r
> 2024/07/31 18: 56: 55.604440  length=249 from=4005 to=4253
POST /v1.41/containers/bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300/stop HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Content-Length: 0\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-ba74fc813d6bbf81-01\r
\r
< 2024/07/31 18: 56: 55.607485  length=170 from=12433 to=12602
HTTP/1.1 204 No Content\r
Api-Version: 1.41\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
                        01 Aug 2024 01: 56: 55 GMT\r
\r
> 2024/07/31 18: 56: 55.608029  length=227 from=4254 to=4480
DELETE /v1.41/containers/bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300 HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-a2578f73f9e824eb-01\r
\r
< 2024/07/31 18: 56: 55.614576  length=170 from=12603 to=12772
HTTP/1.1 204 No Content\r
Api-Version: 1.41\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
                        01 Aug 2024 01: 56: 55 GMT\r
\r
> 2024/07/31 18: 56: 55.615211  length=275 from=4481 to=4755
POST /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/rename?name=test-issue-19338-1 HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Content-Length: 0\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-0462c738d7e536cd-01\r
\r
< 2024/07/31 18: 56: 55.618356  length=170 from=12773 to=12942
HTTP/1.1 204 No Content\r
Api-Version: 1.41\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
                        01 Aug 2024 01: 56: 55 GMT\r
\r
> 2024/07/31 18: 56: 55.619196  length=402 from=4756 to=5157
GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%2C%22com.docker.compose.service%3Dissue-19338%22%3Atrue%7D%7D HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-4eba62b7d00a0032-01\r
\r
< 2024/07/31 18: 56: 55.621533  length=1792 from=12943 to=14734
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
                        01 Aug 2024 01: 56: 55 GMT\r
Content-Length: 1576\r
\r
[
                            {
                                "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd",
                                "Names": [
                                    "/test-issue-19338-1"
                                ],
                                "Image": "docker.io/library/ubuntu:latest",
                                "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                                "Command": "nvidia-smi",
                                "Created": 1722477415,
                                "Ports": [],
                                "Labels": {
                                    "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d",
                                    "com.docker.compose.container-number": "1",
                                    "com.docker.compose.depends_on": "",
                                    "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                                    "com.docker.compose.oneoff": "False",
                                    "com.docker.compose.project": "test",
                                    "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                                    "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                                    "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                                    "com.docker.compose.service": "issue-19338",
                                    "com.docker.compose.version": "2.29.1",
                                    "org.opencontainers.image.ref.name": "ubuntu",
                                    "org.opencontainers.image.version": "24.04"
                                },
                                "State": "created",
                                "Status": "Created",
                                "NetworkSettings": {
                                    "Networks": {
                                        "test_default": {
                                            "IPAMConfig": null,
                                            "Links": null,
                                            "Aliases": [
                                                "test-issue-19338-1",
                                                "issue-19338",
                                                "f4bbe1f14929"
                                            ],
                                            "MacAddress": "",
                                            "NetworkID": "test_default",
                                            "EndpointID": "",
                                            "Gateway": "",
                                            "IPAddress": "",
                                            "IPPrefixLen": 0,
                                            "IPv6Gateway": "",
                                            "GlobalIPv6Address": "",
                                            "GlobalIPv6PrefixLen": 0,
                                            "DriverOpts": null,
                                            "DNSNames": null
                                        }
                                    }
                                },
                                "Mounts": [],
                                "Name": "",
                                "Config": null,
                                "NetworkingConfig": null,
                                "Platform": null,
                                "DefaultReadOnlyNonRecursive": false
                            }
                        ]
> 2024/07/31 18: 56: 55.626484  length=229 from=5158 to=5386
GET /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/json HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-db0f741e43607a1c-01\r
\r
< 2024/07/31 18: 56: 55.627999  length=4572 from=14735 to=19306
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
                        01 Aug 2024 01: 56: 55 GMT\r
Transfer-Encoding: chunked\r
\r
83e\r
{
                            "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd",
                            "Created": "2024-08-01T01:56:55.574628083Z",
                            "Path": "nvidia-smi",
                            "Args": [
                                "nvidia-smi"
                            ],
                            "State": {
                                "Status": "created",
                                "Running": false,
                                "Paused": false,
                                "Restarting": false,
                                "OOMKilled": false,
                                "Dead": false,
                                "Pid": 0,
                                "ExitCode": 0,
                                "Error": "",
                                "StartedAt": "0001-01-01T00:00:00Z",
                                "FinishedAt": "0001-01-01T00:00:00Z"
                            },
                            "Image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                            "ResolvConfPath": "",
                            "HostnamePath": "",
                            "HostsPath": "",
                            "LogPath": "",
                            "Name": "/test-issue-19338-1",
                            "RestartCount": 0,
                            "Driver": "overlay",
                            "Platform": "linux",
                            "MountLabel": "",
                            "ProcessLabel": "",
                            "AppArmorProfile": "",
                            "ExecIDs": [],
                            "HostConfig": {
                                "Binds": [],
                                "ContainerIDFile": "",
                                "LogConfig": {
                                    "Type": "journald",
                                    "Config": null
                                },
                                "NetworkMode": "bridge",
                                "PortBindings": {},
                                "RestartPolicy": {
                                    "Name": "no",
                                    "MaximumRetryCount": 0
                                },
                                "AutoRemove": false,
                                "VolumeDriver": "",
                                "VolumesFrom": null,
                                "ConsoleSize": [
                                    0,
                                    0
                                ],
                                "CapAdd": [],
                                "CapDrop": [],
                                "CgroupnsMode": "",
                                "Dns": [],
                                "DnsOptions": [],
                                "DnsSearch": [],
                                "ExtraHosts": [],
                                "GroupAdd": [],
                                "IpcMode": "shareable",
                                "Cgroup": "",
                                "Links": null,
                                "OomScoreAdj": 0,
                                "PidMode": "private",
                                "Privileged": false,
                                "PublishAllPorts": false,
                                "ReadonlyRootfs": false,
                                "SecurityOpt": [],
                                "UTSMode": "private",
                                "UsernsMode": "",
                                "ShmSize": 65536000,
                                "Runtime": "oci",
                                "Isolation": "",
                                "CpuShares": 0,
                                "Memory": 0,
                                "NanoCpus": 0,
                                "CgroupParent": "user.slice",
                                "BlkioWeight": 0,
                                "BlkioWeightDevice": null,
                                "BlkioDeviceReadBps": null,
                                "BlkioDeviceWriteBps": null,
                                "BlkioDeviceReadIOps": null,
                                "BlkioDeviceWriteIOps": null,
                                "CpuPeriod": 0,
                                "CpuQuota": 0,
                                "CpuRealtimePeriod": 0,
                                "CpuRealtimeRuntime": 0,
                                "CpusetCpus": "",
                                "CpusetMems": "",
                                "Devices": [],
                                "DeviceCgroupRules": null,
                                "DeviceRequests": null,
                                "MemoryReservation": 0,
                                "MemorySwap": 0,
                                "MemorySwappiness": 0,
                                "OomKillDisable": false,
                                "PidsLimit": 2048,
                                "Ulimits": [
                                    {
                                        "Name": "RLIMIT_NPROC",
                                        "Hard": 4194304,
                                        "Soft": 4194304
                                    }
                                ],
                                "CpuCount": 0,
                                "CpuPercent": 0,
                                "IOMaximumIOps": 0,
                                "IOMaximumBandwidth": 0,
                                "MaskedPaths": null,
                                "ReadonlyPaths": null
                            },
                            "GraphDriver": {
                                "Data": {
                                    "LowerDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff"\r
800\r
,
                                    "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/3d08674f5d19b671c4faad6b4f9ad29d77a6e47196e190cb119aa38fdbafe659/diff",
                                    "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/3d08674f5d19b671c4faad6b4f9ad29d77a6e47196e190cb119aa38fdbafe659/work"
                                },
                                "Name": "overlay"
                            },
                            "SizeRootFs": 0,
                            "Mounts": [],
                            "Config": {
                                "Hostname": "f4bbe1f14929",
                                "Domainname": "",
                                "User": "",
                                "AttachStdin": false,
                                "AttachStdout": false,
                                "AttachStderr": false,
                                "Tty": false,
                                "OpenStdin": false,
                                "StdinOnce": false,
                                "Env": [
                                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                                    "container=podman"
                                ],
                                "Cmd": [
                                    "nvidia-smi"
                                ],
                                "Image": "docker.io/library/ubuntu:latest",
                                "Volumes": null,
                                "WorkingDir": "/",
                                "Entrypoint": [],
                                "OnBuild": null,
                                "Labels": {
                                    "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d",
                                    "com.docker.compose.container-number": "1",
                                    "com.docker.compose.depends_on": "",
                                    "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                                    "com.docker.compose.oneoff": "False",
                                    "com.docker.compose.project": "test",
                                    "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                                    "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                                    "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                                    "com.docker.compose.service": "issue-19338",
                                    "com.docker.compose.version": "2.29.1",
                                    "org.opencontainers.image.ref.name": "ubuntu",
                                    "org.opencontainers.image.version": "24.04"
                                },
                                "StopSignal": "15",
                                "StopTimeout": 10
                            },
                            "NetworkSettings": {
                                "Bridge": "",
                                "SandboxID": "",
                                "SandboxKey": "",
                                "Ports": {},
                                "HairpinMode": false,
                                "LinkLocalIPv6Address": "",
                                "LinkLocalIPv6PrefixLen": 0,
                                "SecondaryIPAddresses": null,
                                "SecondaryIPv6Addresses": null,
                                "EndpointID": "",
                                "Gateway": "",
                                "GlobalIPv6Address": "",
                                "GlobalIPv6PrefixLen": 0,
                                "IPAddress": "",
                                "IPPrefixLen": 0,
                                "IPv6Gateway": "",
                                "MacAddress": "",
                                "Networks": {
                                    "test_default": {
                                        "IPAMConfig": null,
                                        "Links": null,
                                        "Aliases": [
                                            "test-issue-19338-1",
                                            "issue-19338",
                                            "f4bbe1f14929"
                                        ],
                                        "MacAddress": "",
                                        "NetworkID":"test_defa\r
a7\r
ult","EndpointID":"","Gateway":"","IPAddress":"","IPPrefixLen":0,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DriverOpts":null,"DNSNames":null}}}}
\r
0\r
\r
> 2024/07/31 18: 56: 55.640836  length=347 from=0 to=346
POST /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/attach?stderr=1&stdin=1&stdout=1&stream=1 HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Content-Length: 0\r
Connection: Upgrade\r
Content-Type: text/plain\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-853e06bb9b4d2ad1-01\r
Upgrade: tcp\r
\r
< 2024/07/31 18: 56: 55.888321  length=561 from=0 to=560
HTTP/1.1 409 Conflict\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0009db718\r
Date: Thu,
                                        01 Aug 2024 01: 56: 55 GMT\r
Content-Length: 340\r
\r
{
                                            "cause": "OCI runtime attempted to invoke a command that was not found",
                                            "message": "preparing container f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd for attach: crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found",
                                            "response": 409
                                        }
> 2024/07/31 18: 56: 55.889157  length=262 from=5387 to=5648
GET /v1.41/containers/f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd/logs?follow=1&stderr=1&stdout=1&tail= HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-85cf00e37654625c-01\r
\r
< 2024/07/31 18: 56: 55.891360  length=181 from=19307 to=19487
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
                                        01 Aug 2024 01: 56: 55 GMT\r
Content-Length: 0\r
\r
> 2024/07/31 18: 56: 55.891696  length=300 from=5649 to=5948
GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-e007044560aca1d3-01\r
\r
> 2024/07/31 18: 56: 55.892578  length=402 from=0 to=401
GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.oneoff%3DFalse%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%2C%22com.docker.compose.service%3Dissue-19338%22%3Atrue%7D%7D HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-2885149299a36c61-01\r
\r
< 2024/07/31 18: 56: 55.894293  length=1828 from=19488 to=21315
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
                                        01 Aug 2024 01: 56: 55 GMT\r
Content-Length: 1612\r
\r
[
                                            {
                                                "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd",
                                                "Names": [
                                                    "/test-issue-19338-1"
                                                ],
                                                "Image": "docker.io/library/ubuntu:latest",
                                                "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                                                "Command": "nvidia-smi",
                                                "Created": 1722477415,
                                                "Ports": [],
                                                "Labels": {
                                                    "com.docker.compose.config-hash":"13075ac5dc2afadcb2de1ca22e075f593c12b18< 2024/07/31 18:56:55.894780  length=1828 from=0 to=18275HdT6TdP3/b14.515 3270d01 0O1Ka\ra
4Abp1if-7V0e4rds"i,o"nc:o m1..d4o1c\rk
eCro.nctoemnpto-sTey.pceo:n taapipnleirc-antuimobne/rj"s:o"n1\r",L"icbopmo.dd-oAcpkie-rV.ecrosmipoons:e .5d.e1p.e2n\rd
sS_eornv"e:r": " ,L"icbopmo.dd/o5c.k1e.r2. c(olmipnousxe).\ri
mXa-gRee"f:e"rsehnac2e5-6I:d3: 5 a08x8c800020595d9ad5d3280\r7
7Dea5t8e4: 3 9T4h4u7,
                                                    1 d0d1a aA1uag2 c250b2f4d 10618: 9536b: 85259 eGaM5T7\r6
1C9o3n0t1eenbt3-9L0e8n"g,t"hc:o m1.6d1o2c\rk
e\rr
.[c{o"mIpdo"s:e".fo4nbeboef1ff"1: 4"9F2a9lasee6"0,
                                                            2"6c4o1m5.4d5o5cck3e2r2.5cao6mdp6o6see4.6parao1j9efc7t4"1: 5"0t3e6sdtd"d,
                                                            2"acaoamb.9d1oac2kdecrd."c,o"mNpaomsees."p:r[o"j/etcets.tc-oinsfsiuge_-f1i9l3e3s8"-: 1""/
                                                            ]h,o"mIem/aggreo"b:e"rdtosc/kdeerv./idoo/clkiebrr/atreys/tu/bduonctkue:rl-actoemspto"s,e".Iymmalg"e,I"Dc"o:m".sdhoac2k5e6r:.3c5oam8p8o8s0e2.5p5r9odjde2c0t7.7weo5r8k4i3n9g4_4d7i1rd"d:a"a/1hao2mce5/bgfrdo1b6e8r9t3sb/8d2e9ve/ad5o7c6k1e9r3/0t1eesbt3"9,
                                                            0"8c"o,m".Cdoomcmkaenrd."c:o"mnpvoisdei.ar-espmlia"c,e""C:r"ebaetce3d2"1: 91e752b234c797b43105f,
                                                            4"4P8o1r1t3s3"b:e[a
                                                            ]1,c"cL6a6bee3lds3"7: 0{
                                                                8"6c7o3mb.9d6occ0kbebr0.3c1o4mdp7o8sbe4.ecdoan3f0i0g"-,h"acsohm".:d"o1c3k0e7r5.acco5mdpco2saef.asdecrbv2idcee1"c:a"2i2ses0u7e5-f1599333c812"b,
                                                                1"8c5odm6.dd3obc4k5e5r3.7cdo1m0p1oasae4.bv1efr7s0i4odn"",:""c2o.m2.9d.o1c"k,e"ro.rcgo.moppoesnec.ocnotnatianienresr.-inmuamgbee.rr"e:f".1n"a,m"ec"o:m".udboucnkteur".,c"oomrpgo.soep.ednecpoenntdasi_noenr"s:."i"m,a"gceo.mv.edroscikoenr".:c"o2m4p.o0s4e".
                                                            }i,m"aSgtea"t:e""s:h"ac2r5e6a:t3e5da"8,
                                                            8"8S0t2a5t5u9sd"d: 2"0C7r7eea5t8e4d3"9,
                                                            4"4N7e1tdwdoarak1Sae2tct5ibnfgds1"6: 8{
                                                                9"3Nbe8t2w9oerak5s7"6: 1{
                                                                    9"3t0e1setb_3d9e0f8a"u,l"tc"o:m{."dIoPcAkMeCro.ncfoimgp"o:sneu.loln,e"oLfifn"k:s""F:anluslel",,""Acloima.sdeosc"k:e[r".tceosmtp-oisses.uper-o1j9e3c3t8"-: 1""t,e"sits"s,u"ec-o1m9.3d3o8c"k,e"rf.4cbobmep1ofs1e4.9p2r9o"j
                                                                        ]e,c"tM.accoAndfdirge_sfsi"l:e"s1"a: : "f/ch:o8m0e:/2g3r:o1b5e:rdt9s"/,d"eNve/tdwoocrkkeIrD/"t:e"stte/sdto_cdkeefra-uclotm"p,o"sEen.dypmoli"n,t"IcDo"m:."d"o,c"keGra.tceowmapyo"s:e".1p0r.o8j9e.c1t..1w"o,r"kIiPnAgd_ddriers"s: "":/"h1o0m.e8/9g.r1o.b6e"r,t"sI/PdPerve/fdioxcLkeenr"/:t2e4s,t""I,P"vc6oGma.tdeowcakye"r:."c"o,m"pGolsoeb.arleIpPlva6cAed"d:r"ebsesc"3: 2"1"9,e"5Gbl3ocb9abl3I0Pfv464P8r1e1f3i3xbLeean1"c:c06,
                                                                        6"eD3rdi3v7e0r8O6p7t3sb"9: 6ncu0lblb,
                                                                        0"3D1N4SdN7a8mbe4se"d:an3u0l0l"},}"
                                                                    }c,o"mM.oduonctkse"r:.[c
                                                                    ]o,m"pNoasmee."s:e"r"v,i"cCeo"n:f"iigs"s:uneu-l1l9,
                                                                    3"3N8e"t,w"ocrokmi.ndgoCcoknefri.gc"o:mnpuolsle,."vPelrastifoonr"m: "": 2n.u2l9l.,
                                                                    1""D,e"foarugl.toRpeeandcOonnltyaNionneRresc.uirmsaigvee."r:effa.lnsaem
                                                                }e
                                                            ]": "ubuntu",
                                                            "org.opencontainers.image.version": "24.04"
                                                        },
                                                        "State": "created",
                                                        "Status": "Created",
                                                        "NetworkSettings": {
                                                            "Networks": {
                                                                "test_default": {
                                                                    "IPAMConfig": null,
                                                                    "Links": null,
                                                                    "Aliases": [
                                                                        "test-issue-19338-1",
                                                                        "issue-19338",
                                                                        "f4bbe1f14929"
                                                                    ],
                                                                    "MacAddress": "1a:fc:80:23:15:d9",
                                                                    "NetworkID": "test_default",
                                                                    "EndpointID": "",
                                                                    "Gateway":"10.89.1.> 2024/07/31 18:56:55.897157  length=250 from=5949 to=61981P"O,S"TI P/Avd1d.r4e1s/sc"o:n"t1a0i.n8e9r.s1/.f64"b,b"eI1PfP1r4e9f2i9xaLee6n0"2:62441,5"4I5P5vc63G2a2t5eaw6ady6"6:e"4"6,a"aG1l9ofb7a4l1I5P0v366Adddddr2easasa"b:9"1"a,2"dGcldo/bsatlaIrPtv 6HPTrTePf/i1x.L1e\rn
"H:o0s,t":D raipvie.rmOopbtys."l:oncualllh,o"sDtN\rS
NUasmeers-"A:gneunltl:} }c}o,m"pMoosunet/sv"2:.[2]9,."1N\ra
mCeo"n:t"e"n,t"-CLoennfgitgh":: n0u\rl
lT,r"aNceetpwaorreknitn:g C0o0n-fci8gd"b:dndudldl2,c"5P2l1a3tffao9rdmd"0: 3neual2la,a"bD2e2f2a3u9l-tbR7eca4d7Obn1l5ydNdo1neRbeaceufr-s0i1v\re
"\r:false
                                                                }
                                                            ]
> 2024/07/31 18: 56: 55.897875  length=236 from=402 to=637
GET /v1.41/events?filters=%7B%22label%22%3A%7B%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-3128fb2c8d3eb8e3-01\r
\r
< 2024/07/31 18: 56: 56.463612  length=477 from=21316 to=21792
HTTP/1.1 500 Internal Server Error\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
                                                            01 Aug 2024 01: 56: 56 GMT\r
Content-Length: 243\r
\r
{
                                                                "cause": "OCI runtime attempted to invoke a command that was not found",
                                                                "message": "crun: executable file `nvidia-smi` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found",
                                                                "response": 500
                                                            }
> 2024/07/31 18: 56: 56.464842  length=297 from=6199 to=6495
GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-c8dbdddd2c5213fa9dd03ea2aab22239-609d6bae7a6c1549-01\r
\r
< 2024/07/31 18: 56: 56.466665  length=227 from=1828 to=2054
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0009da538\r
Date: Thu,
                                                            01 Aug 2024 01: 56: 56 GMT\r
Transfer-Encoding: chunked\r
\r
0\r
\r
2024/07/31 18: 56: 56 socat[
                                                                80739
                                                            ] E write(6,
                                                            0x638cd71cc000,
                                                            227): Broken pipe
< 2024/07/31 18: 56: 56.518496  length=1792 from=21793 to=23584
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc000446220\r
Date: Thu,
                                                            01 Aug 2024 01: 56: 56 GMT\r
Content-Length: 1576\r
\r
[
                                                                {
                                                                    "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd",
                                                                    "Names": [
                                                                        "/test-issue-19338-1"
                                                                    ],
                                                                    "Image": "docker.io/library/ubuntu:latest",
                                                                    "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                                                                    "Command": "nvidia-smi",
                                                                    "Created": 1722477415,
                                                                    "Ports": [],
                                                                    "Labels": {
                                                                        "com.docker.compose.config-hash": "13075ac5dc2afadcb2de1ca22e075f593c12b185d6d3b45537d101aa4b1f704d",
                                                                        "com.docker.compose.container-number": "1",
                                                                        "com.docker.compose.depends_on": "",
                                                                        "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                                                                        "com.docker.compose.oneoff": "False",
                                                                        "com.docker.compose.project": "test",
                                                                        "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                                                                        "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                                                                        "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                                                                        "com.docker.compose.service": "issue-19338",
                                                                        "com.docker.compose.version": "2.29.1",
                                                                        "org.opencontainers.image.ref.name": "ubuntu",
                                                                        "org.opencontainers.image.version": "24.04"
                                                                    },
                                                                    "State": "created",
                                                                    "Status": "Created",
                                                                    "NetworkSettings": {
                                                                        "Networks": {
                                                                            "test_default": {
                                                                                "IPAMConfig": null,
                                                                                "Links": null,
                                                                                "Aliases": [
                                                                                    "test-issue-19338-1",
                                                                                    "issue-19338",
                                                                                    "f4bbe1f14929"
                                                                                ],
                                                                                "MacAddress": "",
                                                                                "NetworkID": "test_default",
                                                                                "EndpointID": "",
                                                                                "Gateway": "",
                                                                                "IPAddress": "",
                                                                                "IPPrefixLen": 0,
                                                                                "IPv6Gateway": "",
                                                                                "GlobalIPv6Address": "",
                                                                                "GlobalIPv6PrefixLen": 0,
                                                                                "DriverOpts": null,
                                                                                "DNSNames": null
                                                                            }
                                                                        }
                                                                    },
                                                                    "Mounts": [],
                                                                    "Name": "",
                                                                    "Config": null,
                                                                    "NetworkingConfig": null,
                                                                    "Platform": null,
                                                                    "DefaultReadOnlyNonRecursive": false
                                                                }
                                                            ]
2024/07/31 18: 56: 56 socat[
                                                                80550
                                                            ] E write(6,
                                                            0x638cd71cc000,
                                                            1792): Broken pipe

X-Reference-Id: 0xc000446220\r
Date: Thu,
                                                            01 Aug 2024 01: 56: 56 GMT\r
Content-Length: 1576\r
\r
[
                                                                {
                                                                    "Id": "f4bbe1f14929ae6026415455c3225a6d66e46aa19f7415036ddd2aaab91a2dcd",
                                                                    "Names": [
                                                                        "/test-issue-19338-1"
                                                                    ],
                                                                    "Image": "docker.io/library/ubuntu:latest",
                                                                    "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                                                                    "Command": "nvidia-smi",
                                                                    "Created": 1722477415,
                                                                    "Ports": [],
                                                                    "Labels": {
                                                                        "com.docker.compose.config-hash":"13075ac^
`docker-compose` + device block capture
2024/07/31 18: 52: 31.865054  length=101 from=0 to=100
HEAD /_ping HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: Docker-Client/unknown-version (linux)\r
\r
< 2024/07/31 18: 52: 31.865863  length=318 from=0 to=317
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Builder-Version: \r
Buildkit-Version: \r
Cache-Control: no-cache\r
Docker-Experimental: true\r
Libpod-Api-Version: 5.1.2\r
Libpod-Buildah-Version: 1.36.0\r
Ostype: linux\r
Pragma: no-cache\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0000fc510\r
Date: Thu,
01 Aug 2024 01: 52: 31 GMT\r
\r
> 2024/07/31 18: 52: 31.866910  length=153 from=101 to=253
GET /v1.41/info HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-bda3b5ba95366261-01\r
\r
> 2024/07/31 18: 52: 32.874226  length=297 from=0 to=296
GET /v1.41/containers/json?all=1&filters=%7B%22label%22%3A%7B%22com.docker.compose.config-hash%22%3Atrue%2C%22com.docker.compose.project%3Dtest%22%3Atrue%7D%7D HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-6b70fe639135b929-01\r
\r
< 2024/07/31 18: 52: 32.904515  length=1696 from=0 to=1695
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0002c4080\r
Date: Thu,
01 Aug 2024 01: 52: 32 GMT\r
Content-Length: 1480\r
\r
[
    {
        "Id": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
        "Names": [
            "/test-issue-19338-1"
        ],
        "Image": "docker.io/library/ubuntu:latest",
        "ImageID": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
        "Command": "nvidia-smi",
        "Created": 1722475833,
        "Ports": [],
        "Labels": {
            "com.docker.compose.config-hash": "0e4992f78ef02e6d2b4d2586b96e15393e23549641a57cbbcd3a5d9b45061d29",
            "com.docker.compose.container-number": "1",
            "com.docker.compose.depends_on": "",
            "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
            "com.docker.compose.oneoff": "False",
            "com.docker.compose.project": "test",
            "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
            "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
            "com.docker.compose.service": "issue-19338",
            "com.docker.compose.version": "2.29.1",
            "org.opencontainers.image.ref.name": "ubuntu",
            "org.opencontainers.image.version": "24.04"
        },
        "State": "created",
        "Status": "Created",
        "NetworkSettings": {
            "Networks": {
                "test_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "test-issue-19338-1",
                        "issue-19338",
                        "bec3219e5b3c"
                    ],
                    "MacAddress": "",
                    "NetworkID": "test_default",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DriverOpts": null,
                    "DNSNames": null
                }
            }
        },
        "Mounts": [],
        "Name": "",
        "Config": null,
        "NetworkingConfig": null,
        "Platform": null,
        "DefaultReadOnlyNonRecursive": false
    }
]
> 2024/07/31 18: 52: 32.907429  length=167 from=297 to=463
GET /v1.41/images/ubuntu/json HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-9814bbfce2115da3-01\r
\r
< 2024/07/31 18: 52: 32.913446  length=1976 from=1696 to=3671
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0002c4080\r
Date: Thu,
01 Aug 2024 01: 52: 32 GMT\r
Content-Length: 1760\r
\r
{
    "Id": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
    "RepoTags": [
        "docker.io/library/ubuntu:latest"
    ],
    "RepoDigests": [
        "docker.io/library/ubuntu@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30",
        "docker.io/library/ubuntu@sha256:c920ba4cfca05503764b785c16b76d43c83a6df8d1ab107e7e6610000d94315c"
    ],
    "Parent": "",
    "Comment": "",
    "Created": "2024-06-07T12:00:09.099611108Z",
    "ContainerConfig": {
        "Hostname": "35a88802559",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": null,
        "Cmd": null,
        "Image": "",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": null
    },
    "DockerVersion": "24.0.5",
    "Author": "",
    "Config": {
        "Hostname": "",
        "Domainname": "",
        "User": "",
        "AttachStdin": false,
        "AttachStdout": false,
        "AttachStderr": false,
        "Tty": false,
        "OpenStdin": false,
        "StdinOnce": false,
        "Env": [
            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
        ],
        "Cmd": [
            "/bin/bash"
        ],
        "Image": "",
        "Volumes": null,
        "WorkingDir": "",
        "Entrypoint": null,
        "OnBuild": null,
        "Labels": {
            "org.opencontainers.image.ref.name": "ubuntu",
            "org.opencontainers.image.version": "24.04"
        }
    },
    "Architecture": "amd64",
    "Os": "linux",
    "OsVersion": "24.0.5",
    "Size": 80563381,
    "VirtualSize": 80563381,
    "GraphDriver": {
        "Data": {
            "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff",
            "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/work"
        },
        "Name": "overlay"
    },
    "RootFS": {
        "Type": "layers",
        "Layers": [
            "sha256:a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea"
        ]
    },
    "Metadata": {
        "LastTagTime": "0001-01-01T00:00:00Z"
    },
    "Container": ""
}
> 2024/07/31 18: 52: 32.917941  length=170 from=464 to=633
GET /v1.41/networks/test_default HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-08dbb05806bdb57a-01\r
\r
< 2024/07/31 18: 52: 32.920532  length=785 from=3672 to=4456
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0002c4080\r
Date: Thu,
01 Aug 2024 01: 52: 32 GMT\r
Content-Length: 570\r
\r
{
    "Name": "test_default",
    "Id": "72257744b1a3a1880aa5397dbc01e790027fe57a942c08c1355d7c2c888725ca",
    "Created": "2024-07-31T18:30:32.237867878-07:00",
    "Scope": "local",
    "Driver": "bridge",
    "EnableIPv6": false,
    "IPAM": {
        "Driver": "default",
        "Options": {
            "driver": "host-local"
        },
        "Config": [
            {
                "Subnet": "10.89.1.0/24",
                "Gateway": "10.89.1.1"
            }
        ]
    },
    "Internal": false,
    "Attachable": false,
    "Ingress": false,
    "ConfigFrom": {
        "Network": ""
    },
    "ConfigOnly": false,
    "Containers": {},
    "Options": {},
    "Labels": {
        "com.docker.compose.network": "default",
        "com.docker.compose.project": "test",
        "com.docker.compose.version": "2.29.1"
    }
}
> 2024/07/31 18: 52: 32.923093  length=156 from=634 to=789
GET /v1.41/version HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-b5534cde8a0dd54e-01\r
\r
< 2024/07/31 18: 52: 32.962099  length=2712 from=318 to=3029
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0000fc510\r
Date: Thu,
01 Aug 2024 01: 52: 32 GMT\r
Transfer-Encoding: chunked\r
\r
800\r
{
    "ID": "18cd51c0-2674-49f3-92eb-95aac5b8db57",
    "Containers": 10,
    "ContainersRunning": 0,
    "ContainersPaused": 0,
    "ContainersStopped": 8,
    "Images": 7,
    "Driver": "overlay",
    "DriverStatus": [
        [
            "Backing Filesystem",
            "btrfs"
        ],
        [
            "Supports d_type",
            "true"
        ],
        [
            "Native Overlay Diff",
            "true"
        ],
        [
            "Using metacopy",
            "false"
        ],
        [
            "Supports shifting",
            "false"
        ],
        [
            "Supports volatile",
            "true"
        ]
    ],
    "Plugins": {
        "Volume": [
            "local"
        ],
        "Network": [
            "bridge",
            "macvlan",
            "ipvlan"
        ],
        "Authorization": null,
        "Log": [
            "k8s-file",
            "none",
            "passthrough",
            "journald"
        ]
    },
    "MemoryLimit": true,
    "SwapLimit": false,
    "CpuCfsPeriod": false,
    "CpuCfsQuota": false,
    "CPUShares": false,
    "CPUSet": false,
    "PidsLimit": true,
    "IPv4Forwarding": true,
    "BridgeNfIptables": true,
    "BridgeNfIp6tables": true,
    "Debug": false,
    "NFd": 16,
    "OomKillDisable": false,
    "NGoroutines": 15,
    "SystemTime": "2024-07-31T18:52:32.959161107-07:00",
    "LoggingDriver": "",
    "CgroupDriver": "systemd",
    "NEventsListener": 0,
    "KernelVersion": "6.5.0-44-generic",
    "OperatingSystem": "ubuntu",
    "OSVersion": "22.04",
    "OSType": "linux",
    "Architecture": "amd64",
    "IndexServerAddress": "",
    "RegistryConfig": {
        "AllowNondistributableArtifactsCIDRs": [],
        "AllowNondistributableArtifactsHostnames": [],
        "InsecureRegistryCIDRs": [],
        "IndexConfigs": {},
        "Mirrors": []
    },
    "NCPU": 64,
    "MemTotal": 472804831232,
    "GenericResources": null,
    "DockerRootDir": "/home/groberts/.local/share/containers/storage",
    "HttpProxy": "",
    "HttpsProxy": "",
    "NoProxy": "",
    "Name": "wcsng-36",
    "Labels": null,
    "ExperimentalBuild": true,
    "ServerVersion": "5.1.2",
    "Runtimes": {
        "crun": {
            "path": "/usr/bin/crun"
        },
        "crun-vm": {
            "path": "/usr/bin/crun-vm"
        },
        "crun-wasm": {
            "path": "/usr/bin/crun-wasm"
        },
        "kata": {
            "path": "/usr/bin/kata-runtime"
        },
        "krun": {
            "path": "/usr/bin/krun"
        },
        "ocijail": {
            "path": "/usr/local/bin/ocijail"
        },
        "runc": {
            "path": "/usr/bin/runc"
        },
        "runj": {
            "path": "/usr/local/bin/runj"
        },
        "runsc": {
            "path": "/usr/bin/runsc"
        },
        "youki": {
            "path": "/usr/local/bin/youki"
        }
    },
    "DefaultRuntime": "crun",
    "Swarm": {
        "NodeID": "",
        "NodeAddr": "",
        "LocalNodeState": "inactive",
        "ControlAvailable": false,
        "Error": "",
        "RemoteManagers": null
    },
    "LiveRestoreEnabled": false,
    "Isolation": "",
    "InitBinary": "",
    "ContainerdCommit": {
        "ID": "","Expec\r1a7\r
ted":""},"RuncCommit":{"ID":"","Expected":""},"InitCommit":{"ID":"","Expected":""},"SecurityOptions":["name=apparmor","name=seccomp,profile=default","name=rootless"],"ProductLicense":"Apache-2.0","CDISpecDirs":null,"Warnings":[],"BuildahVersion":"1.36.0","CPURealtimePeriod":false,"CPURealtimeRuntime":false,"CgroupVersion":"2","Rootless":true,"SwapFree":0,"SwapTotal":0,"Uptime":"1h 34m 19.00s (Approximately 0.04 days)"}
\r
0\r
\r
2024/07/31 18: 52: 32 socat[
            78178
        ] E write(6,
        0x5605b0d1f000,
        2712): Broken pipe
< 2024/07/31 18: 52: 34.003986  length=1108 from=4457 to=5564
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0002c4080\r
Date: Thu,
        01 Aug 2024 01: 52: 34 GMT\r
Content-Length: 893\r
\r
{
            "Platform": {
                "Name": "linux/amd64/ubuntu-22.04"
            },
            "Components": [
                {
                    "Name": "Podman Engine",
                    "Version": "5.1.2",
                    "Details": {
                        "APIVersion": "5.1.2",
                        "Arch": "amd64",
                        "BuildTime": "1969-12-31T16:00:00-08:00",
                        "Experimental": "false",
                        "GitCommit": "",
                        "GoVersion": "go1.22.5",
                        "KernelVersion": "6.5.0-44-generic",
                        "MinAPIVersion": "4.0.0",
                        "Os": "linux"
                    }
                },
                {
                    "Name": "Conmon",
                    "Version": "conmon version 2.1.2, commit: ",
                    "Details": {
                        "Package": "conmon_100:2.1.2~0_amd64"
                    }
                },
                {
                    "Name": "OCI Runtime (crun)",
                    "Version": "crun version 1.15\\ncommit: e6eacaf4034e84185fd8780ac9262bbf57082278\\nrundir: /run/user/3141/crun\\nspec: 1.0.0\\n+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL",
                    "Details": {
                        "Package": "crun_100:1.15-1_amd64"
                    }
                }
            ],
            "Version": "5.1.2",
            "ApiVersion": "1.41",
            "MinAPIVersion": "1.24",
            "GitCommit": "",
            "GoVersion": "go1.22.5",
            "Os": "linux",
            "Arch": "amd64",
            "KernelVersion": "6.5.0-44-generic",
            "BuildTime": "1969-12-31T16:00:00-08:00"
        }
> 2024/07/31 18: 52: 34.006350  length=167 from=790 to=956
GET /v1.41/images/ubuntu/json HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-1ae0b40bcca9b333-01\r
\r
< 2024/07/31 18: 52: 34.008612  length=1976 from=5565 to=7540
HTTP/1.1 200 OK\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0002c4080\r
Date: Thu,
        01 Aug 2024 01: 52: 34 GMT\r
Content-Length: 1760\r
\r
{
            "Id": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
            "RepoTags": [
                "docker.io/library/ubuntu:latest"
            ],
            "RepoDigests": [
                "docker.io/library/ubuntu@sha256:2e863c44b718727c860746568e1d54afd13b2fa71b160f5cd9058fc436217b30",
                "docker.io/library/ubuntu@sha256:c920ba4cfca05503764b785c16b76d43c83a6df8d1ab107e7e6610000d94315c"
            ],
            "Parent": "",
            "Comment": "",
            "Created": "2024-06-07T12:00:09.099611108Z",
            "ContainerConfig": {
                "Hostname": "35a88802559",
                "Domainname": "",
                "User": "",
                "AttachStdin": false,
                "AttachStdout": false,
                "AttachStderr": false,
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": null,
                "Cmd": null,
                "Image": "",
                "Volumes": null,
                "WorkingDir": "",
                "Entrypoint": null,
                "OnBuild": null,
                "Labels": null
            },
            "DockerVersion": "24.0.5",
            "Author": "",
            "Config": {
                "Hostname": "",
                "Domainname": "",
                "User": "",
                "AttachStdin": false,
                "AttachStdout": false,
                "AttachStderr": false,
                "Tty": false,
                "OpenStdin": false,
                "StdinOnce": false,
                "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                ],
                "Cmd": [
                    "/bin/bash"
                ],
                "Image": "",
                "Volumes": null,
                "WorkingDir": "",
                "Entrypoint": null,
                "OnBuild": null,
                "Labels": {
                    "org.opencontainers.image.ref.name": "ubuntu",
                    "org.opencontainers.image.version": "24.04"
                }
            },
            "Architecture": "amd64",
            "Os": "linux",
            "OsVersion": "24.0.5",
            "Size": 80563381,
            "VirtualSize": 80563381,
            "GraphDriver": {
                "Data": {
                    "UpperDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/diff",
                    "WorkDir": "/home/groberts/.local/share/containers/storage/overlay/a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea/work"
                },
                "Name": "overlay"
            },
            "RootFS": {
                "Type": "layers",
                "Layers": [
                    "sha256:a30a5965a4f7d9d5ff76a46eb8939f58e95be844de1ac4a4b452d5d31158fdea"
                ]
            },
            "Metadata": {
                "LastTagTime": "0001-01-01T00:00:00Z"
            },
            "Container": ""
        }
> 2024/07/31 18: 52: 34.012688  length=2835 from=957 to=3791
POST /v1.41/containers/create?name=bec3219e5b3c_test-issue-19338-1 HTTP/1.1\r
Host: api.moby.localhost\r
User-Agent: compose/v2.29.1\r
Content-Length: 2577\r
Content-Type: application/json\r
Traceparent: 00-120dd633fb2e5a2bb1db95db85751ec3-18aadb2ff5c0b9ec-01\r
\r
{
            "Hostname": "",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": null,
            "Cmd": [
                "nvidia-smi"
            ],
            "Image": "ubuntu",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "4a70b70543a273e8ee7f45bf50783bcd3aece1f5ae5bbfba74f61d41f5bcc59b",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:35a88802559dd2077e584394471ddaa1a2c5bfd16893b829ea57619301eb3908",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "test",
                "com.docker.compose.project.config_files": "/home/groberts/dev/docker/test/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/home/groberts/dev/docker/test",
                "com.docker.compose.replace": "bec3219e5b3c9b30f4481133bea1cc66e3d3708673b96c0bb0314d78b4eda300",
                "com.docker.compose.service": "issue-19338",
                "com.docker.compose.version": "2.29.1"
            },
            "HostConfig": {
                "Binds": null,
                "ContainerIDFile": "",
                "LogConfig": {
                    "Type": "",
                    "Config": null
                },
                "NetworkMode": "test_default",
                "PortBindings": {},
                "RestartPolicy": {
                    "Name": "",
                    "MaximumRetryCount": 0
                },
                "AutoRemove": false,
                "VolumeDriver": "",
                "VolumesFrom": null,
                "ConsoleSize": [
                    0,
                    0
                ],
                "CapAdd": null,
                "CapDrop": null,
                "CgroupnsMode": "",
                "Dns": null,
                "DnsOptions": null,
                "DnsSearch": null,
                "ExtraHosts": [],
                "GroupAdd": null,
                "IpcMode": "",
                "Cgroup": "",
                "Links": null,
                "OomScoreAdj": 0,
                "PidMode": "",
                "Privileged": false,
                "PublishAllPorts": false,
                "ReadonlyRootfs": false,
                "SecurityOpt": null,
                "UTSMode": "",
                "UsernsMode": "",
                "ShmSize": 0,
                "Isolation": "",
                "CpuShares": 0,
                "Memory": 0,
                "NanoCpus": 0,
                "CgroupParent": "",
                "BlkioWeight": 0,
                "BlkioWeightDevice": null,
                "BlkioDeviceReadBps": null,
                "BlkioDeviceWriteBps": null,
                "BlkioDeviceReadIOps": null,
                "BlkioDeviceWriteIOps": null,
                "CpuPeriod": 0,
                "CpuQuota": 0,
                "CpuRealtimePeriod": 0,
                "CpuRealtimeRuntime": 0,
                "CpusetCpus": "",
                "CpusetMems": "",
                "Devices": [
                    {
                        "PathOnHost": "nvidia.com/gpu=0",
                        "PathInContainer": "nvidia.com/gpu=0",
                        "CgroupPermissions": "rwm"
                    }
                ],
                "DeviceCgroupRules": null,
                "DeviceRequests": null,
                "MemoryReservation": 0,
                "MemorySwap": 0,
                "MemorySwappiness": null,
                "OomKillDisable": false,
                "PidsLimit": null,
                "Ulimits": null,
                "CpuCount": 0,
                "CpuPercent": 0,
                "IOMaximumIOps": 0,
                "IOMaximumBandwidth": 0,
                "MaskedPaths": null,
                "ReadonlyPaths": null
            },
            "NetworkingConfig": {
                "EndpointsConfig": {
                    "test_default": {
                        "IPAMConfig": null,
                        "Links": null,
                        "Aliases": [
                            "test-issue-19338-1",
                            "issue-19338"
                        ],
                        "MacAddress": "",
                        "DriverOpts": null,
                        "NetworkID": "",
                        "EndpointID": "",
                        "Gateway": "",
                        "IPAddress": "",
                        "IPPrefixLen": 0,
                        "IPv6Gateway": "",
                        "GlobalIPv6Address": "",
                        "GlobalIPv6PrefixLen": 0,
                        "DNSNames": null
                    }
                }
            }
        }
< 2024/07/31 18: 52: 34.021625  length=366 from=7541 to=7906
HTTP/1.1 500 Internal Server Error\r
Api-Version: 1.41\r
Content-Type: application/json\r
Libpod-Api-Version: 5.1.2\r
Server: Libpod/5.1.2 (linux)\r
X-Reference-Id: 0xc0002c4080\r
Date: Thu,
        01 Aug 2024 01: 52: 34 GMT\r
Content-Length: 132\r
\r
{
            "cause": "no such file or directory",
            "message": "container create: stat nvidia.com/gpu=0: no such file or directory",
            "response": 500
        }

@247vanslam
Copy link

Any update on this please? I really enjoy the added security of rootless podman and haven't experienced many other problems with it, but I don't want to run models downloaded from the interwebs in a rootful container.

@giuseppe
Copy link
Member

we miss passing down the device requests from the API to the CLI configuration, something along the lines of:

diff --git a/pkg/api/handlers/compat/containers_create.go b/pkg/api/handlers/compat/containers_create.go
index 14b37804f3..02253b9b2a 100644
--- a/pkg/api/handlers/compat/containers_create.go
+++ b/pkg/api/handlers/compat/containers_create.go
@@ -163,6 +163,11 @@ func cliOpts(cc handlers.CreateContainerConfig, rtc *config.Config) (*entities.C
 	for _, dev := range cc.HostConfig.Devices {
 		devices = append(devices, fmt.Sprintf("%s:%s:%s", dev.PathOnHost, dev.PathInContainer, dev.CgroupPermissions))
 	}
+	for _, r := range cc.HostConfig.Resources.DeviceRequests {
+		if r.Driver == "cdi" {
+			devices = append(devices, r.DeviceIDs...)
+		}
+	}
 
 	// iterate blkreaddevicebps
 	readBps := make([]string, 0, len(cc.HostConfig.BlkioDeviceReadBps))

giuseppe added a commit to giuseppe/libpod that referenced this issue Jan 31, 2025
pass down the devices specifies in the resources block so that CDI
devices in the compose file are honored.

Tested manually with the following compose file:

services:
  testgpupodman_count:
    image: ubuntu:latest
    command: ["nvidia-smi"]
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
  testgpupodman_deviceid:
      image: docker.io/ubuntu:latest
      command: ["nvidia-smi"]
      deploy:
        resources:
          reservations:
            devices:
            - driver: cdi
              device_ids: ['nvidia.com/gpu=all']
              capabilities: [gpu]

Closes: containers#19338

Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe added a commit to giuseppe/libpod that referenced this issue Jan 31, 2025
pass down the devices specifies in the resources block so that CDI
devices in the compose file are honored.

Tested manually with the following compose file:

services:
  testgpupodman_count:
    image: ubuntu:latest
    command: ["nvidia-smi"]
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
  testgpupodman_deviceid:
      image: docker.io/ubuntu:latest
      command: ["nvidia-smi"]
      deploy:
        resources:
          reservations:
            devices:
            - driver: cdi
              device_ids: ['nvidia.com/gpu=all']
              capabilities: [gpu]

Closes: containers#19338

Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe added a commit to giuseppe/libpod that referenced this issue Jan 31, 2025
pass down the devices specifies in the resources block so that CDI
devices in the compose file are honored.

Tested manually with the following compose file:

services:
  testgpupodman_count:
    image: ubuntu:latest
    command: ["nvidia-smi"]
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
  testgpupodman_deviceid:
      image: docker.io/ubuntu:latest
      command: ["nvidia-smi"]
      deploy:
        resources:
          reservations:
            devices:
            - driver: cdi
              device_ids: ['nvidia.com/gpu=all']
              capabilities: [gpu]

Closes: containers#19338

Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe added a commit to giuseppe/libpod that referenced this issue Jan 31, 2025
pass down the devices specifies in the resources block so that CDI
devices in the compose file are honored.

Tested manually with the following compose file:

services:
  testgpupodman_count:
    image: ubuntu:latest
    command: ["nvidia-smi"]
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
  testgpupodman_deviceid:
      image: docker.io/ubuntu:latest
      command: ["nvidia-smi"]
      deploy:
        resources:
          reservations:
            devices:
            - driver: cdi
              device_ids: ['nvidia.com/gpu=all']
              capabilities: [gpu]

Closes: containers#19338

Signed-off-by: Giuseppe Scrivano <[email protected]>
giuseppe added a commit to giuseppe/libpod that referenced this issue Jan 31, 2025
pass down the devices specifies in the resources block so that CDI
devices in the compose file are honored.

Tested manually with the following compose file:

services:
  testgpupodman_count:
    image: ubuntu:latest
    command: ["nvidia-smi"]
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
  testgpupodman_deviceid:
      image: docker.io/ubuntu:latest
      command: ["nvidia-smi"]
      deploy:
        resources:
          reservations:
            devices:
            - driver: cdi
              device_ids: ['nvidia.com/gpu=all']
              capabilities: [gpu]

Closes: containers#19338

Signed-off-by: Giuseppe Scrivano <[email protected]>
mheon pushed a commit to mheon/libpod that referenced this issue Feb 4, 2025
pass down the devices specifies in the resources block so that CDI
devices in the compose file are honored.

Tested manually with the following compose file:

services:
  testgpupodman_count:
    image: ubuntu:latest
    command: ["nvidia-smi"]
    profiles: [gpu]
    deploy:
      resources:
        reservations:
          devices:
          - driver: nvidia
            count: 1
            capabilities: [gpu]
  testgpupodman_deviceid:
      image: docker.io/ubuntu:latest
      command: ["nvidia-smi"]
      deploy:
        resources:
          reservations:
            devices:
            - driver: cdi
              device_ids: ['nvidia.com/gpu=all']
              capabilities: [gpu]

Closes: containers#19338

Signed-off-by: Giuseppe Scrivano <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.