exec_run unexpected empty output when container is running on a remote host #3332
Description
Hello
I am observing incorrect output from exec_run when done on a running container on a remote host. The container is running a standard ubuntu image.
Connection to remote host is over an ssh forwarded unix socket.
Minimal reproducible example
Setup ssh forwarding:
ssh -L /tmp/docker-b.sock:/var/run/docker.sock -NTt -o ServerAliveInterval=30 -o ServerAliveCountMax=10 $DOCKER_HOST_B &
Connect to remote docker and execute command:
>>> import docker
>>> c = docker.DockerClient(base_url="unix:///tmp/docker-b.sock", version="auto", timeout=60, max_pool_size=8)
>>> c.containers.get("42270b507ba3").exec_run("echo hello")
ExecResult(exit_code=0, output=b'hello\n')
>>> c.containers.get("42270b507ba3").exec_run("echo hello")
ExecResult(exit_code=0, output=b'')
>>> c.containers.get("42270b507ba3").exec_run("echo hello")
ExecResult(exit_code=0, output=b'')
>>> c.containers.get("42270b507ba3").exec_run("echo hello")
Troubleshooting
I ran strace on the python process from where I ran exec_run and observed the following,
For unexpected output:
sendto(9, "POST /v1.43/exec/0ab9f918306ddcfdb18b1e36dce81b9ec6ebb2f03e4a26108f5128f73766c8c4/start HTTP/1.1\r\nHost: localhost\r\nUser-Agent: docker-sdk-python/7.1.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: tcp\r\nContent-Type: application/json\r\nContent-Length: 31\r\n\r\n", 286, 0, NULL, 0) = 286
recvfrom(9, "HTTP/1.1 101 UPGRADED\r\nContent-Type: application/vnd.docker.multiplexed-stream\r\nConnection: Upgrade\r\nUpgrade: tcp\r\nApi-Version: 1.43\r\nDocker-Experimental: false\r\nOstype: linux\r\n", 8192, 0, NULL, NULL) = 177
recvfrom(9, "Server: Docker/24.0.5 (linux)\r\n\r\n\1\0\0\0\0\0\0\6hello\n", 8192, 0, NULL, NULL) = 47
For expected output:
sendto(10, "POST /v1.43/exec/832bad59596ae017de0428004e1e9073c2a0e5571ca1ac385df09a27763f6c67/start HTTP/1.1\r\nHost: localhost\r\nUser-Agent: docker-sdk-python/7.1.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: Upgrade\r\nUpgrade: tcp\r\nContent-Type: application/json\r\nContent-Length: 31\r\n\r\n", 286, 0, NULL, 0) = 286
recvfrom(10, "HTTP/1.1 101 UPGRADED\r\nContent-Type: application/vnd.docker.multiplexed-stream\r\nConnection: Upgrade\r\nUpgrade: tcp\r\nApi-Version: 1.43\r\nDocker-Experimental: false\r\nOstype: linux\r\nServer: Docker/24.0.5 (linux)\r\n\r\n", 8192, 0, NULL, NULL) = 210
recvfrom(10, "\1\0\0\0\0\0\0\6", 8, 0, NULL, NULL) = 8
recvfrom(10, "hello\n", 6, 0, NULL, NULL) = 6
So it looks like when the HTTP header is split into two recv calls, the issue is observed.
Full strace output:
docker-issue-strace.log
Additional info
Python version: Python 3.10.12
Docker python SDK version: 7.1.0
Container image: ubuntu:22.04
Docker version:
Client:
Version: 24.0.5
API version: 1.43
Go version: go1.20.3
Git commit: 24.0.5-0ubuntu1~22.04.1
Built: Mon Aug 21 19:50:14 2023
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 24.0.5
API version: 1.43 (minimum version 1.12)
Go version: go1.20.3
Git commit: 24.0.5-0ubuntu1~22.04.1
Built: Mon Aug 21 19:50:14 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.24
GitCommit:
runc:
Version: 1.1.7-0ubuntu1~22.04.2
GitCommit:
docker-init:
Version: 0.19.0
GitCommit: