podman 2.0.5/2.1.1 silently ignores CONTAINER_HOST on centos-8/fedora-32

[ssbarnea]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Setting CONTAINER_HOST should tell podman where to connect to. Setting this variable to a wrong value should make podman return a connection failure and not run with local server. That is very important as it introduces dangerous behaviors where wrong server would be used.

Steps to reproduce the issue:

CONTAINER_HOST=ssh://garbage podman ps

Describe the results you received:
Success listing of running containers from localhost.

Describe the results you expected:
Connection error, something like

Error: Failed to create sshClient: Connection to bastion host (ssh://garbage) failed.: dial tcp: lookup garbage on 192.168.33.1:53: no such host
FAIL: 125

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.14.9
Built:        Wed Sep 30 20:31:11 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

podman info --debug
host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-2.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 81d18b6c3ffc266abdef7ca94c1450e669a6a388'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: journald
  hostname: leno
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.4-200.fc32.x86_64
  linkmode: dynamic
  memFree: 578457600
  memTotal: 33060139008
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 16562778112
  swapTotal: 16601051136
  uptime: 1029h 45m 56.33s (Approximately 42.88 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/ssbarnea/.config/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 1
    stopped: 12
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/ssbarnea/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 63
  runRoot: /run/user/1000/containers
  volumePath: /home/ssbarnea/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1601494271
  BuiltTime: Wed Sep 30 20:31:11 2020
  GitCommit: ""
  GoVersion: go1.14.9
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

▶ rpm -q podman
podman-2.1.1-7.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes/No

Additional environment details (AWS, VirtualBox, physical, etc.):

[ssbarnea]

To be clear the problem seems to reproduce on both fedora and centos with latest rpm versions of podman. As --help documents, the default version this is a bug that needs to addressed and backported.

There is a slight difference in between 2.1.1 and 2.0.5, the newer one that display the value of the environment variable when using --help, the older one displays the hardcoded default. Still, this has no effect on the bug which applies to both: wrong URL must cause error.

[rhatdan]

You need the --remote flag,

CONTAINER_HOST=ssh://garbage podman --remote ps
Login password:
Error: Failed to create sshClient: Connection to bastion host (ssh://garbage) failed.: dial tcp: lookup garbage: no such host

Not sure what the Login password is about.

[rhatdan]

$ CONTAINER_HOST=ssh://garbage podman --remote ps
Error: Failed to create sshClient: Connection to bastion host (ssh://garbage) failed.: dial tcp: lookup garbage: no such host

[ssbarnea]

Why do I need to tell it to use remote when I define this variable? This makes impossible to change which backend is used from outside. The other, unnamed, container tool does not require any arguments to use various backends and the environment variable works not only on cli, but also on python module and andible modules, just defining it it tells it to use specific host.

Is there another environment variable that needs to be defined to achieve that? It essential to be able to change backend without altering the API calls (or the cli arguments) as this would mean: alter the code that may not be under your control.

Another reason for not forcing users to add --remote is that on MacOS they do not need to so, is implicit. I know that on non-Linux host only remote can be used, but that does not seem like a reason for not using remotes on linux, when available.

[edsantiago]

The other unnamed container tool is only remote. It does not have any concept of non-remote operation. Even if the default is localhost, it's always operating on a socket. Podman does not work that way. (Podman-remote does).

[jwhonce]

@ssbarnea Using the latest podman, you should be able to provide login credentials via ssh keys and podman service connection add ... please see podman-system-connection.1.md

[rhatdan]

Then just install podman-remote and alias podman to podman-remote.

There is a field in containers.conf that is not currently exported to allow users to foce the --remote flag to be true, all the time, and then users would have to do podman --remote=false to have local stuff done. I could expose this if people think it would be useful.

[afbjorklund]

We tell minikube users to use podman-remote on Linux, since there were some early bugs with podman --remote...

Otherwise the main problem is similar to #4390, to remember to use different programs on different operating systems.

One approach would be to not rename "podman-remote" on the other operating systems, but then "podman" is missing.

Currently we have OS tabs in the documentation instead, but it does makes it a little more cumbersome to read too:

https://minikube.sigs.k8s.io/docs/handbook/pushing/#3-pushing-directly-to-in-cluster-cri-o-podman-env

It has tabs for Linux | macOS | Windows, similar to the mess that is for the packages on the "install" page.

---

Right now they have to use "podman1-remote" too (for varlink), but hope to support "podman-remote" in next release*.

* See kubernetes/minikube#8596

We ship our own packages for ubuntu, since the official ones were deleted. And refer people to the static release page.

https://github.com/containers/podman/releases/tag/v1.9.3

• podman-remote-darwin.zip
• podman-remote-static.tar.gz
• podman-remote-windows.zip

Just built an arm64 package, too.

Doesn't seem to fit in the naming ?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Podman should support this now, and containers.conf has the remote flag that can default local podman to run in remote mode.