macOS DNS ignored for `login`

[zeha]

/kind bug

Description

This is, unfortunately, #12163 #11976 #10737 once more, but for a specific command (namely, login).

I am trying to log into a container registry, where its DNS name is not resolvable using the resolvers listed in /etc/resolv.conf. However macOS has no problem connecting to it by name. The DNS specifics are pretty much what was explained in #12163 #11976 #10737 already.

I can tell that DNS generally works inside the podman machine, AND it also works for podman pull on the host.

Steps to reproduce the issue:

1. Have a (gitlab) container registry that is not resolvable using the nameservers in /etc/resolv.conf

2. podman login gitlab-registry.example.org

3. Observe podman-remote talking to the nameserver in /etc/resolv.conf, and then reporting "no such host" in the debug logs.

Describe the results you received:

ch@tiq$ podman  --log-level debug login gitlab-registry.example.org
INFO[0000] podman filtering at log level debug
DEBU[0000] Called login.PersistentPreRunE(podman --log-level debug login gitlab-registry.example.org)
DEBU[0000] SSH Ident Key "/Users/ch/.ssh/podman-machine-default" SHA256:XXXXXXXXXXXXXXXXXX ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.3Em3rNCi8h/Listeners", ssh-agent signer(s) enabled
DEBU[0000] SSH Agent Key SHA256:XXXXXXXXXXXXXXXXXX ssh-ed25519
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.4/libpod/_ping
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Found credentials for gitlab-registry.example.org in credential helper containers-auth.json in file /Users/ch/.config/containers/auth.json
Authenticating with existing credentials for gitlab-registry.example.org
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/gitlab-registry.example.org
DEBU[0000] GET https://gitlab-registry.example.org/v2/
DEBU[0000] Ping https://gitlab-registry.example.org/v2/ err Get "https://gitlab-registry.example.org/v2/": dial tcp: lookup gitlab-registry.example.org on [2a02:1748:fad4:5021::1]:53: no such host (&url.Error{Op:"Get", URL:"https://gitlab-registry.example.org/v2/", Err:(*net.OpError)(0x140005b8410)})
DEBU[0000] GET https://gitlab-registry.example.org/v1/_ping
DEBU[0001] Ping https://gitlab-registry.example.org/v1/_ping err Get "https://gitlab-registry.example.org/v1/_ping": dial tcp: lookup gitlab-registry.example.org on [2a02:1748:fad4:5021::1]:53: no such host (&url.Error{Op:"Get", URL:"https://gitlab-registry.example.org/v1/_ping", Err:(*net.OpError)(0x140005b8870)})
Existing credentials are invalid, please enter valid username and password

Describe the results you expected:

I would expect login to find the registry. I believe this worked in the past with a manually built podman 3.4.2-dev and the CGO makefile patch. But hard to tell now.

Additional information you deem important (e.g. issue happens only occasionally):

pull works:

podman  --log-level debug pull gitlab-registry.example.org/buildbox:master
INFO[0000] podman filtering at log level debug
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level debug pull gitlab-registry.example.org/buildbox:master)
DEBU[0000] SSH Ident Key "/Users/ch/.ssh/podman-machine-default" SHA256:XXXXXXXXXXXXXXXXXX ssh-ed25519
DEBU[0000] Found SSH_AUTH_SOCK "/private/tmp/com.apple.launchd.3Em3rNCi8h/Listeners", ssh-agent signer(s) enabled
DEBU[0000] SSH Agent Key SHA256:XXXXXXXXXXXXXXXXXX ssh-ed25519
DEBU[0000] DoRequest Method: GET URI: http://d/v3.4.4/libpod/_ping
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Found credentials for gitlab-registry.example.org in credential helper containers-auth.json in file /Users/ch/.config/containers/auth.json
DEBU[0000] DoRequest Method: POST URI: http://d/v3.4.4/libpod/images/pull
Trying to pull gitlab-registry.example.org/buildbox:master...
Getting image source signatures
Copying blob sha256:495e42c822a03224fc27a04cf05716b82a5f1c4c9252cc7457673be355ab12f2
Copying blob sha256:a2abf6c4d29d43a4bf9fbb769f524d0fb36a2edab49819c1bf3e76f409f953ea
Copying blob sha256:e1769f49f9101e08b259481961baceea032c026337b960a2737ba58d84c08d5f
...

Output of podman version:

podman version
Client:
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.3
Built:        Wed Dec  8 19:41:11 2021
OS/Arch:      darwin/arm64

Server:
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.16.8
Built:        Wed Dec  8 22:48:10 2021
OS/Arch:      linux/arm64

Output of podman info --debug:

host:
  arch: arm64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.30-2.fc35.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 1
  distribution:
    distribution: fedora
    variant: coreos
    version: "35"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.10-200.fc35.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 815165440
  memTotal: 2048368640
  ociRuntime:
    name: crun
    package: crun-1.4-1.fc35.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.4
      commit: 3daded072ef008ef0840e8eccb0b52a7efbd165d
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.aarch64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 13h 6m 21.08s (Approximately 0.54 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1639000090
  BuiltTime: Wed Dec  8 21:48:10 2021
  GitCommit: ""
  GoVersion: go1.16.8
  OsArch: linux/arm64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):

brew info podman
podman: stable 3.4.4 (bottled), HEAD
Tool for managing OCI containers and pods
https://podman.io/
/opt/homebrew/Cellar/podman/3.4.4 (170 files, 41.0MB) *
  Poured from bottle on 2022-01-21 at 23:57:54
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/podman.rb
License: Apache-2.0
==> Dependencies
Build: go ✔, go-md2man ✔
Required: qemu ✔
==> Options
--HEAD
	Install HEAD version
==> Caveats
zsh completions have been installed to:
  /opt/homebrew/share/zsh/site-functions
==> Analytics
install: 9,885 (30 days), 35,309 (90 days), 85,030 (365 days)
install-on-request: 9,882 (30 days), 35,316 (90 days), 85,022 (365 days)
build-error: 0 (30 days)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Darwin tiq 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:29:10 PST 2021; root:xnu-8019.61.5~1/RELEASE_ARM64_T8101 arm64

VPN is Cisco AnyConnect.

[vrothberg]

Thank you for reaching out, @zeha!

@ashley-cui @jwhonce PTAL

[Luap99]

Is podman build with cgo enabled in brew?

[zeha]

As far as I know: yes, and also the podman Makefile tries to ensure this.
If there are specific commands to check if a given binary is built with cgo, I can check on my install.

[zeha]

Note that previously, podman pull would also fail with a DNS error.

[Luap99]

I am not sure if you can check this with the binary, maybe ldd /path/to/podman and check if there are linked libs.
Otherwise you could build from source and check that CGO_ENABLED is set to 1 when you call make podman-remote-darwin

[zeha]

Right, the Makefile does not properly set CGO_ENABLED.

Makefile lines 176-183 (in podman 3.4.4 tarball) try to set DARWIN_GCO=1 for native builds. However, this check relies on NATIVE_GOOS being set. I cannot find where this variable should be set from - the Makefile does not seem to set it.

Homebrew also exports CGO_ENABLED=1 in its formula, but that gets overwritten by the Makefile.

I do not understand what NATIVE_GOOS is/should be, but I think fixing this will help.

[zeha]

Okay, that came as part of my backport from the fix on main. Not sure how that worked locally last time. I'll see about making a new PR then, 3.4-specific.

[zeha]

-> #13020

[zeha]

Got merged, should be fixed in the next 3.4 release.