No connection could be made because the target machine actively refused it

[bwgjoseph]

Bug description

note: realize this could be the same issue as containers/podman-desktop#1173, but I have tried to provide more information and am not sure if it's really the same. Ok to close as duplicate if deem so.

---

I have saved an image from the internet machine and bring into an air-gapped machine to load it in.

I encounter the following error

Error: failed to connect: dial tcp [::1]:49675: connectex: No connection could be made because the target machine actively refused it.

Operating system

Windows 10

Version

0.12.0

Steps to reproduce

// save image on internet machine
docker save busybox:latest > busybox.tar

// load image on air-gapped machine
podman load < busybox.tar

Docker was installed but not running. I did start Docker afterwards, but the same error occur

Initially, I was thinking if this has something to do with DOCKER_HOST, so I followed the guide but when I ran the command

podman machine inspect

The output

[
     {
          "ConfigPath": {
               "Path": "C:\\Users\\bwgjoseph\\.config\\containers\\podman\\machine\\wsl\\podman-machine-default.json"
          },
          "ConnectionInfo": {
               "PodmanSocket": null
          },
          "Created": "2023-02-18T11:25:40.0285425-08:00",
          "Image": {
               "IgnitionFilePath": {
                    "Path": ""
               },
               "ImageStream": "custom",
               "ImagePath": {
                    "Path": "C:\\Users\\bwgjoseph\\.local\\share\\containers\\podman\\machine\\wsl\\podman-image-x64.tar.xz"
               }
          },
          "LastUp": "2023-02-18T13:43:18.4028289-08:00",
          "Name": "podman-machine-default",
          "Resources": {
               "CPUs": 2,
               "DiskSize": 550502400,
               "Memory": 325951488
          },
          "SSHConfig": {
               "IdentityPath": "C:\\Users\\bwgjoseph\\.ssh\\podman-machine-default",
               "Port": 49675,
               "RemoteUsername": "user"
          },
          "State": "running"
     }
]

Relevant log output

> podman system connection list


Name                         URI                                                          Identity                                        Default
podman-machine-default       ssh://user@localhost:49675/run/user/1000/podman/podman.sock  C:\Users\bwgjoseph\.ssh\podman-machine-default  true
podman-machine-default-root  ssh://root@localhost:49675/run/podman/podman.sock            C:\Users\bwgjoseph\.ssh\podman-machine-default  false

podman machine info

Host:
  Arch: amd64
  CurrentMachine: podman-machine-default
  DefaultMachine: podman-machine-default
  EventsDir: C:\Users\bwgjoseph\.local\share\containers\podman\podman
  MachineConfigDir: C:\Users\bwgjoseph\.config\containers\podman\machine\wsl
  MachineImageDir: C:\Users\bwgjoseph\.local\share\containers\podman\machine\wsl
  MachineState: Running
  NumberOfMachines: 1
  OS: windows
  VMType: wsl
Version:
  APIVersion: 4.4.1
  Built: 1675890486
  BuiltTime: Wed Feb  8 13:08:06 2023
  GitCommit: 34e8f3933242f2e566bbbbf343cf69b7d506c1cf
  GoVersion: go1.19.5
  Os: windows
  OsArch: windows/amd64
  Version: 4.4.1

Additional context

• It looks a little related to

  • [Bug]: 4.4.0 on Windows WSL now requires explicit -c <vmname> to perform podman ps etc #17373
  • Podman Windows, Git Bash: connection actively refused #15975
  • Error on inspect podman-desktop#1297
  • "No connection could be made because the target machine actively refused it." after a fresh install on Windows 11 podman-desktop#1173

• I also tried rebooted my machine. This is running on a Virtual Machine if this helps

[benoitf]

cc @lstocchi @jeffmaury do you have faced this issue on Windows ?

@bwgjoseph are you using a firewall or connected to a VPN ?

[jeffmaury]

cc @lstocchi @jeffmaury do you have faced this issue on Windows ?

@bwgjoseph are you using a firewall or connected to a VPN ?

Yes generally happens when podman machine state and wsl are not in sync.
Step to reproduce:

1. podman machine start
2. podman version ==> OK
3. wsl --shutdown
4. podman version ==> error message

[benoitf]

and podman machine inspect command ? It reports running I guess

[bwgjoseph]

cc @lstocchi @jeffmaury do you have faced this issue on Windows ?

@bwgjoseph are you using a firewall or connected to a VPN ?

Don't think there's firewall but I can double check. No VPN.

[bwgjoseph]

cc @lstocchi @jeffmaury do you have faced this issue on Windows ?

@bwgjoseph are you using a firewall or connected to a VPN ?

Yes generally happens when podman machine state and wsl are not in sync.
Step to reproduce:

1. podman machine start
2. podman version ==> OK
3. wsl --shutdown
4. podman version ==> error message

Not sure what does not in sync means. Is there something I can run to check?

I'm pretty sure my wsl is running when I'm testing on podman.

Anything else for me to check?

[jeffmaury]

After I run wsl --shutdown, podman machine inspect reports:

$ podman machine inspect
[
     {
          "ConfigPath": {
               "Path": "C:\\Users\\Jeff\\.config\\containers\\podman\\machine\\wsl\\podman-machine-default.json"
          },
          "ConnectionInfo": {
               "PodmanSocket": null
          },
          "Created": "2023-02-17T08:16:04.8648643+01:00",
          "Image": {
               "IgnitionFilePath": {
                    "Path": ""
               },
               "ImageStream": "35",
               "ImagePath": {
                    "Path": "C:\\Users\\Jeff\\.local\\share\\containers\\podman\\machine\\wsl\\podman-machine-default_fedora-podman-amd64-v36.0.127.tar"
               }
          },
          "LastUp": "2023-02-21T16:25:26.035611+01:00",
          "Name": "podman-machine-default",
          "Resources": {
               "CPUs": 0,
               "DiskSize": 3591372800,
               "Memory": 0
          },
          "SSHConfig": {
               "IdentityPath": "C:\\Users\\Jeff\\.ssh\\podman-machine-default",
               "Port": 54249,
               "RemoteUsername": "user"
          },
          "State": "stopped"
     }
]

[jeffmaury]

cc @lstocchi @jeffmaury do you have faced this issue on Windows ?
@bwgjoseph are you using a firewall or connected to a VPN ?

Yes generally happens when podman machine state and wsl are not in sync.
Step to reproduce:

1. podman machine start
2. podman version ==> OK
3. wsl --shutdown
4. podman version ==> error message

Not sure what does not in sync means. Is there something I can run to check?

I'm pretty sure my wsl is running when I'm testing on podman.

Anything else for me to check?

Yes when you're in this start, try to enter the wsl machine by running wsl if you have a single wsl machine. But as that may restart the wsl machine you need to check if it correctly started so run ps -ef to get the list of processed and you should see at least sshd processes.

[bwgjoseph]

cc @lstocchi @jeffmaury do you have faced this issue on Windows ?
@bwgjoseph are you using a firewall or connected to a VPN ?

Yes generally happens when podman machine state and wsl are not in sync.
Step to reproduce:

1. podman machine start
2. podman version ==> OK
3. wsl --shutdown
4. podman version ==> error message

Not sure what does not in sync means. Is there something I can run to check?
I'm pretty sure my wsl is running when I'm testing on podman.
Anything else for me to check?

Yes when you're in this start, try to enter the wsl machine by running wsl if you have a single wsl machine. But as that may restart the wsl machine you need to check if it correctly started so run ps -ef to get the list of processed and you should see at least sshd processes.

This is windows machine, so there isn't ps -ef command

Anyway, my wsl is alive, but the connection is still null

[benoitf]

could you list all wsl instances ? AFAIK podman brings a fedora wsl instance and here we're seeing Ubuntu so I guess there is something that happened

[bwgjoseph]

I don't see any fedora instance

I have ubuntu installed previously

[jeffmaury]

I don't see any fedora instance

I have ubuntu installed previously

the podman-machine-default should be fedora based

[jeffmaury]

cc @lstocchi @jeffmaury do you have faced this issue on Windows ?
@bwgjoseph are you using a firewall or connected to a VPN ?

Yes generally happens when podman machine state and wsl are not in sync.
Step to reproduce:

1. podman machine start
2. podman version ==> OK
3. wsl --shutdown
4. podman version ==> error message

Not sure what does not in sync means. Is there something I can run to check?
I'm pretty sure my wsl is running when I'm testing on podman.
Anything else for me to check?

Yes when you're in this start, try to enter the wsl machine by running wsl if you have a single wsl machine. But as that may restart the wsl machine you need to check if it correctly started so run ps -ef to get the list of processed and you should see at least sshd processes.

This is windows machine, so there isn't ps -ef command

Anyway, my wsl is alive, but the connection is still null

The ps -ef is to run inside the wsl machine. So list of instructions is:

1. wsl -d podman-machine-default
2. ps -ef

[bwgjoseph]

Alright. It is now the default. I rebooted my machine after I set podman-machine-default as the default wsl instance

Not sure if relevant but podman machine inspect still return null

This is the output for ps -ef in podman-machine-default

[user@DESKTOP-8A256M6 ~]$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 22:02 ?        00:00:00 /lib/systemd/systemd
root        18     1  0 22:02 ?        00:00:00 /usr/lib/systemd/systemd-journald
root        25     1  0 22:02 ?        00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
dbus        28     1  0 22:02 ?        00:00:00 /usr/bin/dbus-broker-launch --scope system --audit
root        29     1  0 22:02 ?        00:00:00 /usr/lib/systemd/systemd-userdbd
root        30    29  0 22:02 ?        00:00:00 systemd-userwork
root        31    29  0 22:02 ?        00:00:00 systemd-userwork
root        32    29  0 22:02 ?        00:00:00 systemd-userwork
dbus        33    28  0 22:02 ?        00:00:00 dbus-broker --log 4 --controller 9 --machine-id 43c24020ff5e49838ebd4554
user        35     1  0 22:02 ?        00:00:00 /usr/lib/systemd/systemd --user
user        37    35  0 22:02 ?        00:00:00 (sd-pam)
user        44    35  0 22:02 ?        00:00:00 /usr/bin/sleep infinity
root        45     1  2 22:04 ?        00:00:00 /usr/lib/systemd/systemd-hostnamed
root        46     0  0 22:04 pts/0    00:00:00 su -l user
root        51     1  1 22:04 ?        00:00:00 /usr/lib/systemd/systemd-logind
user        52    46  0 22:04 pts/0    00:00:00 -bash
user        70    52  0 22:04 pts/0    00:00:00 ps -ef

[jeffmaury]

Alright. It is now the default. I rebooted my machine after I set podman-machine-default as the default wsl instance

Not sure if relevant but podman machine inspect still return null

This is the output for ps -ef in podman-machine-default

[user@DESKTOP-8A256M6 ~]$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 22:02 ?        00:00:00 /lib/systemd/systemd
root        18     1  0 22:02 ?        00:00:00 /usr/lib/systemd/systemd-journald
root        25     1  0 22:02 ?        00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
dbus        28     1  0 22:02 ?        00:00:00 /usr/bin/dbus-broker-launch --scope system --audit
root        29     1  0 22:02 ?        00:00:00 /usr/lib/systemd/systemd-userdbd
root        30    29  0 22:02 ?        00:00:00 systemd-userwork
root        31    29  0 22:02 ?        00:00:00 systemd-userwork
root        32    29  0 22:02 ?        00:00:00 systemd-userwork
dbus        33    28  0 22:02 ?        00:00:00 dbus-broker --log 4 --controller 9 --machine-id 43c24020ff5e49838ebd4554
user        35     1  0 22:02 ?        00:00:00 /usr/lib/systemd/systemd --user
user        37    35  0 22:02 ?        00:00:00 (sd-pam)
user        44    35  0 22:02 ?        00:00:00 /usr/bin/sleep infinity
root        45     1  2 22:04 ?        00:00:00 /usr/lib/systemd/systemd-hostnamed
root        46     0  0 22:04 pts/0    00:00:00 su -l user
root        51     1  1 22:04 ?        00:00:00 /usr/lib/systemd/systemd-logind
user        52    46  0 22:04 pts/0    00:00:00 -bash
user        70    52  0 22:04 pts/0    00:00:00 ps -ef

Seems your WSL instance is fine now. What's about podman ps ?

[bwgjoseph]

Forgot about that. It's the same error though.

[jeffmaury]

Forgot about that. It's the same error though.

Seems the SSH daemon is not initialized correctly. Can you run journalctl --system --unit sshd in the WSL machine ?

[bwgjoseph]

Here's the log. I tried to run podman machine inspect, and restarting podman but it's still the same issue.

Feb 18 11:34:38 DESKTOP-8A256M6 systemd[1]: Starting sshd.service - OpenSSH server daemon...
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[65]: Server listening on 0.0.0.0 port 49675.
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[65]: Server listening on :: port 49675.
Feb 18 11:34:38 DESKTOP-8A256M6 systemd[1]: Started sshd.service - OpenSSH server daemon.
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[68]: PAM unable to dlopen(/usr/lib64/security/pam_sss.so):
/usr/lib64/security/pam_sss.so: cannot open shared object file: No such file or directory
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[68]: PAM adding faulty module: /usr/lib64/security/pam_sss.
so
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[67]: PAM unable to dlopen(/usr/lib64/security/pam_sss.so):
/usr/lib64/security/pam_sss.so: cannot open shared object file: No such file or directory
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[67]: PAM adding faulty module: /usr/lib64/security/pam_sss.
so
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[67]: Accepted publickey for user from ::1 port 34468 ssh2: ED25519 SHA256:RzJAGMQVvbKMy3mY5kKOywth1IauyfpdcYvK0U1iXrE
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[68]: Accepted publickey for user from ::1 port 34470 ssh2: ED25519 SHA256:RzJAGMQVvbKMy3mY5kKOywth1IauyfpdcYvK0U1iXrE
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[68]: pam_unix(sshd:session): session opened for user user(uid=1000) by (uid=0)
Feb 18 11:34:38 DESKTOP-8A256M6 sshd[67]: pam_unix(sshd:session): session opened for user user(uid=1000) by (uid=0)
-- Boot d7483dd70b9f4e78864dc69ddd909545 --
Feb 18 12:31:16 DESKTOP-8A256M6 systemd[1]: Starting sshd.service - OpenSSH server daemon...
Feb 18 12:31:17 DESKTOP-8A256M6 sshd[25]: Server listening on 0.0.0.0 port 49675.
Feb 18 12:31:17 DESKTOP-8A256M6 sshd[25]: Server listening on :: port 49675.
Feb 18 12:31:17 DESKTOP-8A256M6 systemd[1]: Started sshd.service - OpenSSH server daemon.
-- Boot 6ca7a04f859a45e7bb6548ba432f8f50 --
Feb 18 13:47:20 DESKTOP-8A256M6 systemd[1]: Starting sshd.service - OpenSSH server daemon...
Feb 18 13:47:20 DESKTOP-8A256M6 sshd[25]: Server listening on 0.0.0.0 port 49675.
Feb 18 13:47:20 DESKTOP-8A256M6 sshd[25]: Server listening on :: port 49675.
Feb 18 13:47:20 DESKTOP-8A256M6 systemd[1]: Started sshd.service - OpenSSH server daemon.
-- Boot 1fd421016f944fce9172a3e2f63ba5c3 --
Feb 22 00:12:56 DESKTOP-8A256M6 systemd[1]: Starting sshd.service - OpenSSH server daemon...
Feb 22 00:12:57 DESKTOP-8A256M6 sshd[25]: Server listening on 0.0.0.0 port 49675.
Feb 22 00:12:57 DESKTOP-8A256M6 sshd[25]: Server listening on :: port 49675.
Feb 22 00:12:57 DESKTOP-8A256M6 systemd[1]: Started sshd.service - OpenSSH server daemon.
-- Boot fa78c5db2725454aa10ccfcf3e59f047 --
Feb 22 21:39:13 DESKTOP-8A256M6 systemd[1]: Starting sshd.service - OpenSSH server daemon...
Feb 22 21:39:14 DESKTOP-8A256M6 sshd[25]: Server listening on 0.0.0.0 port 49675.
Feb 22 21:39:14 DESKTOP-8A256M6 sshd[25]: Server listening on :: port 49675.
Feb 22 21:39:14 DESKTOP-8A256M6 systemd[1]: Started sshd.service - OpenSSH server daemon.
-- Boot 366a048caec7416abfb4e69a1c430e33 --
Feb 22 21:58:29 DESKTOP-8A256M6 systemd[1]: Starting sshd.service - OpenSSH server daemon...
Feb 22 21:58:30 DESKTOP-8A256M6 sshd[25]: Server listening on 0.0.0.0 port 49675.
Feb 22 21:58:30 DESKTOP-8A256M6 sshd[25]: Server listening on :: port 49675.
Feb 22 21:58:30 DESKTOP-8A256M6 systemd[1]: Started sshd.service - OpenSSH server daemon.
-- Boot bfe7f9e258024e59bc5f42d940ac0f9d --
Feb 22 22:02:31 DESKTOP-8A256M6 systemd[1]: Starting sshd.service - OpenSSH server daemon...
Feb 22 22:02:31 DESKTOP-8A256M6 sshd[25]: Server listening on 0.0.0.0 port 49675.
Feb 22 22:02:31 DESKTOP-8A256M6 systemd[1]: Started sshd.service - OpenSSH server daemon.
Feb 22 22:02:31 DESKTOP-8A256M6 sshd[25]: Server listening on :: port 49675.
Feb 22 22:36:44 DESKTOP-8A256M6 sshd[25]: Received signal 15; terminating.
Feb 22 22:36:44 DESKTOP-8A256M6 systemd[1]: Stopping sshd.service - OpenSSH server daemon...
Feb 22 22:36:44 DESKTOP-8A256M6 systemd[1]: sshd.service: Deactivated successfully.
Feb 22 22:36:44 DESKTOP-8A256M6 systemd[1]: Stopped sshd.service - OpenSSH server daemon.
Feb 22 22:36:53 DESKTOP-8A256M6 systemd[1]: Starting sshd.service - OpenSSH server daemon...
Feb 22 22:36:53 DESKTOP-8A256M6 sshd[25]: Server listening on 0.0.0.0 port 49675.
Feb 22 22:36:53 DESKTOP-8A256M6 sshd[25]: Server listening on :: port 49675.
Feb 22 22:36:53 DESKTOP-8A256M6 systemd[1]: Started sshd.service - OpenSSH server daemon.

ps -ef after running the above command

UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 22:36 ?        00:00:00 /lib/systemd/systemd
root        18     1  0 22:36 ?        00:00:00 /usr/lib/systemd/systemd-journald
root        25     1  0 22:36 ?        00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
dbus        28     1  0 22:36 ?        00:00:00 /usr/bin/dbus-broker-launch --scope system --audit
root        29     1  0 22:36 ?        00:00:00 /usr/lib/systemd/systemd-userdbd
dbus        30    28  0 22:36 ?        00:00:00 dbus-broker --log 4 --controller 9 --machine-id 43c24020ff5e49838ebd4554ff633d1d --max-bytes 536870912 --max-fds 4096 --max-matches 16384 --audit
root        31    29  0 22:36 ?        00:00:00 systemd-userwork
root        32    29  0 22:36 ?        00:00:00 systemd-userwork
root        33    29  0 22:36 ?        00:00:00 systemd-userwork
user        36     1  0 22:36 ?        00:00:00 /usr/lib/systemd/systemd --user
user        37    36  0 22:36 ?        00:00:00 (sd-pam)
user        44    36  0 22:36 ?        00:00:00 /usr/bin/sleep infinity
root        46     0  0 22:37 pts/1    00:00:00 su -l user
root        51     1  0 22:37 ?        00:00:00 /usr/lib/systemd/systemd-logind
user        52    46  0 22:37 pts/1    00:00:00 -bash
user        74    52  0 22:38 pts/1    00:00:00 ps -ef

[jeffmaury]

Seems the sshd service stopped around 22:36:44 so maybe checking why it stopped

[bwgjoseph]

Sorry, I have no idea. I'm not familiar with this, but if there's something for me to run, I can do that.

If it helps, I am running within a VM. Pretty much a fresh VM with WSL and podman installed only.

[mvromer]

For what it's worth, I've been running into this as well from time to time on my Windows 10 machine running WSL 2. Like others, occasionally it seems to resolve itself, but today I haven't been able to figure out how to coax my podman machine back to a working state. That being said, this is what I'm seeing on my end. Not sure if it helps to find a lasting solution or not, but maybe it can help narrow out some other debugging paths.

In my case, a podman system connection list shows that podman is listening for ssh connections on port 49312. I can enter into my machine using wsl -d podman-machine-default and see that sshd is, in fact, listening on that port. I can also run journalctl and see that the sshd service is still running normally.

On the Windows side, I run TCPView and can't see anything listening on port 49312. That's enough to explain the connection refused errors I get from any client on the Windows side. It's almost like WSL isn't exposing to Windows the ports services running inside the Linux distributions are listening on.

Interestingly enough, when I launch my Ubuntu WSL distro with wsl -d Ubuntu, I can run ss -l -t and see the service listening on port 49312. If I try to run ssh user@localhost -p 49312, it does try to connect to the sshd running in my podman machine. I can even run journalctl from the podman machine and see the connection attempt being made.

On top of that, if I start netcat inside my Ubuntu machine using nc -l 8787, I do see it show up in the list of listening ports inside TCPView, which says the wslhost.exe process is doing the listening. I can run the Windows distro of ncat from the NMap 7.92 distro, connect using ncat 127.0.0.1 8787, and successfully echo commands from the Windows side to the Ubuntu side.

At this point I'm not sure why some ports get exposed through WSL vs. not. I'm continuing to try things on my end, and I'd be happy to try to collect any debug output I can on my end. Hopefully though that this helps narrow down where to look for the underlying fault.

[isaacsgi]

I am having this same issue.

Have been a longtime Docker user, but now working for a new firm, they asked that I switch over to Podman.

Note, this is a NEW PC: Windows 11 Pro; Version22H2; OS Build: 22621.1265

I had Podman working fine, shutdown and restarted machine after some of the latest Windows Updates and now can't get Podman to work.

Please let me know what logs or diagnostics I can provide to help resolve.

[jeffmaury]

What is the output of podman system connection ls and podman machine ls

[isaacsgi]

PS C:\Users\GilIsaacs> podman system connection ls
Name                         URI                                                          Identity
                  Default
podman-machine-default       ssh://user@localhost:63320/run/user/1000/podman/podman.sock  C:\Users\GilIsaacs\.ssh\podman-machine-default  false
podman-machine-default-root  ssh://root@localhost:63320/run/podman/podman.sock            C:\Users\GilIsaacs\.ssh\podman-machine-default  true
PS C:\Users\GilIsaacs> podman machine ls
NAME                    VM TYPE     CREATED       LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default  wsl         25 hours ago  Currently running  20          914.7MB     2.294GB

[jeffmaury]

So what kind of error do you have. What happen when you run podman ps or podman images

[isaacsgi]

PS C:\Users\GilIsaacs> podman info
Error: failed to connect: dial tcp [::1]:63320: connectex: No connection could be made because the target machine actively refused it.
PS C:\Users\GilIsaacs> podman ps
Error: failed to connect: dial tcp [::1]:63320: connectex: No connection could be made because the target machine actively refused it.

[isaacsgi]

Seems that my base Windows terminal doesn't connect to the machine, but when I go into the running machine it can run?

PS C:\Users\GilIsaacs> podman run hello-world
Error: failed to connect: dial tcp [::1]:63320: connectex: No connection could be made because the target machine actively refused it.
PS C:\Users\GilIsaacs> wsl -d podman-machine-default

You will be automatically entered into a nested process namespace where
systemd is running. If you need to access the parent namespace, hit ctrl-d
or type exit. This also means to log out you need to exit twice.

[user@gilisaacs ~]$ podman run hello-world
!... Hello Podman World ...!

         .--"--.
       / -     - \
      / (O)   (O) \
   ~~~| -=(,Y,)=- |
    .---. /`  \   |~~
 ~/  o  o \~~~~.----. ~~
  | =(X)= |~  / (O (O) \
   ~~~~~~~  ~| =(Y_)=-  |
  ~~~~    ~~~|   U      |~~

Project:   https://github.com/containers/podman
Website:   https://podman.io
Documents: https://docs.podman.io
Twitter:   @Podman_io
[user@gilisaacs ~]$ podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
[user@gilisaacs ~]$ podman images
REPOSITORY                               TAG         IMAGE ID      CREATED       SIZE
quay.io/podman/hello                     latest      e66ef1031ffc  44 hours ago  82.3 kB
registry.access.redhat.com/ubi8-micro    latest      89765c93bd06  3 weeks ago   28.5 MB

[jeffmaury]

From the Windows shell can you try:

• podman machine stop
• podman machine start
• podman ps

[bwgjoseph]

Hi, was wondering if there's anything I can try with my issue. Thanks

[isaacsgi]

PS C:\Users\GilIsaacs> podman machine stop
Machine "podman-machine-default" stopped successfully

PS C:\Users\GilIsaacs> podman machine start
Starting machine "podman-machine-default"
API forwarding listening on: npipe:////./pipe/docker_engine

Docker API clients default to this address. You do not need to set DOCKER_HOST.
Machine "podman-machine-default" started successfully
PS C:\Users\GilIsaacs> podman ps
Error: failed to connect: dial tcp [::1]:63320: connectex: No connection could be made because the target machine actively refused it.
PS C:\Users\GilIsaacs>

[jeffmaury]

Can you run netstat -n -q | find "LISTENING"

[n1hility]

@tppalani thanks. The error you pasted from Get-NetTCPConnection indicates a problem with WSL on your system, where the WSL loopback forwarding facility is not functioning correctly (it's responsible for mirroring ports that are bound on Linux guests to Windows processes connecting to localhost, the ss output and other commands confirms we have ssh listening, and the Get-NETTCPConnection confirms the port is not mirrored) . If you haven't already I would recommend a full reboot of your system. If the problem persists, I would recommend doing an update to WSL on your system (via wsl --update).

[tppalani]

H @n1hility

I have re-configured podman Now I can able to see all things working as we expected, And also i pulled the image from CLI as well podman desktop GUI both are working fine.

podman --version
podman.exe version 4.5.1

podman system connection ls
Name                         URI                                                          Identity                                      Default
podman-machine-default       ssh://[email protected]:54117/run/user/1000/podman/podman.sock  C:\Users\palani\.ssh\podman-machine-default  false
podman-machine-default-root  ssh://[email protected]:54117/run/podman/podman.sock            C:\Users\palani\.ssh\podman-machine-default  true

Get-NetTCPConnection -LocalPort 54117 -State Listen

LocalAddress                        LocalPort RemoteAddress                       RemotePort State       AppliedSetting OwningProcess
------------                        --------- -------------                       ---------- -----       -------------- -------------
::1                                      54117     ::                                  0          Listen                     8888
127.0.0.1                           54117     0.0.0.0                             0          Listen                     8888

Just one doubt if suppose in future if I'm getting any connection refused state what would be solution how we can handle it. Any how podman will always run until we can stop it explicitly, because using podman i configured kind cluster for my local development. If again i faced similar kind connection issue then i need to re-create my cluster that will make re-work and we will miss out lot of configuration as well. Please share your advise.

[n1hility]

Glad to hear it's working for you now @tppalani. Connection issues don't normally require a rebuild of the podman machine, but there was that bug I mentioned which occurred before 4.5.0 which would have impacted machine instances created after our image stream switched over to F37 (happened earlier this year). While you don't normally have to rebuild/replace you machine instance, they aren't expected to last forever. They are currently tied to Fedora releases, so follow a similar lifespan. Further you may need to update the image to gain access to new features and bug fixes in updated client releases, when the functionality is contingent upon some element in the image. Whenever you have the need to replace/refresh your image, you have a few options on Windows.

1. You can have multiple podman machines, so you could switch between them as needed if you want to avoid removing the state in your current in use machine
2. You can use podman save/export to backup container or volume state to recreate. Although I would take some care when doing this for containers you don't manage (e.g. Kind)
3. You could backup/export your kube artifacts over kind directly, such that kind can be reset from initial state
4. Alternatively, or in combination with 3 you can use a GitOps style approach, and record your kube provisioning actions in a local Git repo up front, so that they can be easily be replayed. Could be useful when moving from your local system to a shared integration env.

[tppalani]

Hi @n1hility , Sorry for the trouble, Again I noted this issue is repeating, even i didn't rebooted my machine.
Still podman is running but when checking pod not able list it. and also using powershell not able to get the connection list.

Trying to understand why and when we ill get this error like connection refused, which connection its actually referring it?

Get-NetTCPConnection -LocalPort 54117 -State Listen
Get-NetTCPConnection : No matching MSFT_NetTCPConnection objects found by CIM query for instances of the
ROOT/StandardCimv2/MSFT_NetTCPConnection class on the  CIM server: SELECT * FROM MSFT_NetTCPConnection  WHERE ((LocalPort = 54117)) AND
((State = 2)). Verify query parameters and retry.
At line:1 char:1
+ Get-NetTCPConnection -LocalPort 54117 -State Listen
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (MSFT_NetTCPConnection:String) [Get-NetTCPConnection], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetTCPConnection


$ wsl -l -v
  NAME                      STATE           VERSION
* podman-machine-default    Running         2
  wsl-vpnkit                Running         2

podman system connection ls
Name                         URI                                                          Identity                                      Default
podman-machine-default       ssh://[email protected]:54117/run/user/1000/podman/podman.sock  C:\Users\palani\.ssh\podman-machine-default  false
podman-machine-default-root  ssh://[email protected]:54117/run/podman/podman.sock            C:\Users\palani\.ssh\podman-machine-default  true

[n1hility]

@tppalani np, so the reason you see this ENOENT on \.\pipe\podman-machine-default is that there is an ssh relay process that forwards traffic on that Windows Named Pipe address (where docker API clients expect to communicate). However that relay can not be established when the wsl instance itself is not network reachable, so the machine instance is running in a substandard operating mode. The only thing you can do at that point is either reboot, or you could try stopping all wsl instances and then running a wsl --shutdown, then starting, but the latter doesnt fully reset all aspects of WSL. Did you try updating your WSL version to the latest (wsl --update)?

[tppalani]

Hi @n1hility Thanks for helping past two days, I'm not sure my power shell not showing any output about this wsl --version command, instead of its displaying all the option (like man page command option).

But what is the recommended version we need use for podman?

[n1hility]

@tppalani So when that happens that means you have the previous generation of WSL that was a fixed / static component of the OS. In the older generation wsl --update will at least update the Linux kernel in use, so you could try that first, and instead of wsl --version, you can use wsl --status on that one to verify the version. Microsoft recently moved to a model where there is an independent update stream of WSL, allowing all recent variants and versions of Windows to stay current. However you need to be on 21H1, 21H2, or 22H2 of Windows 10, or 21H1 on Windows 11. If you run Windows update it should present you with the option to install one of these. Once that is done (and you have rebooted), wsl --install should install the current stream.

[tppalani]

Hi @n1hility here the output of wsl --status

Default Distribution: podman-machine-default
Default Version: 2

Windows Subsystem for Linux was last updated on 4/21/2023
Kernel version: 5.10.16

[jaxor24]

I ran into this same issue after a Windows 10 update this week. Moving from 4.5.0 to 4.5.1 then running the following fixed it:
podman machine rm
podman machine init
podman machine start

[tppalani]

Hi @n1hility

Can you please help me about this

 wsl -l -v
  NAME                                   STATE           VERSION
* podman-machine-default    Running         2
  wsl-vpnkit                              Running         2


podman system connection ls
Name                         URI                                                          Identity                                      Default
podman-machine-default       ssh://[email protected]:64117/run/user/1000/podman/podman.sock  C:\Users\user\.ssh\podman-machine-default  false
podman-machine-default-root  ssh://[email protected]:64117/run/podman/podman.sock            C:\Users\user\.ssh\podman-machine-default  true


Get-NetTCPConnection -LocalPort 64117 -State Listen

LocalAddress                        LocalPort RemoteAddress                       RemotePort State       AppliedSetting OwningProcess
------------                        --------- -------------                       ---------- -----       -------------- -------------
::1                                      64117     ::                                  0          Listen                     40484
127.0.0.1                           64117     0.0.0.0                             0          Listen                     40484

when I'm doing kubectl get pod i'm getting error connection refused with different port number

$ kubectl get pod
E0606 22:02:52.983865   40012 memcache.go:238] couldn't get current server API group list: Get "https://127.0.0.1:51916/api?timeout=32s": dial tcp 127.0.0.1:51916: connectex: No connection could be made because the target machine actively refused it

[TroySchmidt]

I have tried everything I can. Uninstalled, reinstalled, uninstalled WSL and let Podman reinstall that. The connection refused error is still sporadic even with the latest podman install 4.5.1. I got the podman info to connect for me before I did the complete uninstall / reinstall this last time, but port forwarding to sql server container was still failing to connect.

So this last time I uninstalled WSL and now back to the failed to connect connectex errors.

[techert]

I have the same issue. It worked fine until Windows had crashed and restarted.
Windows Version 22H2 Build 19045.3086

podman.exe version 4.5.1
Machine had been created with this version.

wsl -l -v
NAME STATE VERSION

• podman-machine-default Running 2
  Ubuntu-20.04 Stopped 2
  Ubuntu-18.04 Stopped 1

podman system connection ls
Name URI Identity Default
podman-machine-default ssh://[email protected]:11627/run/user/1000/podman/podman.sock C:\Users\me.ssh\podman-machine-default true
podman-machine-default-root ssh://[email protected]:11627/run/podman/podman.sock C:\Users\me.ssh\podman-machine-default false

Get-NetTCPConnection -LocalPort 11627 -State Listen
Get-NetTCPConnection : No matching MSFT_NetTCPConnection objects found by CIM query for instances of the ROOT/StandardCimv2/MSFT_NetTCPConnection class on the CIM server: SELECT * FROM MSFT_NetTCPConnection WHERE ((LocalPort = 11627)) AND ((State =
2)). Verify query parameters and retry.
At line:1 char:1

• Get-NetTCPConnection -LocalPort 11627 -State Listen

•   + CategoryInfo          : ObjectNotFound: (MSFT_NetTCPConnection:String) [Get-NetTCPConnection], CimJobException
    + FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetTCPConnection
  
  
  

Windows event log shows:
Error listening on socket: npipe:////./pipe/docker_engine: open \.\pipe\docker_engine: Access is denied.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name=".NET Runtime" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2023-06-20T16:52:29.2467589Z" /> 
  <EventRecordID>212456</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>mydesktop</Computer> 
  <Security /> 
  </System>
<EventData>
  <Data>[error] podman-machine-default: Error occured in execution group: Error listening on socket: npipe:////./pipe/docker_engine: open \\.\pipe\docker_engine: Access is denied.</Data> 
  </EventData>
  </Event>

What helped me fix it was to disable the "Start on login" in the settings.
Rebooted.
Then started podman desktop as administrator.

[TroySchmidt]

I did get mine resolved by completely going thru the uninstall process again. I also uninstalled WSL as well this time and rebooted. I got it all to install and podman info worked. Then it looked like the sql server container I couldn't connect. But once I changed it from localhost over to 127.0.0.1 then things started to work without an issue.
So, I don't know if that is a port forwarding podman problem, a WSL relay problem, or some other configuration on my machine.

[caaromerogi]

I ran into this same issue after a Windows 10 update this week. Moving from 4.5.0 to 4.5.1 then running the following fixed it: podman machine rm podman machine init podman machine start

This solved the issue for me. Podman v4.5.1 in W10

[fangpenlin]

I encountered the same problem, and I think I found a solution, at least for my case.
tl;dr, run the following commands with admin privilege powershell fix the problem for me

net stop winnat
net start winnat

And here's what I have tried.

• I tried to run SSH to localhost in podman-machine-default, and it worked, so seems like the sshd is working correctly
• Also checked the sshd service and log, didn't see anything looks like it's not working
• I checked my wslconfig, and it seems like localhostForwarding should be enabled by default
• I ran simple http server in a WSL instance with port like 8080, and localhostForwarding is working correctly by running a wslrelay.exe for that port
• Then I wondered, what about the port 60516 assigned by podman for the ssh connection? So I tried to shutdown sshd and run a simple http server instead, somehow the localhostForwarding feature of WSL is not working for that port (no wslrelay.exe started)

After some research, I finally found this WSL issues:

microsoft/WSL#5306

Not sure how WSL works for making the reserved ports, but somehow the port assigned by podman got reserved by WSL. I ran

netsh int ipv4 show excludedportrange protocol=tcp

and the result:

Protocol tcp Port Exclusion Ranges

Start Port    End Port
----------    --------
      5357        5357
     50000       50059     *
     56547       56646
     56647       56746
     59963       60062
     60063       60162
     60263       60362
     60363       60462
     60463       60562
     60563       60662
     60663       60762
     60763       60862
     60866       60965
     61283       61382
     61514       61613
     61614       61713
     61714       61813
     61814       61913
     61914       62013
     62014       62113
     62114       62213
     62214       62313
     62314       62413
     62414       62513
     62514       62613

* - Administered port exclusions.

As you can see my assigned port 60516 is covered by the excluded range 60463 to 60562, and probably that's now wslrelay.exe is not started for the port.

After restarting the service:

net stop winnat
net start winnat

And see the excluded port ranges again:

netsh int ipv4 show excludedportrange protocol=tcp

Protocol tcp Port Exclusion Ranges

Start Port    End Port
----------    --------
      5357        5357
     50000       50059     *

* - Administered port exclusions.

The podman port is not in the excluded list and thus working as expected now. I didn't dig too deep into why WSL is excluding tons of ports like that and somehow those ports covered podman's SSH port, could be a bug. People we are interested can dig deeper into the issue I found. But I guess podman desktop might need to find a way to workaround that issue by selecting a port less likely to be excluded by WSL or maybe come up with a way to dynamically select one not from the excluded range.

[n1hility]

@fangpenlin great observation! We should ensure the port selected is just below the dynamic range, since these can be tied up by WSL / hyperv for internal usage. Will get a fix in for that.

[recodingLife]

For me the root cause was a DNS configuration issue in WSL, which sometimes arises after a Windows reboot.

To address this:

1. Open WSL terminal.
2. Edit the DNS configuration: sudo vim /etc/resolv.conf.
3. Set the nameserver to Google's DNS: nameserver 8.8.8.8.
4. Save and exit the file.

This change redirects DNS requests to Google's nameserver and allowed podman to connect again.

Hope it can help others...

[n1hility]

@recodingLife you were most likely experiencing a different connectivity issue than the primary issue described here since DNS should not be a factor. If you are using a VPN that drops VM traffic you might also be interested in the recently introduced user-mode-networking feature.

[n1hility]

FYI to those impacted by the issue described by the OP: #19557 (just merged) should address this problem.

The new behavior will detect port conflicts on startup and automatically switch to a new available port, which will then be bindable by the wsl relay. You won't need to take any action other than updating to the new release when it's available.

[kid1412621]

after enabled User mode networking, this issue had been resolved.