SSH session does not correctly release file handles after closing

[CJey]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

After establishing a connection, my SSH client reuses this connection and creates a session every second over this connection, sending an exec command on ssh channel to execute some simple script commands, and then closes this session. After my client program ran continuously for about 20 days, my Windows machine hung due to exhausted memory. Analysis revealed that the sshd process had a large number of handles that were not correctly released, as well as a significant number of residual cmd.exe processes.

I tried upgrading to version v9.2.2.0p1-Beta, but the issue still persists. However, after switching to the Bitvise SSH Server software, this problem no longer occurred. As a comparison, the same logic works as expected when running on Linux and macOS servers. I believe this should be a relatively easy case to reproduce, so I won't go into further details.

Expected behavior

no leaks

Actual behavior

file handlers leak

Error details

No response

Environment data

Name                           Value
----                           -----
PSVersion                      5.1.22621.1778
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.22621.1778
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Version

8.6p1

Visuals

too many file handlers hold by sshd.exe

about 2000000 cmd.exe processes, all 32K page table used

[maertendMSFT]

@CJey can you provide repro steps?

[CJey]

@CJey can you provide repro steps?

@maertendMSFT ok，look the full client demo code

golang code

package main

import (
    "fmt"

    "golang.org/x/crypto/ssh"
)

func main() {
    var user = "administrator"
    var pass = "your password"
    var host = "10.1.1.1:22" # your windows sshd entry address
    var cfg = &ssh.ClientConfig{
        User: user,
        Auth: []ssh.AuthMethod{
            ssh.Password(pass),
        },
        HostKeyCallback: ssh.InsecureIgnoreHostKey(),
    }

    var cli, err = ssh.Dial("tcp", host, cfg)
    if err != nil {
        panic(fmt.Errorf("Dial ssh failed, %w", err))
    }
    defer cli.Close()

    fmt.Printf("Infinite looping, session open & session close\n")
    for {
        session, err := cli.NewSession()
        if err != nil {
            panic(fmt.Errorf("New ssh session failed, %w", err))
        }
        session.Run("echo leak")
        session.Close()
    }
}

execute this program，then watch the handlers column of sshd.exe from task manager，you will see the numbers keep growing

[iamswanand]

I'm also facing this issue, is there any updates on this?

[lorenzofattori]

We are experiencing the same issues as well, we notice handles are increasing while the SSH connection is open and not released until the connection is closed, causing a memory leak. @maertendMSFT FYI

[mgkuhn]

How can one reproduce this from the command line quickly?

I tried (from a Linux client):

$ ssh -N -M -o ControlPath=~/fooctl windows-hostname.org &
$ while ssh -o ControlPath=~/fooctl windows-hostname.org tasklist | grep sshd ; do sleep 0 ; done
sshd.exe                      4592 Services                   0      7,600 K
sshd.exe                      8376 Services                   0     10,464 K
sshd.exe                      4868 Services                   0      8,464 K
[...seconds later...]
sshd.exe                      4592 Services                   0      7,600 K
sshd.exe                      8376 Services                   0     10,480 K
sshd.exe                      4868 Services                   0      8,460 K
[...after returning from making a cup of tea...]
sshd.exe                      4592 Services                   0      7,600 K
sshd.exe                      8376 Services                   0     10,444 K
sshd.exe                      4868 Services                   0      8,784 K
[...]

The final column is “Mem Usage” output by tasklist.exe. No spectacular increase ...

[stratfoa]

We too are are facing this issue with a client using a curl script