Non-interactive SSH sessions do not discard stderr from PAM modules

[programmerq]

Expected Behavior

When a user runs a non-interactive SSH command (for example, "tsh ssh node.example.com hostname"), any stderr output generated by PAM or its modules (including scripts invoked by pam_motd.so) should be discarded, as it is not intended for user-facing logs.

Current Behavior

For non-interactive sessions, stderr from any PAM module is displayed to the end-user. The specific scenario is the pam_motd.so module executing a script that outputs an error message to stderr. Other PAM modules writing to stderr would also be incorrectly displayed to the end-user.

Bug Details

Teleport Version

Reported with Teleport version 15.4.7, and reproduced with 17.2.3.

Recreation Steps

Enable PAM integration on a system. Originally reported on Ubuntu 20.04, but reproduced on Debian 12 (bookworm).

ssh_service:
  enabled: yes
  # Enable PAM integration
  pam:
    enabled: yes
    service_name: "sshd"
    use_pam_auth: false

The sshd PAM profile is the Debian/Ubuntu default, and includes:

# Print the message of the day upon successful login.
# This includes a dynamically generated part from /run/motd.dynamic
# and a static (admin-editable) part from /etc/motd.
session    optional     pam_motd.so  motd=/run/motd.dynamic
session    optional     pam_motd.so noupdate

To reproduce the error, a script was created at /etc/update-motd.d/30-stderr. It must be executable.

#!/bin/bash

echo "stderr output" >&2
echo "stdout output"

The expected behavior is that the pam_motd module runs each script, and captures the stdout when building the motd file. Neither stdout nor stderr should appear to the end-user unless the motd is displayed (interactive session with TTY and shell).

jeff@macbook ~ % tsh ssh --no-tty jeff@debhost echo hello world
stderr output
hello world
jeff@macbook ~ % tsh ssh --tty jeff@debhost echo hello world
hello world

With no TTY attached, Teleport will connect the /dev/stderr to the user-facing session immediately, before PAM modules are executed.

With a TTY attached, Teleport doesn't attach the /dev/stdout on /dev/stderr to the user-facing session until after PAM is invoked.

The "stdout output" from the script only displays when the motd is supposed to be displayed.

jeff@macbook ~ % tsh ssh --tty jeff@debhost
Linux debhost 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64
stdout output

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
$

Here is the same test on the same host, but via OpenSSH instead of the Teleport service (note the lack the script's "stderr output".)

jeff@macbook ~ % ssh -T jeff@debhost echo hello world
hello world
jeff@macbook ~ % ssh -t jeff@debhost echo hello world
hello world
jeff@macbook ~ % ssh -t jeff@debhost
Linux debhost 6.1.0-26-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.112-1 (2024-09-30) x86_64
stdout output

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

You have new mail.
Last login: Thu Feb  6 11:49:58 2025 from 10.8.4.82
jeff@debhost:~$

Note that the stderr output is never displayed to the end-user.

When using strace to analyze Teleport's behavior, we noticed that /dev/stderr was connected to /dev/null in all the cases where the unwanted stderr output was not displayed to the user.

I believe this code is where it connects /dev/stderr too early: https://github.com/gravitational/teleport/blob/v17.2.0/lib/srv/exec.go#L147-L165