[Guidance] getting VT sequences without ENABLE_VIRTUAL_TERMINAL_PROCESSING enabled

[therealkenc]

This is part one of a harder question, but I need to get over a basic hump first. I need to give a child process a pty as it's standard output, even though the parent may not have a tty (in Windows parlance, a console) itself. You can assume the parent has no stdin/stdout/stderr. Maybe it's a network daemon.

What I am finding is that even if I write something simple ("hello from child") to the child's stdout handle, in the parent, when I read off the pipe end, I get:

"\x1b[25l\x1b[2J\x1b[m\x1b[Hhello from child\r\n\x1b]0;C:\\Users\\there\\source\\repos\\samples\\ConPTY\\EchoCon\\x64\\Debug\\Ech

The parent doesn't know how to process any of that, because the parent isn't necessarily a terminal emulator. And the child might not even be a console application for all we know, it could be sending house-cat GIFs to stdout. Not our business.

The guidance sought is: how do I coerce the code below such that the parent gets exactly the text "hello from child" from ReadFile(pipe_in). Maybe something silly, but I've run out of ideas except to ask.

This question is in furtherance to finding some elegant solution WSL#3279.

#include "stdafx.h"
#include <Windows.h>
#include <process.h>

void parent(void)
{
  // TL;DR create process with ConPTY pipes, the read from child is further below
  HANDLE pty_in, pty_out, pipe_in, pipe_out;
  CreatePipe(&pty_in, &pipe_out, NULL, 0);
  CreatePipe(&pipe_in, &pty_out, NULL, 0);
  COORD con_sz = { 80, 24 };
  HPCON pty;
  CreatePseudoConsole(con_sz, pty_in, pty_out, 0, &pty);
  CloseHandle(pty_out);
  CloseHandle(pty_in);
  STARTUPINFOEXW startup_info;
  memset(&startup_info, 0, sizeof(startup_info));
  SIZE_T attr_sz = 0;
  startup_info.StartupInfo.cb = sizeof(STARTUPINFOEXW);
  InitializeProcThreadAttributeList(NULL, 1, 0, &attr_sz);
  startup_info.lpAttributeList = (LPPROC_THREAD_ATTRIBUTE_LIST)malloc(attr_sz);
  InitializeProcThreadAttributeList(startup_info.lpAttributeList, 1, 0, &attr_sz);
  UpdateProcThreadAttribute(startup_info.lpAttributeList, 0, 
    PROC_THREAD_ATTRIBUTE_PSEUDOCONSOLE, pty, sizeof(HPCON), NULL, NULL);
  PROCESS_INFORMATION client_info;
  memset(&client_info, 0, sizeof(PROCESS_INFORMATION));
  // self, happened to begin life as EchoCon.exe
  wchar_t cmd[] = L"EchoCon.exe child";
  CreateProcessW(NULL, cmd, NULL, NULL, FALSE, EXTENDED_STARTUPINFO_PRESENT,
    NULL, NULL, &startup_info.StartupInfo, &client_info);
  
  // proof of life hello from parent output
  HANDLE out = GetStdHandle(STD_OUTPUT_HANDLE);
  char buf[256];
  snprintf(buf, sizeof(buf), "hello from parent\n");
  DWORD towrite = (DWORD)strlen(buf);
  DWORD nwritten;
  WriteFile(out, buf, towrite, &nwritten, NULL);
  Sleep(1000);

  // finally the read part
  DWORD nread;
  memset(buf, 0, sizeof(buf));
  // gets: "\x1b[25l\x1b[2J\x1b[m\x1b[Hhello from child\r\n\x1b]0;C:\\Users\\there\\source\\repos\\samples\\ConPTY\\EchoCon\\x64\\Debug\\EchoCon.exe\a\x1b[?25h"
  ReadFile(pipe_in, buf, (DWORD)sizeof(buf), &nread, NULL);
  snprintf(buf, sizeof(buf), "got num bytes: %d\n", nread);
  towrite = (DWORD)strlen(buf);
  WriteFile(out, buf, towrite, &nwritten, NULL);
  Sleep(500);
}

void child()
{
  HANDLE out = GetStdHandle(STD_OUTPUT_HANDLE);
  char buf[256];
  snprintf(buf, sizeof(buf), "hello from child\n");
  DWORD towrite = (DWORD)strlen(buf);
  DWORD nwritten;
  WriteFile(out, buf, towrite, &nwritten, NULL);
}

int main(int argc, const char* argv[])
{
  HANDLE stdout_con = GetStdHandle(STD_OUTPUT_HANDLE);
  DWORD console_mode;
  GetConsoleMode(stdout_con, &console_mode);
  // for good measure
  console_mode &= ~ENABLE_VIRTUAL_TERMINAL_PROCESSING;
  SetConsoleMode(stdout_con, console_mode);
  (argc == 2 && strcmp(argv[1], "child") == 0) ? child() : parent();
  return 0;
}

[n.b. In the example, the parent does have a console, but just for caveman debugging purposes. If you prefer, imagine the parent's WriteFile(out... is debugging out a socket. Or that there are no writes at all in the parent, and I'm setting breakpoints.]

[DHowett-MSFT]

Interesting. Why do you need to give the child process a PTY?

I ask simply because: before ConPTY, people complained that they couldn't get enough information out of a process by simply hooking up a pipe to its output file handle. It was insufficient for terminal emulation.

Since what you're doing is manifestly NOT terminal emulation, you should be able to do the thing that was "insufficient for terminal emulation" and simply hook up a pipe to your child process when you spawn it and avoid ConPTY altogether.

[therealkenc]

Why do you need to give the child process a PTY?

Because the child (over which the parent has no control) may expect a TTY, and probably does.

It was insufficient for terminal emulation.

You might mean ioctl_tty on the pty handle. Contrast terminal emulation; which is data on the handle. That data may or may not be VT sequences. It might be XMODEM, which decidedly needs a TTY not a pipe. [And is, hilariously to me 40 years after my BBS days, integrated into ttyd. Made me smile when I stumbled across that. ;) ]

OpenSSH similarly does not know a VT100 escape sequence from a hole in the ground. It is not a terminal emulator. But the child might (and probably does) need a TTY (it might take a pipe). Before I posted I took at Powershell/openssh-portable on the premise "okay I'll do it however the PowerShell guys do it". Turns out, answer is, ssh.exe doesn't. But Windows ssh.exe hasn't clue-one that it is being spawned from a cmd.exe or powershell.exe. And can not, because it might not. Maybe PuTTY is spawing ssh.exe (and for giggles Microsoft Windows brand telnet.exe too) instead of implementing those wire protocols itself. PuTTY might not even agree with ConPTY's worldview of the VT escape sequences being injected. Parent PuTTY cannot just spawn manifestly-NOT-terminal-emulator child ssh.exe and communicate via pipes, because now foreign sshd on the other end will treat its spawned process (probably login(1) or bash(1)) as a pipe and not a TTY.

[If we want to cut to the chase, the child in question here is decidedly wsl.exe. And we're working up to "how do I receive an out-of-band SIGWINCH equivalent" (contrast sending a ResizePseudoConsole) in a not-a-terminal-emulator win32 program spawning wsl.exe. But first horse before the cart.]

[therealkenc]

Will take this off the books in favor of #1173 and #281.