Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reattached to the wrong terminal #133542

Closed
roblourens opened this issue Sep 21, 2021 · 62 comments · Fixed by #152409, #157684, #186072 or #186947
Closed

Reattached to the wrong terminal #133542

roblourens opened this issue Sep 21, 2021 · 62 comments · Fixed by #152409, #157684, #186072 or #186947
Assignees
Labels
bug Issue identified by VS Code Team member as probable bug important Issue identified as high-priority insiders-released Patch has been released in VS Code Insiders papercut 🩸 A particularly annoying issue impacting someone on the team terminal-persistence Relating to process reconnection or process revive verified Verification succeeded
Milestone

Comments

@roblourens
Copy link
Member

roblourens commented Sep 21, 2021

I just reloaded a window with my vscode-jupyter repo open, and it attached to a terminal which is already open in my vscode repo, so now this terminal is open in two windows:
Sep-21-2021 09-51-02

The right window is the window that originally owned this terminal

One thing about this terminal is that its cwd is some other folder outside of both workspaces, if that matters.

@roblourens
Copy link
Member Author

I reloaded the left window again, and it reattached to the correct terminal.

image

@Tyriar Tyriar added bug Issue identified by VS Code Team member as probable bug terminal General terminal issues that don't fall under another label terminal-persistence Relating to process reconnection or process revive labels Sep 21, 2021
@Tyriar Tyriar added this to the Backlog milestone Sep 21, 2021
@roblourens
Copy link
Member Author

Just so you know, this happens pretty often. I just reloaded a window and got some random terminal from another window. Let me know if I can collect any other info

@JacksonKearl
Copy link
Contributor

Also hit this, I believe it was after restarting insiders for an update.

Recording 2022-02-01 at 10 36 21

@meganrogge
Copy link
Contributor

meganrogge commented Feb 1, 2022

I was able to repro once out of many attempts. I wonder if there's a command to reload all windows to simulate what happens on update? Maybe then I'd see it?

@Tyriar
Copy link
Member

Tyriar commented Feb 4, 2022

@meganrogge exiting the application (cmd+q) and reopening it is essentially what happens on update

@JacksonKearl
Copy link
Contributor

Something similar just happened with the most recent insiders update, but now the terminals aren't synced, it's just that the working directory of one window's terminal is set to the other window's root instead of it's own.

image

@JacksonKearl
Copy link
Contributor

Happened again, accidentally ran a destructive command in the wrong directory :(

cc @Tyriar @meganrogge is there any logging or something I can enable to help you get to the bottom of this?

@meganrogge
Copy link
Contributor

I spent awhile investigating. I don't understand what could be going on here because we store and reconnect to terminals according to the workspaceId. @Tyriar said that he thinks it's a race condition. Setting log level to trace and toggling the shared process would help when it reproes

@alexr00
Copy link
Member

alexr00 commented Feb 11, 2022

Today I had something new happen:

  1. I had 3 VS Code windows open.
  2. I reloaded window 1.
  3. Window 1 now shares a terminal with window 3! Totally different workspaces.

No other windows were reloaded. No windows were closed. Is there a way I can disable terminal reconnect for local windows? I've also run destructive commands when this happens and I don't want to risk it.

@meganrogge
Copy link
Contributor

Yep, it's not working atm for remote terminals, so:

terminal.integrated.enablePersistentSessions: false

@yoyo837
Copy link

yoyo837 commented Mar 1, 2022

Hello everyone, there is any plan to try to fix it in the nearest milestone?

@meganrogge meganrogge modified the milestones: Backlog, June 2022 May 19, 2022
@meganrogge
Copy link
Contributor

meganrogge commented Jun 6, 2022

This code that we added for transferring terminals via dnd between windows looks suspicious

backend.onDidRequestDetach(async (e) => {
const instanceToDetach = this.getInstanceFromResource(getTerminalUri(e.workspaceId, e.instanceId));
if (instanceToDetach) {
const persistentProcessId = instanceToDetach?.persistentProcessId;
if (persistentProcessId && !instanceToDetach.shellLaunchConfig.isFeatureTerminal && !instanceToDetach.shellLaunchConfig.customPtyImplementation) {
if (instanceToDetach.target === TerminalLocation.Editor) {
this._terminalEditorService.detachInstance(instanceToDetach);
} else {
this._terminalGroupService.getGroupForInstance(instanceToDetach)?.removeInstance(instanceToDetach);
}
await instanceToDetach.detachFromProcess();
await this._primaryBackend?.acceptDetachInstanceReply(e.requestId, persistentProcessId);
} else {
// will get rejected without a persistentProcessId to attach to
await this._primaryBackend?.acceptDetachInstanceReply(e.requestId, undefined);
}
}
});

@meganrogge
Copy link
Contributor

Has anyone seen this in awhile? I cannot reproduce 😢

@roblourens
Copy link
Member Author

I don't remember seeing it recently

@meganrogge
Copy link
Contributor

Okay pls reopen if you see it again and I'll be happy to debug with anyone that can reproduce it

@meganrogge meganrogge removed this from the June 2022 milestone Jun 6, 2022
@roblourens
Copy link
Member Author

roblourens commented Jun 7, 2022

Sorry, I cursed myself because this happened again today but I probably won't have consistent repro steps. I reloaded the window with my vscode workspace and got a terminal from this extension sample which is also open in another window, although the terminal panel had not been made visible in that other window's session yet.

I reloaded the window several more times and always got the right terminal after that.

image

@roblourens
Copy link
Member Author

I haven't seen this in a long time. Not sure whether there is anything that can be done to verify it.

@alexr00
Copy link
Member

alexr00 commented Jun 29, 2023

I just ran into this now:

  1. I had ~4 windows open:
    • Window 1 is vscode-pull-request-github2 : terminals cwd were all in the workspace
    • Window 2 is vscode-pull-request-github: terminals cwd were all in the workspace
    • Window 3 is repo1: 2 terminals, both of which had a cwd outside the workspace
    • Window 4 is vscode: terminals cwd were all in the workspace
  2. Updated VS Code
  3. Upon reattach:
    • Window 1's terminals are correct
    • Window 2 has reattached to one of Window 3's terminals and has lost it's original terminal
    • Window 3's terminals are correct
    • Window 4's terminals are correct.

Pty logs from Window 2:

2023-06-29 12:10:40.105 [warning] Couldn't get layout info, a terminal was probably disconnected Could not find pty on pty host
2023-06-29 12:10:40.702 [warning] Persistent process "1": Process had no disconnect runners but was an orphan
2023-06-29 12:10:40.702 [info] Persistent process reconnection "1"
2023-06-29 12:10:41.035 [info] Persistent process "1": Replaying 805 chars and 1 size events
2023-06-29 12:10:44.807 [warning] Persistent process "1": Process had no disconnect runners but was an orphan
2023-06-29 12:10:44.807 [info] Persistent process reconnection "1"
2023-06-29 12:10:44.887 [info] Persistent process "1": Replaying 805 chars and 1 size events

@Tyriar
Copy link
Member

Tyriar commented Jun 29, 2023

@alexr00 thanks for the info! Looking into it

@Tyriar
Copy link
Member

Tyriar commented Jun 29, 2023

I spent a few hours looking into this, here are my findings:

  • This warning in the logs is common across recent failures: Couldn't get layout info, a terminal was probably disconnected Could not find pty on pty host, I tried and tried but could not reproduce this on Insiders or OSS 🙁
  • I'm adding more verbose logs to the info log level that gives me the old/new pty id mapping as well as dumping a bunch of info when the above error happens Add debug logs for reattaching to wrong terminal #186653
  • I looked into whether lastPtyId should be correct and while I found it could cause issues when the pty host restarts (Pty host last pty id is not restored when pty host is restarted #186645), it seems like it should work after a restart
  • I could the cwd being outside the workspace would make a difference as we don't even take that into account when reviving the proc (Revived terminals should take latest command's cwd into account #186635)
  • The management of lastPtyId would be a lot simpler if we moved the id to being a guid instead of a number, but that would be a shame as we then need to send the guid along the wire for pretty much every request to the pty host
  • I looked into possible race conditions where _revivedPtyIdMap could get stale/invalid values. Both a thorough review of the code and injecting timeouts (random, differing by window) into important calls didn't cause problems

Current plan is to wait for #186645 to merge and await a repro with these logs present, then analyze them. I'll also likely be continuing to fiddle with terminal reconnection in debt week to improve perf further and reduce redundancies.

@Tyriar Tyriar modified the milestones: June 2023, July 2023 Jun 29, 2023
@alexr00
Copy link
Member

alexr00 commented Jun 30, 2023

I'll keep an eye out for this and try to maintain a setup similar that one that I just saw it with, but I have to say, this has gotten so much better. I almost never see it.

@alexr00
Copy link
Member

alexr00 commented Jun 30, 2023

I tried to repeat my setup before updating, and it did happen again.
Setup:

  • Window 1 is vscode-pull-request-github2 : terminal cwd is outside the workspace and on a different drive (C vs D)
  • Window 2 is vscode-pull-request-github: terminals cwd were in the workspace
  • Window 3 is repo1: terminal cws was in the workspace
  • Window 4 is vscode: terminals cwd were all in the workspace

After update:

  • Window 1 is vscode-pull-request-github2 : has a new terminal
  • Window 2 is vscode-pull-request-github: correct
  • Window 3 is repo1: has attached to Window 2's terminal and now they share the same terminal
  • Window 4 is vscode: correct

Logs:

2023-06-30 12:19:27.910 [info] Revived process, old id 3 -> new id 1
2023-06-30 12:19:28.183 [info] Expanding terminal instance, old id 1 -> new id undefined
2023-06-30 12:19:28.233 [info] Expanding terminal instance, old id 4 -> new id undefined
2023-06-30 12:19:28.233 [warning] Couldn't get layout info, a terminal was probably disconnected Could not find pty on pty host
2023-06-30 12:19:28.233 [info] Reattach to wrong terminal debug info - layout info by id {"relativeSize":1,"terminal":4}
2023-06-30 12:19:28.236 [info] Reattach to wrong terminal debug info - _revivePtyIdMap [{"newId":1,"state":{"id":3,"shellLaunchConfig":{"icon":{"id":"terminal-powershell"},"executable":"C:\\Users\\alros\\AppData\\Local\\Microsoft\\WindowsApps\\Microsoft.PowerShell_8wekyb3d8bbwe\\pwsh.exe","color":null,"useShellEnvironment":true},"processDetails":{"id":3,"title":"pwsh","titleSource":1,"pid":16904,"workspaceId":"dfe6bca14cf70957a5e0b4d02852452b","workspaceName":"vscode-pull-request-github","cwd":"D:\\repos\\Microsoft\\vscode-pull-request-github","isOrphan":false,"icon":{"id":"terminal-powershell"},"environmentVariableCollections":[["vscode.git",[["GIT_ASKPASS",{"variable":"GIT_ASKPASS","value":"c:\\Program Files\\Microsoft VS Code Insiders\\resources\\app\\extensions\\git\\dist\\askpass.sh","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}],["VSCODE_GIT_ASKPASS_NODE",{"variable":"VSCODE_GIT_ASKPASS_NODE","value":"C:\\Program Files\\Microsoft VS Code Insiders\\Code - Insiders.exe","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}],["VSCODE_GIT_ASKPASS_EXTRA_ARGS",{"variable":"VSCODE_GIT_ASKPASS_EXTRA_ARGS","value":"--ms-enable-electron-run-as-node","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}],["VSCODE_GIT_ASKPASS_MAIN",{"variable":"VSCODE_GIT_ASKPASS_MAIN","value":"c:\\Program Files\\Microsoft VS Code Insiders\\resources\\app\\extensions\\git\\dist\\askpass-main.js","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}],["VSCODE_GIT_IPC_HANDLE",{"variable":"VSCODE_GIT_IPC_HANDLE","value":"\\\\.\\pipe\\vscode-git-54ea0bee87-sock","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}]],[["",{"description":"Enables the following features: git auth provider"}]]]],"hasChildProcesses":false,"shellIntegrationNonce":"cabedd54-a605-4ed8-8c39-1a63e7be2bae"},"processLaunchConfig":{"env":{"ALLUSERSPROFILE":"C:\\ProgramData","APPDATA":"C:\\Users\\alros\\AppData\\Roaming","ChocolateyInstall":"C:\\ProgramData\\chocolatey","ChocolateyLastPathUpdate":"133034006781636031","CHROME_CRASHPAD_PIPE_NAME":"\\\\.\\pipe\\LOCAL\\crashpad_7008_BWXUDWPPZKOQEDJO","CLIENTNAME":"ALROS-3","CommonProgramFiles":"C:\\Program Files\\Common Files","CommonProgramFiles(x86)":"C:\\Program Files (x86)\\Common Files","CommonProgramW6432":"C:\\Program Files\\Common Files","COMPUTERNAME":"ALROS-1","ComSpec":"C:\\WINDOWS\\system32\\cmd.exe","CYGWIN_HOME":"C:\\cygwin","DriverData":"C:\\Windows\\System32\\Drivers\\DriverData","EFC_2220":"0","FOO":"this_is_a_bar","GOPATH":"C:\\Users\\alros\\go","GYP_MSVS_VERSION":"2015","HOMEDRIVE":"C:","HOMEPATH":"\\Users\\alros","INCLUDE":"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\include;C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\atlmfc\\include","LIB":"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\lib","LOCALAPPDATA":"C:\\Users\\alros\\AppData\\Local","LOGONSERVER":"\\\\DB3-RED-DC-08","NUMBER_OF_PROCESSORS":"12","NVM_HOME":"C:\\Users\\alros\\AppData\\Roaming\\nvm","NVM_SYMLINK":"C:\\Program Files\\nodejs","OneDrive":"C:\\Users\\alros\\OneDrive - Microsoft","OneDriveCommercial":"C:\\Users\\alros\\OneDrive - Microsoft","ORIGINAL_XDG_CURRENT_DESKTOP":"undefined","OS":"Windows_NT","Path":"C:\\Program Files\\Common Files\\microsoft shared\\Microsoft Online Services;C:\\Program Files (x86)\\Common Files\\Microsoft Shared\\Microsoft Online Services;C:\\Windows\\System32;C:\\Windows;C:\\Windows\\System32\\wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;D:\\install\\powershell\\6\\;C:\\cygwin\\bin;C:\\Go\\bin;C:\\Program Files\\Microsoft VS Code Exploration\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin;C:\\Program Files\\Microsoft VS Code Insiders\\bin;C:\\Python27;C:\\Program Files\\Microsoft SQL Server\\130\\Tools\\Binn\\;C:\\Users\\alros\\AppData\\Roaming\\nvm;C:\\Program Files\\nodejs;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program Files (x86)\\Yarn\\bin\\;C:\\Program Files\\CMake\\bin;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files\\dotnet\\;C:\\Program Files (x86)\\Gpg4win\\..\\GnuPG\\bin;C:\\Program Files\\GitHub CLI\\;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Program Files\\Git\\cmd;C:\\Program Files\\Docker\\Docker\\resources\\bin;C:\\Users\\alros\\.cargo\\bin;C:\\Users\\alros\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\alros\\go\\bin;C:\\Users\\alros\\.dotnet\\tools;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319;C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\IDE;C:\\Python27;C:\\Users\\alros\\AppData\\Roaming\\nvm;C:\\Program Files\\nodejs;d:\\repos\\microsoft\\vscode\\scripts;C:\\Users\\alros\\AppData\\Local\\Yarn\\bin;C:\\Users\\alros\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\Scripts;C:\\Users\\alros\\.dotnet\\tools;C:\\Users\\alros\\AppData\\Local\\GitHubDesktop\\bin;C:\\ProgramData\\alros\\GitHubDesktop\\bin","PATHEXT":".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC","PROCESSOR_ARCHITECTURE":"AMD64","PROCESSOR_IDENTIFIER":"Intel64 Family 6 Model 85 Stepping 4, GenuineIntel","PROCESSOR_LEVEL":"6","PROCESSOR_REVISION":"5504","ProgramData":"C:\\ProgramData","ProgramFiles":"C:\\Program Files","ProgramFiles(x86)":"C:\\Program Files (x86)","ProgramW6432":"C:\\Program Files","PSModulePath":"C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\Microsoft Azure Information Protection\\Powershell","PUBLIC":"C:\\Users\\Public","PYTHON":"C:\\Python27\\python.exe","SESSIONNAME":"RDP-Tcp#0","SystemDrive":"C:","SystemRoot":"C:\\WINDOWS","TEMP":"C:\\Users\\alros\\AppData\\Local\\Temp","TMP":"C:\\Users\\alros\\AppData\\Local\\Temp","UATDATA":"C:\\windows\\CCM\\UATData\\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77","USERDNSDOMAIN":"REDMOND.CORP.MICROSOFT.COM","USERDOMAIN":"REDMOND","USERDOMAIN_ROAMINGPROFILE":"REDMOND","USERNAME":"alros","USERPROFILE":"C:\\Users\\alros","VS140COMNTOOLS":"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\Tools\\","windir":"C:\\WINDOWS","__COMPAT_LAYER":"DetectorsAppHealth","TERM_PROGRAM":"vscode","TERM_PROGRAM_VERSION":"1.80.0-insider","LANG":"en_US.UTF-8","COLORTERM":"truecolor","GIT_ASKPASS":"c:\\Program Files\\Microsoft VS Code Insiders\\resources\\app\\extensions\\git\\dist\\askpass.sh","VSCODE_GIT_ASKPASS_NODE":"C:\\Program Files\\Microsoft VS Code Insiders\\Code - Insiders.exe","VSCODE_GIT_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_GIT_ASKPASS_MAIN":"c:\\Program Files\\Microsoft VS Code Insiders\\resources\\app\\extensions\\git\\dist\\askpass-main.js","VSCODE_GIT_IPC_HANDLE":"\\\\.\\pipe\\vscode-git-54ea0bee87-sock"},"executableEnv":{"ALLUSERSPROFILE":"C:\\ProgramData","APPDATA":"C:\\Users\\alros\\AppData\\Roaming","ChocolateyInstall":"C:\\ProgramData\\chocolatey","ChocolateyLastPathUpdate":"133034006781636031","CHROME_CRASHPAD_PIPE_NAME":"\\\\.\\pipe\\LOCAL\\crashpad_16072_MPZZDZVVZJBOERYY","CLIENTNAME":"ALROS-3","CommonProgramFiles":"C:\\Program Files\\Common Files","CommonProgramFiles(x86)":"C:\\Program Files (x86)\\Common Files","CommonProgramW6432":"C:\\Program Files\\Common Files","COMPUTERNAME":"ALROS-1","ComSpec":"C:\\WINDOWS\\system32\\cmd.exe","CYGWIN_HOME":"C:\\cygwin","DriverData":"C:\\Windows\\System32\\Drivers\\DriverData","EFC_2220":"0","FOO":"this_is_a_bar","GOPATH":"C:\\Users\\alros\\go","GYP_MSVS_VERSION":"2015","HOMEDRIVE":"C:","HOMEPATH":"\\Users\\alros","INCLUDE":"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\include;C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\atlmfc\\include","LIB":"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\lib","LOCALAPPDATA":"C:\\Users\\alros\\AppData\\Local","LOGONSERVER":"\\\\DB3-RED-DC-08","NUMBER_OF_PROCESSORS":"12","NVM_HOME":"C:\\Users\\alros\\AppData\\Roaming\\nvm","NVM_SYMLINK":"C:\\Program Files\\nodejs","OneDrive":"C:\\Users\\alros\\OneDrive - Microsoft","OneDriveCommercial":"C:\\Users\\alros\\OneDrive - Microsoft","ORIGINAL_XDG_CURRENT_DESKTOP":"undefined","OS":"Windows_NT","Path":"C:\\Program Files\\Common Files\\microsoft shared\\Microsoft Online Services;C:\\Program Files (x86)\\Common Files\\Microsoft Shared\\Microsoft Online Services;C:\\Windows\\System32;C:\\Windows;C:\\Windows\\System32\\wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;D:\\install\\powershell\\6\\;C:\\cygwin\\bin;C:\\Go\\bin;C:\\Program Files\\Microsoft VS Code Exploration\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin;C:\\Program Files\\Microsoft VS Code Insiders\\bin;C:\\Python27;C:\\Program Files\\Microsoft SQL Server\\130\\Tools\\Binn\\;C:\\Users\\alros\\AppData\\Roaming\\nvm;C:\\Program Files\\nodejs;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Program Files (x86)\\Yarn\\bin\\;C:\\Program Files\\CMake\\bin;C:\\ProgramData\\chocolatey\\bin;C:\\Program Files\\dotnet\\;C:\\Program Files (x86)\\Gpg4win\\..\\GnuPG\\bin;C:\\Program Files\\GitHub CLI\\;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Program Files\\Git\\cmd;C:\\Program Files\\Docker\\Docker\\resources\\bin;C:\\Users\\alros\\.cargo\\bin;C:\\Users\\alros\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\alros\\go\\bin;C:\\Users\\alros\\.dotnet\\tools;C:\\Windows\\Microsoft.NET\\Framework\\v4.0.30319;C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\VC\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\IDE;C:\\Python27;C:\\Users\\alros\\AppData\\Roaming\\nvm;C:\\Program Files\\nodejs;d:\\repos\\microsoft\\vscode\\scripts;C:\\Users\\alros\\AppData\\Local\\Yarn\\bin;C:\\Users\\alros\\AppData\\Local\\Packages\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\LocalCache\\local-packages\\Python39\\Scripts;C:\\Users\\alros\\.dotnet\\tools;C:\\Users\\alros\\AppData\\Local\\GitHubDesktop\\bin;C:\\ProgramData\\alros\\GitHubDesktop\\bin","PATHEXT":".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC","PROCESSOR_ARCHITECTURE":"AMD64","PROCESSOR_IDENTIFIER":"Intel64 Family 6 Model 85 Stepping 4, GenuineIntel","PROCESSOR_LEVEL":"6","PROCESSOR_REVISION":"5504","ProgramData":"C:\\ProgramData","ProgramFiles":"C:\\Program Files","ProgramFiles(x86)":"C:\\Program Files (x86)","ProgramW6432":"C:\\Program Files","PSModulePath":"C:\\Program Files\\WindowsPowerShell\\Modules;C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\Microsoft Azure Information Protection\\Powershell","PUBLIC":"C:\\Users\\Public","PYTHON":"C:\\Python27\\python.exe","SESSIONNAME":"RDP-Tcp#0","SystemDrive":"C:","SystemRoot":"C:\\WINDOWS","TEMP":"C:\\Users\\alros\\AppData\\Local\\Temp","TMP":"C:\\Users\\alros\\AppData\\Local\\Temp","UATDATA":"C:\\windows\\CCM\\UATData\\D9F8C395-CAB8-491d-B8AC-179A1FE1BE77","USERDNSDOMAIN":"REDMOND.CORP.MICROSOFT.COM","USERDOMAIN":"REDMOND","USERDOMAIN_ROAMINGPROFILE":"REDMOND","USERNAME":"alros","USERPROFILE":"C:\\Users\\alros","VS140COMNTOOLS":"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\Tools\\","VSCODE_CODE_CACHE_PATH":"C:\\Users\\alros\\AppData\\Roaming\\Code - Insiders\\CachedData\\c1bca6d7cc2c5f71ae04deda90c0cf50819ffde4","VSCODE_CWD":"C:\\Program Files\\Microsoft VS Code Insiders","VSCODE_IPC_HOOK":"\\\\.\\pipe\\72671c30d05ce318932621dac30dfd34-1.80.0-insider-main-sock","VSCODE_NLS_CONFIG":"{\"locale\":\"en\",\"osLocale\":\"en-us\",\"availableLanguages\":{},\"_languagePackSupport\":true}","VSCODE_PID":"16072","windir":"C:\\WINDOWS","__COMPAT_LAYER":"DetectorsAppHealth"},"options":{"shellIntegration":{"enabled":true,"suggestEnabled":false,"nonce":"cabedd54-a605-4ed8-8c39-1a63e7be2bae"},"windowsEnableConpty":true,"environmentVariableCollections":[["vscode.git",[["GIT_ASKPASS",{"variable":"GIT_ASKPASS","value":"c:\\Program Files\\Microsoft VS Code Insiders\\resources\\app\\extensions\\git\\dist\\askpass.sh","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}],["VSCODE_GIT_ASKPASS_NODE",{"variable":"VSCODE_GIT_ASKPASS_NODE","value":"C:\\Program Files\\Microsoft VS Code Insiders\\Code - Insiders.exe","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}],["VSCODE_GIT_ASKPASS_EXTRA_ARGS",{"variable":"VSCODE_GIT_ASKPASS_EXTRA_ARGS","value":"--ms-enable-electron-run-as-node","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}],["VSCODE_GIT_ASKPASS_MAIN",{"variable":"VSCODE_GIT_ASKPASS_MAIN","value":"c:\\Program Files\\Microsoft VS Code Insiders\\resources\\app\\extensions\\git\\dist\\askpass-main.js","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}],["VSCODE_GIT_IPC_HANDLE",{"variable":"VSCODE_GIT_IPC_HANDLE","value":"\\\\.\\pipe\\vscode-git-54ea0bee87-sock","type":1,"options":{"applyAtProcessCreation":true,"applyAtShellIntegration":false}}]],[["",{"description":"Enables the following features: git auth provider"}]]]],"workspaceFolder":{"uri":{"$mid":1,"fsPath":"d:\\repos\\Microsoft\\vscode-pull-request-github","_sep":1,"external":"file:///d%3A/repos/Microsoft/vscode-pull-request-github","path":"/d:/repos/Microsoft/vscode-pull-request-github","scheme":"file"},"name":"vscode-pull-request-github","index":0}}},"unicodeVersion":"11","replayEvent":{"events":[{"cols":258,"rows":17,"data":"PS D:\\repos\\Microsoft\\vscode-pull-request-github> \u001b[93mgit \u001b[0mpull\r\nremote: Enumerating objects: 1, done.\r\nremote: Counting objects: 100% (1/1), done.\r\nremote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0\r\nUnpacking objects: 100% (1/1), 614 bytes | 61.00 KiB/s, done.\r\nFrom https://github.com/microsoft/vscode-pull-request-github\r\n   8cbac164..c6f00d59  main       -> origin/main\r\nUpdating 8cbac164..c6f00d59\r\nFast-forward\r\n src/commands.ts | 13 \u001b[32m++++++++++\u001b[31m---\r\n\u001b[0m 1 file changed, 10 insertions(+), 3 deletions(-)\r\nPS D:\\repos\\Microsoft\\vscode-pull-request-github> \u001b[93mgit \u001b[0mpull\r\nAlready up to date.\r\nPS D:\\repos\\Microsoft\\vscode-pull-request-github> \u001b[93mgit \u001b[0mcheckout \u001b[90m-b \u001b[0mrelease/0.68\r\nSwitched to a new branch 'release/0.68'\r\nPS D:\\repos\\Microsoft\\vscode-pull-request-github> \u001b[93mgit \u001b[0mpush\r\nfatal: The current branch release/0.68 has no upstream branch.\r\nTo push the current branch and set the remote as upstream, use\r\n\r\n    git push --set-upstream origin release/0.68\r\n\r\nTo have this happen automatically for branches without a tracking\r\nupstream, see 'push.autoSetupRemote' in 'git help config'.\r\n\r\nPS D:\\repos\\Microsoft\\vscode-pull-request-github> \u001b[93mgit \u001b[0mpush \u001b[90m--set-upstream \u001b[0morigin release/0.68\r\nTotal 0 (delta 0), reused 0 (delta 0), pack-reused 0\r\nremote:\r\nremote: Create a pull request for 'release/0.68' on GitHub by visiting:\r\nremote:      https://github.com/microsoft/vscode-pull-request-github/pull/new/release/0.68\r\nremote:\r\nTo https://github.com/microsoft/vscode-pull-request-github.git\r\n * [new branch]        release/0.68 -> release/0.68\r\nbranch 'release/0.68' set up to track 'origin/release/0.68'.\r\nPS D:\\repos\\Microsoft\\vscode-pull-request-github>\u001b[1C\u001b[93mgit \u001b[0mcheckout main\r\nSwitched to branch 'main'\r\nYour branch is up to date with 'origin/main'.\r\nPS D:\\repos\\Microsoft\\vscode-pull-request-github>\u001b[1C\u001b[93mgit \u001b[0mcheckout \u001b[90m-b \u001b[0malexr00/disablePrerelease \u001b[38;5;238m    \r\n\u001b[0mSwitched to a new branch 'alexr00/disablePrerelease'\r\nPS D:\\repos\\Microsoft\\vscode-pull-request-github>\u001b[1C"}],"commands":{"isWindowsPty":true,"commands":[{"startLine":0,"endLine":11,"executedLine":0,"command":"git pull","isTrusted":true,"cwd":"D:\\repos\\Microsoft\\vscode-pull-request-github","exitCode":0,"commandStartLineContent":"PS D:\\repos\\Microsoft\\vscode-pull-request-github> git pull --unshallow","timestamp":1688104913850},{"startLine":11,"endLine":13,"executedLine":11,"command":"git pull","isTrusted":true,"cwd":"D:\\repos\\Microsoft\\vscode-pull-request-github","exitCode":0,"commandStartLineContent":"PS D:\\repos\\Microsoft\\vscode-pull-request-github> git pull --unshallow","timestamp":1688104992406},{"startLine":13,"endLine":14,"executedLine":13,"command":"git checkout -b release/0.68","isTrusted":true,"cwd":"D:\\repos\\Microsoft\\vscode-pull-request-github","exitCode":0,"commandStartLineContent":"PS D:\\repos\\Microsoft\\vscode-pull-request-github> git checkout -b release/0.68","timestamp":1688105048380},{"startLine":15,"endLine":23,"executedLine":15,"command":"git push","isTrusted":true,"cwd":"D:\\repos\\Microsoft\\vscode-pull-request-github","exitCode":1,"commandStartLineContent":"PS D:\\repos\\Microsoft\\vscode-pull-request-github> git push --set-upstream origin release/1.80","timestamp":1688105055685},{"startLine":24,"endLine":32,"executedLine":24,"command":"git push --set-upstream origin release/0.68","isTrusted":true,"cwd":"D:\\repos\\Microsoft\\vscode-pull-request-github","exitCode":0,"commandStartLineContent":"PS D:\\repos\\Microsoft\\vscode-pull-request-github> git push --set-upstream origin release/0.68","timestamp":1688105063631},{"startLine":33,"endLine":35,"executedLine":33,"command":"git checkout main","isTrusted":true,"cwd":"D:\\repos\\Microsoft\\vscode-pull-request-github","exitCode":0,"commandStartLineContent":"PS D:\\repos\\Microsoft\\vscode-pull-request-github> git checkout main","timestamp":1688113116127},{"startLine":36,"endLine":37,"executedLine":36,"command":"git checkout -b alexr00/disablePrerelease","isTrusted":true,"cwd":"D:\\repos\\Microsoft\\vscode-pull-request-github","exitCode":0,"commandStartLineContent":"PS D:\\repos\\Microsoft\\vscode-pull-request-github> git checkout -b alexr00/disablePrerelease     ","timestamp":1688113158174},{"startLine":37,"startX":52,"command":"","isTrusted":true,"cwd":"D:\\repos\\Microsoft\\vscode-pull-request-github","timestamp":0}]}},"timestamp":1688120353450}}]
2023-06-30 12:19:28.488 [info] Expanding terminal instance, old id 3 -> new id 1
2023-06-30 12:19:28.865 [warning] Persistent process "1": Process had no disconnect runners but was an orphan
2023-06-30 12:19:28.866 [info] Persistent process reconnection "1"
2023-06-30 12:19:29.410 [info] Persistent process "1": Replaying 2108 chars and 1 size events
2023-06-30 12:19:29.487 [info] Revived process, old id 2 -> new id 2
2023-06-30 12:19:29.572 [info] Expanding terminal instance, old id 2 -> new id 2
2023-06-30 12:19:32.741 [warning] Persistent process "2": Process had no disconnect runners but was an orphan
2023-06-30 12:19:32.742 [info] Persistent process reconnection "2"
2023-06-30 12:19:32.762 [warning] Persistent process "1": Process had no disconnect runners but was an orphan
2023-06-30 12:19:32.763 [info] Persistent process reconnection "1"
2023-06-30 12:19:33.420 [info] Persistent process "2": Replaying 5919 chars and 1 size events
2023-06-30 12:19:33.428 [info] Persistent process "1": Replaying 2108 chars and 1 size events

@Tyriar
Copy link
Member

Tyriar commented Jun 30, 2023

Thanks @alexr00, it seems there's some inconsistency with the layout info and the revived procs, so I suspect this silently error swallowing now:

Added logs for that here: #186732, I'll merge for July so you might not be able to test for a week but when you do please also grab the Terminal output log as this is renderer side.

@Tyriar
Copy link
Member

Tyriar commented Jul 3, 2023

Cross post #186922 (comment)

I think I've identified a problem, since revive can happen after startup (eg. open a new window with a folder), this could introduce conflicts in _revivedPtyIdMap. _revivedPtyIdMap should include the workspace ID in its key

@Tyriar
Copy link
Member

Tyriar commented Jul 3, 2023

With @meganrogge's logs in #186922, I was able to reproduce the problem!

image

The repro is very complicated, it is essentially trying to get into a state where 2+ windows have "old pty ids" that conflict with one another, and then increasing the latency of the pty host in order to force the race condition to show itself.

  1. Set these settings:
    "terminal.integrated.persistentSessionReviveProcess": "onExitAndWindowClose",
    "terminal.integrated.developer.ptyHost.latency": 500
    
  2. Set default log level to trace
  3. Close all windows
  4. Open window 1 (W1) with folder A
  5. W1: Open 2 terminals
  6. W1: Close window
  7. W1: Open window
  8. W1: Open pty host log, you should see this:
    2023-07-03 12:54:57.963 [info] Expanding terminal instance, old id 1 -> new id 1
    2023-07-03 12:54:57.963 [info] Expanding terminal instance, old id 2 -> new id 2
    
  9. Open window 2 (W2) with folder B
  10. W2: Open 2 terminals
  11. W1: Close window
  12. W2: Close window
  13. Open VS Code (W2 should open)
  14. W2: Open pty host log, you should something like this:
    2023-07-03 12:54:57.963 [info] Expanding terminal instance, old id 3 -> new id 1
    2023-07-03 12:54:57.963 [info] Expanding terminal instance, old id 4 -> new id 2
    
  15. W2: Close window
  16. Open an external terminal and type code
  17. Open another external terminal and type code <folder A>
  18. Run the commands in both terminals in quick succession
  19. W1|W2: Open pty host log, you should see something like this:
    2023-07-03 13:01:56.461 [info] Expanding terminal instance, old id 1 -> new id 3
    2023-07-03 13:01:56.461 [info] Expanding terminal instance, old id 2 -> new id 4
    2023-07-03 13:01:57.086 [info] Expanding terminal instance, old id 1 -> new id undefined
    2023-07-03 13:01:57.086 [info] Expanding terminal instance, old id 2 -> new id undefined
    

The terminal with new id undefined end up attaching to the wrong pty.


The fix for this is simple, just add workspace ID as part of the key of the old id, so 1 becomes <folder B hash>-1. This should make conflicts impossible to occur.

Tyriar added a commit that referenced this issue Jul 3, 2023
@vscodenpa vscodenpa added unreleased Patch has not yet been released in VS Code Insiders insiders-released Patch has been released in VS Code Insiders and removed unreleased Patch has not yet been released in VS Code Insiders labels Jul 3, 2023
@github-actions github-actions bot locked and limited conversation to collaborators Aug 17, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Issue identified by VS Code Team member as probable bug important Issue identified as high-priority insiders-released Patch has been released in VS Code Insiders papercut 🩸 A particularly annoying issue impacting someone on the team terminal-persistence Relating to process reconnection or process revive verified Verification succeeded
Projects
None yet