Jupyter over remote ssh sometimes becomes slow and/or unresponsive #138784
Assignees
Labels
bug
Issue identified by VS Code Team member as probable bug
notebook-perf
verified
Verification succeeded
Milestone
Comments
murphycj2
changed the title
|
Thanks for filing this issue & I'm sorry you're running into this issue, please could you try the following:
If that doesn't make any difference please could you:
|
|
Thanks @DonJayamanne . Not displaying large amounts of output does help. I experimented with the code a bit and here are my observations: (1) If I disable the cell with just the (2) If I still create the figures but no longer display them (see below), this helps immensely with performance... there are basically no issues. (3) While still hiding the figures as in (2), but I go back to displaying (4) If I do the same as (3), but take it even further by displaying 500 rows and columns of
|
|
Thanks for the feedback. Please could you:
Please note, I'll need two logs, one without any data frames, no plots, just print statements. |
|
Here is the text for Even if vscode totally froze up while trying to run something in jupyter, I was still always able to ssh into the server no problem. Here is the Jupyter output from just the print statements: LogHere is the Jupyter output where I print large dataframes and plots: LogFew other things I've observed: (1) In the second log above, I see there are some lines like (2) Here are some additional logs that I observed while it took 15 minutes to execute a cell with a simple print statement (but it was executed after cells with the dataframes and plots): Log from `Log (Window)`Log from `Remote - SSH` |
|
Thanks for the logs. |
|
Also having this issue with jupyter over ssh, in a notebook that has a decent number of plots. |
|
This issue happened with me also (especially when i am insider a docker container inside the remote machine) |
|
I have the same issue, even when I freshly open a notebook with some outputs it is very slow and unresponsive.
I have a stable connection to the server:
|
|
Same problem, sometimes it works fine, sometimes it is unusable, stable and insider |
2 tasks
greazer
added
under-discussion
|
After discussion in sync. Possibly add logging for every message that's only turned on if the user explicitly is told to (for debugging purposes). |
greazer
removed
under-discussion
|
experiencing similar issues when working on jupyter via ssh to an ubuntu server from an ubuntu pc |
|
ALso had the issue where running a cell lags(or starts after some time) for jupyter notebooks when using remote SSH
|
|
Still the same problem |
|
I also experience the same. I observed that:
I hope these may help diagnose the problem. |
roblourens
added a commit
that referenced
this issue
Jan 12, 2023
roblourens
added a commit
that referenced
this issue
Jan 12, 2023
vscodenpa
added
the
unreleased
|
I pushed a change which should mostly fix this slowness with large outputs. I think there are more optimizations we could do. The issue comes when a save happens, and this blocks the remote extension host process for some period of time. Then a symptom of that might be that an execution appears to hang or take a long time. Would appreciate anyone trying this out in tomorrow's vscode insiders build. If you're seeing an issue related to the variables view, that wouldn't be fixed here, however I recently pushed a different fix that may help with that, and you could try it out in insiders + the jupyter pre-release extension. |
vscodenpa
added
insiders-released
|
I have been experiencing these slowdowns (specifically on an AWS server) and I don't use the variables view. I definitely found that clearing cell outputs eliminates the hangs. I'm confused how this is caused by saving though, because I just found that the notebook I was working on wasn't being saved. I was doing some work tethered to my phone as I rode a bus into remote mountains. Certainly noticed that the bad connectivity meant even more waiting for things to execute than usual. Unfortunately, when I got back to stronger internet vscode was complaining it couldn't connect and was hanging when it tried to save. I figured it was fine since I saw @roblourens 's comment and naively interpreted that as meaning I should feel comfortable that my notebook was getting saved each time it hung. Nope. When I finally had to reload window to get things back to working I found that all the code I wrote on the bus was no longer in the notebook (no worries, I had copied it out just in case, so no loss). I'm trying the 2023-01-13T07:49:05.588Z insiders build right now (is the "tomorrow's" build that should have the changes?). I think it is still too soon for me to be able to offer an opinion on how much of an improvement this fix might be. It is hard because the problem is erratic: sometimes cells execute immediately and sometimes it takes 20 seconds or longer. |
|
Yeah, that version has the latest fix, you might have hit a different issue. And the issue might still have been kicked off by the notebook attempting to save. I'll leave this issue open to track
namely, we still send notebook data between the processes more often than we technically have to, and I think that can be improved.
Sorry about that, glad you had it saved, the intent is that the unsaved changes would have been persisted on the frontend. |
vscodenpa
removed
the
insiders-released
|
I can confirm that this change is a big improvement for my use case. If those additional optimizations ever get prioritized that would be amazing and appreciated, but I'm already grateful for this improvement. Thank you! |
|
On second thought, closing this issue, and moving other work to #172345 |
|
I have tried the insiders version on a podman container accessed on a machine over ssh (so I do vscode remote via ssh to the machine and then I use remote containers to access the container). It does not open a terminal and seems to be spinning forever with the following error in the logs: 2023-01-27 10:40:00.790 [info] [LocalProcess0][resolveAuthority(attached-container,27)][1013ms] waiting... May this be related to the recent changes or shall I seek for another bug report and eventually open a new one? |
|
Not related to this change- you can open an issue against the Remote Containers extension, I'm not sure how to interpet that error |
|
Hello, I'm using jupyter/r-notebook docker image to run on AWS EC2 instance and have 2 options to acces it:
So if I compare those 2 options there is a huge difference in UI when notebook has some data and charts. It's so sad that I'm unable to use VS Code Jupyter Notebooks for my work. But if I create a brand new empty notebook in VS Code - everything works like a charm! |
|
@bo44arov @paul-brenner We have added experimental saving logic for Remote SSH, it would be great if you can give this a try and see if it improves the performance on large notebooks
Thanks in advance! |
|
@rebornix The improvements are noticeable, thank you for your great work! However, the latency is still there and appears to be significant from time to time. I love this ssh-notebook function, and I believe there must be more folks like me. Hope your team can prioritize this issue. Appreciate! |
|
Hi,
I just checked VS Code Insiders 1.81.0 version with "Remote Save" feature enabled in settings.
I have generated in a loop 30 charts in first cell and after this I run code in a new cell. It still hangs for a while after generating charts for the first run. If I run second cell code again, it executes immediately as expected.
So we still have an issue when a lot of data (or visual data) was generated recently.
I checked the same code in web UI of Jupyter - and it works fast.
Anton.
From: Peng Lyu ***@***.***>
Sent: Friday, June 30, 2023 12:33 AM
To: microsoft/vscode ***@***.***>
Cc: bo44arov ***@***.***>; Mention ***@***.***>
Subject: Re: [microsoft/vscode] Jupyter over remote ssh sometimes becomes slow and/or unresponsive (Issue #138784)
@bo44arov<https://github.com/bo44arov> @paul-brenner<https://github.com/paul-brenner> We have added experimental saving logic for Remote SSH, it would be great if you can give this a try and see if it improves the performance on large notebooks
* Install latest VS Code Insiders
* Remote SSH into your remote machine
* Add "notebook.experimental.remoteSave": true to your Remote/User settings
* Reload the window
* Then test the scenarios which used to slow or block the network
Thanks in advance!
-
Reply to this email directly, view it on GitHub<#138784 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANC3VYFTDGIGL3WSDQVYMLDXNXX7FANCNFSM5JXJYFJA>.
You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>
|
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
and others
Environment data
Expected behaviour
I expect code cells to begin executing immediately and (for simple code snippets) to finish executing immediately (e.g. print('hello'))
Actual behaviour
Once the jupyter notebook starts to contain a non-trivial amount of stuff, I start to observe the following behaviors: (1) code cells that contain simple tasks (e.g.
print('hello')) start taking several seconds or more to complete, (2) it may take several seconds or even minutes for vscode to even visually show that it will start executing a code cell, and (3) in severe cases, the whole vscode editor will become unresponsive and I will need to force shut it down.Steps to reproduce:
Below is code for a jupyter notebook that contains a minimal example. The problems start to appear once you've cycled through the code cells 2 or 3 times. I think one of the main contributing factors to this issue are just notebooks that have a lot of "stuff" in them. To demonstrate that, the last code block prints 50 scatter plots as pngs, which the native jupyter lab server over the browser handles just fine... but vscode seems to have issues. Moreover, in this simple example I have observed instances where it starts to take a while to save the notebook and the editor may even become unresponsive.
Other things to note:
ami-083ac7c7ecf9bb9b0).Jupyter example
Logs
Here is a screenshot where it takes almost 3 seconds for a simple print statement. I've seen worse in some of my code related to real projects, though. This is just what I was able to reproduce with a minimal example.
The text was updated successfully, but these errors were encountered: