Jupyter cannot connect to a remote server using a custom ServerPicker #13112

mtreca · 2023-03-20T15:15:11Z

Applies To

Notebooks (.ipynb files)
Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?

Hi there!

I created a custom VSCode extension in order to interact with my company's remote JupyterHub clusters, based on the examples provided on your repository here. My extension defines, among other things, a custom Jupyter Server Picker which is then registered with the Jupyter extension.

The custom server picker is registered and functions properly since it is properly displayed when clicking the Select kernel button after activating my extension. The Server picker also seems to return proper IJupyterServerUri objects, which is confirmed by logs:

authorizationHeader: {Authorization: 'Bearer MYTOKEN'}
baseUrl: 'https://jh-cluster.mycompany.com/'
expiration: undefined
token: 'MYTOKEN'

However, the actual resulting connexion fails immediately with the following traceback (true sensitive values are edited out in all caps):

~/.myenv/bin/python ~/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/pythonFiles/vscode_datascience_helpers/getServerInfo.py
info 15:55:44.004: Connecting local server kernel http://localhost:8888/
info 15:55:44.005: Creating server with url : http://localhost:8888/
info 15:55:44.006: Server started.
info 15:55:59.363: Creating server with url : https://jh-cluster.mycompany.com/
warn 15:55:59.409: DataScience Error [eo [Error]: Failed to connect to the remote Jupyter Server 'https://remote/'. Verify the server is running and reachable. (Not Found).
	at lo.setJupyterURIToRemote (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:1679005)
	at async /Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:24:216365] {
  category: 'remotejupyterserverconnection',
  url: 'https://remote/?id=NAME OF MY REMOTE SERVER PICKER&uriHandle=live',
  serverId: 'HEX VALUE',
  originalError: [t [Error]: Not Found
  	at t.create (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:68133)
  	at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
  	at async Object.listRunning (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:55998)
  	at async mg.getRunningKernels (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:17:237122)
  	at async Promise.all (index 0)
  	at async Ti.validateRemoteConnection (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:1664096)
  	at async lo.setJupyterURIToRemote (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:1678698)
  	at async /Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:24:216365] {
    response: L [Response] {
      size: 0,
      timeout: 0,
      [Symbol(Body internals)]: [Object],
      [Symbol(Response internals)]: [Object]
    },
    traceback: ''
  },
  baseUrl: 'https://remote/'
}
error 15:55:59.415: SessionManager:getKernelSpecs failure:  [Error: Poll (@jupyterlab/services:KernelSpecManager#specs) is disposed.
	at e.dispose (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:171555)
	at u.dispose (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:62984)
	at mg.dispose (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:17:236236)
	at process.processTicksAndRejections (node:internal/process/task_queues:96:5)]

Which seems to indicate that the url provided by the server picker is not properly read or parsed.

Furthermore, manually registering the remote cluster using the https://jh-cluster.mycompany.com/token?=MYTOKEN functions perfectly well.

Any idea what is going on? Thanks!

VS Code Version

1.76.2

Jupyter Extension Version

v2023.3.1000641008

Jupyter logs

~/.myenv/bin/python ~/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/pythonFiles/vscode_datascience_helpers/getServerInfo.py
info 15:55:44.004: Connecting local server kernel http://localhost:8888/
info 15:55:44.005: Creating server with url : http://localhost:8888/
info 15:55:44.006: Server started.
info 15:55:59.363: Creating server with url : https://jh-cluster.mycompany.com/
warn 15:55:59.409: DataScience Error [eo [Error]: Failed to connect to the remote Jupyter Server 'https://remote/'. Verify the server is running and reachable. (Not Found).
	at lo.setJupyterURIToRemote (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:1679005)
	at async /Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:24:216365] {
  category: 'remotejupyterserverconnection',
  url: 'https://remote/?id=NAME OF MY REMOTE SERVER PICKER&uriHandle=live',
  serverId: 'HEX VALUE',
  originalError: [t [Error]: Not Found
  	at t.create (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:68133)
  	at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
  	at async Object.listRunning (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:55998)
  	at async mg.getRunningKernels (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:17:237122)
  	at async Promise.all (index 0)
  	at async Ti.validateRemoteConnection (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:1664096)
  	at async lo.setJupyterURIToRemote (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:1678698)
  	at async /Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:24:216365] {
    response: L [Response] {
      size: 0,
      timeout: 0,
      [Symbol(Body internals)]: [Object],
      [Symbol(Response internals)]: [Object]
    },
    traceback: ''
  },
  baseUrl: 'https://remote/'
}
error 15:55:59.415: SessionManager:getKernelSpecs failure:  [Error: Poll (@jupyterlab/services:KernelSpecManager#specs) is disposed.
	at e.dispose (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:171555)
	at u.dispose (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:2:62984)
	at mg.dispose (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000641008/out/extension.node.js:17:236236)
	at process.processTicksAndRejections (node:internal/process/task_queues:96:5)]



### Coding Language and Runtime Version

Python v3.8

### Language Extension Version (if applicable)

_No response_

### Anaconda Version (if applicable)

_No response_

### Running Jupyter locally or remotely?

N/A or Not sure

The text was updated successfully, but these errors were encountered:

mtreca · 2023-03-20T15:16:50Z

Tagging @DonJayamanne since they wrote the module I base my code off :)

DonJayamanne · 2023-03-30T19:34:13Z

@mtreca
Thanks for filing this issue and trying to create an extension based off the API.
Please could you try this again with the latest pre-release bits of the Jupyter extension.
Please try the following

Enable Verbose logging as follows
- Open settings & go into Jupyter->Logging
- Change the value to verbose
- Reload VS Code,
Remove all remote URIs from VS Code (ensure nothing is displayed in the list), I can see that you have http://localhost:8888, try to remove that too
Reload VS Code
Now go through the steps you ran to replicate this issue
Then please share the logs again

mtreca · 2023-04-03T13:58:26Z

Hi @DonJayamanne, thank you for your reply!

I followed the steps you mentioned, but I do not think the output changes a lot:

info 15:52:45.105: Creating server with url : https://jh-cluster.mycompany.com/
debug 15:52:45.202: Disposing session manager
debug 15:52:45.202: SessionManager - dispose contents manager
debug 15:52:45.202: ShutdownSessionAndConnection - dispose session manager
warn 15:52:45.203: DataScience Error [co [Error]: Failed to connect to the remote Jupyter Server 'https://remote/'. Verify the server is running and reachable. (Not Found).
	at bo.setJupyterURIToRemote (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:1720809)
	at async /Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:24:217068] {
  category: 'remotejupyterserverconnection',
  url: 'https://remote/?id=MyCompany JupyterHub Clusters&uriHandle=ServerName',
  serverId: '606cbe91b29e660c7dc4cb9d5e7cc5493b5381f1cadf148f39083b52b39559e9',
  originalError: [t [Error]: Not Found
  	at t.create (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:68133)
  	at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
  	at async t.listRunning (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:55998)
  	at async jg.getRunningKernels (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:17:237503)
  	at async Promise.all (index 0)
  	at async Ki.validateRemoteConnection (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:1705868)
  	at async bo.setJupyterURIToRemote (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:1720502)
  	at async /Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:24:217068] {
    response: L [Response] {
      size: 0,
      timeout: 0,
      [Symbol(Body internals)]: [Object],
      [Symbol(Response internals)]: [Object]
    },
    traceback: ''
  },
  baseUrl: 'https://remote/'
}
debug 15:52:45.211: Finished disposing jupyter session manager
error 15:52:45.212: SessionManager:getKernelSpecs failure:  [Error: Poll (@jupyterlab/services:KernelSpecManager#specs) is disposed.
	at e.dispose (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:171555)
	at u.dispose (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:2:62984)
	at jg.dispose (/Users/maximet/.vscode-oss/extensions/ms-toolsai.jupyter-2023.3.1000892223/out/extension.node.js:17:236617)
	at process.processTicksAndRejections (node:internal/process/task_queues:96:5)]

DonJayamanne · 2023-04-05T19:58:36Z

@mtreca I will need to build a custom version of the VSIX with additional logging to see whats going on
Will send that some time today.

mtreca · 2023-04-06T15:25:32Z

Thanks!
I did a little digging through the codebase (also I am not fluent in TS), and it does seem that the baseUrl from my kernel picker is never properly to the extension since it seems that the url it is trying to connect to is defined in the constants file.

DonJayamanne · 2023-04-17T20:55:31Z

picker is never properly to the extension since it seems that the ur

Thanks, unfortunately I'm not entirely sure I understand this statement.
Is it possible to share some more information on this, some screenshots of teh debug sesssion and the values or the like

mtreca · 2023-05-15T12:21:37Z

Hi again. Sorry for the delay and not being clear in my last message. Regardless of how I define the remote server configuration, the baseUrl defined by my kernel picker is always equal to "https://remote/", which seems to be hardcoded in the Jupyter extension. So I guess my custom kernel picker does not properly pass the remote url to the Jupyter extension.

After thinking about it, I would also be happy with a much simpler mechanism which would just allow my extension to write URLs to the "Existing Jupyter Servers" section of the Kernel Picker (i.e. as if the user had manually entered this url before). No need to define a custom kernel picker in this case. Do you know if such a thing is possible? Thanks a lot for your help!

DonJayamanne · 2023-05-15T12:27:46Z

I recently fixed a few issues and also created a sample extension that uses this api
Please can you try the your extension once again in around 1/2 hour with three latest pre-release version of thr Jupyter extension invscode insiders

Let met know goes that goes

Thanks for your patience

mtreca · 2023-05-15T12:46:36Z

Good to hear, thanks for the quick update!

My work env does not let me use insiders or pre-release. Will test when the new version of the plugin is released then!

Will keep you updated, thanks

DonJayamanne · 2023-05-15T12:52:38Z

I haven't tried using a custom Jupyter hub though, will try that soon

DonJayamanne · 2023-06-02T01:37:06Z

Found a few issues and have addressed them, however it will take a while to ship those fixes, will keep you posted.
Here's a PR with all of the changes and fixes #13588

DonJayamanne · 2023-06-23T02:16:05Z

This should be addressed in the latest pre-release version, please let me know how that goes

brandonjbjelland · 2023-07-09T07:26:09Z

Hi @DonJayamanne 👋

I'm using the pre-release version and followed your steps above, deleting any existing connections + reload, and I'm still hitting this issue. Maybe I'm using the API incorrectly but the https://remote/ prefix has me think the same problem originally encountered is persisting:

00:23:21.724 [error] Server URL cannot be used. Did you uninstall an extension that provided a Jupyter server connection?. Provider Id=b4eb6a42-a9e9-4702-8466-1228f6b31afa and handle=http://127.0.0.1:8888/lab
00:23:21.725 [warn] DataScience Error vo [Error]: Failed to connect to the remote Jupyter Server 'https://remote/'. Verify the server is running and reachable. (Server URL cannot be used. Did you uninstall an extension that provided a Jupyter server connection?).
    at Eo.addJupyterServerOld (~/.vscode/extensions/ms-toolsai.jupyter-2023.7.1001901100-darwin-arm64/out/extension.node.js:2:1711335)
    at ~/.vscode/extensions/ms-toolsai.jupyter-2023.7.1001901100-darwin-arm64/out/extension.node.js:2:1741740 {
  category: 'remotejupyterserverconnection',
  isJupyterError: true,
  url: 'https://remote/?id=b4eb6a42-a9e9-4702-8466-1228f6b31afa&uriHandle=http://127.0.0.1:8888/lab',
  serverId: '21b39e5db145d6ed0196da211c0cfcf52d09cadc8a485b4f914a13b119541613',
  originalError: mo [Error]: Server URL cannot be used. Did you uninstall an extension that provided a Jupyter server connection?
      at So.getJupyterServerUri (~/.vscode/extensions/ms-toolsai.jupyter-2023.7.1001901100-darwin-arm64/out/extension.node.js:2:1706960)
      at So.validateRemoteUriOld (~/.vscode/extensions/ms-toolsai.jupyter-2023.7.1001901100-darwin-arm64/out/extension.node.js:2:1705859)
      at Eo.addJupyterServerOld (~/.vscode/extensions/ms-toolsai.jupyter-2023.7.1001901100-darwin-arm64/out/extension.node.js:2:1711023)
      at ~/.vscode/extensions/ms-toolsai.jupyter-2023.7.1001901100-darwin-arm64/out/extension.node.js:2:1741740 {
    category: 'remotejupyterserveruriprovider',
    isJupyterError: true,
    providerId: 'b4eb6a42-a9e9-4702-8466-1228f6b31afa',
    handle: 'http://127.0.0.1:8888/lab',
    originalError: Error: Server URL cannot be used. Did you uninstall an extension that provided a Jupyter server connection?
        at mf.getJupyterServerUri (~/.vscode/extensions/ms-toolsai.jupyter-2023.7.1001901100-darwin-arm64/out/extension.node.js:17:103132)
        at So.getJupyterServerUri (~/.vscode/extensions/ms-toolsai.jupyter-2023.7.1001901100-darwin-arm64/out/extension.node.js:2:1706810)
        at So.validateRemoteUriOld (~/.vscode/extensions/ms-toolsai.jupyter-2023.7.1001901100-darwin-arm64/out/extension.node.js:2:1705859),
    serverId: '21b39e5db145d6ed0196da211c0cfcf52d09cadc8a485b4f914a13b119541613'
  },
  baseUrl: 'https://remote/'
}

Any suggestions or further debug information that would help you?

I ... also created a sample extension that uses this api

It'd be super useful to see that code and be sure I'm using the API as expected! 😄

DonJayamanne · 2023-07-09T11:07:34Z

I’m sorry you are running into this
I have a sample extension that I created to use this api, will publish that so you can use that for reference purposes
Leaving this issue open for now

brandonjbjelland · 2023-07-10T03:00:40Z

Thank you @DonJayamanne , your responsiveness is most appreciated! 🥇

DonJayamanne · 2023-07-18T20:56:43Z

Just and update for everyone here, we're working on stabilizing the API in the current iteration
#13894
This iteration we will spend time reviewing the API

I will post the sample later once we have a stable API, this prevents issues with users using an old API and then having to switch again to another API.
I'm sorry for the delay in getting this addressed and thank you for your patience

DonJayamanne · 2023-09-02T12:46:05Z

@brandonjbjelland are you still running into this issue with the current API.
FYI, #13894 should end up getting completed next month, i.e. the API should be ready for use then.
YOu are welcome to use it now and provide feedback, though I haven't got a sample for anyone yet.

brandonjbjelland · 2023-09-02T15:34:44Z

I'd put the component that depended on this functionality on hold until hearing back. I'll dig back in within a week or so. Thanks @DonJayamanne !

DonJayamanne · 2023-10-01T23:48:47Z

A new API has been published and this new API will be available for use in the next version of the Jupyter exetnsion (to be published sometime this week).

Here's a sample https://github.com/microsoft/vscode-extension-samples/tree/main/jupyter-server-provider-sample
The JupyterHub extension also makes use of this same API https://github.com/microsoft/vscode-jupyter-hub

DonJayamanne · 2023-10-01T23:49:43Z

You can try this new API in vscode insiders + pre-relese version of the Jupyter extension,
optionally please wait for the official release

Do let me know how things go,
If you run into issue, please file a new issue.

mtreca · 2023-10-12T12:53:13Z

Hi again! Thanks a lot for working on this feature and providing an example use of your new API. Following your example works really nicely and the ability to add custom functions is also appreciated. I managed to get the new registration API working pretty quickly.
Quick note / nitpick: I do get greeted by a JupyterHub prompt when the new version of the Jupyter extension starts, although I do not intend to use it nor are any notebooks open (see screenshot). Is there any way to disable this? Thanks!

DonJayamanne · 2023-10-12T13:22:07Z

This happens if you had connected to Jupyter hub from within the Jupyter extension in the past;
You can go into the list of Jupyter servers and remove it

Open a notebook
Go into the kernel picker
Select Select Another Kernel
Select Existing Jupyter Servers
Remove the server that you think contributes this prompt,

We have created a new JupyterHub extension that would make this unnecessary going forward
My suggestion is to just delete the servers and add them again

DonJayamanne · 2023-10-12T13:22:28Z

Hi again! Thanks a lot for working on this feature and providing an example use of your new API. Following your example works really nicely and the ability to add custom functions is also appreciated. I managed to get the new registration API working pretty quickly.
Quick note / nitpick: I do get greeted by a JupyterHub pro

Thats great new, please do let me know if you run into any issues.

mtreca · 2023-10-12T16:41:31Z

That fixed it, thanks again!

mtreca added the bug Issue identified by VS Code Team member as probable bug label Mar 20, 2023

github-actions bot assigned amunger Mar 20, 2023

amunger assigned DonJayamanne and unassigned amunger Mar 20, 2023

DonJayamanne added info-needed Issue requires more information from poster notebook-remote Applies to remote Jupyter Servers labels Mar 30, 2023

DonJayamanne mentioned this issue Jun 2, 2023

Explore how to simplify the code for managing connections to local and remote Jupyter Servers #12832

Closed

DonJayamanne closed this as completed Jun 23, 2023

DonJayamanne reopened this Jul 9, 2023

DonJayamanne closed this as completed Oct 1, 2023

github-actions bot locked as resolved and limited conversation to collaborators Nov 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jupyter cannot connect to a remote server using a custom ServerPicker #13112

Jupyter cannot connect to a remote server using a custom ServerPicker #13112

mtreca commented Mar 20, 2023

mtreca commented Mar 20, 2023

DonJayamanne commented Mar 30, 2023

mtreca commented Apr 3, 2023

DonJayamanne commented Apr 5, 2023

mtreca commented Apr 6, 2023

DonJayamanne commented Apr 17, 2023

mtreca commented May 15, 2023

DonJayamanne commented May 15, 2023 •

edited

Loading

mtreca commented May 15, 2023 •

edited

Loading

DonJayamanne commented May 15, 2023

DonJayamanne commented Jun 2, 2023

DonJayamanne commented Jun 23, 2023

brandonjbjelland commented Jul 9, 2023 •

edited

Loading

DonJayamanne commented Jul 9, 2023

brandonjbjelland commented Jul 10, 2023

DonJayamanne commented Jul 18, 2023 •

edited

Loading

DonJayamanne commented Sep 2, 2023

brandonjbjelland commented Sep 2, 2023

DonJayamanne commented Oct 1, 2023

DonJayamanne commented Oct 1, 2023

mtreca commented Oct 12, 2023 •

edited

Loading

DonJayamanne commented Oct 12, 2023

DonJayamanne commented Oct 12, 2023

mtreca commented Oct 12, 2023

Jupyter cannot connect to a remote server using a custom ServerPicker #13112

Jupyter cannot connect to a remote server using a custom ServerPicker #13112

Comments

mtreca commented Mar 20, 2023

Applies To

What happened?

VS Code Version

Jupyter Extension Version

Jupyter logs

mtreca commented Mar 20, 2023

DonJayamanne commented Mar 30, 2023

mtreca commented Apr 3, 2023

DonJayamanne commented Apr 5, 2023

mtreca commented Apr 6, 2023

DonJayamanne commented Apr 17, 2023

mtreca commented May 15, 2023

DonJayamanne commented May 15, 2023 • edited Loading

mtreca commented May 15, 2023 • edited Loading

DonJayamanne commented May 15, 2023

DonJayamanne commented Jun 2, 2023

DonJayamanne commented Jun 23, 2023

brandonjbjelland commented Jul 9, 2023 • edited Loading

DonJayamanne commented Jul 9, 2023

brandonjbjelland commented Jul 10, 2023

DonJayamanne commented Jul 18, 2023 • edited Loading

DonJayamanne commented Sep 2, 2023

brandonjbjelland commented Sep 2, 2023

DonJayamanne commented Oct 1, 2023

DonJayamanne commented Oct 1, 2023

mtreca commented Oct 12, 2023 • edited Loading

DonJayamanne commented Oct 12, 2023

DonJayamanne commented Oct 12, 2023

mtreca commented Oct 12, 2023

DonJayamanne commented May 15, 2023 •

edited

Loading

mtreca commented May 15, 2023 •

edited

Loading

brandonjbjelland commented Jul 9, 2023 •

edited

Loading

DonJayamanne commented Jul 18, 2023 •

edited

Loading

mtreca commented Oct 12, 2023 •

edited

Loading