Component: IPyWidgets
IPyWidgets (both the default and custom widgets) are supported.
TOC
IPyWidgets in general is a very complex piece of functionality hence requires a lot of moving parts and work to get it working in VS Code. Following is a summary of what's involved in adding support for IPyWidgets in VS Code:
- The sample project https://github.com/jupyter-widgets/ipywidgets/blob/master/examples/web3 is used as a basis for hosting Widgets in VS Code. The source for that repo can be found here https://github.com/microsoft/vscode-jupyter-ipywidgets. More information about that repo can be found in the corresponding README.md file.
- The
Widget ManagerfromJupyter Labis hosted in the UI layer (renderer). - Introduce a kernel in the UI layer (renderer).
Widget Managers are responsible for rendering widgets, and they require access to the kernel (
Kernel.IKernelConnection). In VS Code the real kernel (Kernel.IKernelConnection) is hosted in the extension host process which does not have access to the UI layer (dom/renderer). Thus, a kernel needs to be re-introduced in the UI layer. - Synchronize kernel messages between the extension host & renderer process.
Essentially the kernel in the UI layer is no different from the kernel in the extension host.
Widgets can use this same kernel (
Kernel.IKernelConnection) to execute code against the backend kernel and use all of its methods. - Base Widgets sources are bundled and shipped with the VS Code extension. All built-in Widgets such as sliders, textboxes, and the like work out of the box.
- The UI layer (renderer) is sandboxed and does not allow execution and loading of resources from any arbitrary resource. All 3rd party widget scripts are downloaded and saved into a specific location and then loaded into the UI layer (renderer).
- 3rd party scripts are loaded from the following 3 sources:
- CDN (jsdelivr, unpkg)
- Local file system
- Remote Jupyter Server
- Widget Manager - This is the Jupyter Lab Widget manager found in the following package @jupyter-widgets/jupyterlab-manager. We have our own version we load from here.
- Jupyter Lab Kernel - This is the Kernel connection interface found here Kernel.IKernelConnection in @jupyter-widgets/jupyterlab-manager
- Jupyter Notebook - This is the Notebook (loosely refers to the UI component & all of the corresponding backend components) in Jupyter Notebooks associated with a single notebook file.
- Proxy Web Socket - Getting access to the Kernel connection interface Kernel.IKernelConnection from @jupyter-widgets/jupyterlab-manager requires a WebSocket class. As VS Code doesn't always start Jupyter Servers, we end up providing an object that looks like a Web Socket object (hence the term proxy web socket). VS Code starts the kernel process and communicates with it using zmq using the
encodeanddecodemethods found in @nteract/messaging/lib/wire-protocol
Why do we need to do this? IPyWidgets require talking to a kernel within the UI. Since our kernel actually resides on the Extension side, we need to forward messages from our real kernel to a dummy kernel on the UI side:
We achieve this syncing by replacing the websocket in each kernel and making each one wait for the other.
Under the covers this is done with two classes and bunch of messages between them.
On the UI side there's a file here:
This contains a ProxyKernel that listens for messages from the extension. It forwards these messages onto a real kernel implementation, making sure its state matches the state of the kernel on the extension side.
On the Extension side there's two files:
The IPyWidgetMessageDispatcher takes messages from the extension's kernel and forwards them onto the UI side. It accomplishes this by adding a couple of hooks into the JupyterWebSocket.
- Send Hook - this hook listens to when the extension is sending a message to the real kernel. This hook allows us to setup the kernel on the UI side to have the same set of futures that we have on the extension side. This is necessary so that message hooking on the UI side finds those futures. Message hooks are used in a number of ipyWidgets to control data and where it's displayed.
- Receive Hook - this hook listens to messages that come back from the real kernel but before the extension receives them. We send each message to the UI side so that any listeners for messages in the UI will get them before we use them in the extension. This is necessary so that events in the UI (which can generate more requests) happen before the extension finishes a request.
This sequence diagram outlines how that works:
The hooks allow the proxy kernel to handle messages before the CellExecution layer does. This allows the UI to register comm targets and other hooks prior to any output messages coming through.
Besides kernel syncing, IPyWidgets requires loading JS files into the UI that are not shipped with our extension. We handle this with some special hooks when we create our UI side WidgetManager.
These hooks essentially do the following:
- Try loading the widget source from memory. This works for the standard jupyter widgets.
- Try loading the widget source from a known CDN (as of right now we're using https://www.jsdelivr.com/ & https://unpkg.com) if the user is okay with that.
- Try loading the widget from where jupyter finds it as best we can (this has problems with dependencies not getting loaded)
- Try loading the widget from remote Jupyter Notebook (/nbextensions//index.js). Though this doesn't work with Jupyter Labs.
Loading Widgets and registering with requirejs
Diagram
# To recreate, open an issue and paste this code into the issue. Wikis don't support mermaid yet but issues do.
```mermaid
sequenceDiagram
participant WidgetManager
participant Notebook (WebView)
Notebook (WebView)->>WidgetManager: Render Widget
WidgetManager->>Notebook (WebView): Request Widget Source
Notebook (WebView)->>IPyWidgetScriptSource: Request Widget Source
IPyWidgetScriptSource<<->>ScriptUriConverter: Convert to Webview URI
IPyWidgetScriptSource->>Notebook (WebView): Return converted Widget Source
loop RequireJs
Notebook (WebView)->>Notebook (WebView): Register Scripts with requirejs
end
Notebook (WebView)->>WidgetManager: Return Widget Source
loop Load Widget
WidgetManager->>WidgetManager: Load using requirejs
end
Note right of WidgetManager: If widget is not
registered, then
loading will fail,
and error displayed.
```
Loading from various sources
Diagram
```mermaid
sequenceDiagram
participant IPyWidgetScriptSource
participant CDN
participant LocalFS
participant RemoteJupyter
IPyWidgetScriptSource->>CDN: Request Widget Source
loop Look for Widget
CDN->>CDN: jsDelivr.com
CDN->>CDN: unpkg.com
end
CDN->>IPyWidgetScriptSource: Return Widget Source
Note right of IPyWidgetScriptSource: If we got everything
return to WebView.
Else try others.
IPyWidgetScriptSource->>LocalFS: Request Widget Source
loop Look for Widget
LocalFS->>LocalFS: Look for widgets in folder.
end
Note right of LocalFS: If widgest are found
copy .js to
/tmp
folder. Requirement
of WebView
LocalFS->>IPyWidgetScriptSource: Return Widget Source
Note right of IPyWidgetScriptSource: If we got everything
return to WebView.
Else try others.
IPyWidgetScriptSource->>RemoteJupyter: Request Widget Source
loop Look for Widget
RemoteJupyter->>RemoteJupyter: Look for widgets in
/nbextension//index.js
end
RemoteJupyter->>IPyWidgetScriptSource: Return Widget Source
```
Notes:
- The widget sources located from (CDN/localFS/remote jupyter) needs to be registered with
requirejs.- IPyWidgets uses
requirejsto locate/load widget sources.
- IPyWidgets uses
- Loading from local file system (widgets located in
<python sysprefix>/share/jupyter/nbextensions/<widget>) does not always work due to the fact that:- Widgets might have other dependencies defined in
extension.js. (extension.jsis the main entrypoint for the widget UI in jupyter. It usually has anindex.jsas well) - Widgets might have style sheets embedded in
extension.jsfile. - Loading
extension.jsis not possible due to the following reasons:- It is specific to jupyter lab/notebooks.
- These files can add module dependencies only available in jupyter lab/notebooks
require(["jupyterlab/ui", etc]) - These files expect DOM elements to be available that are available in jupyter lab/notebooks.
- Some widgets do not have plain js in
extension.js, they have js as strings that gets evaluated.
- Widgets might have other dependencies defined in
- Loading from remote Jupyter server doesn't work in the case of Jupyter Labs.
- Jupyter labs creates new bundles as and when widgest (python packages) are installed.
To add more confusion to how this all works, there are actually 2 WidgetManagers.
What's the difference?
The Extension's WidgetManager is a wrapper around the npm WidgetManager. It provides the messaging infrastructure in order to forward messages appropriately from the inner WidgetManager to the Extension host.
Most of its functionality is then forwarded onto the inner Widget Manager.
This WidgetManager actually handles the rendering of widgets. It's a subclass of @jupyter-widgets/jupyterlab-manager that allows us to hook loading of the widget class objects. This is where our custom script loading is injected.
If for some reason ipywidgets isn't working, the way to debug this is to :
- Setup logpoints in the extension side in either jupyterNotebook.ts onHandleIOPub or in the send/receive hooks setup in ipyWidgetDispatcher. (The ipyWidgetDispatcher hooks are at a lower level)
- Setup similar logpoints in the kernel.ts file on the UI side
- Run your scenario
- Compare the messages each side gets and the order.
Alternatively you might also run 'jupyter lab' or 'jupyter notebook' with the same code and see what the difference is their implementation. Generally you can set breakpoints in chrome in their kernel implementation too.
Due to the complex nature of IPyWidgets, writing highlevel unit tests isn't sufficient. We have end to end tests that test the following functionality:
- Rendering widgets
- Interacting with interactive widgets (verifying updates to the UI)
- Rendering & interaction of complex 3rd party widgets such as matplotlib, ipyvolume, ipysheets, etc.
Note: A large number of the tests haven't been migrated to support the new Notebook UI in vscode. This is currently a WIP.
Outdated
There are some basic parts involved in IPyWidgets support
On the extension side, there's a number of classes involved in talking to jupyter:
IPyWidgets is supported by rewiring the WebSocket in the picture above like so:
This websocket rerouting allows us to intercept both inbound and outbound messages to the kernel.