Integration of kernels and extensions with jupyter

[costrouc]

Motivation for this change

this main comment has been edited to reflect progress

Anyone who has used jupyterlab and notebooks has run into the issue of adding kernels and extensions to the UI. This is tricky because often times the kernels run in a different language e.g. julia, haskell, etc. Nix can handle this extremely well and would make jupyter notebook installations much easier. This PR provides intructions on a way to "auto-installkernels and extensions into the nix environment. Any input would be much appreciated. Here is what my research has shown me so far (I dont understand extensions yet wherestatic` directory is put).

Jupyter Kernels

The environment variable JUPYTER_PATH is a list of : separated paths that have a kernels directory. Every directory within contains kernels kernels/<kernel-name>/kernel.json. Here is an example kernel.json.

{
    "argv":["/nix/store/bwfygfcdvis9wd1c1v51xwnwhw1hx0a0-python3-3.6.6/bin/python3.6m","-m","jupyter_c_kernel","-f","{connection_file}"],
    "display_name":"C",
    "language":"c"
}

I would like to automatically add a set of kernels via the JUPYTER_PATH variable to the jupyterlab and notebook packages. Not sure exactly how to do this here is an example of me setting the variable after the fact in a nix-shell. It has been show below how to automatically do this but the issue is that kernels exist for many languages not just python. What I have settled on is that kernel.json and the images should be included at share/jupyter/kernels/<kernel-name>/.

JupyterLab Extensions

Jupyterlab is the future so I have put the most effort into installation instructions. Additionally notebook's installation paths are not as well documented and often times work without any additional steps. Jupyterlab has a command line option --app-dir and environment variable JUPYTERLAB_DIR that control where the extensions are installed usually share/jupyter/lab. These extensions are npm packages that after being combined through a package.json must be "compiled" with webpack to produce compressed static files. The npm package dependencies can easily be in the thousands... thanks Javascript leading to about 100MB - 1GB of dependencies that must be compiled for each additional extension added. This step cannot be pure unless we override the jupyterlab package management and handle all of the npm dependencies. Currently it seems that npm packages are not a good fit for nixpkgs. So I have come up with an unpure but working way to package jupyterlab extensions. Each package that comes with a jupyterlab extension uses passthru = { jupyterlabExtensions = [ .... ]; };. Then we include the following lines

# make temp directory for extensions
TEMPDIR=$(mktemp -d -p /tmp)
mkdir -p $TEMPDIR
# copy base jupyter lab setup
cp -r ${pkgs.python36Packages.jupyterlab}/share/jupyter/lab/* $TEMPDIR
chmod -R 755 $TEMPDIR

# gather unique jupyterlab extensions
# for each extension
#      jupyter labextension install --no-build --app-dir=$TEMPDIR <extension>
${pkgs.stdenv.lib.concatMapStrings
     (s: "jupyter labextension install --no-build --app-dir=$TEMPDIR ${s}; ")
     (pkgs.lib.unique
       ((pkgs.lib.concatMap
           (d: pkgs.lib.attrByPath ["passthru" "jupyterlabExtensions"] [] d)
           buildInputs) ++ additionalExtensions))  }
# finally build jupyterlab npm extensions with webpack
jupyter lab build --app-dir=$TEMPDIR

# start jupyterlab service
jupyter lab --app-dir=$TEMPDIR

Working Example with 20 jupyerlab extension + 4 kernels (c, python, go, ansible)

All that needs to be edited by user is kernels, additionalExtensions, and buildInputs. The rest is automatic and will launch a jupyterlab instance for you.

shell.nix

{ pkgs ? import <nixpkgs> {}, pythonPackages ? pkgs.python36Packages }:

let kernels = [
      pkgs.python36Packages.ansible-kernel
      pythonPackages.jupyter-c-kernel
      pkgs.gophernotes
    ];

    additionalExtensions = [
      "@jupyterlab/toc"
      "@jupyterlab/fasta-extension"
      "@jupyterlab/geojson-extension"
      "@jupyterlab/katex-extension"
      "@jupyterlab/mathjax3-extension"
      "@jupyterlab/plotly-extension"
      "@jupyterlab/vega2-extension"
      "@jupyterlab/vega3-extension"
      "@jupyterlab/xkcd-extension"
      "jupyterlab-drawio"
      "@jupyterlab/hub-extension"
      "jupyterlab_bokeh"
    ];
in
pkgs.mkShell rec {
  buildInputs = [
    ### Base Packages
    pythonPackages.jupyterlab pkgs.nodejs

    ### Extensions
    pythonPackages.ipywidgets
    pythonPackages.ipydatawidgets
    pythonPackages.ipywebrtc
    pythonPackages.pythreejs
    pythonPackages.ipyvolume
    pythonPackages.jupyterlab-git
    pythonPackages.jupyterlab-latex
    pythonPackages.ipyleaflet
    pythonPackages.ipympl
  ] ++ kernels;

  shellHook = ''
    TEMPDIR=$(mktemp -d -p /tmp)
    mkdir -p $TEMPDIR
    cp -r ${pkgs.python36Packages.jupyterlab}/share/jupyter/lab/* $TEMPDIR
    chmod -R 755 $TEMPDIR
    echo "$TEMPDIR is the app directory"

    # kernels
    export JUPYTER_PATH="${pkgs.lib.concatMapStringsSep ":" (p: "${p}/share/jupyter/") kernels}"

# labextensions
${pkgs.stdenv.lib.concatMapStrings
     (s: "jupyter labextension install --no-build --app-dir=$TEMPDIR ${s}; ")
     (pkgs.lib.unique
       ((pkgs.lib.concatMap
           (d: pkgs.lib.attrByPath ["passthru" "jupyterlabExtensions"] [] d)
           buildInputs) ++ additionalExtensions))  }
jupyter lab build --app-dir=$TEMPDIR

# start jupyterlab
jupyter lab --app-dir=$TEMPDIR
  '';

}

Things done

Not all of these kernels work currently (gnuplot, zeus-cling)

Adding four kernels:

• jupyter_c_kernel (C)
• ansible_kernel (ansible)
• gophernotes (Go)

Want to add extension support for python/jupyterlab extension:

• ipywebrtc (webrtc in jupyterlab)
• pythreejs (three.js in jupyterlab)
• ipyvolume (3d volumetric rendering in jupyterlab(
• jupyterlab-latex (latex document editing in jupyterlab)
• ipywidgets (widgets alot of packages depend on this one)
• jupyterlab-git (git support)
• ipyleaflet (leaflet extension)
• ipympl (matplotlib extension)
• ipydatawidgets (A set of widgets to help facilitate reuse of large datasets)

This will require input on how to properly include extensions and kernels within jupyter derivations.

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Fits CONTRIBUTING.md.

---

[FRidh]

You have seen the top-level jupyter and jupyter-kernel attributes that are used by the jupyter NixOS module?

[costrouc]

I had not seen that before and thank you for pointing that out! However, I see some issues with declaring this in nixos modules in that only nixos can use that. Ideally I would like to say "start a jupyter notebook with these 3 kernels (julia, C, and python) and these 3 extensions (ipywidget, ipyvolume, jupyterlab-drawio)" for example in any linux distribution. This currently is not possible and a pain point I have had with jupyter. Welcome to any ideas.

I have learned that the JUPYTER_PATH environment variable can point to each kenel json definition and JUPYTERLAB_DIR points to the javascript extensions static files. This will require a meta package or something to collect all of the extensions in one location and set JUPYTER_PATH correctly.

[FRidh]

I had not seen that before and thank you for pointing that out! However, I see some issues with declaring this in nixos modules in that only nixos can use that. Ideally I would like to say "start a jupyter notebook with these 3 kernels (julia, C, and python) and these 3 extensions (ipywidget, ipyvolume, jupyterlab-drawio)" for example in any linux distribution. This currently is not possible and a pain point I have had with jupyter. Welcome to any ideas.

The NixOS module only provides an interface and a service, but the actual logic for running a notebook with specified kernels is implemented in the two top-level attributes I referred to.

[FRidh]

Original PR #33673

[costrouc]

Thank you and I should read more carefully the expressions. next time. I will try it out with the new kernels with the derivations and also would be interested in adding extensions support to the nixpkgs/pkgs/applications/editors/jupyter/default.nix.

[FRidh]

Should we not somewhere hardcode a path to gnuplot?

[costrouc]

Yes we should. Currently I am trying to get the gnuplot_kernel to work. It freezes on me for an unknown reason.

[veprbl]

I would like to automatically add a set of kernels via the JUPYTER_PATH variable to the jupyterlab and notebook packages.

This should be easy to achieve by adding a setupHook to jupyterlab.
Examples are

• cmake:

  nixpkgs/pkgs/development/tools/build-managers/cmake/setup-hook.sh

  Lines 1 to 3 in a6809c0

  	addCMakeParams() {
  	addToSearchPath CMAKE_PREFIX_PATH $1
  	}

  
  

  nixpkgs/pkgs/development/tools/build-managers/cmake/setup-hook.sh

  Line 94 in a6809c0

  addEnvHooks "$targetOffset" addCMakeParams

• python:
  

  nixpkgs/pkgs/development/interpreters/python/setup-hook.sh

  Lines 1 to 3 in a6809c0

  	addPythonPath() {
  	addToSearchPathWithCustomDelimiter : PYTHONPATH $1/@sitePackages@
  	}

  
  

  nixpkgs/pkgs/development/interpreters/python/setup-hook.sh

  Line 15 in a6809c0

  addEnvHooks "$hostOffset" addPythonPath

[costrouc]

ansible-kernel depends on #49764 being move to python-modules for python 2.7 and 3+ support.

Here is the error I get (due to using python2.7 for 3 packages)

  File "/nix/store/bwfygfcdvis9wd1c1v51xwnwhw1hx0a0-python3-3.6.6/lib/python3.6/re.py", line 142, in <module>
    class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'

[costrouc]

@veprbl I am not very knowledgeable on how to properly do this. Could you give some further advice? Additionally I should state that jupyterlab kernels are not necessarily for python only. For example this PR adds support for Go and C.

[costrouc]

I would like some feedback of using passthru. Otherwise I do not see anything controversial about this PR.

...
passthru ={
   jupyterlabExtensions = [ "jupyterlab-datawidgets" "@jupyter-widgets/jupyterlab-manager
" ];
};
...

Is the correct way to pass that the package requires an npm dependency that cannot be included in the build requirements. Is this a good way to pass these dependencies? This expression is then used to select the necessary extensions to install (additionallExtensions is a list of strings of extensions that do not require a python package).

(pkgs.lib.unique
       ((pkgs.lib.concatMap
           (d: pkgs.lib.attrByPath ["passthru" "jupyterlabExtensions"] [] d)
           buildInputs) ++ additionalExtensions))

[costrouc]

I was unaware of link and how it works but if it were able to isolate each extension this that might just solve it. Looks promising. I would really appreciate help in this area but I will be looking into it as well. Anything we can do to make the extensions a derivation is perfect

edit: @MMesch after looking more carefully https://github.com/jupyterlab/jupyterlab/blob/master/jupyterlab/commands.py#L659-L684 it looks link jupyter labextension link <path to extension>. Would work. I would like to work on this but I think that it deserves another PR. As this PRs goal was to add some of the packages. I will open another PR that we should discus in.

This PR is still ready for merge.

[MMesch]

agree. I'll help you with the other PR ...

[GrahamcOfBorg]

Success on aarch64-linux (full log)

Attempted: gophernotes

Partial log (click to expand)

}
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin
shrinking /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin/bin/gophernotes
strip is /nix/store/6dpnd5aniypn8124mmy8f88s4mq2zl07-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin/bin
patching script interpreter paths in /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin
checking for references to /build in /nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin...
strip is /nix/store/6dpnd5aniypn8124mmy8f88s4mq2zl07-binutils-2.30/bin/strip
/nix/store/ir0wnj34nnxjwr2j4n8d4ncvn4y6yn2c-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin

[GrahamcOfBorg]

Success on x86_64-linux (full log)

Attempted: gophernotes

Partial log (click to expand)

}
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin
shrinking /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin/bin/gophernotes
strip is /nix/store/rpbg8gmqxhz8g61p1plz5d2srs84pvmv-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin/bin
patching script interpreter paths in /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin
checking for references to /build in /nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin...
strip is /nix/store/rpbg8gmqxhz8g61p1plz5d2srs84pvmv-binutils-2.30/bin/strip
/nix/store/lwkcg6bzksgpl9fnqx7sw97b8s1kblk8-gophernotes-unstable-b58dd906c43c1ef0b5976b150872c79f82daafcd-bin

[costrouc]

I will also be closing this along with two other PRs to combine with this one. #49807, #49721, and #50858.

This will lead to a PR with better python 3 support for jupyter (newest releases are python 3.5+ only). Addition of 3 kernels (c, go, ansible) and 20 jupyterlab extensions. Also will have an editor that has a clean way to add kernels and extensions. This will be a big PR but I am finding that they all depend on each other.

[ariutta]

This will lead to a PR...

Did the new PR land?

Related: it appears similar functionality can be found in the JupyterWith repo.