Improve the wrapper kernel experience for downstreams

[bollwyvl]

Problem

Today, it's possible to reuse the kernel labextension to get up to something that runs on top of all of the pyodide machinery, but it's still pretty non-trivial.

Upstream, ipykernel provides KernelBase, which makes it quite easy to depend on ipykernel, and create another, separately-branded kernel which only has to override the parts you want (or the full IPyKernel).

Proposed Solution

In this repo, or a new one, add, document, and test a cookiecutter or copier that makes it easier to build new, PyPI- and npm-distributable kernels that depend on jupyterlite-pyodide-kernel.

Additional context

The two in-the-wild downstream kernels I've worked on are

• pidgy (lead by @tonyfast)
• robotkernel (lead by @datakurre)

Today, they still pretty much need to be hard forks, but with the split from jupyterlite(_core), it should be much easier to do this simply now.

If in a new repo, it might be a bit simpler, but runs the risk of getting out-of-date.

If this was included in-tree, it could be tested (up to CI, which is tricky without act-in-GHA, which is kinda nasty). copier makes it a good deal easier to properly package these things, and could be made to extend the as-shipped CLI (behind an [extra], i guess).

A Future Family Tree of Kernels

flowchart LR

classDef abstract stroke-dasharray:4px

kernel:::abstract --> worker:::abstract --> js --> pyodide

js --> p5
worker --> xeus-*
kernel --> webr
pyodide --> pidgy
pyodide --> robot
pyodide --> lite-builder[#33: lite-in-pyodide]
kernel --> remote:::abstract
remote --> binder

subgraph third-party
    pidgy
    robot
    webr
    remote
    binder
end

subgraph legend
    abstract[abstract\nimplementation]:::abstract
end

Loading

[tonyfast]

stoked to see work on this! thanks!

so pidgy is basically a few ipython extensions in a trench coat. it ships an IPyKernel & kernelspec that only invokes a magic https://github.com/deathbeds/pidgy/blob/main/src/pidgy/kernel/__init__.py#L15. it builds completely on the IPython.InteractiveShell.

my hope for my jupyterlite is that the pidgy kernel is your default python kernel. they would then effectively be programming in markdown.

an option i have, but havent tried, is to put the pidgy configuration information the in the ipython_config.py. in this case, i'd lose some branding with the kernel logo and familiarity across lite and lab.

[bollwyvl]

kernel is your default python kernel

Not sure what "your" means in this case... but can certainly image for a site that only has pidgy content in it... "plain old python" still doesn't quite work they way one would expect.

pidgy configuration information the in the ipython_config.py

it... probably wouldn't work today. as mentioned above, we kinda need to make sure python works before overloading with uploaded content... one thing that is in the kernelpec "schema" is environment variables, but again, it might be too early in the lifecycle.

So, coupling that, plus branding, help links, and, eventually, syntax highlighting, and downstream installability, it seems there's a fair amount of value in shipping a custom kernel that couldn't be config only.

invokes a magic

Right, for my reference, as I haven't looked in a while: the critical chunk of loadPyodideAndPackages:

    import pyolite
    await piplite.install(['pidgy'])
    import IPython
    __import__("pidgy").load_ipython_extension(IPython.get_ipython())

We've already made that part much more granular: instead of loadPyodideAndPackages, we now have:

• initRuntime
• initFilesystem
• initPackageManager
• initKernel
  • likely, pidgy will only have to override this,
    • super.initKernel will get it up to a running PyodideKernel with all patches applied
    • and chdir'd into the correct folder for shared contents to work
• initGlobals

So the worker.ts should be pretty short indeed, and seldom need many changes.

It looks like kernel.js will be a single class that declares initWorker

And finally, we'll probably need to do a fair amount of work on the extension index.ts side, much of which (like config parsing, etc) should probably be lifted to a IPyodideKernelManager.

[tonyfast]

Not sure what "your" means in this case... but can certainly image for a site that only has pidgy content in it... "plain old python" still doesn't quite work they way one would expect.

that is true. i think most folks would have a direct link (eg https://deathbeds.github.io/pidgy/branch/feb-demo/run/lab/?path=basics.ipynb). in that case, the kernel is defined in the notebook format and pidgy will work.

otherwise, i'm happy with the current launcher (see figure) that will open an IPykernel that invokes the pidgy extensions, or using the pidgy kernel directly. basically, it'll do what the code snippet you shared does.

a nice to have, this may be too far, is a lite repl with the pidgy kernel. could these changes help pidgy have a demo with:

<iframe
  src="https://deathbeds.github.io/pidgy/repl/index.html?kernel=pidgy"
  width="100%"
  height="100%"
></iframe>

[bollwyvl]

is a lite repl with the pidgy kernel

Right: this is the advantage to a "real" kernelspec, etc. as this should already work, provided a new-enough jupyterlite that ships the REPL app.

Pursuing this line of thought: really anything that changes the mime type from text/x-ipython (as defined by the standard IPython globals like display and %%magics) should probably be another mimetype, and therefore another kernel.

Indeed, pyodide-kernel doesn't implement all of the magics... but then neither does ipykernel-on-windows, or ipykernel-deep-in-a-container.

For a truly lightweight, no-webpack kernel, I could imagine a "Put a single JSON in {lite_dir}/kernelspecs/{my-kernel}/kernel.json".

We'd have the "well-known" (but not schema-described) kernelspec to play with, as well as the (also not schema-constrained)
kernel_info_reply message:

# kernlspecs/pidgy/kernel.json
{
  "name": "pidgy",
  "display_name": "Pidgy (Pyodide)",
  "language": "markdown",
  "mimetype": "text/pidgy+markdown",
  "argv": [],
  "resource_dir": {
    "logo-32x32": "data:...",
    "logo-64x64": "data:..."
  },
  "env": {
    "PIDGY_START_KERNEL_ON_IMPORT": "1"
  },
  "metadata": {
    "@jupyterlite/pyodide-kernel-extension": {
      "kernelImportPackage": "pidgy",
      "kernelImportName": "pidgy.lite",
      "kernelInfoReply": {
        "language_info": {},
        "debugger": false,
        "help_links": {}
      }
    }
  }
}

The above-mentioned future IPyodideKernelManager could then find kernelspecs that included the well-known metadata name, and launch them on behalf of the user.

Of course, for downstreams, this could still be made packageable for PyPI with a minimal, flat repo structure which would preserve all the lovely aspects of a real package manager and entry_points:

pyproject.toml
LICENSE
README.md
kernel.json
jupyterlite_pidgy_kernel.py

... which could also choose to handle resolving piplite Warehouse index, repodata, etc.

[tonyfast]

is a lite repl with the pidgy kernel

Right: this is the advantage to a "real" kernelspec, etc. as this should already work, provided a new-enough jupyterlite that ships the REPL app.

super cool to know. pidgy has a kernelspec, so i guess i should be able to try that out now. thanks fo the heads up.

Pursuing this line of thought: really anything that changes the mime type from text/x-ipython (as defined by the standard IPython globals like display and %%magics) should probably be another mimetype, and therefore another kernel.

pidgy transpiles to IPython, so i chose text/x-ipython, the target language, as the mimetype. the pidgy isn't as much of kernel as it is an enhanced shell. so i thought it made sense, and text/markdown feels weird if i have markdown cells already.

i guess without a schema it is hard to know what mimetype means.

For a truly lightweight, no-webpack kernel, I could imagine a "Put a single JSON in {lite_dir}/kernelspecs/{my-kernel}/kernel.json".

jupyter packaging has a declarative way of sharing the kernel directory.

[tool.hatch.build.targets.wheel.shared-data]
"src/kernelspec" = "share/jupyter/kernels/pidgy"

would it be possible to export/configure the kernel spec in a pyproject.toml (or with args). something like

[tool.doit.lite]
kernelspecs = ["src/pidgy-lite-spec"]]

We'd have the "well-known" (but not schema-described 😆 ) kernelspec to play with, as well as the (also not schema-constrained 💀) kernel_info_reply message:

# kernlspecs/pidgy/kernel.json
{
...
}

yea i dig this. i could imagine shipping two different kernelspecs, one for lite and for IPython.

for this thin wrapper over pyolite is there a way to include some IPythonic configuration like InteractiveShellApp.extensions: [pidgy]

Of course, for downstreams, this could still be made packageable for PyPI with a minimal, flat repo structure which would preserve all the lovely aspects of a real package manager and entry_points:

pyproject.toml
LICENSE
README.md
kernel.json
jupyterlite_pidgy_kernel.py

... which could also choose to handle resolving piplite Warehouse index, repodata, etc.

seems cool to get there. i think for the pidgy case i'll be most likely to maintain two different kernels.

1. the user facing normal kernel
2. the docs facing lite kernel

so i'll still have the nested directory structure somewhere.

[bollwyvl]

different kernelspecs, one for lite and for IPython.

Oh, indeed. These definitely would not go into a real {x-different-places}/kernels/{foo}. Need to keep the scope real small, and very safe from breaking any other interactive computing experiences...

[tool.hatch.build.targets.wheel.shared-data]

Here and on core, we've explicitly avoided lite changing anything in jupyter --paths (with data-files) for this reason, but labextensions are a big boat to turn, so we get stuff like #32.

[tool.doit.lite]

We have the addon API in place through a PEP621-compliant, versioned entry-point, and it's been pretty stable. Adding more toml until it's really in stdlib, unless traitlets goes there first, is not something i see a lot of benefits heading towards.

InteractiveShellApp.extensions: [pidgy]

Probably not going to go down the traitlets-in-js road.

The intent is to get exactly one PEP423 installable package and one dotted-import-with-sideffects that replaces the current pyodide_kernel, into which the author can put whatever they want. Then, when the tracebacks come in because of upstream version drift, they'll be scoped to the exact line of code that created the problem.

maintain two different kernels.

Right, for the most part that would be the point, where a lite kernel depends on the real kernel. The real kernel can gate things with platform_machine == 'wasm-32', if not covered by the pyodide-kernel patches for ipykernel, etc.