Rethinking the notebook cells weekly meeting

[tonyfast]

The Rethinking the Notebook Cells spun out of the nbformat workshop. During these weekly meeting we are discussing Jupyter Enhancement Proposals to submit that update the nbformat schema.

The meeting is currently scheduled for Tuesday 8AM Pacific Time and they are temporarily hosted on Zoom.

Meeting Notes

• March 7, 2023

[tonyfast]

March 7th, 2023

Name	Affiliation	GitHub	Favorite Schema Key
tonyfast		@tonyfast	properties
fcollonval	QuantStack	@fcollonval
Angus Hollands	Princeton University	@agoose77	😄
Rowan	Curvenote / ExecutableBooks	@rowanc1
Nick Bollweg	Georgia Tech	@bollwyvl

Agenda

first meeting of the notebook cells schema group outside of the nbformat workshop.

• Meeting logistics

  • use hackmd for notes
  • use google meet for video because jovyan is crowded
    • ⚠️ this account is limited to our hour so we have a real hard stop.
  • the textual format team is working in other channels to submit their jeps.

• Research

  • which schema draft are we using?
  • should only be adding cells and metadata
  • how is this file format going to be reused?
  • introduction of notebook mimetype. how do we carry around the mimebundle across documents and use that information.
  • how do we use attachments better? where do attachments belong?
    • could attachments just be a cell? hold the whole mimebundle
  • Distinguish between saving and reading - always uphold $schema, but not extraSchemas?
  • Should extraSchemas allow embedding schema?
  • Do we include @context?
    • Probably a separate JEP because the value proposition is a different learning curve.

• Interests

  • Rowan - standardization of notebooks in scientific publishing. dealing with authorship, title, subtitles, scholarship.

to do

• follow up with JEP shepherd
• post an issue to the team compass
• add the event to the community calendar

$vocabulary

• does this provide the convention (and therefore the tools) we need

https://gregsdennis.github.io/Manatee.Json/usage/schema/vocabs.html

"$vocabulary": {
    "https://json-schema.org/draft/2019-WIP/vocab/core": true,              // 2
    "https://json-schema.org/draft/2019-WIP/vocab/applicator": true,
    "https://json-schema.org/draft/2019-WIP/vocab/validation": true,
    "https://json-schema.org/draft/2019-WIP/vocab/meta-data": true,
    "https://json-schema.org/draft/2019-WIP/vocab/format": true,
    "https://json-schema.org/draft/2019-WIP/vocab/content": true,
    "https://myserver.net/my-vocab": true
  },

• Angus' understanding of vocabulary¹:
  • Vocabularies allow meta schemas to define custom keywords, e.g. a units keyword that adds units to an integer:

     {
         "type": "number",
         "units": "kg/s"
     }

  • One must create a new metaschema that defines these vocabularies, and copies the meta-schema that it "inherits" from (or use allOf?)
  • The $vocabulary section of a metaschema lists the vocabularies, and a boolean flag of whether they constitute a failure if they cannot be located. The units keyword above does not affect validation, so it can safely be ignored if the validator cannot find the URI (it's metadata). Other keyword schemas might not be so permissive:

     {
         "type": "number",
         "isEven": True
     }

    This schema would incorrectly validate documents with odd integers, but the essence is still upheld. A keyword that changed the "type" would not be ignorable if the validator is at-all to be useful.

    Modern JSON Schema introduces vocabularies, which allow you to define a group of keywords and identify them with a URI. Schema authors can then use that URI to tell implementations that the need to support the vocabulary in order to use the schema. If they can't, instead of failing validation, the implementation refuses to run the schema and indicates which vocabularies it doesn't understand.²

  • i.e. $vocabulary solves the problem of "is this failure a 'unrecoverable' error?".
  • We could use this to introduce a top-level extraSchemas field (?)
    • Crucially, it means that validators that don't understand what to do with extraSchemas don't try and validate the document.

Challenges

flowchart
    mimetypes --> IANA
    multiple_schema[multiple schema]
    validation --> validation_report[validation report]
    JEP --> end_meeting[end this meeting]

Loading

• Extra schemas: Failure modes
  • How can our approaches fail?
    • two conflicting extra schemas
  • How can users save themselves if we break stuff? what happens code/clients break?

Reference

• Notes from the workshop: https://docs.google.com/document/d/1DMMUOYEhFxoAEKITOrCUK9x0vkTy68mfZ9clof3UrMc/edit#heading=h.2q8mfjoa85k9

JEP Drafts

• $schema - https://hackmd.io/@u1M5398WTl6qOUg8YdOH0Q/r1ZInYjCi
• extraSchemas - https://hackmd.io/9QZ8YibfQHm9l1B6JPSQsg
  Pre-proposal JEP is out
• Cell types - https://hackmd.io/EmDM0wm1Tli3VVW7KrTwJQ
• schema keyword: Add JEP for adding $schema to notebook format jupyter/enhancement-proposals#97

References

Footnotes

1. https://json-schema.org/learn/glossary.html ↩

2. https://modern-json-schema.com/what-is-modern-json-schema ↩

[ellisonbg]

Thanks for posting this here, I heard that the notebook format community workshop was great. I looked through the notes and the linked JEP drafts, but I am having a tough time connecting the low level technical details (schema, vocab, etc.) to the ambitious title of the meeting "rethinking notebook cells." Is there somewhere that has a high level overview or summary of what this working group is working on to help others know if they might want to attend? Thanks for working on this!

[tonyfast]

in this hackmd, there is a heading called JEP drafts that collects the documents we are referencing from the workshop. i'm including those links below.

• $schema - https://hackmd.io/@u1M5398WTl6qOUg8YdOH0Q/r1ZInYjCi
• extraSchemas - https://hackmd.io/9QZ8YibfQHm9l1B6JPSQsg
  Pre-proposal JEP is out
• Cell types - https://hackmd.io/EmDM0wm1Tli3VVW7KrTwJQ

@jasongrout has collected all of the different efforts into a single google doc

[andrii-i]

Should this be added to Project Jupyter calendar for more visibility?

[tonyfast]

this is one of the to do items. if you can add it for us that would help. i don't know how to do that.

[JasonWeill]

@tonyfast Added!

[fcollonval]

@tonyfast I created a channel on Jupyter Zoom - ⚠️ it is not the Jovyan channel

@JasonWeill would you mind updating the Jupyter calendar event to update the zoom link?

[JasonWeill]

@fcollonval Updated the Google Calendar event with the new Zoom link.

[tonyfast]

March 14th, 2023

Name	Affiliation	GitHub
tonyfast		@tonyfast
Steve Purves	Curvenote	@stevejpurves
Jason Grout	Databricks	@jasongrout
Angus Hollands	Princeton University	@agoose77
Nick Bollweg	GTech	@bollwyvl

Agenda

• discuss open jeps

  • Add JEP for adding $schema to notebook format jupyter/enhancement-proposals#97
    • top level $schema is not contentious
  • pre-proposal: add extraSchemas to notebook format jupyter/enhancement-proposals#96
    • extra schemas might be contentious
      • need to resolve the root notebook schema and extra schemas can fail
      • we need to be able to turn things on and off in case of failure

• deprecation notes

  • bump the metaschema to draft 2020/12, currently version 4 doesn't support deprecation, it wasn't introduced until draft 2019.
  • at least a year for the deprecation. find a good reference for the deprecation cycle as precedent
  • old validators will feel when additionalProperties: false, which will require updating existing nbformat schema
  • precedence in nbformat for changes
  • $schema takes precedence over nbformat and nbformat_minor
    • Require that $schema validates against a URI-template that captures major, minor version
    • Encode this in the schema with const
    • Can also do this in the metaschema, though it's less important.

• what is the jep process

  • ask SSC what this process will look like
  • software steering council is still being formed. jeps will be priority

• discuss work in progress

  • Text based Format - https://hackmd.io/CmAhY_3tRK6ge4tqANflTg
  • Cell's Markdown Format - https://docs.google.com/document/d/1B8mhaHud7DMY55q1mg5sSDhZ96FGC6cbJpypYO1BocA
  • Persist user expressions - https://docs.google.com/document/d/110OJnl7baNeCz6Y5KnKaA4dpLdltB_fvpr2Q0Rf_36M

[fcollonval]

@tonyfast I did not pay attention but could we move this, in the JEP repo: jupyter/enhancement-proposals#95

That repo is a better place and will ease discoverability for the broader Jupyter community.

[tonyfast]

sure no problem. ill do that work after today's meeting.

[tonyfast]

these conversations have outgrown the JEP at this point.....

June 13, 2023

Name	Affiliation	GitHub
tonyfast		@tonyfast
Afshin T. Darian	QuantStack	@afshin
Gabriel Fouasnon	Quansight Labs	@gabalafou

Agenda

• If there's time, maybe walk through the nbconvert variants

  • @tonyfast response to @gabalafou: We are testing different representations of notebooks and cells with automated and manual testing. The notebook variants allow us to track the different versions of notebooks we've been testing for accessibility purposes. Some of the variants were specifically designed for user testing. Other experiments are designed to explore idealized representations of the notebooks and their annotation object model.
    • @gabalafou to @tonyfast: Thanks! What I was really trying to find out when I put this on the agenda was not so much a walk through to understand the architecture and how these variants are generated, but more specifically, because I don't have time to test every variation, which variants I should explore and test. Perhaps we can cover this in the next meeting.

  each variant is defined by a jupyter configuration file. the configuration files we used to generate notebooks are found in this list https://github.com/Iota-School/notebooks-for-all/blob/main/pyproject.toml#L132

  Some of the variants are from a parametric study to explore how cells would be configured as ordered lists, unordered lists, definition list we represent them as tables and feeds, too. Through the parametric study we could explored the space of possible semantics

Notes

• Discussion of work related to scrolling and virtual windowing.
• Analogy: notebook as feed.
  • late edit: ordered lists might have preferred semantics over a feed, but we can address this when we test with a screen reader.
• There's a separate push to make JupyterLab (Notebook?) completely usable by keyboard only.
• top level main > feed
• we hope modify the semantics for of the jupyter notebook interface. there would be no vision changes. we will add roles and aria to improve the primary navigation of page with assistive technology.

Summary: we spent this session discussing what a quality annotation object model.

we spent this session discussing what it would take to implement a more explicit accessibility object model based for the new jupyter notebook like. we reviewed the accessibility affordances of the notebooks for all project. our goal is try to capture a similar annotation object model for jupyter notebook release and live up the accessible v7 promise. this effort would knock some items on the @manfromjupyter audit jupyter/notebook#6800

in the near term, it would help to split up this issue like we did 9399.

cc: @steff456