Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Does not work with Sphinx's internationalization workflow #587

Open
tonywu6 opened this issue Mar 5, 2024 · 3 comments · May be fixed by #600
Open

Does not work with Sphinx's internationalization workflow #587

tonywu6 opened this issue Mar 5, 2024 · 3 comments · May be fixed by #600
Labels
bug Something isn't working

Comments

@tonywu6
Copy link

tonywu6 commented Mar 5, 2024

Describe the bug

Sphinx uses gettext for internationalization: https://www.sphinx-doc.org/en/master/usage/advanced/intl.html

With this workflow, when translations are provided for notebooks, MyST-NB fails with a nbformat.reader.NotJSONError:

python -m sphinx -T -E -b html -D language=de docs docs/_build/html
Full traceback 
Traceback (most recent call last):
  File ".venv/lib/python3.11/site-packages/nbformat/reader.py", line 20, in parse_json
    nb_dict = json.loads(s, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.rye/py/[email protected]/install/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.rye/py/[email protected]/install/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/.rye/py/[email protected]/install/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venv/lib/python3.11/site-packages/sphinx/cmd/build.py", line 298, in build_main
    app.build(args.force_all, args.filenames)
  File ".venv/lib/python3.11/site-packages/sphinx/application.py", line 355, in build
    self.builder.build_update()
  File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 293, in build_update
    self.build(to_build,
  File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 313, in build
    updated_docnames = set(self.read())
                           ^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 420, in read
    self._read_serial(docnames)
  File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 441, in _read_serial
    self.read_doc(docname)
  File ".venv/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 498, in read_doc
    publisher.publish()
  File ".venv/lib/python3.11/site-packages/docutils/core.py", line 236, in publish
    self.apply_transforms()
  File ".venv/lib/python3.11/site-packages/docutils/core.py", line 216, in apply_transforms
    self.document.transformer.apply_transforms()
  File ".venv/lib/python3.11/site-packages/sphinx/transforms/__init__.py", line 83, in apply_transforms
    super().apply_transforms()
  File ".venv/lib/python3.11/site-packages/docutils/transforms/__init__.py", line 182, in apply_transforms
    transform.apply(**kwargs)
  File ".venv/lib/python3.11/site-packages/sphinx/transforms/i18n.py", line 397, in apply
    patch = publish_msgstr(self.app, msgstr, source,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/sphinx/transforms/i18n.py", line 73, in publish_msgstr
    doc = reader.read(
          ^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/docutils/readers/__init__.py", line 70, in read
    self.parse()
  File ".venv/lib/python3.11/site-packages/docutils/readers/__init__.py", line 76, in parse
    self.parser.parse(self.input, document)
  File ".venv/lib/python3.11/site-packages/myst_nb/sphinx_.py", line 89, in parse
    notebook = nb_reader.read(inputstring)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/myst_nb/core/read.py", line 38, in standard_nb_read
    return nbf.reads(text, as_version=NOTEBOOK_VERSION)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/nbformat/__init__.py", line 89, in reads
    nb = reader.reads(s, **kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/nbformat/reader.py", line 76, in reads
    nb_dict = parse_json(s, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/nbformat/reader.py", line 26, in parse_json
    raise NotJSONError(message) from e
nbformat.reader.NotJSONError: Notebook does not appear to be JSON: 'Franz jagt im komplett verwahrlosten Ta...

Exception occurred:
  File ".venv/lib/python3.11/site-packages/nbformat/reader.py", line 26, in parse_json
    raise NotJSONError(message) from e
nbformat.reader.NotJSONError: Notebook does not appear to be JSON: 'Franz jagt im komplett verwahrlosten Ta...

Translations look like this

#: ../../index.ipynb:10002
msgid "The quick brown **fox** jumps over the lazy dog"
msgstr "**Franz** jagt im komplett verwahrlosten Taxi quer durch Bayern"

Sphinx will attempt to parse each msgstr using the parser configured for the corresponding source file (in this case the Notebook parser) because translations might contain markups:

https://github.com/sphinx-doc/sphinx/blob/ae51974e217cc7590e1b9c68f18af3e043027c6d/sphinx/transforms/i18n.py#L72-L78

Perhaps fall back to MyST-Parser in this case?

Reproduce the bug

https://github.com/tonywu6/issues-repro-myst-nb-intl

List your environment

python==3.11
sphinx==7.2.6
docutils==0.20.1
myst-nb==1.0.0
myst-parser==2.0.0
@tonywu6 tonywu6 added the bug Something isn't working label Mar 5, 2024
@welcome Welcome
Copy link

welcome bot commented Mar 5, 2024

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

@OriolAbril OriolAbril linked a pull request May 7, 2024 that will close this issue
Open
@OriolAbril
Copy link
Contributor

@tonywu6 I have added a PR #600 which should fix this. It would be great if you could try it out and ensure it works for you too.

@tonywu6
Copy link
Author

tonywu6 commented May 7, 2024

@OriolAbril Can confirm that i18n is working as expected with this patch (tried on some of our repos)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support translated notebooks OriolAbril/MyST-NB
2 participants
@OriolAbril @tonywu6