fix(skills): read OOXML parts as bytes and form JSON as UTF-8 in office skill scripts - #70289
Closed
solyanviktor-star wants to merge 1 commit into
Closed
solyanviktor-star wants to merge 1 commit into
solyanviktor-star wants to merge 1 commit into
Conversation
…ce skill scripts The bundled office skills (NousResearch#68595) read user documents and agent-authored payloads with the locale-default codec: - docx/powerpoint validators/base.py opened OOXML part XML in text mode before handing it to lxml. On Windows (cp1251/GBK) the bytes decode to mojibake that lxml then parses, so validation runs against silently corrupted document text; on locales where the UTF-8 bytes don't decode the validator crashes with UnicodeDecodeError instead of validating. Opening as bytes lets lxml honor the encoding declared in the XML prolog. - The pdf form scripts (fill_fillable_fields, fill_pdf_form_with_annotations, create_validation_image, check_bounding_boxes) read the fields JSON the agent authors — UTF-8 by construction — with the locale codec, so non-ASCII form values (any Cyrillic/CJK/accented input) either crash or get written into the user's PDF as mojibake. The json.dump writers use ensure_ascii=True and were already safe; only the readers needed pinning. Adds a contract test asserting every document/payload reader is locale-independent, plus a live regression test that runs check_bounding_boxes.py on a non-ASCII fields.json under a forced non-UTF-8 locale — it fails without the fix on both POSIX (C locale) and Windows (cp1251 chokes on the 0x98 byte of U+2018).
Collaborator
|
Merged via PR #71078 — your commit(s) were cherry-picked onto current main with your authorship preserved in git log (rebase merge). This PR was part of the class-wide close-out of bare |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
The office skills bundled in #68595 read user documents and agent-authored payloads with the locale-default codec, which breaks them on any non-UTF-8 host locale (stock Windows cp1251/GBK/cp932):
OOXML validators (
docxandpowerpointscripts/office/validators/base.py): part XML was opened in text mode and handed tolxml.etree.parse(). Live repro (cp1251): the UTF-8 bytes decode to mojibake that lxml successfully parses — validation silently runs against corrupted document text (Отчёт за июль→Р[icon]тчёт Р·Р° июль). On locales where the bytes don't decode, the validator crashes withUnicodeDecodeErrorinstead of validating. Opening as bytes lets lxml honor the encoding declared in the XML prolog — the canonical way to feed lxml.PDF form scripts (
fill_fillable_fields.py,fill_pdf_form_with_annotations.py,create_validation_image.py,check_bounding_boxes.py): the fields JSON is authored by the agent as UTF-8, but was read back with the locale codec — non-ASCII form values (any Cyrillic/CJK/accented input) either crash the script or get silently written into the user's PDF as mojibake. Thejson.dumpwriters useensure_ascii=Trueand were already safe; only the readers needed pinning.This is the same unwired-reader class as the merged .env/UTF-8 fixes (#60895 lineage), applied to the freshly bundled skills.
Related Issue
No open issue; found by auditing #68595 for locale-dependent I/O. Dedup: searched PRs/issues for office-skill encoding fixes — none exist.
Type of Change
Changes Made
skills/productivity/{docx,powerpoint}/scripts/office/validators/base.py: open part XML as"rb"(lxml reads the prolog encoding).skills/productivity/pdf/scripts/{fill_fillable_fields,fill_pdf_form_with_annotations,create_validation_image,check_bounding_boxes}.py: read the fields JSON withencoding="utf-8".tests/skills/test_office_document_skills.py: contract test pinning every document/payload reader to a locale-independent mode, plus a live regression test that runscheck_bounding_boxes.pyon a non-ASCIIfields.jsonunder a forced non-UTF-8 locale (LC_ALL=C,PYTHONUTF8=0). The payload includes U+2018, whose 0x98 byte is unmapped in cp1251, so the test also fails-without-fix on Windows dev machines, not only under the POSIX C locale.How to Test
git stashtheskills/changes and runpytest tests/skills/test_office_document_skills.py -q→ the 7 new tests fail (6 contract, 1 live subprocess repro).open(p, "r")+lxml.etree.parseunder cp1251 → parsed text ≠ original (silent mojibake); viaopen(p, "rb")→ equal.