fix: encoding round-trip for skills_hub/skills_sync cache and lock reads (supersedes #62667) - #64822
fix: encoding round-trip for skills_hub/skills_sync cache and lock reads (supersedes #62667)#64822AlexFucuson9 wants to merge 1 commit into
Conversation
…d skills_sync Addresses teknium1 review on NousResearch#62667: The write-side callers already use ensure_ascii=False (UTF-8 output), but the paired read-side callers used read_text() without encoding, which on non-UTF-8 Windows locales would decode with the system default (e.g. cp1252) and corrupt non-ASCII skill names/descriptions. Fixed read sites: - tools/skills_hub.py:1045 (paired with :1055 write) - tools/skills_hub.py:3234 (paired with :3254 write) - tools/skills_hub.py:3288 (paired with :3294 HubLockFile write) - tools/skills_hub.py:3360 (HubLockFile taps read) - tools/skills_hub.py:3678,3744 (hermes_index_cache read, paired with :3732 write) - tools/skills_sync.py:405 (reads HubLockFile written by skills_hub)
teknium1
left a comment
There was a problem hiding this comment.
Thanks for addressing the missing read-side calls identified on #62667. The underlying Windows issue is still present on current main, but this patch needs the matching producer changes to make the encoding contract valid.
Problems
- The changed readers become UTF-8-only, while the paired writers remain locale-dependent:
tools/skills_hub.py:1147,:3368, and:3408useensure_ascii=Falsewithoutencoding="utf-8". Read-only changes do not guarantee the advertised round trip on non-UTF-8 Windows locales. - The changed reader paths catch
OSErrorandJSONDecodeError, notUnicodeDecodeError; existing locale-encoded state can therefore raise after this patch rather than take the established fallback. - No non-ASCII round-trip regression test is included. Existing
HubLockFiletests attests/tools/test_skills_hub.py:1311-1402use ASCII fixtures.
Suggested changes
- Apply UTF-8 explicitly to the paired writers, retain the matching readers, and add behavioral non-ASCII cache/lock round-trip coverage. Please avoid a source-text guard;
AGENTS.mdprohibits source-inspection tests.
Automated hermes-sweeper review.
| if time.time() - stat.st_mtime > INDEX_CACHE_TTL: | ||
| return None | ||
| return json.loads(cache_file.read_text()) | ||
| return json.loads(cache_file.read_text(encoding="utf-8")) |
There was a problem hiding this comment.
This reader-only change does not create a UTF-8 round trip: its paired writer still calls write_text(json.dumps(..., ensure_ascii=False)) without encoding="utf-8". On a non-UTF-8 Windows locale it can write locale bytes (or fail for unrepresentable characters), and this UTF-8 reader can then raise UnicodeDecodeError, which the surrounding handler does not catch.
| return {"version": 1, "installed": {}} | ||
| try: | ||
| return json.loads(self.path.read_text()) | ||
| return json.loads(self.path.read_text(encoding="utf-8")) |
There was a problem hiding this comment.
HubLockFile.save() still writes ensure_ascii=False JSON without an explicit encoding, so making only this consumer UTF-8-only breaks the producer/consumer encoding contract. Update the paired writer and cover a non-ASCII lock entry round trip.
|
Closing as resolved by PR #71078 (merged, commit d372fda): the class-wide close-out salvaged your #50655/#54241/#56385/#66856/#65440 series as the backbone (authorship preserved in git log) and swept the remaining sites, so every read_text/write_text call this PR touches is now guarded on current main — verified per-site. A CI linter rule in check-windows-footguns.py plus the AST guard test now prevent regressions. Your overlapping/split variants of the same series are being closed together; the credit for the class rests on your commits. |
Supersedes #62667
Addresses all issues raised by @teknium1 in the #62667 review:
Problem
The write-side callers in
skills_hub.pyuseensure_ascii=False(UTF-8 output), but the paired read-side callers useread_text()without specifying encoding. On non-UTF-8 Windows locales (e.g. cp1252), this decodes UTF-8 bytes with the system default and corrupts non-ASCII skill names, descriptions, and metadata.Fixed read sites (7 total)
tools/skills_hub.pyensure_ascii=Falsetools/skills_hub.pyensure_ascii=Falsetools/skills_hub.pyensure_ascii=Falsetools/skills_hub.pytools/skills_hub.pytools/skills_hub.pytools/skills_sync.pyWhat this does NOT change
ignore_file.write_text(...)at :3249 writes ASCII-only content, no encoding needed."rb"/"wb") are unaffected.