fix: add encoding="utf-8" to write_text() with ensure_ascii=False in skills_hub - #62667
fix: add encoding="utf-8" to write_text() with ensure_ascii=False in skills_hub#62667AlexFucuson9 wants to merge 1 commit into
Conversation
… in skills_hub Path.write_text() without encoding= defaults to the platform encoding (cp1252 on Windows). When combined with ensure_ascii=False, non-ASCII characters in skill names, descriptions, and metadata are silently corrupted on Windows — mojibake in cache files that then fail to parse. Affected sites: - _write_index_cache(): cache_file.write_text(json.dumps(data, ensure_ascii=False)) - _search_catalog_fallback(): same pattern with default=str All other write_text() calls in the codebase already pass encoding="utf-8".
Related: this narrow fix (2
Not marking |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating a real Windows encoding issue.
Problems
- The changed writers at
tools/skills_hub.py:1055and:3254now emit UTF-8, but their paired readers at:1045and:3234still useread_text()withoutencoding. The proposed write-only change therefore does not guarantee the advertised round trip on a non-UTF-8 Windows locale. - The same state path remains in
HubLockFile:tools/skills_hub.py:3294writesensure_ascii=FalseJSON, while:3288reads without an encoding;tools/skills_sync.py:405also reads that lock file without one. - The diff adds no regression test. The existing
HubLockFilecoverage attests/tools/test_skills_hub.py:1334-1402does not exercise non-ASCII persistence.
Suggested changes
- Update the paired cache reads, lock-file producer/consumers, and add a non-ASCII round-trip regression test. The verified open #47489 diff already covers these related paths.
Automated hermes-sweeper review.
| cache_file = index_cache_dir / f"{key}.json" | ||
| try: | ||
| cache_file.write_text(json.dumps(data, ensure_ascii=False)) | ||
| cache_file.write_text(json.dumps(data, ensure_ascii=False), encoding="utf-8") |
There was a problem hiding this comment.
Please update the paired read_text() calls too (_read_cache at current main line 1045 and _read_index_cache at line 3234). Writing UTF-8 while reading through the Windows default locale does not preserve the non-ASCII metadata this change is intended to protect.
|
Superseded by #64822 — adds encoding='utf-8' to all paired cache/lock reads in skills_hub and skills_sync. |
Summary
Path.write_text()withoutencoding=defaults to the platform encoding —cp1252on Windows,utf-8on Linux/macOS. Two call sites inskills_hub.pypassensure_ascii=Falsetojson.dumps()(allowing non-ASCII characters through) but omitencoding="utf-8"on the subsequentwrite_text(). On Windows, this silently corrupts non-ASCII skill names, descriptions, and metadata into mojibake, which then fails to parse on read.All other
write_text()calls in the codebase already specifyencoding="utf-8"(checked:curator.py,curator_backup.py,skill_bundles.py,agent_init.py,skills_sync.py,checkpoint_manager.py).Changes
write_text(json.dumps(data, ensure_ascii=False))..., encoding="utf-8")write_text(json.dumps(data, ensure_ascii=False, default=str))..., encoding="utf-8")Test Plan
ruff check tools/skills_hub.pypasses