fix: report the document's real line ending in the status bar - #540
Merged
Conversation
The status bar rendered `t('editor.status.crlf')` — the literal string
"CRLF", for every document, on every platform. `samples/stress-test.md`
has not got one CR byte in it and the app said CRLF about it.
The label is now the model's own `getEOL()`, taken in
`syncStatusFromModel` alongside the language and the word count, which is
where the other document-wide readings are refreshed. No new detection:
Monaco's buffer builder already counts the endings in the source and
rewrites the WHOLE buffer to the majority one before the model exists, so
`getEOL()` is both what the buffer holds and what a save writes out. A
mixed file is therefore not mixed once it is open, and the label says
which ending won (a tie goes to LF). A document with no line break at all
— an empty untitled tab — takes the model's platform default, CRLF on
Windows and LF elsewhere, which is what Enter will insert.
`detectLineEnding` in `utils/frontMatter.ts` is deliberately left where it
is and not lifted into a shared helper. It answers a different question,
about a string it is handed rather than about the open document, and the
two agree wherever both can see the same content.
Not translated, like the language id and the zoom level beside it: LF and
CRLF are acronyms, and the `crlf` key held the same ASCII in all five
locales that defined it, so the key is deleted rather than given
`LF`/`CRLF` twins in 26 languages.
The encoding indicator is left hardcoded on purpose, with a comment
naming its source: `fix/non-utf8-documents` (#372) detects the real
encoding and puts it on `Tab.encoding`. That field does not exist on
master, so wiring the label to it now would mean declaring it here too —
a second, always-'UTF-8' answer to a question that branch already
answers, and a conflict when it lands.
The two harnesses that lift `syncStatusFromModel` out of the component
gain the new binding and a fixed `getEOL` on their fake models; guessing
an EOL from a fake buffer would re-introduce exactly what this removes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PathGao
pushed a commit
that referenced
this pull request
Aug 8, 2026
What review of the detection work turned up, all of it on the surface the user actually meets rather than in the decoding itself. **The status bar was still saying `UTF-8`.** It had been the literal string since before Markpad could open a non-UTF-8 file at all, which made it true. Detection made it a lie for exactly the documents an encoding indicator is for: a GBK file now opens, reads correctly and saves back as GBK, and the one place a reader could have caught a misdetection was asserting the opposite. It reads `tab.encoding` now — derived rather than synced, because the encoding belongs to the file the buffer came from and the store already carries it. Same defect #540 fixed one slot to the left. **The refusal was an English sentence from Rust.** Pasting an emoji into a GBK document is refused, correctly — but the reader got "auto-save failed, unsaved changes still in memory" in their own language and the reason in English, in a different corner of the screen. The reason is the half that has to be understood. So `encode_text` returns a marker and this side writes the sentence. The translation that matters is not English-to-Chinese; it is a fact about the program (`ENCODING_UNMAPPABLE`) turned into a fact about what to do next (this file is GBK, GBK cannot hold that character, write a UTF-8 copy). That the result exists in six languages is the cheap half, on machinery the app already has — and it makes this refusal the peer of `lossySaveBlocked`, which has been translated all along. The rule that follows is in the comment: a failure the user can act on gets a code and a sentence; the 91 that hand back an OS error keep the OS's own words, which are the whole of the information and the only string worth searching for. The marker carries nothing. It held the encoding label at first, and the frontend sliced it back out — a round trip that told us what we had passed into the call ourselves. Two claims in the Rust docs were also wrong: - "bytes the guess reproduces exactly are unchanged" overstates it. Several legacy encodings spell one character more than one way — Shift_JIS reaches U+2160 at both 0x8754 and 0xFA4A — so a decode/encode pair normalises to whichever the encoder prefers, with neither `lossy` nor `unmappable` set. The guarantee is over the document's TEXT. The test that pinned it is `..._reproduces_its_canonical_bytes` now, because its fixtures are encoded by `encoding_rs` itself and that is all they can prove. - BOM-less UTF-16 is not detected, by `chardetng`'s design, and reaches the single-byte guess. It round-trips, so nothing is destroyed. Named in `decode_text` rather than left for whoever reads the next bug report.
PathGao
added a commit
that referenced
this pull request
Aug 8, 2026
) (#544) * fix: detect a document's encoding and save it back in the same one (#372) A document in a legacy encoding was decoded as UTF-8 with U+FFFD substituted for every byte the decoder disagreed with. #371 closed the data-loss half of that — the buffer is flagged and refused a write over its own file — but the document was still unreadable, so a GBK, Big5, Shift-JIS or CP-1252 file could be opened and not used. The decode now sniffs a BOM, then tries UTF-8, then asks chardetng (the detector Firefox ships) and decodes with its answer. The label travels to the frontend with the text, lives on the tab, and comes back with the save, so an unedited legacy document written back is byte-for-byte the file that was opened. UTF-8 BOM and UTF-16 LE/BE are kept too: the BOM is taken out of the buffer, where `\u{FEFF}# Title` silently stops being a heading, and put back on the save. Two refusals rather than a guess, both leaving the buffer dirty with the reason in a toast: - a file no encoding can read is still `lossy`, and still cannot be written over itself. Detection narrows that case, it does not remove it. - a character the document's own encoding cannot represent — an emoji typed into a Shift-JIS file — fails the save instead of writing encoding_rs's `😀` HTML escape into a Markdown file. Save As is unchanged and stays the way out of both: it writes UTF-8 and repoints the tab at it. encoding_rs was already in the lock file (reqwest), so the new crate is chardetng alone. Its legacy encoders are left on the default slow tables rather than the `fast-*-encode` features, which cost ~180KB each. Also deletes `open_markdown`, the last strict `read_to_string` path, which had no call site left. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(encoding): name the encoding, and say why a save was refused What review of the detection work turned up, all of it on the surface the user actually meets rather than in the decoding itself. **The status bar was still saying `UTF-8`.** It had been the literal string since before Markpad could open a non-UTF-8 file at all, which made it true. Detection made it a lie for exactly the documents an encoding indicator is for: a GBK file now opens, reads correctly and saves back as GBK, and the one place a reader could have caught a misdetection was asserting the opposite. It reads `tab.encoding` now — derived rather than synced, because the encoding belongs to the file the buffer came from and the store already carries it. Same defect #540 fixed one slot to the left. **The refusal was an English sentence from Rust.** Pasting an emoji into a GBK document is refused, correctly — but the reader got "auto-save failed, unsaved changes still in memory" in their own language and the reason in English, in a different corner of the screen. The reason is the half that has to be understood. So `encode_text` returns a marker and this side writes the sentence. The translation that matters is not English-to-Chinese; it is a fact about the program (`ENCODING_UNMAPPABLE`) turned into a fact about what to do next (this file is GBK, GBK cannot hold that character, write a UTF-8 copy). That the result exists in six languages is the cheap half, on machinery the app already has — and it makes this refusal the peer of `lossySaveBlocked`, which has been translated all along. The rule that follows is in the comment: a failure the user can act on gets a code and a sentence; the 91 that hand back an OS error keep the OS's own words, which are the whole of the information and the only string worth searching for. The marker carries nothing. It held the encoding label at first, and the frontend sliced it back out — a round trip that told us what we had passed into the call ourselves. Two claims in the Rust docs were also wrong: - "bytes the guess reproduces exactly are unchanged" overstates it. Several legacy encodings spell one character more than one way — Shift_JIS reaches U+2160 at both 0x8754 and 0xFA4A — so a decode/encode pair normalises to whichever the encoder prefers, with neither `lossy` nor `unmappable` set. The guarantee is over the document's TEXT. The test that pinned it is `..._reproduces_its_canonical_bytes` now, because its fixtures are encoded by `encoding_rs` itself and that is all they can prove. - BOM-less UTF-16 is not detected, by `chardetng`'s design, and reaches the single-byte guess. It round-trips, so nothing is destroyed. Named in `decode_text` rather than left for whoever reads the next bug report. --------- Co-authored-by: PathGao <gaoyanbo@gaoyanbodeMacBook-Air.local> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The editor's status bar ends with two hardcoded strings:
Neither is measured. The app tells every user their file is CRLF and UTF-8, whatever it actually is —
samples/stress-test.mdcontains zero CR bytes and the status bar said CRLF.Found while fixing #148, which was a bug about line endings the whole way down: the write-back mis-resolved a line because a pattern ate a
\r, three fixes missed it because they were verified on macOS, and the suite's own convention (readSourceinstead ofreadFileSync) exists because a CRLF checkout on Windows once turned fifteen test files red. Through all of that, the one place in the app that claims to report a document's line ending was reporting a constant.The line ending
model.getEOL(), surfaced throughlineEndingLabeland rendered next to the language and word count insyncStatusFromModel— the existing home for document-wide readings, already re-run on tab switch and on content change.Not a fourth detector. Monaco's buffer builder already decides this:
createTextBufferFactorycounts CR/LF/CRLF and, withnormalizeEOLat its default, rewrites the whole buffer to the majority ending before the model exists. SogetEOL()is not an opinion sitting beside the bytes — it is the bytes, and it is what a save writes out.detectLineEndinginfrontMatter.tsstays where it is. It answers a different question — which ending to emit when rewriting a front-matter block in a string it was handed — and lifting it into a shared helper would have meant more code for a less accurate answer.Mixed endings need no third label, because by the time anything can be displayed the document is no longer mixed.
a\r\nb\r\nc\nnormalizes to CRLF and reports CRLF; a tie goes to LF, since Monaco requires a CR majority rather than a plurality. The test asserts the normalized buffer contents alongside the label, so it pins the claim and not just the string.A document with no line break at all — an empty untitled tab, a one-line file — has nothing to detect and falls back to the model's
defaultEOL, which the standalone services resolve per platform. So the same empty tab honestly reads CRLF on Windows and LF on a Mac, which is what Enter will insert and what the save will write. Covered by a test.The encoding indicator is deliberately untouched
fix/non-utf8-documents(#372) implements real detection — BOM sniff, thenchardetng— and carries the answer onTab.encodingso a legacy document can be written back in its own encoding. That branch declares the field as required, deliberately, so a construction site cannot forget it and silently rewrite a GBK file as UTF-8.Wiring the label to it here would mean declaring the field on
Tabin this PR too: a duplicate, always-'UTF-8'answer to a question that branch answers properly, four more construction sites to keep in sync, and a guaranteed conflict in the interface when it lands. The guard would never fire, so the label would readUTF-8exactly as it does now — the same lie with more code behind it.The markup carries a comment naming the branch, the issue and the field. Wiring it once #372 lands is a one-line follow-up.
i18n: five keys removed, none added
LFandCRLFare acronyms. The existingcrlfkey held the identical ASCII'CRLF'in the five locales that defined it, and the other 21 fell through to English — it was carrying zero information. The neighbouring status items ({zoomLevel}%,{currentLanguage}) render raw, so raw is the local convention.editor.status.utf8stays until the encoding branch lands.Validation
npm test850 ·npm run check667 files, 0 errors ·cargo test141 (Rust untouched)scripts/editorLineEnding.test.tsbuilds real Monaco piece-tree buffers the waycreateModeldoes and pushes them through the reallineEndingLabel: CRLF →CRLF, LF →LF, three mixed fixtures with their normalized contents asserted, plus the no-line-break case.Two existing harnesses (
undoHistoryPerTab,imageUndoKeepsFile) liftsyncStatusFromModelout of the component and eval it; they now receive the reallineEndingLabeland a fixedgetEOLon their fake models. Fixed that way on purpose — a fake that guessed an EOL from its own buffer would be a second implementation of the thing this removes.Not verified: no run of the packaged app, so "the status bar visibly reads LF on stress-test.md" is inference from the wiring plus the Monaco behaviour exercised directly. The Windows-default case is read out of
standaloneServices.jsand driven through an explicitdefaultEOL, not on a Windows machine.🤖 Generated with Claude Code