Skip to content

fix: report the document's real line ending in the status bar - #540

Merged
PathGao merged 1 commit into
masterfrom
fix/status-bar-truth
Aug 8, 2026
Merged

fix: report the document's real line ending in the status bar#540
PathGao merged 1 commit into
masterfrom
fix/status-bar-truth

Conversation

@PathGao

@PathGao PathGao commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

The editor's status bar ends with two hardcoded strings:

<div class="status-item">{t('editor.status.crlf')}</div>
<div class="status-item">{t('editor.status.utf8')}</div>

Neither is measured. The app tells every user their file is CRLF and UTF-8, whatever it actually is — samples/stress-test.md contains zero CR bytes and the status bar said CRLF.

Found while fixing #148, which was a bug about line endings the whole way down: the write-back mis-resolved a line because a pattern ate a \r, three fixes missed it because they were verified on macOS, and the suite's own convention (readSource instead of readFileSync) exists because a CRLF checkout on Windows once turned fifteen test files red. Through all of that, the one place in the app that claims to report a document's line ending was reporting a constant.

The line ending

model.getEOL(), surfaced through lineEndingLabel and rendered next to the language and word count in syncStatusFromModel — the existing home for document-wide readings, already re-run on tab switch and on content change.

Not a fourth detector. Monaco's buffer builder already decides this: createTextBufferFactory counts CR/LF/CRLF and, with normalizeEOL at its default, rewrites the whole buffer to the majority ending before the model exists. So getEOL() is not an opinion sitting beside the bytes — it is the bytes, and it is what a save writes out.

detectLineEnding in frontMatter.ts stays where it is. It answers a different question — which ending to emit when rewriting a front-matter block in a string it was handed — and lifting it into a shared helper would have meant more code for a less accurate answer.

Mixed endings need no third label, because by the time anything can be displayed the document is no longer mixed. a\r\nb\r\nc\n normalizes to CRLF and reports CRLF; a tie goes to LF, since Monaco requires a CR majority rather than a plurality. The test asserts the normalized buffer contents alongside the label, so it pins the claim and not just the string.

A document with no line break at all — an empty untitled tab, a one-line file — has nothing to detect and falls back to the model's defaultEOL, which the standalone services resolve per platform. So the same empty tab honestly reads CRLF on Windows and LF on a Mac, which is what Enter will insert and what the save will write. Covered by a test.

The encoding indicator is deliberately untouched

fix/non-utf8-documents (#372) implements real detection — BOM sniff, then chardetng — and carries the answer on Tab.encoding so a legacy document can be written back in its own encoding. That branch declares the field as required, deliberately, so a construction site cannot forget it and silently rewrite a GBK file as UTF-8.

Wiring the label to it here would mean declaring the field on Tab in this PR too: a duplicate, always-'UTF-8' answer to a question that branch answers properly, four more construction sites to keep in sync, and a guaranteed conflict in the interface when it lands. The guard would never fire, so the label would read UTF-8 exactly as it does now — the same lie with more code behind it.

The markup carries a comment naming the branch, the issue and the field. Wiring it once #372 lands is a one-line follow-up.

i18n: five keys removed, none added

LF and CRLF are acronyms. The existing crlf key held the identical ASCII 'CRLF' in the five locales that defined it, and the other 21 fell through to English — it was carrying zero information. The neighbouring status items ({zoomLevel}%, {currentLanguage}) render raw, so raw is the local convention. editor.status.utf8 stays until the encoding branch lands.

Validation

  • npm test 850 · npm run check 667 files, 0 errors · cargo test 141 (Rust untouched)
  • scripts/editorLineEnding.test.ts builds real Monaco piece-tree buffers the way createModel does and pushes them through the real lineEndingLabel: CRLF → CRLF, LF → LF, three mixed fixtures with their normalized contents asserted, plus the no-line-break case.
  • Verified it fails against the old code — restoring the hardcoded string turns the wiring test red.

Two existing harnesses (undoHistoryPerTab, imageUndoKeepsFile) lift syncStatusFromModel out of the component and eval it; they now receive the real lineEndingLabel and a fixed getEOL on their fake models. Fixed that way on purpose — a fake that guessed an EOL from its own buffer would be a second implementation of the thing this removes.

Not verified: no run of the packaged app, so "the status bar visibly reads LF on stress-test.md" is inference from the wiring plus the Monaco behaviour exercised directly. The Windows-default case is read out of standaloneServices.js and driven through an explicit defaultEOL, not on a Windows machine.

🤖 Generated with Claude Code

The status bar rendered `t('editor.status.crlf')` — the literal string
"CRLF", for every document, on every platform. `samples/stress-test.md`
has not got one CR byte in it and the app said CRLF about it.

The label is now the model's own `getEOL()`, taken in
`syncStatusFromModel` alongside the language and the word count, which is
where the other document-wide readings are refreshed. No new detection:
Monaco's buffer builder already counts the endings in the source and
rewrites the WHOLE buffer to the majority one before the model exists, so
`getEOL()` is both what the buffer holds and what a save writes out. A
mixed file is therefore not mixed once it is open, and the label says
which ending won (a tie goes to LF). A document with no line break at all
— an empty untitled tab — takes the model's platform default, CRLF on
Windows and LF elsewhere, which is what Enter will insert.

`detectLineEnding` in `utils/frontMatter.ts` is deliberately left where it
is and not lifted into a shared helper. It answers a different question,
about a string it is handed rather than about the open document, and the
two agree wherever both can see the same content.

Not translated, like the language id and the zoom level beside it: LF and
CRLF are acronyms, and the `crlf` key held the same ASCII in all five
locales that defined it, so the key is deleted rather than given
`LF`/`CRLF` twins in 26 languages.

The encoding indicator is left hardcoded on purpose, with a comment
naming its source: `fix/non-utf8-documents` (#372) detects the real
encoding and puts it on `Tab.encoding`. That field does not exist on
master, so wiring the label to it now would mean declaring it here too —
a second, always-'UTF-8' answer to a question that branch already
answers, and a conflict when it lands.

The two harnesses that lift `syncStatusFromModel` out of the component
gain the new binding and a fixed `getEOL` on their fake models; guessing
an EOL from a fake buffer would re-introduce exactly what this removes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@PathGao
PathGao merged commit b3a2569 into master Aug 8, 2026
4 checks passed
@PathGao
PathGao deleted the fix/status-bar-truth branch August 8, 2026 05:16
PathGao pushed a commit that referenced this pull request Aug 8, 2026
What review of the detection work turned up, all of it on the surface the
user actually meets rather than in the decoding itself.

**The status bar was still saying `UTF-8`.** It had been the literal string
since before Markpad could open a non-UTF-8 file at all, which made it true.
Detection made it a lie for exactly the documents an encoding indicator is
for: a GBK file now opens, reads correctly and saves back as GBK, and the
one place a reader could have caught a misdetection was asserting the
opposite. It reads `tab.encoding` now — derived rather than synced, because
the encoding belongs to the file the buffer came from and the store already
carries it. Same defect #540 fixed one slot to the left.

**The refusal was an English sentence from Rust.** Pasting an emoji into a
GBK document is refused, correctly — but the reader got "auto-save failed,
unsaved changes still in memory" in their own language and the reason in
English, in a different corner of the screen. The reason is the half that
has to be understood.

So `encode_text` returns a marker and this side writes the sentence. The
translation that matters is not English-to-Chinese; it is a fact about the
program (`ENCODING_UNMAPPABLE`) turned into a fact about what to do next
(this file is GBK, GBK cannot hold that character, write a UTF-8 copy). That
the result exists in six languages is the cheap half, on machinery the app
already has — and it makes this refusal the peer of `lossySaveBlocked`,
which has been translated all along.

The rule that follows is in the comment: a failure the user can act on gets
a code and a sentence; the 91 that hand back an OS error keep the OS's own
words, which are the whole of the information and the only string worth
searching for.

The marker carries nothing. It held the encoding label at first, and the
frontend sliced it back out — a round trip that told us what we had passed
into the call ourselves.

Two claims in the Rust docs were also wrong:

  - "bytes the guess reproduces exactly are unchanged" overstates it. Several
    legacy encodings spell one character more than one way — Shift_JIS
    reaches U+2160 at both 0x8754 and 0xFA4A — so a decode/encode pair
    normalises to whichever the encoder prefers, with neither `lossy` nor
    `unmappable` set. The guarantee is over the document's TEXT. The test
    that pinned it is `..._reproduces_its_canonical_bytes` now, because its
    fixtures are encoded by `encoding_rs` itself and that is all they can
    prove.
  - BOM-less UTF-16 is not detected, by `chardetng`'s design, and reaches the
    single-byte guess. It round-trips, so nothing is destroyed. Named in
    `decode_text` rather than left for whoever reads the next bug report.
PathGao added a commit that referenced this pull request Aug 8, 2026
) (#544)

* fix: detect a document's encoding and save it back in the same one (#372)

A document in a legacy encoding was decoded as UTF-8 with U+FFFD
substituted for every byte the decoder disagreed with. #371 closed the
data-loss half of that — the buffer is flagged and refused a write over
its own file — but the document was still unreadable, so a GBK, Big5,
Shift-JIS or CP-1252 file could be opened and not used.

The decode now sniffs a BOM, then tries UTF-8, then asks chardetng (the
detector Firefox ships) and decodes with its answer. The label travels
to the frontend with the text, lives on the tab, and comes back with the
save, so an unedited legacy document written back is byte-for-byte the
file that was opened. UTF-8 BOM and UTF-16 LE/BE are kept too: the BOM
is taken out of the buffer, where `\u{FEFF}# Title` silently stops being
a heading, and put back on the save.

Two refusals rather than a guess, both leaving the buffer dirty with the
reason in a toast:

  - a file no encoding can read is still `lossy`, and still cannot be
    written over itself. Detection narrows that case, it does not remove
    it.
  - a character the document's own encoding cannot represent — an emoji
    typed into a Shift-JIS file — fails the save instead of writing
    encoding_rs's `&#128512;` HTML escape into a Markdown file.

Save As is unchanged and stays the way out of both: it writes UTF-8 and
repoints the tab at it.

encoding_rs was already in the lock file (reqwest), so the new crate is
chardetng alone. Its legacy encoders are left on the default slow tables
rather than the `fast-*-encode` features, which cost ~180KB each.

Also deletes `open_markdown`, the last strict `read_to_string` path,
which had no call site left.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(encoding): name the encoding, and say why a save was refused

What review of the detection work turned up, all of it on the surface the
user actually meets rather than in the decoding itself.

**The status bar was still saying `UTF-8`.** It had been the literal string
since before Markpad could open a non-UTF-8 file at all, which made it true.
Detection made it a lie for exactly the documents an encoding indicator is
for: a GBK file now opens, reads correctly and saves back as GBK, and the
one place a reader could have caught a misdetection was asserting the
opposite. It reads `tab.encoding` now — derived rather than synced, because
the encoding belongs to the file the buffer came from and the store already
carries it. Same defect #540 fixed one slot to the left.

**The refusal was an English sentence from Rust.** Pasting an emoji into a
GBK document is refused, correctly — but the reader got "auto-save failed,
unsaved changes still in memory" in their own language and the reason in
English, in a different corner of the screen. The reason is the half that
has to be understood.

So `encode_text` returns a marker and this side writes the sentence. The
translation that matters is not English-to-Chinese; it is a fact about the
program (`ENCODING_UNMAPPABLE`) turned into a fact about what to do next
(this file is GBK, GBK cannot hold that character, write a UTF-8 copy). That
the result exists in six languages is the cheap half, on machinery the app
already has — and it makes this refusal the peer of `lossySaveBlocked`,
which has been translated all along.

The rule that follows is in the comment: a failure the user can act on gets
a code and a sentence; the 91 that hand back an OS error keep the OS's own
words, which are the whole of the information and the only string worth
searching for.

The marker carries nothing. It held the encoding label at first, and the
frontend sliced it back out — a round trip that told us what we had passed
into the call ourselves.

Two claims in the Rust docs were also wrong:

  - "bytes the guess reproduces exactly are unchanged" overstates it. Several
    legacy encodings spell one character more than one way — Shift_JIS
    reaches U+2160 at both 0x8754 and 0xFA4A — so a decode/encode pair
    normalises to whichever the encoder prefers, with neither `lossy` nor
    `unmappable` set. The guarantee is over the document's TEXT. The test
    that pinned it is `..._reproduces_its_canonical_bytes` now, because its
    fixtures are encoded by `encoding_rs` itself and that is all they can
    prove.
  - BOM-less UTF-16 is not detected, by `chardetng`'s design, and reaches the
    single-byte guess. It round-trips, so nothing is destroyed. Named in
    `decode_text` rather than left for whoever reads the next bug report.

---------

Co-authored-by: PathGao <gaoyanbo@gaoyanbodeMacBook-Air.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant