[JamBalaya56562]

From #5399. std::fs::read_to_string rejects any file that is not UTF-8, so a UTF-16 checksum file aborts the install:

1: failed read_to_string: .../powershell-7.4.10-linux-x64.tar.gz.checksum
2: stream did not contain valid UTF-8

What this actually costs today

I want to be precise rather than oversell it: nobody is hitting that crash right now, and the reason is the point.

PowerShell's release tooling regressed and published hashes.sha256 as UTF-16LE (PowerShell/PowerShell#25693). The fix was aquaproj/aqua-registry#38202, which disabled checksum verification for the affected range. So the standing cost of mise not reading UTF-16 is that verification stays switched off for a whole band of releases.

Fetching the raw bytes from GitHub shows the registry's version boundary is exactly an encoding boundary:

release	first bytes
v7.4.7	61 32 39 37 33 65 …	UTF-8
v7.4.10	FF FE 36 00 64 00 …	UTF-16LE BOM, not valid UTF-8
v7.5.0	35 38 31 33 34 31 …	UTF-8

which lines up with the vendored registry: semver("<= 7.4.7") keeps asset: hashes.sha256, and everything above it has no checksum block at all.

It is not PowerShell-specific either — Windows PowerShell 5.1's Out-File writes UTF-16LE by default, so this recurs for anything generating checksums that way.

The change

file::decode_text + file::read_to_string_bom, used at the three aqua sites that feed parse_checksum_from_content:

• verify_checksum_file_matches_expected — the install path, and the frame in powershell-core install in Fedora (WSL) fails #5399's stack
• the lockfile checksum binding after download
• fetch_checksum_from_file — switches get_text → get_bytes + decode_text. get_text decodes with the response charset and replaces what it cannot map instead of failing, so that path could not fail the way the file path does; it would record whatever the lossy decode produced. I have not reproduced that — converting it just makes all three agree.

Only a BOM selects the encoding. Detecting UTF-16 without one means guessing from the density of NUL bytes, which can misfire on binary input, and Out-File always writes a BOM, so the guess buys nothing. Input with no BOM is decoded as UTF-8 exactly as before — it just gets an error message naming the problem instead of std's bare "stream did not contain valid UTF-8".

encoding_rs is only a transitive dep (via reqwest's charset), and BOM'd UTF-16 is ~25 lines of std, so no new direct dependency.

Deliberately not done

• file::read_to_string itself is untouched. Over a hundred call sites, config files and plugin scripts among them; that is a much bigger behaviour change than this bug warrants. Happy to widen it if you'd rather have it everywhere.
• Other backends read checksums too (http.rs, pkgx.rs, static_helpers.rs, all via get_text). Same defect class, left out to keep this reviewable — decode_text is public so they can adopt it.

Tests

Decoder: UTF-16LE, UTF-16BE, a UTF-8 BOM, plain UTF-8 and empty input unchanged, an odd trailing byte, and an unpaired surrogate.

End to end: the real line for the artifact in #5399, taken from the v7.4.10 hashes.sha256 that broke it —

d91b9172668f4b6aef4abce8c780cd298872c7a0f4487cc47444d26877ba49f6 *powershell-7.4.10-linux-x64.tar.gz

— written as UTF-16LE with BOM, verified against the correct expectation and then against a wrong one, so the passing case cannot be a file that was silently never parsed.

Verification

No local cargo run; rustfmt --edition 2024 only. lint (clippy -D warnings), linux cargo test, unit-macos and windows-unit all cover these files.

Summary by CodeRabbit

• Bug Fixes
  • Improved checksum verification for files encoded as UTF-8 or UTF-16.
  • Added support for detecting byte-order marks when reading text files.
  • Improved handling of invalid or truncated text data while preserving clear failure behavior.
  • Prevented valid UTF-16 checksum files from being rejected during installation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: f1111b8f-4876-4932-8b3f-0030caad83fc

📥 Commits

Reviewing files that changed from the base of the PR and between 2d56687 and c2286e6.

📒 Files selected for processing (2)

• src/backend/aqua.rs
• src/file.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• src/backend/aqua.rs
• src/file.rs

---

📝 Walkthrough

Walkthrough

The change adds BOM-aware decoding for UTF-8, UTF-16LE, and UTF-16BE text. Aqua checksum verification and fetching use this decoder. Tests cover valid encodings, invalid input, and mismatched digests.

Changes

Checksum decoding

Layer / File(s)	Summary
BOM-aware text decoding src/file.rs	Adds decode_text and read_to_string_bom. The decoder supports UTF-8, UTF-16LE, and UTF-16BE BOMs. Tests cover valid, invalid, truncated, and no-BOM input.
Aqua checksum integration src/backend/aqua.rs	Checksum verification and metadata fetching use BOM-aware decoding. Tests cover UTF-16LE checksum files, valid digest verification, and incorrect expected digests.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Poem

I’m a rabbit with bytes in my burrow,
UTF-16 now hops without sorrow.
BOMs point the way,
Checksums pass their day,
And wrong little hashes get told no.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: support for checksum files with byte-order marks.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Greptile Summary

Adds BOM-aware checksum decoding for Aqua.

• Decodes UTF-8 BOM, UTF-16LE, and UTF-16BE checksum content while retaining strict UTF-8 behavior for BOM-less input.
• Applies the decoder to Aqua’s local verification, lockfile binding, and remote checksum-fetch paths.
• Adds decoder and end-to-end UTF-16LE checksum verification tests.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/backend/aqua.rs	Routes all three Aqua checksum parsing paths through BOM-aware decoding and adds an end-to-end UTF-16LE verification test.
src/file.rs	Introduces strict BOM-aware text decoding with coverage for supported encodings and malformed input.

_{Reviews (2): Last reviewed commit: "fix(aqua): read checksum files that carr..." | Re-trigger Greptile}