Repository navigation
perf: dictionary coverage, build staging, and the fingerprint index - #439
Merged
Merged
Conversation
fingerprints.json held `stem -> {a,j,s}` while the only question anything
asks it is the reverse: photo_recipe hashes an uploaded image and wants to
know which published photo those bytes are. So the lookup was a nested scan
over all 474 entries with an Object.entries() allocation per call.
Keyed by digest, JSON.parse hands back the index itself and the lookup is one
property access: 0.0205 ms -> 0.000039 ms worst case. That is a cold path (an
MCP tool, called rarely), so this is about the shape being right and degrading
linearly as photos grow rather than about the microseconds.
Deliberately a plain object rather than a Map parsed at the loadJson boundary.
Building a 474-entry Map costs ~0.072 ms, more than the scan it would replace,
so the boundary parse would spend the entire win.
The wire cost goes UP by 214 bytes brotli (16,134 -> 16,348), because sha256 is
incompressible and inverting duplicates the tier suffix. Worth recording why no
encoding change is coming: 474 digests sit on a ~15.2 KB entropy floor and
brotli already recovers the hex alphabet's waste, so hex lands at 16.1 KB
against base64's 16.4 and raw bytes' 15.4. Only truncating the digest moves
that number, and that is a separate decision about how strong a claim the tool
is making.
Two invariants come with it, because an inverted index can lose an entry in
ways the old shape could not:
- the generator REFUSES a repeated digest instead of picking one. Two published
tiers with identical bytes were always ambiguous; stem-first merely hid it,
since the scan returned whichever it reached first.
- check-photo-pipeline.mjs asserts the digest count. Controls established what
each check actually catches rather than what it looked like it caught: a
corrupted digest and a collision both trip the per-stem loop as drift, and
the count is the only witness for a STALE digest still naming a published
stem. The first draft of that comment claimed the opposite and the control
is what corrected it.
The per-stem check now asserts in the read direction too: hash the published
bytes, and the index must name that exact stem and tier.
build.mjs staged www/ verbatim into .build/, and www/scripts is 89% of that tree's bytes: the zenc cargo target/ and the uv venv put it at ~232 MB across 767 files, against ~27 MB for everything that actually ships. So every build copied a quarter of a gigabyte in order for .assetsignore to then exclude it from upload. .build/www drops 292 MB -> 34 MB. Cold build 3.0s -> 2.3s, measured over 3 runs each. It lands three times per CI run rather than once, since wrangler.jsonc carries build.command and both `wrangler deploy --dry-run` and `routes:check` re-run it (confirmed by watching .build/www/index.html's mtime move under each). routes:check 11.6s -> 10.5s. The staged tree is BYTE-IDENTICAL: 1444 files, 0 differing against a baseline built from the previous commit. That is the bar rather than a nicety, because /a/ and /i/ are content-addressed, so one differing byte mints a new URL and orphans the a-dict snapshots naming the old hash. Safe because every consumer imports from the SOURCE tree (build.mjs, contract-tests.mjs, gen-manifest.mjs, and the package.json script paths), so nothing has ever read .build/www/scripts. link-integrity is what catches a page that starts to, and the control is recorded at the code: a link to /scripts/shell-data.mjs injected into page prose fails the build by name. Worth keeping the first attempt at that control, which was INVALID. It added a <script> before </body> and tripped the desktop-partial tripwire instead, and the tell was that a path which IS staged failed identically. A control that fires for the wrong reason reads exactly like a control that works. Not done, and measured so nobody re-does it: raising UV_THREADPOOL_SIZE past its default 4 buys ~10% (2777ms -> 2503ms at 8 on a 14-core machine) and then flattens, so the q11 fan-out is not what the build waits on.
…ionary
`dict:roll --live` discovered the shell by fetching 4 documents and scanning
them for /a/ refs. An asset loaded lazily FROM JavaScript appears in no
document, so it was structurally invisible: measured against production, the
HTML-only scan found 8 assets and the graph holds 19.
The 11 it could not see had therefore never been given a dictionary at all.
Six of them are on the every-page path, because nav.js pulls nav-run.js,
nav-tray.js and infotip.js plus their CSS. lens.js is another, at 22.6 KB
brotli the largest asset on the site. A returning visitor re-downloaded each in
full where a delta is a few hundred bytes.
References are transitive (/lens -> lens-boot -> lens -> lens-tools is three
hops), so this closes over the graph rather than following one level and
calling it done. Termination is the visited set against a finite /a/ namespace,
one fetch per name.
Bodies are cached for the adoption loop below, which wanted the same bytes.
That saves a request per asset and closes a real window: fetching twice lets a
deploy land between the scan and the adopt, so the roll would store bytes this
run never read.
Coverage goes 8/19 bases to 19/19, with all 19 current-hash snapshots on hand.
Note the payoff is on the NEXT change to each asset rather than today: a
dictionary has to be bytes the browser already holds, so adopting what
production serves right now is what makes the following deploy cheap.
This is the same failure CLAUDE.md's dcz section already records ("Not urgent
was wrong, and the cost of believing it was 161 commits"), arriving through a
different door. That one was a roll nobody ran; this is a roll that ran nightly
and could not see two thirds of what it was rolling.
Separate commit because a roll writes into the source tree, which the build never may, and because this one is DATA where the parent is code. Adopts the 11 assets the previous commit's discovery gap kept invisible (nav-run, nav-tray and infotip js+css, lens and its four lazy panes), and prunes two stale lens snapshots past the 3-per-base keep. Sourced from https://aadhar.sh, so these are bytes browsers are holding right now.
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
aadhar-sh | eac808e | Aug 17 2026, 02:57 AM |
Contributor
Wire-size diff vs merge base
Brotli q11, the quality the edge serves. Every number here is deterministic: Client assetsNo change. 21 files, 74.97 KiB Brotli total. PagesNo change. 48 files, 500.45 KiB Brotli total. Worker bundleNo change. 228.47 KiB gzip. Dictionary deltas (
|
oddharsh
added a commit
that referenced
this pull request
Aug 17, 2026
The rebase onto main brought #439/#440/#441 (the 200px tier, packed histograms, the fingerprint index and one fewer copy of the photo data), and that code was written against www/. Four integration fixes it needed: - build.mjs's meta-twin derivation wrote to ${OUT}/www/images/meta and read exif.json + histograms.json from the same place, so it generated 0 files. Its own floor caught it, which is the tripwire doing exactly its job. - the homepage bake lost its `histograms` load in a conflict resolution and failed with a bare ReferenceError. Restored from main. - contract-tests imported ../www/scripts/build-histogram-index.mjs and read www/images/histograms.json. - tooltip.js cited www/scripts/build-histogram-index.mjs in a comment. That last one was caught by the served-path contract test added earlier on this branch, which is the first time it has found something it was not written for. Also un-resurrected 158 files. #441 DELETED www/images/meta and gitignored it: the per-photo twins are derived from exif.json + histograms.json now, and a pure function of committed bytes should not be a second committed copy. The bulk conflict resolution added them back at their moved path, which would have undone that silently. The .gitignore rule and build.mjs's STAGE_SKIP both survive, repointed to public/images/meta. 307 contract tests (main's 12 coverage tests came across), typecheck, lint, check-wrangler, pages:check, tools:check and perf-budget all clean. Bundle 669.37 KiB, up from 666.62 because main's photo work is now in it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Started as "are there alternatives to JSON that would be faster on the client."
The answer there was no, and the measurements are at the bottom. Chasing it
turned up three real things elsewhere, which is what this PR is.
1. The dictionary roll could only see 8 of 19 shell assets
dict:roll --liveruns nightly and discovers the shell by fetching 4 documentsand scanning them for
/a/refs. An asset loaded lazily FROM JavaScript appearsin no document, so it was structurally invisible. Measured against production:
The 11 it could not see had therefore never been given a dictionary at all. Six
are on the every-page path, because
nav.jspullsnav-run.js,nav-tray.jsand
infotip.jsplus their CSS.lens.jsis another, at 22.6 KB brotli thelargest asset on the site. A returning visitor re-downloaded each in full where
a delta is a few hundred bytes.
References are transitive (
/lenstolens-boottolenstolens-toolsisthree hops), so discovery now closes over the graph rather than following one
level. Termination is the visited set against a finite
/a/namespace, onefetch per name. Bodies are cached for the adoption loop, which saves a request
per asset and closes a window where a deploy could land between the scan and the
adopt.
Coverage goes 8/19 bases to 19/19, all 19 current-hash snapshots on hand.
When this pays is worth being precise about. A dictionary has to be bytes the
browser already holds, so adopting what production serves today creates no delta
today. It makes the NEXT change to each of those assets cheap. Before this, that
next change was going to cost full weight indefinitely.
Same failure the dcz section of CLAUDE.md already records ("Not urgent was wrong,
and the cost of believing it was 161 commits"), arriving through a different
door. That one was a roll nobody ran. This is a roll that ran nightly and could
not see two thirds of its subject.
2. The build staged 232 MB so the deploy could refuse it
build.mjsstagedwww/verbatim, andwww/scriptsis 89% of that tree's bytes:the zenc cargo
target/and the uv venv, 767 files. Every build copied a quarterof a gigabyte in order for
.assetsignoreto then exclude it from upload.It lands three times per CI run rather than once, since
wrangler.jsonccarriesbuild.commandand bothwrangler deploy --dry-runandroutes:checkre-runit. Confirmed by watching
.build/www/index.html's mtime move under each.The staged tree is byte-identical: 1444 files, 0 differing against a baseline
built from the previous commit. That is the bar rather than a nicety, because
/a/and/i/are content-addressed, so one differing byte mints a new URL andorphans the a-dict snapshots naming the old hash.
Safe because every consumer imports from the SOURCE tree, so nothing has ever
read
.build/www/scripts.link-integrityis what catches a page that startsto, verified with a link injected into page prose.
Ruled out and recorded at the code so nobody redoes it: raising
UV_THREADPOOL_SIZEpast its default 4 buys about 10% (2777ms to 2503ms at 8 on14 cores) and then flattens, so the q11 fan-out is not what the build waits on.
3. The fingerprint index is stored in the direction it is read
fingerprints.jsonheldstem -> {a,j,s}while the only question anything asksit is the reverse:
photo_recipehashes an upload and wants to know which photothose bytes are. So the lookup was a nested scan over 474 entries with an
Object.entries()allocation per call. Keyed by digest,JSON.parsehands backthe index itself: 0.0205 ms to 0.000039 ms worst case.
Cold path (an MCP tool, called rarely), so this is about the shape being right
and degrading linearly as photos grow rather than about the microseconds.
Deliberately a plain object rather than a
Mapparsed at theloadJsonboundary: building a 474-entry Map costs ~0.072 ms, more than the scan it would
replace, so the boundary parse would spend the entire win.
The wire cost goes UP 214 bytes brotli, and the reason is recorded at the
generator so nobody "optimises" the encoding later. sha256 is incompressible, so
474 digests sit on a ~15.2 KB entropy floor and brotli already recovers the hex
alphabet's waste: hex 16.1 KB, base64 16.4 KB, raw bytes 15.4 KB. Only truncating
the digest moves that number, which is a separate decision about how strong a
claim the tool is making.
Two new invariants, since an inverted index can lose an entry in ways the old
shape could not: the generator refuses a repeated digest instead of picking one,
and
check-photo-pipeline.mjsasserts the digest count.What the controls corrected
Both worth knowing, because in each case the first version looked right.
link-integritycontrol was INVALID. It added a<script>before</body>and tripped the desktop-partial tripwire instead; the tell was that apath which IS staged failed identically. A control that fires for the wrong
reason reads exactly like one that works.
It does not: a collision trips the per-stem loop first, because the surviving
entry names the other stem and that reads as drift. What the count uniquely
catches is a stale digest still naming a published stem.
The JSON question that started it
Format was the wrong axis. Every payload the browser parses is under 0.3 ms, and
any replacement needs a userland decoder while
JSON.parseis C++ in V8:JSON.parse(today)splitThe delimited version reads 42% fewer raw bytes and still loses. Two shapes did
measure better and are deliberately NOT in this PR, because the absolute numbers
are below the noise until the photo count grows roughly 10x: column+dictionary
exif.json(2,617 to 2,312 brotli, parse 0.149 to 0.032 ms) and bundling the 12rendered histograms as bytes (2,880 br across 12 requests to 1,432 br in 1).
That second one needs a hover-rate number the repo does not collect yet.
Verification
Build, 291/291 contract tests, lint (oxlint incl. type-aware), typecheck,
photos:check,pages:check,routes:check(136 routes, 0 failures),perf-budget, andwrangler deploy --dry-runall green locally.One freshness note: the
a-dictcommit is data sourced from productiontoday. If this sits and a shell asset ships meanwhile, re-run
pnpm run dict:rollbefore merging so it adopts current bytes.