Skip to content

perf: dictionary coverage, build staging, and the fingerprint index - #439

Merged
oddharsh merged 4 commits into
mainfrom
fingerprints-inverted-index
Aug 17, 2026
Merged

oddharsh merged 4 commits into
mainfrom
fingerprints-inverted-index

Conversation

@oddharsh

Copy link
Copy Markdown
Owner

Started as "are there alternatives to JSON that would be faster on the client."
The answer there was no, and the measurements are at the bottom. Chasing it
turned up three real things elsewhere, which is what this PR is.

1. The dictionary roll could only see 8 of 19 shell assets

dict:roll --live runs nightly and discovers the shell by fetching 4 documents
and scanning them for /a/ refs. An asset loaded lazily FROM JavaScript appears
in no document, so it was structurally invisible. Measured against production:

discovery assets found
HTML scan (today) 8
closing over the reference graph 19

The 11 it could not see had therefore never been given a dictionary at all. Six
are on the every-page path, because nav.js pulls nav-run.js, nav-tray.js
and infotip.js plus their CSS. lens.js is another, at 22.6 KB brotli the
largest asset on the site. A returning visitor re-downloaded each in full where
a delta is a few hundred bytes.

References are transitive (/lens to lens-boot to lens to lens-tools is
three hops), so discovery now closes over the graph rather than following one
level. Termination is the visited set against a finite /a/ namespace, one
fetch per name. Bodies are cached for the adoption loop, which saves a request
per asset and closes a window where a deploy could land between the scan and the
adopt.

Coverage goes 8/19 bases to 19/19, all 19 current-hash snapshots on hand.

When this pays is worth being precise about. A dictionary has to be bytes the
browser already holds, so adopting what production serves today creates no delta
today. It makes the NEXT change to each of those assets cheap. Before this, that
next change was going to cost full weight indefinitely.

Same failure the dcz section of CLAUDE.md already records ("Not urgent was wrong,
and the cost of believing it was 161 commits"), arriving through a different
door. That one was a roll nobody ran. This is a roll that ran nightly and could
not see two thirds of its subject.

2. The build staged 232 MB so the deploy could refuse it

build.mjs staged www/ verbatim, and www/scripts is 89% of that tree's bytes:
the zenc cargo target/ and the uv venv, 767 files. Every build copied a quarter
of a gigabyte in order for .assetsignore to then exclude it from upload.

.build/www     292 MB -> 34 MB
cold build     3.0s -> 2.3s   (3 runs each)
routes:check   11.6s -> 10.5s

It lands three times per CI run rather than once, since wrangler.jsonc carries
build.command and both wrangler deploy --dry-run and routes:check re-run
it. Confirmed by watching .build/www/index.html's mtime move under each.

The staged tree is byte-identical: 1444 files, 0 differing against a baseline
built from the previous commit. That is the bar rather than a nicety, because
/a/ and /i/ are content-addressed, so one differing byte mints a new URL and
orphans the a-dict snapshots naming the old hash.

Safe because every consumer imports from the SOURCE tree, so nothing has ever
read .build/www/scripts. link-integrity is what catches a page that starts
to, verified with a link injected into page prose.

Ruled out and recorded at the code so nobody redoes it: raising
UV_THREADPOOL_SIZE past its default 4 buys about 10% (2777ms to 2503ms at 8 on
14 cores) and then flattens, so the q11 fan-out is not what the build waits on.

3. The fingerprint index is stored in the direction it is read

fingerprints.json held stem -> {a,j,s} while the only question anything asks
it is the reverse: photo_recipe hashes an upload and wants to know which photo
those bytes are. So the lookup was a nested scan over 474 entries with an
Object.entries() allocation per call. Keyed by digest, JSON.parse hands back
the index itself: 0.0205 ms to 0.000039 ms worst case.

Cold path (an MCP tool, called rarely), so this is about the shape being right
and degrading linearly as photos grow rather than about the microseconds.

Deliberately a plain object rather than a Map parsed at the loadJson
boundary: building a 474-entry Map costs ~0.072 ms, more than the scan it would
replace, so the boundary parse would spend the entire win.

The wire cost goes UP 214 bytes brotli, and the reason is recorded at the
generator so nobody "optimises" the encoding later. sha256 is incompressible, so
474 digests sit on a ~15.2 KB entropy floor and brotli already recovers the hex
alphabet's waste: hex 16.1 KB, base64 16.4 KB, raw bytes 15.4 KB. Only truncating
the digest moves that number, which is a separate decision about how strong a
claim the tool is making.

Two new invariants, since an inverted index can lose an entry in ways the old
shape could not: the generator refuses a repeated digest instead of picking one,
and check-photo-pipeline.mjs asserts the digest count.

What the controls corrected

Both worth knowing, because in each case the first version looked right.

  • The first link-integrity control was INVALID. It added a <script> before
    </body> and tripped the desktop-partial tripwire instead; the tell was that a
    path which IS staged failed identically. A control that fires for the wrong
    reason reads exactly like one that works.
  • My comment on the new digest-count assertion claimed it backstops a COLLISION.
    It does not: a collision trips the per-stem loop first, because the surviving
    entry names the other stem and that reads as drift. What the count uniquely
    catches is a stale digest still naming a published stem.

The JSON question that started it

Format was the wrong axis. Every payload the browser parses is under 0.3 ms, and
any replacement needs a userland decoder while JSON.parse is C++ in V8:

format raw brotli decode
JSON.parse (today) 53,185 2,617 0.142 ms
hand-rolled binary 53,182 2,814 0.177 ms
delimited + split 30,983 2,461 0.241 ms

The delimited version reads 42% fewer raw bytes and still loses. Two shapes did
measure better and are deliberately NOT in this PR, because the absolute numbers
are below the noise until the photo count grows roughly 10x: column+dictionary
exif.json (2,617 to 2,312 brotli, parse 0.149 to 0.032 ms) and bundling the 12
rendered histograms as bytes (2,880 br across 12 requests to 1,432 br in 1).
That second one needs a hover-rate number the repo does not collect yet.

Verification

Build, 291/291 contract tests, lint (oxlint incl. type-aware), typecheck,
photos:check, pages:check, routes:check (136 routes, 0 failures),
perf-budget, and wrangler deploy --dry-run all green locally.

One freshness note: the a-dict commit is data sourced from production
today. If this sits and a shell asset ships meanwhile, re-run
pnpm run dict:roll before merging so it adopts current bytes.

fingerprints.json held `stem -> {a,j,s}` while the only question anything
asks it is the reverse: photo_recipe hashes an uploaded image and wants to
know which published photo those bytes are. So the lookup was a nested scan
over all 474 entries with an Object.entries() allocation per call.

Keyed by digest, JSON.parse hands back the index itself and the lookup is one
property access: 0.0205 ms -> 0.000039 ms worst case. That is a cold path (an
MCP tool, called rarely), so this is about the shape being right and degrading
linearly as photos grow rather than about the microseconds.

Deliberately a plain object rather than a Map parsed at the loadJson boundary.
Building a 474-entry Map costs ~0.072 ms, more than the scan it would replace,
so the boundary parse would spend the entire win.

The wire cost goes UP by 214 bytes brotli (16,134 -> 16,348), because sha256 is
incompressible and inverting duplicates the tier suffix. Worth recording why no
encoding change is coming: 474 digests sit on a ~15.2 KB entropy floor and
brotli already recovers the hex alphabet's waste, so hex lands at 16.1 KB
against base64's 16.4 and raw bytes' 15.4. Only truncating the digest moves
that number, and that is a separate decision about how strong a claim the tool
is making.

Two invariants come with it, because an inverted index can lose an entry in
ways the old shape could not:

- the generator REFUSES a repeated digest instead of picking one. Two published
  tiers with identical bytes were always ambiguous; stem-first merely hid it,
  since the scan returned whichever it reached first.
- check-photo-pipeline.mjs asserts the digest count. Controls established what
  each check actually catches rather than what it looked like it caught: a
  corrupted digest and a collision both trip the per-stem loop as drift, and
  the count is the only witness for a STALE digest still naming a published
  stem. The first draft of that comment claimed the opposite and the control
  is what corrected it.

The per-stem check now asserts in the read direction too: hash the published
bytes, and the index must name that exact stem and tier.
build.mjs staged www/ verbatim into .build/, and www/scripts is 89% of that
tree's bytes: the zenc cargo target/ and the uv venv put it at ~232 MB across
767 files, against ~27 MB for everything that actually ships. So every build
copied a quarter of a gigabyte in order for .assetsignore to then exclude it
from upload. .build/www drops 292 MB -> 34 MB.

Cold build 3.0s -> 2.3s, measured over 3 runs each. It lands three times per CI
run rather than once, since wrangler.jsonc carries build.command and both
`wrangler deploy --dry-run` and `routes:check` re-run it (confirmed by watching
.build/www/index.html's mtime move under each). routes:check 11.6s -> 10.5s.

The staged tree is BYTE-IDENTICAL: 1444 files, 0 differing against a baseline
built from the previous commit. That is the bar rather than a nicety, because
/a/ and /i/ are content-addressed, so one differing byte mints a new URL and
orphans the a-dict snapshots naming the old hash.

Safe because every consumer imports from the SOURCE tree (build.mjs,
contract-tests.mjs, gen-manifest.mjs, and the package.json script paths), so
nothing has ever read .build/www/scripts. link-integrity is what catches a page
that starts to, and the control is recorded at the code: a link to
/scripts/shell-data.mjs injected into page prose fails the build by name.

Worth keeping the first attempt at that control, which was INVALID. It added a
<script> before </body> and tripped the desktop-partial tripwire instead, and
the tell was that a path which IS staged failed identically. A control that
fires for the wrong reason reads exactly like a control that works.

Not done, and measured so nobody re-does it: raising UV_THREADPOOL_SIZE past
its default 4 buys ~10% (2777ms -> 2503ms at 8 on a 14-core machine) and then
flattens, so the q11 fan-out is not what the build waits on.
…ionary

`dict:roll --live` discovered the shell by fetching 4 documents and scanning
them for /a/ refs. An asset loaded lazily FROM JavaScript appears in no
document, so it was structurally invisible: measured against production, the
HTML-only scan found 8 assets and the graph holds 19.

The 11 it could not see had therefore never been given a dictionary at all.
Six of them are on the every-page path, because nav.js pulls nav-run.js,
nav-tray.js and infotip.js plus their CSS. lens.js is another, at 22.6 KB
brotli the largest asset on the site. A returning visitor re-downloaded each in
full where a delta is a few hundred bytes.

References are transitive (/lens -> lens-boot -> lens -> lens-tools is three
hops), so this closes over the graph rather than following one level and
calling it done. Termination is the visited set against a finite /a/ namespace,
one fetch per name.

Bodies are cached for the adoption loop below, which wanted the same bytes.
That saves a request per asset and closes a real window: fetching twice lets a
deploy land between the scan and the adopt, so the roll would store bytes this
run never read.

Coverage goes 8/19 bases to 19/19, with all 19 current-hash snapshots on hand.
Note the payoff is on the NEXT change to each asset rather than today: a
dictionary has to be bytes the browser already holds, so adopting what
production serves right now is what makes the following deploy cheap.

This is the same failure CLAUDE.md's dcz section already records ("Not urgent
was wrong, and the cost of believing it was 161 commits"), arriving through a
different door. That one was a roll nobody ran; this is a roll that ran nightly
and could not see two thirds of what it was rolling.
Separate commit because a roll writes into the source tree, which the build
never may, and because this one is DATA where the parent is code.

Adopts the 11 assets the previous commit's discovery gap kept invisible
(nav-run, nav-tray and infotip js+css, lens and its four lazy panes), and
prunes two stale lens snapshots past the 3-per-base keep. Sourced from
https://aadhar.sh, so these are bytes browsers are holding right now.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
aadhar-sh eac808e Aug 17 2026, 02:57 AM

@github-actions

Copy link
Copy Markdown
Contributor

Wire-size diff vs merge base

5ac6509 → 9f01862

Brotli q11, the quality the edge serves. Every number here is deterministic:
an unchanged file produces no row. Advisory — this check fails on nothing.

Client assets

No change. 21 files, 74.97 KiB Brotli total.

Pages

No change. 48 files, 500.45 KiB Brotli total.

Worker bundle

No change. 228.47 KiB gzip.

Dictionary deltas (pd/)

No change. 144 deltas, 448.45 KiB total.

@oddharsh
oddharsh merged commit da7f047 into main Aug 17, 2026
7 checks passed
@oddharsh
oddharsh deleted the fingerprints-inverted-index branch August 17, 2026 02:58
oddharsh added a commit that referenced this pull request Aug 17, 2026
The rebase onto main brought #439/#440/#441 (the 200px tier, packed histograms,
the fingerprint index and one fewer copy of the photo data), and that code was
written against www/. Four integration fixes it needed:

- build.mjs's meta-twin derivation wrote to ${OUT}/www/images/meta and read
  exif.json + histograms.json from the same place, so it generated 0 files. Its
  own floor caught it, which is the tripwire doing exactly its job.
- the homepage bake lost its `histograms` load in a conflict resolution and
  failed with a bare ReferenceError. Restored from main.
- contract-tests imported ../www/scripts/build-histogram-index.mjs and read
  www/images/histograms.json.
- tooltip.js cited www/scripts/build-histogram-index.mjs in a comment.

That last one was caught by the served-path contract test added earlier on this
branch, which is the first time it has found something it was not written for.

Also un-resurrected 158 files. #441 DELETED www/images/meta and gitignored it:
the per-photo twins are derived from exif.json + histograms.json now, and a pure
function of committed bytes should not be a second committed copy. The bulk
conflict resolution added them back at their moved path, which would have undone
that silently. The .gitignore rule and build.mjs's STAGE_SKIP both survive,
repointed to public/images/meta.

307 contract tests (main's 12 coverage tests came across), typecheck, lint,
check-wrangler, pages:check, tools:check and perf-budget all clean. Bundle
669.37 KiB, up from 666.62 because main's photo work is now in it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant