Scaffold pnpm-based monorepo with TypeScript, Vite, Vitest, Biome, Zod, zarrita, and Docusaurus - #1
Merged
xinaesthete merged 6 commits intoOct 9, 2025
Conversation
Co-authored-by: xinaesthete <674834+xinaesthete@users.noreply.github.com>
Copilot
AI
changed the title
Scaffold a pnpm based monorepo for a TypeScript library mirroring the Python SpatialData package.
It should use vite, vitest, zarrita, biome, and zod. There should be a 'core' package containing the main interfacing code, and also a docusaurus site.
Scaffold pnpm-based monorepo with TypeScript, Vite, Vitest, Biome, Zod, zarrita, and Docusaurus
Oct 7, 2025
xinaesthete
previously requested changes
Oct 7, 2025
Contributor
There was a problem hiding this comment.
@copilot We should get rid of the boilerplate tutorial stuff in docs/authors etc.
xinaesthete
marked this pull request as ready for review
October 9, 2025 08:48
Contributor
|
This isn't in any actual working state in terms of real functionality, but no reason to keep it in a PR. |
xinaesthete
added a commit
that referenced
this pull request
Jul 16, 2026
* Consume Resource Resolvers in useLayerData (Step 1, all 3 increments) Make useLayerData *consume* the resolvers #85 landed unconsumed, per docs/plans/step1-consumption-tactics.md. The 17-member public surface is unchanged; useLayerData.spec.tsx stays green throughout. Net -724/+394 lines. - Inc 1 (shapes): ShapesResolver drives geometry/tooltip/fill-colour-row loads; vis-side projection memos handle the tooltip->geometry patch (coupling #1) and keep prebuilt/fill-colour lazy. Fill-colour entry is withheld until rows load so the feature-state runtime rebuilds and fill colours actually appear. - Inc 2 (images + labels): ImagesResolver/LabelsResolver consumed via getLoadedData; LabelsLoaderData retyped to LabelsChannelDefaults (tooltip is now a separate resource). Physical-size world-bounds compute kept in the hook (coupling #2). - Inc 3 (store): one SpatialEntryStore + one reconcile() commit-effect replace the per-kind driving effects (hook now has two useEffects total). Points is wrapped in a non-owning proxy so the stable PointsDataEngine the panels subscribe to survives a store rebuild on dataset swap; points row-codes/matching stay on the render-phase engine calls in getLayers (Track A). Verified: vis typecheck + build clean, full suite 549 passing, Biome gate clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * disable tsgo extension in this workspace, pending upgrade to TS7 * Document the non-owning points-resolver lifecycle + guard it with a test The store is designed to own its resolvers (subscribe + dispose), but points is owned by the stable PointsDataEngine the panels subscribe to — so the store borrows it through createNonOwningResolver (no-op dispose) to avoid clearing the engine's cache on a dataset-swap rebuild. Expand the rationale where it goes against the store's ownership grain: the two-owner problem, why the no-op is correct not just safe, why it doesn't reintroduce "points is special" in the store, alternatives rejected, and the exit condition. Add the ownership model at the construction site and note the StrictMode useMemo-subscribe caveat. Add a lifecycle test: a spatialData swap rebuilds the raster resolvers and the store, and the points cache/render-resource identity must survive it — the test that fails if the proxy ever regresses to a real dispose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs: reflect Resource Resolver Step 1 as landed + add changeset - resource-resolver-handoff.md: flip Status off "ready for implementation", add a Progress section and mark Step 0/1 landed (contracts, four resolvers, useLayerData consumption via SpatialEntryStore.reconcile); note the non-owning points proxy and the deferred render-phase points calls (Track A). - spatial-canvas-status.mdx: replace the stale "minimal ScatterplotLayer" points description with the PointsDataEngine reality; retitle useLayerData; add a Resource Resolver entry to "Recently landed"; refresh the feature/table roadmap item (tooltip/pick routing done, ping-pong remains). - Add a changeset for the vis-side resolver consumption. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Extract non-owning proxy + shapes projection out of useLayerData Intermediate decomposition pass, no behaviour change. Moves two cohesive, kind-local chunks out of the 1.6kloc hook: - resolvers/nonOwningResolver.ts — `createNonOwningResolver` + its lifecycle rationale (the store-ownership exception for points). - shapesProjection.ts — the shapes feature-state / fill-colour projection helpers and their cache-entry types (`ShapePrebuiltEntry`, `ShapeFillColorEntry`, `getStableShapeFeatureStateRuntime`, signature/serialise helpers). This is the `project()` half of ADR 0004 §4; a vis-local waypoint before Step 3 relocates it into @spatialdata/layers. useLayerData.ts drops from 1623 to 1446 lines and imports both. The shapes read path now lives in a small dedicated module, so Track B / Step 3 touch it rather than the hook. Behaviour-preserving: control-char signature separators kept byte-identical; full suite (550 tests) green, vis typecheck + build clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Review fixes: preserve explicit shape stroke; replan on element-map change Two code-review findings, verified against current code: - shapesProjection: mergeShapeFeatureStateForRender no longer clobbers an explicit per-feature strokeColorByFeatureId when a fill-by-column encoding is active. It now mirrors the fill map only when the caller has NOT set an explicit stroke override (the schema allows both together, e.g. via SpatialLayerProps). Signature unchanged and deliberately so: it already hashes the explicit stroke, which is what drives the render; dropping that term would stale the runtime when the stroke changes, and in the mirroring case the term is already empty. - useLayerData: the reconcile effect now depends on elementMapValue, so it replans when element resolution changes without layers/store changing — e.g. a coordinate system switch that makes a previously unavailable element resolvable. The map is memoised on availableElements, so no per-render churn. Skipped: wiring reconcile into reloadElement (the finding's other suggestion) — reloadElement has zero runtime callers (dead surface, per the Step 3 punchlist), so it would fix nothing observable. Verified: vis typecheck + build clean, Biome gate clean, full suite 550 passing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
xinaesthete
added a commit
that referenced
this pull request
Aug 24, 2026
The overlay would mark a tile as errored while deck was holding good content for it. Two loads for one tile can be in flight at once. deck restarts a tile whenever needsReload is set — after an abort, or when a getTileData update trigger changes, which the feature filter does on every toggle — and Tile2DHeader.loadData does not await the attempt it replaces. deck guards its own state against the loser with a _loaderId compared after its await, and discards the stale result. The debug hooks are called from inside getTileData, upstream of that check, so they had no guard of their own: whichever attempt reported last won the entry. When the abandoned one rejected last, the tile went red even though the replacement had resolved. onTileLoadStart now returns an attempt id and onTileLoadEnd requires it back; a report whose id is no longer current is dropped, returning the state object unchanged so the store's signature comparison does not even notify. Required rather than optional, so forgetting to pass it cannot quietly reinstate the race. attemptByTileId joins the state signature for the same reason — a bumped attempt discarded as a no-op would leave the guard matching a stale report. The mirror case is covered too, and is why the guard is not simply "never overwrite an error": a stale SUCCESS landing while the replacement is still in flight would claim the tile is done when nothing has drawn it. Reproduced headlessly first — start, start, end(success, #2), end(error, #1) — and verified non-vacuous: with the guard disabled both new tests fail. Checked in the app on the 12.1M-point element by flipping the feature selection eight times in under two seconds, which is the abort-and-restart path: 0 errors afterwards, and no tile the overlay called an error that deck held content for. Behaviour outside the overlay is unchanged; the points themselves were always correct. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
xinaesthete
added a commit
that referenced
this pull request
Aug 24, 2026
* Points: probe for a Morton-tiled artifact before preloading (D5 step 1)
`PointsResolver` gains a `tiling` resource — a one-key `RequestSlot` holding the
element's tileable Morton metadata, or `null` when it cannot drive viewport tiles
(no artifact, no row-group range reads, no bounds, or a failed probe). `plan()`
asks `planPointsLoads` for the probe and preload decisions together, so a tileable
element no longer schedules a full-table read it would immediately throw away.
The row-codes and matching tasks are deferred on the same question. Gating them on
`isTiled` alone is not enough: planning them while the probe is still in flight
does the wasted read AND settles the codes, so the next pass reads "already
loaded" and the waste becomes invisible — the exact shape of bug the deferral
exists to prevent.
A failed probe reads two ways on purpose: `failed` and retryable in the snapshot,
but `null` ("cannot tile") to the planner, so the layer falls through to the
ordinary preload rather than stranding — and `isTilingSettled` counts it as
answered, so a persistent failure does not re-probe every reconcile.
Opt-in and inert by default: `PointsResolveConfig.pointsTiling` defaults to
`'off'`, collapsing planning to exactly today's behaviour. Nothing renders through
the tiled path yet.
`planPointsLoads` moves to core (core cannot import from layers, and a second copy
is how the two drift); layers re-exports it, so no consumer import moves.
Plan: docs/plans/points-morton-tiled-viewport-loading.md, harvested from
backup/points-wip-20260702.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points (Morton): cap the z-cover depth, dedup concurrent row-group probes
Diagnosing holes in the tiled points render ruled out the store: on a real 12.1M
-point Xenium artifact the stored morton codes reproduce exactly from the sentinel
bbox under the reader's own formula (215k rows sampled, 100% match), row-group
statistics are monotonic and non-overlapping, byte ranges are contiguous, and the
reader's row-group selection for a viewport rectangle is exact — 92 of 92, nothing
missed, nothing wasted. What is wrong is how much work it does to get there.
`zcoverRectangle` recursed to the full 16 bits per axis, resolving the rectangle to
individual quantised cells when its only job is picking row groups: 38,014 intervals
for one viewport-sized query, each driving two bisects. Capped at depth 10 that is
521 intervals selecting the SAME 92 row groups — verified over a viewport tile, the
whole slide and a zoomed-in box. Stopping early only ever widens a cell, and the
extra rows are filtered against the exact bounds after the read, so the cover stays
complete; the tests pin completeness rather than the interval count alone.
The extent cache held the settled value, so it deduped nothing while a read was in
flight — and this index is built under exactly that load, every viewport tile
bisecting over the same row groups at once. Each tile started its own full row-group
fetch for an entry the others were already fetching: 8 concurrent callers cost 18
range reads, now 4. A failed probe is evicted instead of cached so a transient error
cannot strand the bisect for the life of the source.
The real fix for that probe is to read the column statistics already sitting in the
footer we parse, rather than range-reading and decoding the row group twice. The
vendored parquet-wasm build exposes no statistics accessor, so it needs a wasm
rebuild or a minimal Thrift footer read; the constraint is now documented at the
call site.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points (Morton): fix the row-group bisect dropping groups (the render holes)
The tiled points render had holes. The store was not at fault: on the real 12.1M
-point artifact the stored morton codes reproduce exactly from the sentinel bbox
under the reader's own formula, every row group decodes, and all 12,165,021 rows are
readable. The reader was losing them.
`readParquetRowGroupColumnExtent` took a row group's last value with
`readParquetRowGroup(..., { offset: rowCount - 1, limit: 1 })`. The vendored
parquet-wasm ignores `offset` on a row-group read and hands back the FIRST row again,
so every row group reported max === min — that it spanned a single code. Measured
against the parquet statistics: row group 1 reported (437752573, 437752573) for a
true span of (437752573, 724881652).
Nothing errored, because an understated max is still a valid answer to a monotone
predicate. The bisect asks "first row group whose max >= target", so it landed one
group late and never read the group CONTAINING the interval start. One viewport query
lost 11 of the 92 matching row groups — 187,990 points, 6% — in Z-order-shaped bands,
which is exactly what the holes were. Verified end to end: the same query now scans
every matching row group and returns 3,128,988 points, matching an independent
full-table scan (the residual 2 against pyarrow are float32/float64 boundary
comparisons, not losses).
The bound now comes from the sort order the format already guarantees: a row group's
values lie at or below the next group's first value. Conservative where equal codes
span a boundary, needs only the read that works, and halves the reads since each
first value is cached and shared with its neighbour. The last row group keeps an open
bound, which the bisect already reads as "may contain the target".
The regression test asserts an exact count for a bounded query — under-selection is
silent by construction and only a total can see it. Confirmed it fails against the
old behaviour (224 of 260 points).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: render Morton-tiled elements from viewport tiles (D5 step 2)
A layer with `pointsTiling: 'auto'` on a Morton artifact now draws through
`mortonTiledStrategy` — deck's TileLayer reading row groups for the viewport —
instead of a memory-capped resident preload. Verified on a real 12.1M-point Xenium
element: tiles load and pan, the layer frames itself, and the memory cap stops
applying to it.
The snapshot is what makes it drawable. A tiled entry plans no preload, so a preload
resolution left sitting at `idle` reads as blocking forever — and auto-fit rides the
isBlocking true->false transition, so the layer would never frame either. Instead the
entry reports only the resources it HAS: no preload (absent, and `isBlocking` already
skips a resource that is not there), plus world bounds derived from the artifact's own
extent, which is the only thing that lets auto-fit run before a single tile has
loaded. `blockingResources` grows to cover the probe, since until it answers we do not
know which path the entry is on.
The adapter's tiled resource is memoised on (element, metadata). That is not a
nicety: the resource's loader is what TileLayer keys `getTileData` on, so a fresh
identity per project() refetches every visible tile and turns a pan into a full
reload.
Tiling is per LAYER but the probe's answer is cached per ELEMENT, and reading the
probe alone is wrong — found in the browser, not by a test: switching tiling off left
the layer still drawing tiles while plan() went back to preloading, so it did both.
Every consumer now combines the two (`isTiledFor` in core, `usesTiledPath` in vis).
Known and deliberate, tracked in the plan doc: a tiled layer draws flat-coloured and
ignores the feature filter (the tile scan does not return per-point codes yet — step
3); switching to tiling does not evict the preload already performed, so the panel's
truncation notice still reports that resident memory over a tiled render; and the
tiling control's status line reads the engine without subscribing, so it can lag the
probe by a render.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: release the preload on a tiled layer, and stop mis-reporting it
Three follow-ons from watching a tiled layer in the app, plus the sizing choice
that made a real acquisition boundary look like a rendering fault.
Releasing the resident window. A layer switched to tiling mid-session has usually
already preloaded, and nothing gave those rows back: plan() stops ASKING for a
preload, which is not the same as evicting one, so up to the full memory cap stayed
held behind a render that never reads it. The probe's settle now drops the preload,
its row-aligned codes and its feature-index scan — all three are defined against that
window, so keeping them would leave state describing a batch that no longer exists.
The catalog stays: it describes the element's features, not the window. Eviction runs
once, on the settle, so a second layer reading the same element un-tiled re-requests
it on its next plan pass rather than ping-ponging.
The truncation notice. It read "4,000,000 of 12,165,021 points in memory — capped;
raise the cap for more" directly above "the memory cap does not apply" — each true,
the pair nonsense. A tiled layer draws from the viewport, so a resident count says
nothing about what is on screen; the panel now says nothing instead.
The tiling status line. It read engine.isTiled() directly, outside the engine
subscription, so it kept insisting an element had no Morton index while tiling it —
the probe settles asynchronously and nothing re-rendered the line. `tiled` now comes
through usePointsFeatureState, which carries the subscription.
Point sizing is one behaviour instead of two. Tiles sized in fixed pixels while the
preloaded path used world units, so pointSize meant different things depending on a
checkbox, and a zoomed-out tiled layer drew every one of millions of points as a fixed
screen dot: density saturated to a flat mass and every tile seam and density edge
hardened into an artefact. Both paths now size in world units with the model-matrix
scale folded in, so points shrink as you zoom out and overdraw self-limits.
Verified in the app: the capped notice is gone from a tiled layer and the status line
is correct on first paint. The sizing change is pinned by unit tests on the layer
props rather than pixels.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: colour a Morton-tiled layer by feature (D5 step 3)
Viewport tiles now carry a feature code per point, so a tiled layer colours, takes
per-feature overrides and answers Feature Highlight exactly like the preloaded path.
Until now it drew flat, and the same element looked like two different datasets
depending on a checkbox.
The codes were already being read and discarded: the tile scan consults the feature
column in order to filter on it, then returned bare coordinates. So the change is
mostly about not throwing them away — `scanMortonTableInBounds` takes an optional
Int32PointBuffer and appends to it in lockstep with the geometry, and the worker's
tile-scan handler builds one and returns it. The worker boundary needed nothing: the
protocol already declared an optional featureCodes on the columnar result and already
transferred its buffer.
One real gate had to move. `loadMortonPointsInBounds` projected the code column only
when a filter was active, which is exactly backwards for colour: the no-filter "all
features" view is the common case and was the one arriving without codes. It now
projects whenever the artifact HAS a code column, and both the worker and main-thread
returns carry the result.
Short codes are dropped rather than padded, on both paths. A partial array would leave
the tail reading code 0 — a VALID feature — and mis-colour it with conviction, which
is worse than the flat fallback.
Verified on the real 12.1M-point element: a tiled layer draws per-feature colours
matching the preloaded path. Tests pin one code per point, every code a real catalog
entry, and codes still returned (all equal to the selection) under a filter.
The feature FILTER still does not narrow tiles — that is step 4 — so a tiled layer
draws every feature in the viewport regardless of the selection. The panel now says
so rather than claiming flat colour.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: apply the feature filter to Morton-tiled layers (D5 step 4)
A tiled layer drew every feature in the viewport whatever was selected. The selection
now reaches getTileData — and its updateTriggers, so changing it refetches instead of
serving the previous selection's cached tiles — and is applied INSIDE the row-group
scan, so a tile arrives holding only what was asked for. On the real 12.1M-point
element one gene takes a viewport tile from 3,128,988 points to 87,594.
The composite's own filter machinery is untouched and needs to be: it is gated on the
preloaded-columnar kind, so a tiled layer passes straight through rather than
filtering twice. renderCap stays unset — a resident-window notion, and a tile is
already bounded by its viewport.
The catalog is now planned for a tiled entry. On the preloaded path it arrives free as
a preview off the geometry decode; a tiled entry never decodes a resident batch, so
nothing built one — and a selection persists as feature NAMES, which cannot become the
codes the scan filters on without it. A saved config with a selection would draw
every feature until someone happened to open the panel. A failed catalog is
deliberately not re-planned: it reports undefined exactly as one that never ran, so a
gate on the value alone re-emits the task on every reconcile forever.
The feature-row panel needed a tiled case of its own. Every other signal it reads
describes a resident batch a tiled layer does not have, so its rows fell through to
"beyond the resident window; select it to fetch its points" — greyed, and wrong twice
over: the points are available, and no feature-index scan is involved.
I had this wrong in the plan and briefly in the code: the filter does NOT narrow I/O.
Measured on the same query, unfiltered vs one gene — 92 row groups and 158.1MB either
way, 3,128,988 points vs 87,594. Row groups are chosen SPATIALLY on a Morton artifact
and a gene's points are spread across all of them, so no feature filter can skip one.
The win is 36x fewer points leaving the worker, not less reading. Narrowing the fetch
by feature needs a feature-primary index — the open index-selection question in ADR
0002/0003, and what the `*_feature_then_morton` permutations in the test store exist
to explore. The plan now records the measurement rather than the expectation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: distrust a Morton sentinel box that is not the code domain (D5 step 5)
The sentinel rows are a claim a Parquet artifact makes about itself, and nothing
else in the file forces them to be true. Believing a wrong one does not fail: the
box is both the TileLayer extent and the domain mortonIntervalsForBounds
normalises against, so a wrong one clips the tile grid — whole regions are never
requested, no pending tile, nothing to wait for — and mis-maps every viewport to
row groups. Points are never misplaced, because the reader re-filters to the query
bounds, so the only symptom is that part of the map is missing. That is the third
silent under-selection on this path after the zcover depth and the row-group
bisect.
The probe now recomputes morton_code_2d from x/y for a sample of real rows and
refuses to tile unless a majority agree. That tests the invariant that matters —
is this box the quantisation domain? — not a convention, so an artifact with a
deliberately padded domain still tiles. Samples come from the middle of the file:
a truncated box can agree with the true one near the origin by coincidence, never
in the interior. A rejected element reports supportsRowGroupRangeReads: false with
no bounds, the same pair the oversized-sentinel case already produced, so it
degrades to the capped preload through the existing resolver gate — and warns,
since a silent downgrade is what got us here.
Found by way of the index-permutations store, whose *_feature elements recorded a
box a quarter the size of their own domain: 0/320 sampled rows reproduced their
codes from it, 320/320 from the true extent. Those files have been regenerated in
place; the writer never had the bug, they were just older than it.
Cost is one extra row-group read per element, cached with the metadata — 3 range
reads / 0.37 MB / 414 ms becomes 4 / 2.16 MB / 523 ms on a 12.1M-point element,
about one step of the bisect a single viewport query already runs eight of.
The plan doc also records what the tile grid actually does, measured rather than
assumed: minZoom/maxZoom -1 with tileSize 512 pins every tile at 1024 local units
at every zoom, so the whole element is one fixed 11x4 grid that never subdivides.
Zooming in gets no more detail and zooming out reads all 44 tiles unbudgeted. The
pointsTiling default cannot flip until that is fixed, which is not the probe cost
the open question assumed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: refuse to Morton-tile a file whose code column is not sorted
Having a morton_code_2d column does not make a file Morton-ordered. A
feature-primary artifact — sorted (feature, morton) — carries the identical
column with identical, correct values, a correct sentinel box, and every field the
tiling probe looks for. Only the order is wrong, and nothing in the file said so.
The row-group bisect binary-searches that index assuming it ascends, so on such a
file it landed arbitrarily and a tile came back holding whichever feature blocks
happened to live in the row groups it picked: some tiles showed one or two genes,
most showed none. That is what `transcripts_feature_then_morton` has been doing.
The probe now reads the per-row-group [min, max] for the Morton column out of the
parquet footer and requires it to be non-decreasing. On the permutations store
that element descends at 185 of its 244 row-group boundaries; both morton-primary
elements descend at none — INCLUDING transcripts_morton_then_feature, so a
secondary feature key stays supported and the verdict is on the file rather than
on the element's name. The store's index-manifest does say `experimental` for the
feature-primary condition, but a reader cannot rely on a manifest; now the file
answers for itself.
The check is free: datasetMetadata.parts already carries the footer bytes when the
probe runs, and the statistics are complete on all 245 row groups. Failing it also
short-circuits the sentinel sampling read, since the outcome can no longer change,
so a rejected element now costs less than it did before either guard. The element
still loads through the capped preload, and the feature-code row-group index it
exists to exercise lives on that path and is untouched.
decodeUnsignedIntStat is new because morton_code_2d is uint32, which parquet
stores as INT32 with a UINT_32 annotation: Morton codes use the top bit for real,
so decodeIntStat reads the far corner of a slide as negative.
Those statistics also retire a standing TODO. loadParquetRowGroupColumnExtent says
it should be reading row-group statistics and instead range-reads and decodes each
group twice for a few boundary values; they are right here, complete, and free.
Noted against step 6, where a subdividing grid will issue many more bisects.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: index row groups from footer stats, size the tile grid from the data (D5 step 6)
Two things, and the first is what makes the second affordable.
**Row-group selection stops reading the file.** selectMortonRowGroups picks row
groups from the per-row-group [min, max] the tiling probe already parsed out of the
parquet footer. The bisect it replaces range-read the row group's BYTES — every
column, ~2MB on a real transcripts artifact — to recover two boundary values,
log2(rowGroups) steps per Morton interval, a few hundred intervals per query. That
is how a viewport query could pull most of a 439MB file to answer a question the
footer had already answered. On one 1024um tile of the 12.1M-point element, both
returning the same 643,961 points:
bisect 97 range reads / 175.12 MB / 2911 ms
footer index 32 range reads / 57.83 MB / 1035 ms
The remaining 32 reads are the row-group data itself. The new path is also stricter
— the bisect tested only `max` and assumed the groups tile the code space without
gaps, this intersects both ends — and the bisect stays as the fallback when
statistics will not parse, so it is an optimisation, not a new requirement.
**The grid comes from the artifact.** It was one fixed level, so every tile was 1024
local units at every zoom: zooming in read a 1024-unit tile to look at 50 units of
it, and 1024 came from deck's defaults rather than from the data. mortonTileGrid
derives both ends from the point density — finest stays at least one row group's
footprint, since below that four tiles fetch what one used to for the same bytes;
coarsest holds at most 400k rows, so one request is a fraction of the layer. And
zoomOffset = log2(modelMatrixScale) couples deck's z, chosen from a WORLD-space
zoom, to tile spans expressed in LOCAL units — the model matrix is exactly that
difference, and without it a tile lands at whatever size the transform implied.
For the Xenium element that is two levels, 1024um and 512um. The narrowness is the
finding, not a shortfall: the row-group size is the floor, and 50k-row groups put it
at ~402um. The old fixed 1024 was accidentally near-optimal for this file and would
not be for one an order of magnitude smaller or denser.
**The tile cache is budgeted in rows.** maxCacheSize now comes from a row budget
rather than deck's `5 x the selected tile count`, which on a coarse viewport of this
element could retain ~220 tiles / ~71M rows against a 4M resident cap; it is 16
tiles / ~5.2M rows, stated. That answers the plan's open question 3 at the level
ADR 0005 asks for — the second pool is accounted, not managed. Nothing evicts by
bytes, and a tile's real footprint is whatever its points weigh. maxRequests stays
at 6, now as a decision rather than an inheritance.
Verified in the app: at viewport zoom -7.8, 44 tiles at z -1; at -2.8, 6 tiles at
z 0, with the cache trimmed from 44 to exactly 16.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: Morton viewport tiling on by default (D5 step 7, closes D5)
pointsTiling defaults to 'auto' (DEFAULT_POINTS_TILING), read everywhere through
the new pointsTilingEnabled() rather than compared to 'auto' — the default has to
mean the same thing to the resolver deciding what to load, the hook deciding what
to render, and the panel drawing the checkbox, and three literal comparisons is
how those drift apart.
The argument for flipping is not that the probe is cheap, which is how the plan's
open question framed it. It is that on a Morton artifact the capped preload is
NOT a neutral alternative: it keeps the first `cap` rows in FILE order, and file
order there is a prefix of the Z-curve, so the 4M-row default shows a spatially
skewed chunk of the slide rather than a sample of it. Tiles read what is in view.
What it costs, measured rather than assumed: at the default zoomed-out framing of
the 12.1M-point element the tiled path loads all 44 tiles — 12,165,029 points /
~158 MB, the whole artifact — against a 4M-row prefix for the preload. First
paint on a fully zoomed-out view is ~3x the rows, in exchange for a correct
picture that streams in 44 pieces instead of blocking on one decode. Zooming OUT
is the one direction viewport tiling does not help: there is no coarser
representation to read. That wants a multi-resolution points pyramid, not a finer
index, and it is now the loudest thing in the punch-list. `pointsTiling: 'off'`
restores the old behaviour per layer, and the panel keeps the toggle precisely so
that comparison can be made.
An element that cannot be tiled pays one probe (4 range reads / ~2.16 MB, cached;
on a non-Morton element it is footer metadata the preload reads anyway) and is
otherwise untouched. The three guards decline loudly and fall through.
The panel hides the memory-cap control on a tiled layer — it governs nothing
there and sat directly above a line saying the cap does not apply.
Twelve resolver tests changed, which is the honest cost of the flip: a fresh
entry now plans ['tiling'] and defers the preload until the probe answers. The
tests that were about preload/rowCodes mechanics pin pointsTiling: 'off' and say
why; the default gets its own tests rather than being asserted incidentally forty
times.
Docs: D5 closed in the punch-list, per-element path table in
points-preload-feature-filter-status, and the plan marked complete with the no-LOD
gap and the byte-level cache accounting recorded as deliberately open.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Consolidate the footer walks and the probe memos after ADR 0005 landed
Housekeeping against main, no behaviour change.
rowGroupFeatureCodeExtents and rowGroupMortonExtents were the same walk twice —
strip the footer's trailing 8 bytes, parse, flatten across parts, verify the
count against the dataset — differing only in which column they name and how they
decode it. The walk moves to parquetFooterStats as rowGroupColumnStats, where the
footer format is already understood. The decode deliberately does NOT move with
it: it is the part that depends on the column's logical type, and getting it
wrong is silent, since a uint32 Morton code read signed comes back negative past
2^31.
The two row-group index probes each hand-rolled "memoize the in-flight promise,
but forget a null or a rejection" — fourteen near-identical lines apiece for a
rule whose failure mode is invisible. They now share one memoizeProbe, evicting
through the existing evictIfCurrent so a retry that already superseded an entry
is not clobbered by the old promise settling late. That is the plain-Map form of
the discipline ADR 0005 rung 2b applies to parquetTableCache; these values are
numbers, far too small to be worth a byte budget, so ByteLruCache would be the
wrong tool rather than the shared one.
Adds the test for the half of that memo nothing covered. Dedup was pinned; the
retry was not, and it is the half that fails silently — one transient read would
leave the bisect treating a readable row group as unbounded for the life of the
source, visible only as viewport queries returning fewer points. Verified
non-vacuous: without the null eviction the new test fails.
Also corrects loadParquetRowGroupColumnExtent's comment, which still said footer
statistics were out of reach of the vendored parquet-wasm build and named a
minimal Thrift read as the way out. That read is parquetFooterStats, row-group
selection goes through it, and this path is now the fallback.
Considered and not done: expressing the tile-cache budget in bytes to match
MemoryReporting. The grid holds nothing — it is a plan, not a resident cache —
and its budget is derived from DEFAULT_POINTS_MEMORY_CAP, which is rows, so bytes
would make it less comparable to its own input rather than more.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: drop tile debug reports from a superseded load
The overlay would mark a tile as errored while deck was holding good content
for it.
Two loads for one tile can be in flight at once. deck restarts a tile whenever
needsReload is set — after an abort, or when a getTileData update trigger changes,
which the feature filter does on every toggle — and Tile2DHeader.loadData does not
await the attempt it replaces. deck guards its own state against the loser with a
_loaderId compared after its await, and discards the stale result. The debug hooks
are called from inside getTileData, upstream of that check, so they had no guard of
their own: whichever attempt reported last won the entry. When the abandoned one
rejected last, the tile went red even though the replacement had resolved.
onTileLoadStart now returns an attempt id and onTileLoadEnd requires it back; a
report whose id is no longer current is dropped, returning the state object
unchanged so the store's signature comparison does not even notify. Required
rather than optional, so forgetting to pass it cannot quietly reinstate the race.
attemptByTileId joins the state signature for the same reason — a bumped attempt
discarded as a no-op would leave the guard matching a stale report.
The mirror case is covered too, and is why the guard is not simply "never overwrite
an error": a stale SUCCESS landing while the replacement is still in flight would
claim the tile is done when nothing has drawn it.
Reproduced headlessly first — start, start, end(success, #2), end(error, #1) — and
verified non-vacuous: with the guard disabled both new tests fail. Checked in the
app on the 12.1M-point element by flipping the feature selection eight times in
under two seconds, which is the abort-and-restart path: 0 errors afterwards, and no
tile the overlay called an error that deck held content for.
Behaviour outside the overlay is unchanged; the points themselves were always
correct.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: tile debug tooltips, and stop drawing tiles deck has dropped
Two things, found together: the tooltip you asked for, and — while verifying it —
the actual cause of red-looking tiles over resolved data during pan and zoom.
TOOLTIPS. formatPointsTileDebugTooltip and isPointsTileDebugPickObject were both
written, exported and tested, and the overlay's polygons already carried the pick
kind. Nothing in vis consumed any of it, so hovering a tile did nothing — the
formatter had a test and zero production callers, the same lost-wiring shape as
the rest of D5. getFeatureTooltip now recognises a tile-debug pick and returns the
entry: tile id, status, viewport batch progress, index, bbox, clipped bbox,
duration or elapsed, point count, load mode, and the error message. That last one
is the point. The overlay can only paint a rectangle a colour; "which tile, and
why" had nowhere to appear.
STALE TILES. The overlay only ever grew. completedTilesById is one of the sources
the active set is rebuilt from and nothing pruned it, so every tile ever loaded
stayed painted — including tiles from a zoom level the view had left, drawn over
ground the current tiles had since rendered. Measured in the app: 62 rectangles
for a 44-tile viewport, the extra 18 at a zoom level no longer in use, four of
them aborted and not even in deck's cache any more. aborted fills dusty red
([180,80,80]) against error's [220,60,60] — at a glance the same thing. Panning
and zooming alone is enough, which is exactly the report.
My own step 6 made it visible: with the old single fixed zoom level there was
never a tile at another z to leave behind.
Fixed by wiring deck's onTileUnload rather than inventing a pruning heuristic.
That is the right boundary — while a tile is in deck's cache it can be reused and
belongs on the overlay; once evicted it does not. Forgetting a tile also takes it
out of lastViewportTiles, or the rebuild that follows every event puts it straight
back as a pending tile nothing will ever load. Verified live after a zoom in and
out: nothing drawn that deck is not holding, where before four such tiles were.
Also raises the timeout on the feature-catalog tests that build a parquet fixture
through `uv run python`, matching the 120s their own beforeAll already uses.
vitest's 5s default is not a budget for that; they timed out intermittently under
full-suite parallelism while passing in isolation. Pre-existing — reproduced on a
stashed clean tree before changing it, so this is not my flake being papered over.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: recolour the tile overlay's aborted state away from red
aborted was a dusty red, [180,80,80] against error's [220,60,60]. A cancelled
request is not a failure, but at a glance the two were the same thing — and since
an abandoned tile stays drawn until deck unloads it, that red could be sitting on
top of ground that had loaded perfectly. It is violet now: an unoccupied hue (the
palette otherwise runs neutral / amber / green / blue / red) at an alpha near
pending's, because a request the viewport moved on from is the least interesting
thing on the overlay. Red now means exactly one thing.
error is marked out by weight as well, via tileDebugStatusLineWidth: 4px against
everyone else's 2px. loaded green against error red is the classic red-green pair,
so hue alone does not carry that difference for everyone. Opacity cannot do that
job — loading is deliberately fully opaque too, so an in-flight tile stays legible
— which is why the width accessor exists rather than another alpha tweak.
An abort no longer sets errorMessage either. It was the literal string 'aborted',
which the tooltip rendered as a row labelled "error" reading "aborted": the same
red-herring as the colour, in words. Found by hovering the recoloured tile, which
is the tooltip earning its keep immediately.
The tests assert properties rather than channels — error is the only status that
reads as red, every outline is distinct, error is uniquely the widest — so
retuning the palette later does not mean rewriting them. Two of those tests failed
when first written and were right to: amber is also red-dominant (the green
channel is what separates it), and error is not the only fully opaque outline.
Verified in the app on a real abort: the tile draws violet next to green loaded
neighbours, and its tooltip reads status "aborted" with no error row.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: consolidate the D5 changesets, 15 into 3
The 15 were step-scoped — each describing the state after one commit — and
changesets are concatenated into a single published release entry, so a reader
got intermediate claims alongside the final behaviour. Two were flatly
contradicted by later commits in the same PR: the metadata-probe entry announced
`pointsTiling` defaulting to 'off' with "nothing renders through the tiled path
yet", and the tiled-render entry listed feature colours, tiled filtering and the
resident-memory notice as known gaps, all of which this release closes.
Regrouped by what a changelog reader needs rather than by how the work was
sequenced: the feature end to end (core minor / vis minor), the row-group read
correctness and cost work behind it (core patch), and the tile debug overlay
(layers + vis patch). Same bump set — `changeset status` is unchanged.
492 lines to 236.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: thin the D5 comments, keeping the invariants
The branch added 860 comment lines against 1,057 of code — 45%, where the
repository sits at 21% and the comparable engine files at 22%. A reviewer pays
for all of it, and a good deal was narration rather than knowledge.
Cut: history ("used to be one fixed level", "tiles used to size in fixed
pixels"), measurements that belong in the plan doc, and facts stated twice
(PointsRendererAdapter explained the TileLayer identity requirement in both the
memo's doc and the getter's).
Kept: the things that will be re-broken by someone who does not know them — the
deck `_loaderId` race, `+`/`*2` rather than `|`/`<<1` in the Morton spread, the
sentinel box being a claim rather than a fact, `[]` meaning "conclude nothing"
in rowGroupColumnStats, the zcover depth table that justifies the constant.
Two comments were also wrong. The palette doc claimed `error` is the only status
with a fully opaque outline — `loading` is too, which is why the width channel
exists — and VTableSource still said footer statistics were out of reach of the
vendored parquet-wasm, in a file that now reads them.
860 comment lines to 691; 45% to 39%, against PointsResolver's own 36% and
RequestSlot's 44%. No code changed: build, 998 tests, biome and eslint unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Points: work through the CodeRabbit review on the D5 branch
Six code findings, plus one the bot had closed as addressed and had not been.
Fail-closed guards (the sort gate exists to keep tiling away from a mis-indexed
file, so its own failure modes have to fail the same direction):
- "No statistics" was read as "sorted". `mortonRowGroupExtentsAreSorted` skips
null extents, so an empty list — no footer, a parse failure, a row-group count
that disagreed — and an all-null list both answered `true`, a verdict on zero
evidence. A feature-primary artifact with no statistics passed the one gate
built to stop it, and the all-null case is worse still: `selectMortonRowGroups`
includes every unknown extent, so every tile scans the whole file. Now a
three-state `mortonRowGroupOrderVerdict`, with the warning naming which
happened. CodeRabbit marked this thread resolved; nothing had implemented it.
- An extent that is not a range (non-finite, or `min > max`) is rejected rather
than skipped. It cannot come from healthy statistics, so it means the decode is
wrong — and as the first or only extent it slid past unnoticed, after which
`selectMortonRowGroups` treats the inverted range as intersecting nothing and
drops that row group from every query.
- Tile-debug stores are pruned. deck's `Tileset2D.finalize()` clears its cache
WITHOUT firing the `onTileUnload` that prunes a store, so the fix for stale
overlay tiles covered a live layer but not a departing one: a layer hidden,
removed or switched to `pointsTiling: 'off'` mid-request left its in-flight
count in an append-only map, holding the global spinner on for a layer that no
longer exists. Progress is now driven by the live layers, and a layer leaving
the tiled path (including via `reloadElement`) loses its store, so toggling
tiling back on cannot repaint a previous session's tiles.
- `ensureTilingMetadata` is idempotent once ready. A settled `RequestSlot`
answers with a FRESH resolved promise while `pending` stays undefined, so the
"did a new request start?" test read true every time: status churned
loading→ready and `releaseResidentBatch` ran again, per call. `plan()` gates on
`isTilingSettled` and never reached it twice, but it is public on
`PointsDataEngine`. A failed probe stays retryable.
- `mortonTileGrid` applies its documented default cache row budget instead of
falling through to the flat tile-count fallback.
Two tests that could not fail. The unfiltered-view assertion read
`calls[0]?.featureCodes`, undefined both for "no filter" and for "no scan ran at
all"; and the scatter sizing props omitted the required `color`/`opacity`, inert
only because test files sit outside the build tsconfig's `include`.
Not fixed here: the `PointsElement` identity finding. The staleness is real but
not tiling's — `pointsEngine` is created once and never rebuilt when
`spatialData` swaps, so preload, catalog, rowCodes and matching carry the same
exposure. Re-probing only the tiling slot would hide a whole-engine problem.
Each new test verified to fail without its fix. 1005 tests, build and lint clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR scaffolds a complete pnpm-based monorepo structure for the SpatialData.ts library, mirroring the architecture of the Python SpatialData package.
What's New
📦 Monorepo Structure
@spatialdata/core- Core library for interfacing with SpatialData storesdocs- Docusaurus documentation site with custom branding🛠️ Core Package (
@spatialdata/core)The core package provides TypeScript/JavaScript functionality for reading and validating SpatialData stores:
Features:
🎨 Development Tools
📚 Documentation Site
Commands
Tech Stack
All build, test, and lint checks pass successfully. The monorepo is ready for development! 🚀
Original prompt
💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.