diff --git a/.changeset/morton-bisect-cost.md b/.changeset/morton-bisect-cost.md new file mode 100644 index 00000000..08bcde4e --- /dev/null +++ b/.changeset/morton-bisect-cost.md @@ -0,0 +1,32 @@ +--- +"@spatialdata/core": patch +--- + +Points (Morton): stop the row-group bisect from doing orders of magnitude more work +than the query needs. + +Two independent costs, both measured against a real 12.1M-point Xenium `transcripts` +artifact (245 row groups) with a viewport-sized query rectangle: + +- **`zcoverRectangle` now stops at `MORTON_ZCOVER_MAX_DEPTH` (10).** It recursed to + the full 16 bits per axis, resolving the rectangle to individual quantised cells + when its only job is picking row groups — **38,014 intervals**, each driving two + bisects, to select 92 row groups. At the cap that is 521 intervals selecting the + **same 92 row groups**, verified over a viewport tile, the whole slide and a + zoomed-in box. A coarser cell can only widen the covered code range, and the rows + it brings in are filtered against the exact bounds after the read, so the cover + stays complete. +- **Concurrent extent probes for the same row group are deduped.** The cache held the + settled value, so it dedups nothing while a read is in flight — and the index is + built under exactly that load, with every viewport tile bisecting over the same row + groups at once. Each tile started its own full row-group fetch for an entry the + others were already fetching (8 concurrent callers: 18 range reads, now 4). A + failed probe is evicted rather than cached, so a transient error does not strand + the bisect permanently. + +Not fixed, and now documented where it bites: the extent probe should read the row +group's **column statistics**, which are already in the footer we parse, instead of +range-reading and decoding the row group's bytes twice. The vendored parquet-wasm +build exposes no statistics accessor (`RowGroupMetaData` offers only `numRows` / +`fileOffset` / `compressedSize` / `column`), so this needs a wasm rebuild or a minimal +Thrift read of the footer. diff --git a/.changeset/morton-rowgroup-extent-max.md b/.changeset/morton-rowgroup-extent-max.md new file mode 100644 index 00000000..52ab8d2a --- /dev/null +++ b/.changeset/morton-rowgroup-extent-max.md @@ -0,0 +1,30 @@ +--- +"@spatialdata/core": patch +--- + +Points (Morton): fix viewport queries silently dropping row groups — the holes in the +tiled render. + +`readParquetRowGroupColumnExtent` built the bisect index by reading a row group's +first value and its last value, taking the last with +`readParquetRowGroup(..., { offset: rowCount - 1, limit: 1 })`. The vendored +parquet-wasm **ignores `offset` on a row-group read** and returned the first row +again, so every row group reported `max === min` — claiming to span a single value. + +Nothing errored. The bisect asks "first row group whose max >= target", and an +understated max moves that answer one group too far forward, so the row group that +actually *contained* the interval start was never read. On a real 12.1M-point Xenium +artifact one viewport query silently lost **11 of the 92 matching row groups — +187,990 points (6%)**, which is what the Z-order-shaped holes in the tiled points +render were. + +The upper bound now comes from the sort order instead: the file is sorted on this +column, so a row group's values all lie at or below the next row group's first value. +That is conservative (equal codes spanning a boundary keep both groups in range), +needs only the one read that works, and halves the reads — each row group's first +value is cached and shared with its neighbour. The last row group keeps an open +bound. + +The regression test asserts an exact point count for a bounded query rather than +"more than zero": this class of bug is silent by construction, and only a total sees +it. diff --git a/.changeset/points-morton-sentinel-guard.md b/.changeset/points-morton-sentinel-guard.md new file mode 100644 index 00000000..3c27d65b --- /dev/null +++ b/.changeset/points-morton-sentinel-guard.md @@ -0,0 +1,30 @@ +--- +"@spatialdata/core": patch +--- + +Points: refuse to Morton-tile an artifact whose sentinel bounding box is not the domain +its codes were quantised against. + +The sentinel rows are a claim a Parquet artifact makes about itself, and nothing else in +the file forces them to be true. Believing a wrong one does not fail — it silently clips +the tile grid to the bogus box, so whole regions are never even requested, and +`mortonIntervalsForBounds` normalises viewports against it and selects the wrong row +groups. Points are never misplaced (the reader re-filters to the query bounds), so the +only symptom is that part of the map is missing. + +`getPointsTilingMetadata` now recomputes `morton_code_2d` from x/y for a sample of real +rows and requires a majority to agree. On a real 12.1M-point element a sound artifact +matches 320/320 sampled rows and one with a stale sentinel box matches 0/320, so the test +is not marginal; a majority rather than an exact match tolerates coordinates landing on a +cell boundary. A rejected element reports `supportsRowGroupRangeReads: false` with no +`bounds` — the same pair the oversized-sentinel case already produced — so it degrades to +the capped preload through the existing path, and warns rather than downgrading silently. + +Cost is one extra row-group read per element, cached with the metadata: the probe goes +from 3 range reads / 0.37 MB / 414 ms to 4 / 2.16 MB / 523 ms on that element. The sample +is taken from the middle of the file, because a truncated box can agree with the true one +near the origin by coincidence but never in the interior. + +`mortonCode2dForPoint` / `mortonBoundsAgreeWithCodes` are exported for this, and pin the +interleave convention (x in the even bits) that `zcoverRectangle` and the writer both +already assumed without anything checking they stayed in step. diff --git a/.changeset/points-morton-sort-guard.md b/.changeset/points-morton-sort-guard.md new file mode 100644 index 00000000..86cd4398 --- /dev/null +++ b/.changeset/points-morton-sort-guard.md @@ -0,0 +1,30 @@ +--- +"@spatialdata/core": patch +--- + +Points: refuse to Morton-tile a file whose `morton_code_2d` column is not sorted. + +Having the column does not make a file Morton-ordered. A feature-primary artifact — +sorted `(feature, morton)` — carries the identical column with identical, correct values, +a correct sentinel box, and every field the tiling probe looks for. Only the order is +wrong, and nothing in the file said so. The row-group bisect binary-searches that index +assuming it ascends, so it landed arbitrarily and a tile came back holding whichever +feature blocks happened to live in the row groups it picked: some tiles showed one or two +genes, most showed none. + +The probe now reads the per-row-group `[min, max]` for the Morton column out of the +parquet footer and requires it to be non-decreasing. On the permutations store, +`transcripts_feature_then_morton` descends at 185 of its 244 row-group boundaries while +both morton-primary elements descend at none — including `transcripts_morton_then_feature`, +so a *secondary* feature key stays supported and the test is on the file rather than the +element's name. + +The check is free: `datasetMetadata.parts` already carries the footer bytes when the probe +runs, and the statistics are complete. Failing it also skips the sentinel sampling read, +since the outcome can no longer change, so a rejected element now costs less than before. +The element still loads through the capped preload, and its feature-code row-group index +on that path — a separate mechanism — is unaffected. + +Adds `decodeUnsignedIntStat`: `morton_code_2d` is `uint32`, which parquet stores as INT32 +with a UINT_32 annotation, and Morton codes use the top bit for real, so `decodeIntStat` +would read the far corner of a slide as negative. diff --git a/.changeset/points-morton-tile-grid.md b/.changeset/points-morton-tile-grid.md new file mode 100644 index 00000000..79afdb44 --- /dev/null +++ b/.changeset/points-morton-tile-grid.md @@ -0,0 +1,47 @@ +--- +"@spatialdata/core": patch +"@spatialdata/layers": patch +--- + +Points: select Morton row groups from footer statistics, and size the tile grid from the +artifact (D5 step 6). + +**Row-group selection no longer reads the file.** `selectMortonRowGroups` picks row +groups from the per-row-group `[min, max]` the tiling probe already parsed out of the +parquet footer. The bisect it replaces range-read the row group's BYTES — every column, +~2MB on a real transcripts artifact — to recover two boundary values, `log2(rowGroups)` +steps per Morton interval, for a few hundred intervals per query. Measured on one +1024 um viewport tile of a 12.1M-point element, both returning the same 643,961 points: + +| row-group selection | range reads | bytes | wall | +|---|---|---|---| +| bisect | 97 | 175.12 MB | 2911 ms | +| footer index | 32 | 57.83 MB | 1035 ms | + +The remaining 32 reads are the row-group data itself. The new path is also stricter: the +bisect tested only `max` and assumed row groups tile the code space without gaps, while +this intersects both ends. The bisect stays as the fallback when statistics will not +parse, so this is an optimisation rather than a new requirement. + +**The tile grid is derived instead of hardcoded.** It was one fixed level +(`minZoom/maxZoom: -1`), so every tile was 1024 local units at every zoom: zooming in +read a 1024-unit tile to look at 50 units of it, and 1024 came from deck's defaults +rather than from the data. `mortonTileGrid` now derives both ends from the point +density: the finest level stays at least one row group's footprint (below that, four +tiles fetch what one used to, for the same bytes and more requests), the coarsest holds +at most 400k rows, and `zoomOffset = log2(modelMatrixScale)` couples deck's `z` — chosen +from a world-space zoom — to tile spans expressed in local units. + +For the Xenium element that is two levels, 1024 um and 512 um, and the narrowness is the +point: the row-group size is the floor, so 50k-row groups put it at ~402 um. The old +fixed 1024 was accidentally near-optimal for that file and would not be for one an order +of magnitude smaller or denser. + +**The tile cache is budgeted in rows.** `maxCacheSize` comes from a row budget rather +than deck's default of `5 x the selected tile count`, which on a coarse viewport of this +element could retain ~220 tiles / ~71M rows against a 4M resident cap. It is now 16 tiles +/ ~5.2M rows, stated. `maxRequests` stays at 6, but as a decision rather than an +inheritance. Accounting only — nothing evicts by bytes yet (ADR 0005). + +`PointsLoaderCapabilities` gains `totalRows` and `maxRowsPerGroup`, which is what the +grid is derived from. diff --git a/.changeset/points-morton-tiled-render.md b/.changeset/points-morton-tiled-render.md new file mode 100644 index 00000000..9c9b23df --- /dev/null +++ b/.changeset/points-morton-tiled-render.md @@ -0,0 +1,37 @@ +--- +"@spatialdata/core": patch +"@spatialdata/layers": patch +"@spatialdata/vis": patch +--- + +Points: render Morton-tiled elements from viewport tiles (D5 step 2). + +A points layer with `pointsTiling: 'auto'` on a Morton artifact now draws through +`mortonTiledStrategy` — deck's `TileLayer` reading row groups for the viewport — +instead of a memory-capped resident preload. A 12.1M-point transcripts element can be +explored at full detail, and the memory cap no longer applies to it. + +- **`@spatialdata/core`** — `PointsResolver` reports what a tiled entry actually has: + no `preload` resource (absent, not idle, so `isBlocking` skips it), world `bounds` + derived from the artifact's own extent so auto-fit can frame the layer before a + single tile loads, and geometry status driven by the probe. `blockingResources` + covers `tiling` as well as `preload`. New `transformAxisAlignedBounds` helper. +- **`@spatialdata/layers`** — `PointsRendererAdapter.getTiledResource`, memoised on + (element, metadata): a new resource identity would make `TileLayer` refetch every + visible tile, so a pan would become a full reload. Exposed via `PointsDataEngine` + alongside `isTiled` / `getTilingMetadata` / `ensureTilingMetadata`. +- **`@spatialdata/vis`** — the tiled branch in `getLayers`, tiled world bounds, + `hasRenderableLayerData` counting a tiled element as drawable, per-layer tile-debug + stores feeding viewport-tile progress into `isLoading`, and `pointsTiling` / + `showTileDebugOverlay` controls on the points layer panel. + +Tiling is per LAYER but the probe's answer is cached per ELEMENT, so every consumer +combines the two (`usesTiledPath`, `isTiledFor`). Reading the probe alone left a layer +rendering tiles after the user switched tiling off, while planning went back to +preloading — both at once. + +Known gaps, tracked in `docs/plans/points-morton-tiled-viewport-loading.md`: a tiled +layer draws flat-coloured and ignores the feature filter (the tile scan does not yet +return per-point codes — step 3), switching a layer to tiling does not evict the +preload it already did, and the panel's truncation notice still reports resident +memory on a tiled layer. diff --git a/.changeset/points-morton-tiling-default.md b/.changeset/points-morton-tiling-default.md new file mode 100644 index 00000000..46608c9e --- /dev/null +++ b/.changeset/points-morton-tiling-default.md @@ -0,0 +1,36 @@ +--- +"@spatialdata/core": minor +"@spatialdata/vis": minor +--- + +Points: Morton viewport tiling is now on by default (D5 step 7, closes D5). + +`pointsTiling` defaults to `'auto'` (`DEFAULT_POINTS_TILING`), so every points element +is probed once and takes the tiled path if — and only if — it can. Read the config +through the new `pointsTilingEnabled(...)` rather than comparing to `'auto'`: the +default has to mean the same thing to the resolver deciding what to load, the hook +deciding what to render, and the panel drawing the checkbox. + +**Why on rather than opt-in.** On a Morton artifact the capped preload is not a neutral +alternative: it keeps the first `cap` rows in FILE order, and file order there is a +prefix of the Z-curve — a spatially skewed chunk of the slide rather than a sample of +it. Tiles read what is actually in view, colour by feature, honour the feature filter +inside the row-group scan, and subdivide with zoom. + +**What it costs, measured.** At the default zoomed-out framing of a 12.1M-point Xenium +element, the tiled path loads all 44 tiles — 12,165,029 points / ~158 MB, the whole +artifact — against a 4M-row prefix for the preload. So first paint on a fully +zoomed-out view is ~3x the rows, in exchange for a correct picture that streams in 44 +pieces instead of blocking on one decode. Zooming OUT is the one direction viewport +tiling does not help, because there is no coarser representation to read; that wants a +multi-resolution points pyramid, not a finer index. `pointsTiling: 'off'` restores the +previous behaviour per layer. + +An element that cannot be tiled is unaffected beyond one probe (4 range reads / ~2.16 MB +on a 12.1M-point element, cached with the metadata; on a non-Morton element it is footer +metadata the preload reads anyway). The three guards — no usable sentinel row group, a +sentinel box that is not the code domain, an unsorted Morton column — decline loudly and +fall through to the preload. + +The panel now hides the memory-cap control on a tiled layer: it governs nothing there, +and it sat directly above a line saying the cap does not apply. diff --git a/.changeset/points-tiled-coherence.md b/.changeset/points-tiled-coherence.md new file mode 100644 index 00000000..3247c8e0 --- /dev/null +++ b/.changeset/points-tiled-coherence.md @@ -0,0 +1,34 @@ +--- +"@spatialdata/core": patch +"@spatialdata/layers": patch +"@spatialdata/vis": patch +--- + +Points: make a tiled layer stop reporting — and looking like — a capped preload. + +Three things a tiled layer got wrong once it was actually drawing: + +- **The resident window is released.** A layer switched to tiling mid-session had + usually already preloaded, and nothing gave those rows back: `plan()` stops *asking* + for a preload, which is not the same as evicting one. The probe's settle now drops + the preload, its row-aligned codes and its feature-index scan (all defined against + that window); the catalog stays, since it describes the element rather than the + window. +- **The truncation notice no longer contradicts itself.** It read "4,000,000 of + 12,165,021 points in memory — capped; raise the cap for more" directly above "the + memory cap does not apply". A tiled layer draws from the viewport, so a resident + count is not a statement about what is on screen, and the panel now says nothing + rather than something true and misleading. +- **The tiling status line is live.** It read `engine.isTiled(...)` directly, outside + the engine subscription, so it kept saying "this element has no Morton index" about + an element it was already tiling — the probe settles asynchronously and nothing + re-rendered it. `tiled` now comes through `usePointsFeatureState`, which carries the + subscription. + +Point sizing is also now **one behaviour instead of two**: the Morton tile path sized +points in fixed pixels while the preloaded path used world units, so `pointSize` meant +something different depending on a checkbox, and a zoomed-out tiled layer drew every +one of its millions of points as a fixed screen dot. Density saturated into a flat +mass and every tile seam and acquisition boundary hardened into what looked like a +rendering fault. Both paths now size in world units with the model-matrix scale folded +in, so points shrink as you zoom out and overdraw self-limits. diff --git a/.changeset/points-tiled-feature-colours.md b/.changeset/points-tiled-feature-colours.md new file mode 100644 index 00000000..d524de03 --- /dev/null +++ b/.changeset/points-tiled-feature-colours.md @@ -0,0 +1,36 @@ +--- +"@spatialdata/core": patch +"@spatialdata/layers": patch +"@spatialdata/vis": patch +--- + +Points: colour a Morton-tiled layer by feature (D5 step 3). + +Viewport tiles now carry a feature code per point, so a tiled layer colours, takes +per-feature overrides and responds to Feature Highlight exactly like the preloaded +path. Until now it drew flat — the same element looked like two different datasets +depending on a checkbox. + +The codes were already being read and thrown away: the tile scan consults the feature +column to filter on it, then returned bare coordinates. + +- **`@spatialdata/core`** — `scanMortonTableInBounds` takes an optional + `Int32PointBuffer` and appends to it in lockstep with the geometry; the worker's + tile-scan handler builds one and returns it (the protocol already carried an + optional `featureCodes` and transferred its buffer, so the worker boundary needed no + change). `loadMortonPointsInBounds` now projects the code column whenever the + artifact **has** one rather than only when a filter is active — the no-filter "all + features" view was precisely the case that arrived without codes — and both its + worker and main-thread returns carry them. +- **`@spatialdata/layers`** — `mortonTiledStrategy` forwards `colorByFeature`, + `featureCodeSpaceSize`, `featureColorOverrides` and `highlightFeatureCode` to its + per-tile scatter layers, and rebuilds them when those change. +- **`@spatialdata/vis`** — the tiled branch stops forcing `colorByFeature: false` and + reads the same element-scoped colour inputs the preloaded branch does. + +A short codes array is dropped rather than padded, on both paths: the remaining points +would read code 0 — a *valid* feature — and be confidently mis-coloured, which is +worse than no colour at all. + +The feature **filter** still does not narrow tiles (step 4): a tiled layer draws every +feature in the viewport regardless of the selection, and the panel now says so. diff --git a/.changeset/points-tiled-feature-filter.md b/.changeset/points-tiled-feature-filter.md new file mode 100644 index 00000000..1a832562 --- /dev/null +++ b/.changeset/points-tiled-feature-filter.md @@ -0,0 +1,35 @@ +--- +"@spatialdata/core": patch +"@spatialdata/vis": patch +--- + +Points: apply the feature filter to Morton-tiled layers (D5 step 4). + +A tiled layer used to draw every feature in the viewport whatever was selected. The +selection now reaches `getTileData` — and its `updateTriggers`, so changing it +refetches instead of serving the previous selection's cached tiles — and is applied +**inside** the row-group scan, so a tile arrives holding only the selected features. +On a real 12.1M-point transcripts element, selecting one gene takes a viewport tile +from 3,128,988 points to 87,594: 36x fewer points uploaded and drawn. + +It does **not** reduce I/O, and the plan's original expectation that it would has been +corrected. The same query read the same 92 row groups and the same 158MB either way: +row groups are chosen *spatially* on a Morton artifact, and a gene's points are spread +across all of them, so no feature filter can skip one. Narrowing the fetch by feature +needs a feature-primary index — the open index-selection question in ADR 0002/0003. + +Two supporting fixes: + +- **The catalog is planned for a tiled entry.** On the preloaded path it arrives free + as a preview off the geometry decode; a tiled entry never decodes a resident batch, + so nothing built one — and a selection is stored as feature NAMES, which cannot + become the codes the scan filters on without it. A saved config with a selection + would draw every feature until someone happened to open the filter panel. A failed + catalog is not re-planned, or the task re-emits on every reconcile forever. +- **Feature rows read honestly on a tiled layer.** Every other signal the panel uses + describes a resident batch a tiled layer does not have, so its rows fell through to + "beyond the resident window; select it to fetch its points" — greyed, and wrong + twice over: the points are available, and no feature-index scan is involved. + +`renderCap` stays unset on the tiled path: it is a resident-window notion, and a tile +is already bounded by its viewport. diff --git a/.changeset/points-tiling-metadata-probe.md b/.changeset/points-tiling-metadata-probe.md new file mode 100644 index 00000000..9a356350 --- /dev/null +++ b/.changeset/points-tiling-metadata-probe.md @@ -0,0 +1,28 @@ +--- +"@spatialdata/core": patch +"@spatialdata/layers": patch +--- + +Points: probe for a Morton-tiled artifact before committing to a full-table preload +(D5 step 1). + +`PointsResolver` gains a `tiling` resource — a one-key `RequestSlot` holding the +element's **tileable** Morton metadata, or `null` when the element cannot drive +viewport tiles (no Morton artifact, no row-group range reads, no bounds, or a failed +probe). `plan()` now asks `planPointsLoads` for both decisions at once, so a tileable +element no longer schedules a full-table preload it would immediately throw away, and +the row-codes / feature-index-scan tasks — both defined against the resident batch — +wait for the same answer. + +This is **opt-in and inert by default**: the new `PointsResolveConfig.pointsTiling` +defaults to `'off'`, which collapses planning to exactly today's behaviour. Nothing +renders through the tiled path yet — that is the next step of +`docs/plans/points-morton-tiled-viewport-loading.md`. + +- **`@spatialdata/core`** additionally exports `planPointsLoads` (moved from + `@spatialdata/layers` so the resolver can call it) and the resolver reads + `getTilingMetadata` / `isTiled` / `isTilingSettled`. A failed probe is a retryable + `failed` resolution that still reports "cannot tile", so the layer falls through to + the ordinary preload rather than stranding. +- **`@spatialdata/layers`** re-exports `planPointsLoads` from core; no consumer import + moves. diff --git a/docs/plans/points-morton-tiled-viewport-loading.md b/docs/plans/points-morton-tiled-viewport-loading.md new file mode 100644 index 00000000..b483add2 --- /dev/null +++ b/docs/plans/points-morton-tiled-viewport-loading.md @@ -0,0 +1,593 @@ +# Points: Morton-tiled viewport-driven loading (D5) + +Status: **complete** (2026-08-12) — all seven steps implemented, D5 closed in the +punch-list. What is deliberately still open is listed under +[Remaining work](#remaining-work) and the open questions. +Implements [points-redesign-punchlist](./points-redesign-punchlist.md) **D5** and the +"Morton is still dark" line in [points-mvp-and-roadmap](./points-mvp-and-roadmap.md). +Format contract: [ADR 0002](../adr/0002-spatially-aware-vector-loading.md). +Encoding → strategy mapping: [ADR 0003](../adr/0003-points-render-resource.md). +Structural substrate: [ADR 0004](../adr/0004-resource-resolver-owned-by-core.md) / +[layer-data-engine-decomposition](./layer-data-engine-decomposition.md). +Per-element path table: [points-preload-feature-filter-status](./points-preload-feature-filter-status.md). + +Harvest source: branch `claude/quizzical-roentgen-3ee079`, preserved as tag +**`backup/points-wip-20260702`** (tip `a724230`). Key commits `42dbf21` (points tiling), +`42c3ece` (WIP tiling + visualization integration), `3517646` (points render resource). + +--- + +## The point + +A Morton-sorted points artifact lets us read **only the row groups whose Morton +interval intersects the viewport**, so a 12M-row transcripts element renders without +ever holding 12M rows in memory. Today we hold the first *N* rows (memory cap) in a +resident preload and draw those, whatever the viewport is. That is the wrong axis: +zooming into a corner should get *more* detail there, not the same truncated prefix. + +This plan lights up the `morton-tiled` encoding end to end. It is **not** a rewrite — +almost everything below already exists and is tested; what was lost in the harvest is +the *wiring*, and the wiring's home moved from a React hook to a resolver. + +--- + +## Why it is dark today + +Three facts, in order of how directly they block: + +1. **The renderer adapter hardcodes the path off.** + [`PointsRendererAdapter.ts:80`](../../packages/layers/src/adapters/PointsRendererAdapter.ts:80) + is `const RESOLVE_OPTIONS = { experimentalOptimizations: 'off' as const }`, and every + resolve site passes `metadataKnown: false` + ([:118](../../packages/layers/src/adapters/PointsRendererAdapter.ts:118), + [:252](../../packages/layers/src/adapters/PointsRendererAdapter.ts:252), + [:272](../../packages/layers/src/adapters/PointsRendererAdapter.ts:272)). + `resolvePointsEncoding` ([`pointsLoader.ts:65`](../../packages/core/src/pointsLoader.ts:65)) + therefore cannot return anything but `'preloaded-columnar'`. +2. **Nothing probes the metadata.** `getPointsTilingMetadata` + ([`VPointsSource.ts:2256`](../../packages/core/src/models/VPointsSource.ts:2256), + surfaced on the element at [`models/index.ts:693`](../../packages/core/src/models/index.ts:693)) + has no caller outside tests. `PointsResolver` has slots for `preload`, `rowCodes`, + `catalog` and `matching` — and none for tiling metadata. The probe lived in the + deleted god-hook and did not come across with the decomposition. +3. **So `mortonTiledStrategy` is unreachable.** The adapter says so itself at + [`PointsRendererAdapter.ts:263`](../../packages/layers/src/adapters/PointsRendererAdapter.ts:263): + *"No path reaches this today… It is wrong the moment a tiled strategy is pointed at + a growing resource, which is what D5 does."* + +## What already exists (do not rebuild) + +| Layer | Piece | Where | +|---|---|---| +| core | Morton interval computation, sentinel handling, metadata type | [`pointsTiling.ts`](../../packages/core/src/pointsTiling.ts) | +| core | Row-group bisect + range reads + in-bounds scan (worker & main-thread) | [`VPointsSource.ts:2488`](../../packages/core/src/models/VPointsSource.ts:2488) `loadMortonPointsInBounds` | +| core | Metadata probe, cached per element path | [`VPointsSource.ts:2256`](../../packages/core/src/models/VPointsSource.ts:2256) | +| core | `morton-tiled` loader factory | `createMortonTiledPointsLoader`, [`pointsLoader.ts`](../../packages/core/src/pointsLoader.ts) | +| core | Worker tile scan + protocol | [`pointsWorkerScan.ts`](../../packages/core/src/workers/pointsWorkerScan.ts), `points-worker.ts` | +| layers | `TileLayer` strategy w/ abort, clipping, per-tile scatter, debug overlay | [`mortonTiledStrategy.ts`](../../packages/layers/src/mortonTiledStrategy.ts) | +| layers | Encoding → strategy table | [`pointsRenderStrategies.ts`](../../packages/layers/src/pointsRenderStrategies.ts) | +| layers | Tile debug store + hooks + polygon data | `pointsTileDebug.ts`, `pointsTiledDebugHooks.ts` | +| layers | Probe-vs-preload decision helpers | `planPointsLoads`, `shouldPreloadAfterMetadataProbe`, [`pointsLoadPlan.ts`](../../packages/layers/src/pointsLoadPlan.ts) | +| layers | Blocked-preload user messages | `pointsPreloadBlockedMessage`, `pointsTilingUnavailableMessage` | +| tests | `mortonPointsTiling.spec.ts`, `pointsMortonScanFilter.spec.ts`, `pointsTiling.spec.ts`, `pointsTileDebug.spec.ts`, `pointsLoadPlan.spec.ts`, `pointsRenderStrategies.spec.ts` | core/layers | + +## What to harvest from the WIP branch + +The branch is the only place this ran end to end. Its wiring is **hook-shaped** — a +`Map` on `loadedDataRef` plus an `await` inside a giant `Promise.all` — and must be +re-expressed as resolver slots. Harvest the *logic and its edge cases*, not the shape. + +| From `backup/points-wip-20260702` | Verdict | Notes | +|---|---|---| +| `useLayerData.ts:958–1010` — plan gate (`wantsOptimized`, `metadataKnown`, `planPointsLoads`) | **Port** | Becomes the `tiling` branch of `PointsResolver.plan()`. The helper it calls already exists on main. | +| `useLayerData.ts:1191–1237` — the probe, its `renderableMetadata` gate (`supportsRowGroupRangeReads && bounds`), preload-cache eviction on success, and *both* fallback paths (probe says "not tileable" → preload; probe throws → preload) | **Ported (step 1)** | These branches are the whole reason the path degrades safely. The "probe failed ⇒ still try preload" arm is kept, as a `failed`-but-reads-as-`null` slot. Its `getParquetRowCount` fallback was **not** ported: `probedTotalRows` fed `shouldPreloadAfterMetadataProbe`, which ignores `totalRows` entirely — the read was vestigial. | +| `useLayerData.ts:1576` — `hasRenderableLayerData` counting `tilingMetadata.bounds` as renderable | **Port** | On main this is `pointsEngine.hasData` ([`useLayerData.ts:931`](../../packages/vis/src/SpatialCanvas/useLayerData.ts:931)); a tiled element has no resident data and would read as "nothing to draw". | +| `useLayerData.ts:1621–1640` — world bounds from `tilingMetadata.bounds` when there is no preload | **Port** | Main's points branch ([`useLayerData.ts:990`](../../packages/vis/src/SpatialCanvas/useLayerData.ts:990)) returns `null` without preloaded data → no framing, dead "Center on layer". | +| `pointsTileProgress.ts` (whole file) | **Take nearly as-is** | Already imports `TileDebugStore`/`TiledPointsDebugState` from `@spatialdata/layers`, which main still exports. Gives `Loading points… (3/12 tiles, 1,204,993 points)`. | +| `PointsStylePanel.tsx` | **Do not take** | Superseded by main's `PointsLayerPanel` + `PointsFeatureFilterPanel`. Lift only the tile-debug toggle if we want one. | +| `renderers/pointsRenderer.ts` tiled branch | **Do not take** | Dead on main (deleted in `dd290db`); the composite + strategies replaced it. | +| `resolvePointsRenderResource.ts` (vis copy) | **Do not take** | Already relocated to `layers` (`f704a58`). | + +## Target design + +### 1. A `tiling` slot on `PointsResolver` + +Add a fifth `RequestSlot` next to `preload`/`rowCodes`/`catalog`/`matching` +([`PointsResolver.ts:94`](../../packages/core/src/engine/PointsResolver.ts:94)): + +```ts +tiling: RequestSlot<'probe', PointsTilingMetadata | null>; +``` + +- Key `'probe'` — the element path is fixed, so there is exactly one request. `null` is + a **settled fact** ("this element is not tileable"), not an absence, exactly like the + catalog's `null` for a `feature_key`-less element. +- `plan()` emits `{ id: \`${key}#tiling\`, resource: 'tiling' }` when + `wantsOptimized && !tilingSettled(key)`. +- The **preload task becomes conditional on the probe's answer**, via the existing + `planPointsLoads({ wantsOptimized, metadataKnown, tiledMetadata, hasPreloaded })`. Until + the probe settles, plan neither — that is the whole point of probing first, and it is + why `planPointsLoads` returns two independent booleans. +- A failed probe settles `null` **and is retryable**, so `retry()` re-runs it; the + fallback preload still runs (WIP branch's catch arm). + +`wantsOptimized` needs a home in `PointsResolveConfig` +([`PointsResolver.ts:88`](../../packages/core/src/engine/PointsResolver.ts:88)) — a +serialisable `pointsTiling?: 'auto' | 'off'` entry prop (default decided in step 4; +see open question 1). + +### 2. `blockingResources` must stop being a constant + +`readonly blockingResources = ['preload']` +([`PointsResolver.ts:178`](../../packages/core/src/engine/PointsResolver.ts:178)) and +`isBlocking` treats `status === 'idle'` as blocking +([`SpatialEntryStore.ts:157`](../../packages/core/src/engine/SpatialEntryStore.ts:157)). +A tiled entry never plans a preload, so its `preload` slot stays `idle` **forever** and +the canvas sits on "Loading layer data…" with auto-fit never firing. + +This is load-bearing and easy to get wrong: **auto-fit piggybacks on the +`isBlocking` true→false transition** — the same trap that bit the shapes +non-blocking pass. Two candidate fixes: + +- **(a)** `blockingResources` becomes a method of the entry's state — `['tiling']` until + the probe settles, then `['preload']` or `[]`. ADR 0004 already calls it *"data, not a + switch"* ([`resolver.ts:128`](../../packages/core/src/engine/resolver.ts:128)), and this + is the first case that needs it to vary. +- **(b)** Keep the array and have the tiled path settle `preload` to a sentinel + "not applicable" resolution. + +**Recommend (a).** (b) puts a lie in the preload slot and will confuse every later +reader of the resident-batch invariants. + +### 3. Adapter: a real tiled resource, identity-stable + +`PointsRendererAdapter` gains a `getTiledResource(element, key, metadata)` memo keyed on +`(element, metadata)`, resolving with `{ tilingMetadata, metadataKnown: true }` and +`experimentalOptimizations: 'auto'`. Identity stability is not cosmetic here: a fresh +resource per `project()` tears down the `TileLayer` and **re-fetches every visible +tile**. The existing memo (`resolve()`, keyed on batch identity + signature) is the +model; `pointsRenderResourceSignature` already includes `parquetPath` and the `rg` flag. + +Delete `RESOLVE_OPTIONS` as a module constant and thread the option through from config. + +### 4. Bounds, framing and status + +- Resolver `EntryResources.bounds` for a tiled entry comes from + `tilingMetadata.bounds` (transformed), not from a resident batch. +- `pointsEngine.hasData` / `hasRenderableLayerData` must count "tileable metadata + settled" as renderable. +- Tile progress: mount `pointsTileProgress.ts` in vis, feed `isLoading` and the footer + message from `pointsTileLoadingMessage(...)`. `isBlocking` stays false once metadata is + known — tiles refine an already-framed layer, they do not gate first paint. + +### 5. Feature filter, colour, and the code column — the real gap + +The tiled path filters by feature (`loadPointsInBounds({ featureCodes })` → row-group +scan), and `mortonTiledStrategy` already threads `featureCodes` into `getTileData` and +into its `updateTriggers`. But: + +- **`loadMortonPointsInBounds` returns geometry only** — `data` is `[xs, ys(, zs)]` and + the per-point feature code is *used for filtering and then discarded* + ([`VPointsSource.ts:2488`](../../packages/core/src/models/VPointsSource.ts:2488); the + worker path returns `workerResult.data` and the main path builds from + `Float32PointBuffer`s only). So a tiled batch has **no `featureCodes`**, and + colour-by-feature, the palette LUT and Feature Highlight — all of which key on the + per-point code — silently degrade to flat colour. +- **`mortonTiledStrategy` does not forward the colour props** it would need even if the + codes were there: its `scatterStyleProps` carries only `color`, sizes, opacity, + `modelMatrix`, `use3d`, whereas `preloadedScatterStrategy` forwards `colorByFeature`, + `featureCodeSpaceSize`, `featureColorOverrides` and `highlightFeatureCode`. +- **`rowCodes` is preload-shaped.** `plan()`'s `needsRowCodes` gate reads the first + `min(rowCount, cap)` rows *in file order* to align with the resident batch. On a tiled + element there is no resident batch and that read is both meaningless and expensive — + the gate must exclude the tiled path. Per-tile codes ride the tile batch instead. +- **The catalog's preview phase disappears.** The resident-subset preview falls out of + the preload decode; with no preload, only the full scan can produce a catalog. For a + Morton artifact with `{feature_key}_codes` that is the cheap row-group dictionary-page + scan, so this is acceptable — but the catalog task must be planned explicitly rather + than arriving as a side effect of a decode that no longer happens. +- **`renderCap`** is a whole-batch notion; on the tiled path it has to be per tile (or + retired in favour of the tile budget). Decide, don't inherit silently. + +Steps 3–4 below own this; step 2 ships **flat-coloured** tiles on purpose. + +--- + +## Implementation sequence + +Each step builds, passes tests, and leaves the branch shippable. + +**Step 1 — Probe (no render change). ✅ done.** +`tiling` slot + `plan()` gate + `PointsResolveConfig.pointsTiling` (default `'off'`, so +planning is byte-for-byte today's when nobody opts in). `planPointsLoads` moved to +`core` — `core` cannot import from `layers` and duplicating the decision is how the two +drift — with a re-export left behind. 13 headless specs in +`core/tests/pointsResolver.spec.ts`. + +One thing the design above got wrong, found by the tests: gating only on `isTiled` is +not enough. **Row codes and the matching scan have to wait on the *pending* probe too.** +Planning them while it is in flight does the wasted read *and settles the codes*, so the +next pass reads "already loaded" and the waste becomes invisible — the exact shape of +bug this deferral exists to prevent. `plan()` returns early on `probeMetadata || isTiled`. + +*Acceptance met: no behaviour change with tiling off; full suite (854 tests), build and +biome green.* + +**Step 2 — Draw tiles (flat colour).** +Adapter `getTiledResource`; `blockingResources` becomes state-derived; bounds and +`hasData` from metadata; `pointsTileProgress` wired to the footer. Tiled elements render +through `mortonTiledStrategy` with flat colour and no feature filter. +*Acceptance: a Morton fixture frames correctly on load, pans/zooms with tiles loading in, +"Center on layer" works, no "Loading layer data…" hang, non-Morton elements unchanged.* + +**Step 3 — Per-point codes on tile batches. ✅ done.** +`scanMortonTableInBounds` takes an optional codes buffer and appends in lockstep with +the geometry; the worker handler builds one and returns it (the protocol's +`PointsWorkerColumnarResult.featureCodes` and its transferable already existed, so the +boundary needed nothing). `loadMortonPointsInBounds` projects the code column whenever +the artifact HAS one rather than only when filtering — the no-filter "all features" +view was exactly the case arriving without codes — and both its worker and +main-thread returns carry them. `mortonTiledStrategy` forwards the colour props, and +vis stops forcing `colorByFeature: false`. + +Short codes are dropped rather than padded, on both paths: a partial array would leave +the tail reading code 0 — a *valid* feature — and mis-colour it with conviction. + +*Acceptance met: on a real 12.1M-point element a tiled layer draws per-feature colours +matching the preloaded path; tests pin one code per point, every code a real catalog +entry, and codes still returned (and all equal to the selection) under a filter.* + +**Step 4 — Feature filter + catalog on the tiled path. ✅ done.** +The selection reaches `getTileData` (and its `updateTriggers`, so a change refetches +rather than serving the previous selection's tiles) and is applied inside the +row-group scan. The composite's filter machinery is untouched: it is gated on +`preloaded-columnar`, so a tiled layer passes straight through. `renderCap` stays +unset — it is a resident-window notion, and a tile is already bounded by its viewport. + +The catalog is now **planned** for a tiled entry. On the preloaded path it arrives +free as a preview off the geometry decode; a tiled entry never decodes a resident +batch, so nothing built one — and the selection is stored as feature NAMES, which +cannot become codes without it. A saved config with a selection would otherwise draw +every feature until someone opened the panel. A failed catalog is not re-planned +(`retry()` is the way back), or the task re-emits forever. + +The feature-row panel also needed a tiled case. Every other signal it reads describes +a resident batch a tiled layer does not have, so its rows fell through to "beyond the +resident window; select it to fetch its points" — greyed, and wrong twice: the points +are available, and no feature-index scan is involved. + +**The original acceptance criterion here was wrong, and the measurement is worth +keeping.** "Fewer row-group range reads" does not happen on a Morton artifact: + +| viewport query on a 12.1M-point element | points returned | row groups | bytes | +|---|---|---|---| +| all 541 features | 3,128,988 | 92 | 158.1 MB | +| one gene (EPCAM) | 87,594 | 92 | 158.1 MB | + +Row groups are chosen **spatially**, and a gene's points are spread across all of +them, so no feature filter can skip one. What the filter buys is 36x fewer points +leaving the worker — the GPU, memory and overdraw win — not less I/O. Narrowing the +*fetch* by feature needs a feature-primary index; that is exactly what the +`transcripts_feature_then_morton` / `transcripts_morton_then_feature` permutations +exist to explore, and it is the open index-selection question in ADR 0002/0003 rather +than something this step could deliver. + +*Acceptance met, restated: filtering a tiled element narrows what each tile returns +(36x for one gene), a selection change refetches rather than reusing cached tiles, and +a tiled element resolves a name-based selection without the panel ever opening.* + +--- + +## The tile grid, measured + +Steps 1–4 made the path *work*; they never examined what deck is actually asked to +tile. Measured against the live `xenium_2.q0.001.htj2k.index-permutations` store +(2026-08-12), reading `TileLayer.state.tileset` directly in the browser. + +### It is one fixed grid of ~44 tiles that never subdivides + +[`mortonTiledStrategy.ts:97`](../../packages/layers/src/mortonTiledStrategy.ts:97) pins +`minZoom: -1, maxZoom: -1` with `tileSize: 512`. deck's non-geospatial traversal +([`tileset-2d/utils.js` `getIdentityTileIndices`](../../node_modules/.pnpm/@deck.gl+geo-layers@9.3.7_@deck.gl+core@9.3.7_@deck.gl+extensions@9.3.7_@deck.gl+core@9_95b707e63fcfb29d74e361206c081c66/node_modules/@deck.gl/geo-layers/dist/tileset-2d/utils.js)) +computes `scale = 2^z * 512 / tileSize`, so **every tile is 1024 element-local units, +at every zoom**. For `transcripts_morton` (bounds 10871 x 3627 µm) that is an +11 x 4 = **44-tile grid, fixed for the life of the layer**: + +``` +extent [3.42, 2.45, 10874.72, 3629.29] +selected 44 tiles (x 0..10, y 0..3), cacheSize 44, all loaded, scheduler idle +``` + +Consequences — **all addressed in step 6**, kept here as the statement of the problem: + +- **Zooming in never got more detail.** The same 1024-unit tile was re-used at every + scale; a 50 µm viewport still read a 1024 µm tile. Loading followed the viewport's + *position* but not its *scale*. +- **Nothing budgeted the tile cache.** deck's default `maxCacheSize` is `5 x the + selected tile count`, so a coarse viewport of this element could retain ~220 tiles — + ~71M rows, against a resident cap of 4M. +- **1024 units was arbitrary.** It fell out of `tileSize: 512` and `z = -1`, not out of + the data. + +### Where the "regions that never get queued" come from + +Not scheduling, and not our loader. The tile grid is clipped to +`resource.loader.capabilities.bounds` — the **sentinel bounding box**, read from the +first row group by `extractSentinelBoundingBox` +([`pointsTiling.ts:231`](../../packages/core/src/pointsTiling.ts:231)) — and on two of +the four elements in the permutations store that box is wrong: + +| element | sentinel bbox (= `TileLayer.extent`) | true x/y extent | tiles that can exist | +|---|---|---|---| +| `transcripts_morton` | x 3.42–10874.72, y 2.45–3629.29 | same | 11 x 4 = 44 | +| `transcripts_morton_then_feature` | x 3.42–**10550.23**, y **2144.59–3138.84** | x 3.42–10874.72, y 2.45–3629.29 | 11 x 2 = **22** | +| `transcripts_feature_then_morton` | identical to the above | as above | — | + +*(Fixed by the regeneration in step 5 — all four now report the true extent, and +`transcripts_morton_then_feature` selects the full 11 x 4 grid. Kept here because it is +the worked example of how a bad box presents.)* + +The observed symptom is exactly that: on `transcripts_morton_then_feature` the tileset +only ever holds y = 2 and y = 3, so the top half of the tissue is never *requested* — +no pending tile, no debug rectangle, nothing to wait for. Switch the same view to +`transcripts_morton` and all 44 tiles select and load. + +**The store is at fault, not the reader.** Reproducing the stored `morton_code_2d` +from x/y confirms which box the codes were quantised against — 320 sampled rows per +element, x-first interleave: + +| element | matches under sentinel bbox | matches under true extent | +|---|---|---| +| `transcripts_morton` | 0 / 320 | **320 / 320** | +| `transcripts_morton_then_feature` | 0 / 320 | **309 / 320** (rest are float-boundary ties) | + +So the codes in the `*_feature` permutations were quantised against the full extent +while their sentinel rows record a sub-box: the two disagree *inside the same file*. +Today's writer does not reproduce this — running `morton_sort_points` over all three +sort orders yields the true bbox every time, because `_extreme_positions` is taken +before the sort and the sentinels are prepended after +([`points.py:43`](../../python/spatialdata-js-util/src/spatialdata_js_util/points.py:43)). +**The fixture on disk is stale and needs regenerating.** + +That also means the damage is wider than the tile grid: `metadata.bounds` is the +Morton quantisation domain for `mortonIntervalsForBounds` +([`pointsTiling.ts:208`](../../packages/core/src/pointsTiling.ts:208)), so a wrong box +also maps every viewport to the wrong Morton intervals and therefore the wrong row +groups. Points are never *misplaced* — `loadMortonPointsInBounds` re-filters the +decoded rows to the requested bounds — so the failure is silently subtractive, the same +shape as the bisect bug fixed in `6cfe7bb`. + +### The reader gap this exposes — now guarded + +We trusted the sentinel box completely, on a claim the artifact makes about itself, and +a wrong claim degraded into "some of the map is missing" with no error anywhere. + +The probe now **recomputes `morton_code_2d` from x/y** for a sample of real rows and +refuses to tile unless a majority agree +([`pointsTiling.ts` `mortonBoundsAgreeWithCodes`](../../packages/core/src/pointsTiling.ts), +called from `VPointsSource.mortonBoundsMatchStoredCodes`). That tests the invariant that +actually matters — *is this box the quantisation domain?* — rather than a convention, so +an artifact with a deliberately padded domain still tiles. A rejected element drops its +`bounds` and reports `supportsRowGroupRangeReads: false`, which is the same pair the +"oversized sentinel row group" case already produces, so the resolver's existing probe +gate falls straight through to the capped preload. It also `console.warn`s: a silent +downgrade is what got us here. + +Cost, measured on the 12.1M-point / 245-row-group element: + +| probe | wall | range reads | bytes | +|---|---|---|---| +| before | 414 ms | 3 | 0.37 MB | +| with the guard | 523 ms | 4 | 2.16 MB | + +One extra row-group read, once per element, cached with the metadata — about one step of +the bisect that a single viewport query already runs eight of. The sample is taken from +the **middle** of the file: a truncated box can agree with the true one near the origin +by coincidence, never in the interior. Only positive evidence of disagreement disables +tiling; an unreadable sample keeps today's behaviour rather than losing the feature to an +unrelated failure. + +A cheaper check may become possible: `parquetFooterStats.ts` now parses per-row-group +column statistics out of the footer (that is what the feature-code index uses), so the +x/y extent could be read directly once a float stat decoder exists — `decodeIntStat` only +handles integer physical types today. That would compare the box against the data's real +extent for no extra I/O, but it tests the *convention* (box == exact min/max) rather than +the invariant, so it belongs alongside this check, not instead of it. + +### The second claim we were taking on trust: the sort + +A `morton_code_2d` column does not make a file Morton-**sorted**. A feature-primary +artifact — `transcripts_feature_then_morton`, sorted `(feature, morton)` — carries the +identical column with identical, correct values, a correct sentinel box, and every field +the probe looks for. Only the order is wrong, and nothing in the file says so. The +row-group bisect binary-searches that index assuming it ascends, so on this element it +lands somewhere arbitrary and a tile comes back holding whichever feature blocks happened +to live in the row groups it picked. That is what "some tiles just pick up one or other +feature, most miss" was. + +Read from footer statistics, the two indexes could not look more different — each row +group of the feature-primary file spans nearly the whole code range, because one gene is +scattered across the whole slide: + +| element | descents (`min[i] < max[i-1]`) | first six row groups `[min, max]` | +|---|---|---| +| `transcripts_morton` | **0** / 244 | `[0,0] [437752573,724881652] [724881909,732597813] …` | +| `transcripts_morton_then_feature` | **0** / 244 | identical — morton is still the primary key | +| `transcripts_feature_then_morton` | **185** / 244 | `[0,0] [450484663,4193473654] [443527237,4288997289] …` | + +Note the second row: a *secondary* feature key is harmless, which is why this has to be +measured rather than inferred from the element's name. The index-manifest does record +`tiling_kind: "experimental"` for the feature-primary condition, but a store's manifest is +not something a reader can rely on; the file itself now answers the question. + +The probe checks it, and the check is **free** — `datasetMetadata.parts` already carries +the footer bytes by the time it runs, and `morton_code_2d` is INT32 with complete +statistics on all 245 row groups. Failing it skips the sentinel sampling read as well, +since the outcome can no longer change, so a rejected element costs *less* than before. +One subtlety: the column is `uint32` and Morton codes use the top bit for real, so the +statistics must be decoded unsigned — `decodeIntStat` would read the far corner of the +slide as negative. + +**This also retires the bisect's standing TODO.** `loadParquetRowGroupColumnExtent` says +it "should be reading the row group's column statistics" and instead range-reads and +decodes each row group twice to recover a few boundary values. Those statistics are right +here, complete, and free. Folding them in belongs with step 6. + +Under-selection has now bitten this path four times (`zcover` depth, the row-group bisect, +the sentinel box, the sort). Standing rule for the tiled path: prefer to fail loudly over +returning fewer points. + +--- + +## Remaining work + +**Step 5 — Trustworthy grid. ✅ done.** +The permutations store was regenerated in place (2026-08-12): all four elements now +report the true extent, and their codes reproduce 320/320 from it. Today's writer never +had the bug — `_extreme_positions` is taken before the sort and the sentinels prepended +after — so the file was simply older than the writer. The reader-side guard above landed +with it, so the next stale artifact degrades to preload with a warning instead of +silently drawing part of the map. + +**Step 6 — A grid that follows zoom. ✅ done.** + +*The row-group index moved off the file.* `selectMortonRowGroups` picks row groups from +the footer statistics the probe already read, instead of bisecting. The bisect it +replaces range-read ~2MB **per step** to recover two numbers, `log2(rowGroups)` steps +per interval, for a few hundred intervals — which is how a viewport query could pull +most of a 439MB file to answer a question the footer had already answered. Measured on +one 1024 µm tile of the 12.1M-point element, both returning the same 643,961 points: + +| row-group selection | range reads | bytes | wall | +|---|---|---|---| +| bisect | 97 | 175.12 MB | 2911 ms | +| footer index | **32** | **57.83 MB** | **1035 ms** | + +The remaining 32 reads *are* the row-group data. It is also stricter than the bisect, +which tested only `max` and assumed the groups tile the code space without gaps. The +bisect stays as the fallback for an artifact whose statistics will not parse. + +*The grid comes from the artifact.* [`pointsTileGrid.ts`](../../packages/core/src/pointsTileGrid.ts) +derives both ends from one number, the point density: + +- **finest** — a tile must stay at least as big as one row group's footprint, + `sqrt(maxRowsPerGroup / density)`. Reads round up to whole row groups, so below that + four tiles fetch what one used to, for the same bytes and more requests. Same + argument as `MORTON_ZCOVER_MAX_DEPTH`: resolution finer than the storage granularity + is pure cost. +- **coarsest** — a tile should hold at most `POINTS_TILE_TARGET_ROWS` (400k), so one + request stays a fraction of the layer instead of most of it. +- `zoomOffset = log2(modelMatrixScale)` couples `z` to the viewport: deck picks + `z = ceil(viewport.zoom + zoomOffset)` from a zoom in **world** units while tile spans + are in **local** units, and the model matrix is exactly that difference. It lands a + tile at 256–512 screen pixels instead of at whatever the transform implied. + +For this element that is `minZoom -1 … maxZoom 0` — 1024 µm (~324k rows) and 512 µm +(~81k rows) — with `zoomOffset 2.234`. Two levels, which is worth knowing rather than +hiding: **the row-group size is the floor**, and 50k-row groups on this artifact put it +at ~402 µm. The old fixed 1024 was accidentally near-optimal *for this file* and would +not be for one an order of magnitude smaller or denser. Verified in the app: at +viewport zoom −7.8 it selects 44 tiles at z −1; at −2.8 it selects 6 at z 0. + +*The tile cache is budgeted in rows.* `maxCacheSize` is now +`cacheRowBudget / estimatedRowsPerTile`, clamped to [16, 512], against +`DEFAULT_POINTS_MEMORY_CAP` — 16 tiles here, a stated worst case of ~5.2M rows, versus +deck's default reaching ~220 tiles / ~71M rows. Observed shrinking from 44 to exactly +16 on zoom-in. `maxRequests` stays at 6, but as a decision rather than an inheritance. +This answers **open question 3** at the level ADR 0005 asks for — the second pool is now +*accounted*, not managed: nothing evicts by bytes, and a tile's real footprint is +whatever its points weigh. + +**Step 7 — Defaults + docs. ✅ done.** +`pointsTiling` defaults to `'auto'` (`DEFAULT_POINTS_TILING`), read everywhere through +`pointsTilingEnabled` so `undefined` cannot mean one thing to the resolver and another +to the panel. The per-element table is in +[points-preload-feature-filter-status](./points-preload-feature-filter-status.md), and +D5 is closed in the punch-list. + +The argument for flipping is not "the probe is cheap" — that was open question 1's +framing, and it was the wrong one. It is that **on a Morton artifact the preload is +not a neutral alternative**: it keeps the first `cap` rows in FILE order, and file +order there is a prefix of the Z-curve, so the 4M-row default shows a spatially skewed +chunk of the slide rather than a sample of it. Tiles read what is in view. + +The panel keeps the toggle — that comparison is worth being able to make — and now +hides the memory cap on a tiled layer, which governs nothing there and sat directly +above a line saying so. + +**The cost of the flip, measured, because it is not free.** At the default zoomed-out +framing of the 12.1M-point element, the tiled path selects all 44 tiles and loads +**12,165,029 points / ~158 MB** — the whole artifact. The capped preload it replaces +reads a 4M-row prefix. So the first paint on this element is now ~3x the rows and +more I/O than before, in exchange for a picture that is *correct* rather than a +Z-curve prefix, and one that streams in 44 pieces instead of blocking on one decode. + +That is the **no-LOD gap** stated plainly: zooming out is the one direction viewport +tiling does not help, because there is no coarser representation to read — every tile +is full resolution. The fix is a multi-resolution points pyramid (the writer already +has a `points multiscale` command), not a finer index, and it is the strongest +argument for doing that work next. If the trade proves wrong for a given deployment, +`pointsTiling: 'off'` is one config key away. + +Twelve resolver tests changed, which is the honest cost of the flip: a fresh entry now +plans `['tiling']` and defers the preload until the probe answers. The tests that were +about preload/rowCodes mechanics pin `pointsTiling: 'off'` and say why; the default has +its own tests instead of being asserted incidentally forty times. + +--- + +## Verification + +- **Fixture.** A `transcripts_morton` element (Morton sort + `feature_name_codes` + + multiple row groups) written by `spatialdata-js-util` — see + `python/spatialdata-js-util/src/spatialdata_js_util/index_permutations.py` and + `scripts/benchmark_points_index.py`. Needs to be small enough to live under the demo's + `/test-fixtures` but with **enough row groups that a viewport touches a strict subset** + — a single-row-group fixture proves nothing. +- **Beware the fixture-proxy trap**: the vis demo's `/test-fixtures` proxy 502s when a + launcher sets `PORT`, and worktrees need the fixture symlink. +- The permutations store was regenerated on 2026-08-12; all four points elements are + now sound. If you are on an **older copy**, `transcripts_morton_then_feature` and + `transcripts_feature_then_morton` carry the stale sentinel box — the probe will now + refuse to tile them and say so in the console, rather than drawing half the slide. +- **`transcripts_feature_then_morton` is not a tiling fixture** and never was: it is + feature-primary, so the probe declines it by design. Use it to exercise the + feature-code row-group index on the *preload* path, which is a different mechanism + and unaffected. `transcripts_morton` and `transcripts_morton_then_feature` are the + tiling fixtures. +- **The debug overlay is the instrument.** `showTileDebugOverlay` already colours tiles by + status; use it plus `read_network_requests` to confirm range reads are bounded by the + viewport rather than fetching the whole file. +- **Verify on both surfaces** — the full-UI `SpatialCanvas` and `SpatialCanvasViewer` own + separate handlers; a tiled layer must work in both, with real data. + +## Open questions + +1. **Default for `pointsTiling`.** `'auto'` costs one metadata probe (a footer read) on + every points element before anything renders; `'off'` means nobody gets the feature + without opting in. Leaning `'auto'`, with the probe made cheap and its failure + arm falling straight through to preload — but measure the probe on a real Xenium + store first. **Answered in step 7: `'auto'`** — and the framing above was wrong. + The probe cost was never the deciding factor (4 range reads / 2.16 MB / ~520 ms, + once, cached, and on a non-Morton element it is footer metadata the preload reads + anyway). What decides it is that the preload is *not* a neutral alternative on a + Morton artifact: file order is a prefix of the Z-curve, so the capped preload shows + a spatially skewed chunk of the slide. Both guards make the failure arms loud, so + `'auto'` degrades visibly rather than silently. +2. **Preload *and* tiles?** A tiled element could still preload a small resident window + for instant zoomed-out context while tiles fill in. Attractive, but it re-introduces + two batches with different code spaces and revives the alignment invariants D5 was + supposed to escape. Default: no. +3. **Tile cache eviction / memory accounting.** *Answered in step 6, at the accounting + level.* `maxCacheSize` is derived from a row budget rather than left at deck's + `5 x selected`, so the second pool has a stated worst case (~16 tiles / ~5.2M rows + on the Xenium element). Still open, and deliberately so per + [ADR 0005](../adr/0005-memory-accounting-before-management.md): nothing evicts by + *bytes*, and a tile's real footprint is whatever its points weigh, so a dense + viewport can still exceed the estimate. Manage after accounting, not before. +4. **Multi-layer worker contention (D6).** Two tiled layers multiply concurrent row-group + reads through one points worker. Out of scope here, but D5 makes it reachable. +5. **Does the resident/matched machinery apply at all on the tiled path?** The + feature-index scan (`matching`) exists because the resident window truncates the + dataset. Viewport tiling is a different answer to the same problem. Likely: `matching` + is not planned for tiled elements — confirm rather than leave both running. diff --git a/docs/plans/points-preload-feature-filter-status.md b/docs/plans/points-preload-feature-filter-status.md index 33f6f561..61c0da6e 100644 --- a/docs/plans/points-preload-feature-filter-status.md +++ b/docs/plans/points-preload-feature-filter-status.md @@ -1,7 +1,7 @@ # Points preload & feature filter — status and plan **Status:** work in progress (branch/worktree, not yet on `main`) -**Last updated:** 2026-06-20 +**Last updated:** 2026-08-12 **Related:** [ADR 0002](../adr/0002-spatially-aware-vector-loading.md), [ADR 0003](../adr/0003-points-render-resource.md) This document captures what we built, what broke, what we fixed, and what still @@ -86,6 +86,43 @@ Feature-primary or compound-indexed stores are still experimental. They may make filter changes part of the structural load key because the point of the index is to fetch only selected features. +### Which path an element takes (as of D5 step 7, 2026-08-12) + +**The choice is automatic.** `pointsTiling` defaults to `'auto'` +(`DEFAULT_POINTS_TILING`), so every points element is probed once and takes the tiled +path if — and only if — it can. Resolve the config with `pointsTilingEnabled(...)` +rather than comparing to `'auto'`, so `undefined` means the same thing everywhere. + +Measured against `xenium_2.q0.001.htj2k.index-permutations.zarr`: + +| element | index | path | why | +|---|---|---|---| +| `transcripts` | none (dict-only `feature_name`) | preloaded scatter | no `morton_code_2d`; the probe returns `null` from the schema alone | +| `transcripts_morton` | `(morton)` | **viewport tiles** | sorted, sentinel box verified against the codes | +| `transcripts_morton_then_feature` | `(morton, feature)` | **viewport tiles** | a *secondary* feature key is harmless — morton is still primary and still sorted | +| `transcripts_feature_then_morton` | `(feature, morton)` | preloaded scatter | morton column present but unsorted; the bisect would land arbitrarily, so the probe declines it and warns | + +Three ways an element with a Morton column still ends up on the preload, all of them +loud rather than silent — see +[points-morton-tiled-viewport-loading](./points-morton-tiled-viewport-loading.md): + +1. **No usable sentinel row group** (missing, or 5+ rows) — no bounds, so no + quantisation domain. +2. **Sentinel box is not the code domain** — recomputing `morton_code_2d` from x/y + disagrees for a majority of sampled rows. A stale artifact that would otherwise + have rendered part of the slide with no error. +3. **`morton_code_2d` is not sorted across row groups** — read from footer statistics, + free. + +The probe costs 4 range reads / ~2.16 MB / ~520 ms on a 12.1M-point element, once, and +is cached with the metadata. On an element that cannot be tiled it is the footer +metadata the preload path reads anyway. + +**Why on by default rather than opt-in:** on a Morton artifact the preload is not a +neutral alternative. It keeps the first `cap` rows in FILE order, and file order here +is a prefix of the Z-curve — a spatially skewed chunk of the slide, not a sample of +it. Tiles read what is actually in view. + --- ## What we changed (summary) diff --git a/docs/plans/points-redesign-punchlist.md b/docs/plans/points-redesign-punchlist.md index 2673012b..d1b8ec63 100644 --- a/docs/plans/points-redesign-punchlist.md +++ b/docs/plans/points-redesign-punchlist.md @@ -132,9 +132,22 @@ Each notes *why* it's coupled to the state-model / decode rework. completes.** Today stats only appear once the whole-dataset catalog settles; there's real use in showing progressive/active counts. Tied to D3 (progressive catalog build) and the stats state model (F3's proper fix). -- **D5 — Tiled (Morton) viewport-driven loading.** The tiled path isn't exercised; - viewport-driven load is a major feature and exactly the kind of demand-driven - state the new model should own (Morton tiling is still "dark" per the roadmap). +- **D5 — Tiled (Morton) viewport-driven loading. ✅ DONE (2026-08-12).** Shipped in + seven steps — + [points-morton-tiled-viewport-loading.md](./points-morton-tiled-viewport-loading.md). + A Morton element is probed, framed from its own extent, and drawn from viewport + tiles that colour by feature, honour the feature filter inside the row-group scan, + and subdivide with zoom; row groups are selected from footer statistics rather than + by bisecting the file (97 range reads / 175 MB → 32 / 58 MB for one tile). **On by + default**: the preload it replaces keeps the first `cap` rows in *file* order, which + on a Morton artifact is a prefix of the Z-curve — a skewed chunk of the slide. + Three guards decline a malformed artifact loudly and fall back to the preload. + *Left open, deliberately:* the tile cache is accounted but not managed by bytes + (ADR 0005), and there is **no LOD** — a zoomed-out view reads every tile, measured + at 12.17M points / ~158 MB for the default framing of the Xenium element, against a + 4M-row prefix for the preload it replaces. That wants a multi-resolution points + pyramid (the writer has `points multiscale`), not a finer index, and it is the next + thing worth doing. - **D6 — Worker contention with multiple layers.** Multiple point layers share one worker; the engine keys by element and assumes single-demand-per-element. Multi-layer sharing / a work queue belongs with the engine redesign. diff --git a/packages/core/src/engine/PointsResolver.ts b/packages/core/src/engine/PointsResolver.ts index 56626619..8b8a12f1 100644 --- a/packages/core/src/engine/PointsResolver.ts +++ b/packages/core/src/engine/PointsResolver.ts @@ -1,8 +1,16 @@ +import type { Matrix4 } from '@math.gl/core'; import type { PointsElement } from '../models/index.js'; import { featureCodeMapFromCatalog, remapRowFeatureCodes } from '../pointsFeatures.js'; import { DEFAULT_POINTS_MEMORY_CAP } from '../pointsLimits.js'; import type { PointsLoadProgress, PointsLoadResult } from '../pointsLoadOptions.js'; -import type { PointsFeatureCatalog } from '../pointsTiling.js'; +import { planPointsLoads } from '../pointsLoadPlan.js'; +import { + type PointsFeatureCatalog, + type PointsTilingMetadata, + type PointsTilingMode, + pointsTilingEnabled, +} from '../pointsTiling.js'; +import { type AxisAlignedBounds, transformAxisAlignedBounds } from '../spatialViewFit.js'; import type { EntryNotice, SpatialEntryError } from './errors.js'; import { RequestSlot } from './RequestSlot.js'; import { Resolution } from './resolution.js'; @@ -89,6 +97,15 @@ export interface PointsResolveConfig { pointsMemoryCap?: number; colorByFeature?: boolean; featureCodes?: number[]; + /** + * Whether to probe for a Morton-tiled artifact before falling back to the resident + * preload (D5). `'auto'` — {@link DEFAULT_POINTS_TILING}, and now the default — + * probes; `'off'` never does. + * + * Read it through {@link pointsTilingEnabled} rather than comparing to `'auto'`, so + * the default lives in exactly one place. + */ + pointsTiling?: PointsTilingMode; } interface PointsEntry { @@ -152,8 +169,28 @@ interface PointsEntry { * streaming `partial` is the scan's growing buffer. */ matching: RequestSlot; + /** + * Morton tiling metadata (D5), as a one-key slot — the element path is fixed, so + * there is exactly one request to make (`'probe'`). + * + * The value is **tileable metadata or `null`**, not raw metadata: the probe applies + * the same renderability gate the render resolver would + * (`supportsRowGroupRangeReads && bounds`), so `null` is the settled fact "this + * element cannot be tiled" rather than "we have not looked". A *failed* probe reads + * as `null` too — it must fall through to the preload rather than strand the layer — + * while staying `failed`, and therefore retryable, in the snapshot. + */ + tiling: RequestSlot; + /** World bounds of a tiled entry, memoised on the metadata AND the transform it + * was computed with — bounds are transform-relative. */ + bounds?: AxisAlignedBounds | null; + boundsSource?: PointsTilingMetadata; + boundsTransform?: unknown; } +/** The tiling slot's only key. The element path is fixed, so one request exists. */ +type TilingProbeKey = 'probe'; + /** A settled or in-flight matched batch, tagged with the selection it covers. */ interface MatchingValue { readonly signature: string; @@ -187,9 +224,22 @@ export interface PointsMatchingLoadState { export class PointsResolver implements ResourceResolver { readonly kind = 'points' as const; - /** Only the resident preload gates a first paint. The catalog, row codes and - * feature scan all refine an already-drawable layer. */ - readonly blockingResources = ['preload'] as const; + /** + * What gates a first paint: the tiling probe (until it answers, we do not know + * which path this entry is even on) and the resident preload. The catalog, row + * codes and feature scan all refine an already-drawable layer. + * + * This stays a constant — *the snapshot varies instead*. `isBlocking` skips a + * resource the entry does not have, so a tiled entry (which never plans a preload) + * simply omits `preload` from its resources, and an entry with tiling off omits + * `tiling`. That keeps the list what ADR 0004 asks for — data describing this + * resolver's kind — rather than a switch, and avoids the alternative of settling a + * fake `preload` resolution that lies about a load nobody ran. + * + * Getting this wrong is not cosmetic: a tiled entry whose `preload` stayed `idle` + * blocks forever, and auto-fit rides the `isBlocking` true→false transition. + */ + readonly blockingResources = ['tiling', 'preload'] as const; private readonly entries = new Map(); private readonly listeners = new Set<() => void>(); @@ -252,6 +302,19 @@ export class PointsResolver implements ResourceResolver({ + context: { + elementKey: key, + kind: 'points', + resource: 'tiling', + fallback: 'load-failed', + }, + onChange, + // Nothing is drawable from a probe, so its start is not a re-render. Its + // SETTLE still notifies (a settle always does), which is what re-plans — + // and re-planning is how the deferred preload gets scheduled. + notifyOnLoading: false, + }), }; this.entries.set(key, entry); } @@ -274,12 +337,60 @@ export class PointsResolver implements ResourceResolver 0; @@ -359,6 +470,9 @@ export class PointsResolver implements ResourceResolver): EntryResources { const key = ctx.elementKey; - // Key the memo by everything the snapshot embeds: the entry (several layers - // may share one element), and the selection (it drives the truncation notice). - // Points bounds are not wired in Step 1, so the transform is not part of the key. - const configSig = (ctx.config.featureCodes ?? []).join(','); + // Key the memo by everything the snapshot embeds: the entry (several layers may + // share one element), the selection (it drives the truncation notice), and + // whether tiling is on (it decides which resources the entry even has, and a + // config flip alone bumps no version). + const configSig = `${(ctx.config.featureCodes ?? []).join(',')}|${pointsTilingEnabled(ctx.config.pointsTiling)}`; const cached = this.snapshots.get(ctx.entryId, this.version, ctx.transform, configSig); if (cached) return cached; + // Which resources this entry HAS — see `blockingResources`. A tiled entry has no + // resident preload (not an idle one: none), and an entry with tiling off never + // asked the tiling question. + const tiled = this.isTiledFor(ctx.config, key); + const resources: Record> = { + catalog: this.catalogResolution(key), + rowCodes: this.rowCodesResolution(key), + matching: this.matchingResolution(key), + }; + if (!tiled) { + resources.preload = this.preloadResolution(key); + } + if (pointsTilingEnabled(ctx.config.pointsTiling)) { + resources.tiling = this.tilingResolution(key); + } + const value: EntryResources = { entryId: ctx.entryId, elementKey: key, - resources: { - preload: this.preloadResolution(key), - catalog: this.catalogResolution(key), - rowCodes: this.rowCodesResolution(key), - matching: this.matchingResolution(key), - }, + resources, notices: this.notices(key, ctx.config.featureCodes), - bounds: null, // Points bounds come from the tiling metadata; not wired in Step 1. + // A tiled entry can be framed from the artifact's own extent, with no geometry + // in memory — which is the only thing that makes auto-fit possible before a + // single tile has loaded. The preloaded path's bounds stay with the host (it + // caches them against the resident batch); see the D5 plan step 2. + bounds: tiled ? this.tiledBounds(key, ctx.transform) : null, revision: this.version, }; @@ -417,6 +547,30 @@ export class PointsResolver implements ResourceResolver { + return this.entries.get(key)?.tiling.resolution ?? Resolution.idle(); + } + + /** + * World bounds for a tiled entry, from the artifact's extent. Memoised on + * (metadata, transform) like the shapes resolver memoises on (data, transform): the + * snapshot returns bounds by identity, so recomputing per call would be a fresh + * object every reconcile. + */ + private tiledBounds(key: string, transform: Matrix4): AxisAlignedBounds | null { + const entry = this.entries.get(key); + const metadata = entry?.tiling.value; + if (!entry || !metadata?.bounds) return null; + if (entry.boundsSource === metadata && entry.boundsTransform === transform) { + return entry.bounds ?? null; + } + const computed = transformAxisAlignedBounds(metadata.bounds, transform); + entry.bounds = computed; + entry.boundsSource = metadata; + entry.boundsTransform = transform; + return computed; + } + private matchingResolution(key: string): Resolution { // Unwrap the slot's Resolution into Resolution // — the resource surface is the batch, the signature is internal bookkeeping. @@ -489,6 +643,56 @@ export class PointsResolver implements ResourceResolver { + const { key, layerId, element } = target; + const slot = this.ensureEntry(key).tiling; + const before = slot.pending; + const loading = slot.request('probe', async () => { + const metadata = await element.getPointsTilingMetadata(); + if (!metadata?.supportsRowGroupRangeReads || !metadata.bounds) { + return null; + } + return metadata; + }); + + // Same contract as the preload's: 'loading' only when a NEW request starts (not + // on a dedup). The only terminal state the probe owns is a *tileable* answer — + // that entry is now drawable. Every other outcome hands off to the preload, which + // reports its own; claiming 'ready' here would clear the spinner while the real + // geometry load had not started. + if (loading !== before) { + this.callbacks.onStatus?.(layerId, 'loading'); + void loading.then(() => { + if (this.isTiled(key)) { + this.releaseResidentBatch(key); + this.callbacks.onStatus?.(layerId, 'ready'); + } + }); + } + return loading; + } + + /** + * Drop the resident window once an element is known to be tiled. + * + * A layer switched to tiling mid-session has usually already preloaded — up to the + * full memory cap, tens of millions of rows the tile path will never read. Nothing + * else releases it: `plan()` stops ASKING for a preload, which is not the same as + * giving one back, so the memory stayed held and the panel went on reporting "4M of + * 12.1M in memory — capped" over a render that has no cap. + * + * The row codes and the feature-index scan go with it: both are defined against + * that window (codes are row-aligned to it, the scan exists only because it + * truncates the dataset), so keeping them would leave state describing a batch that + * no longer exists. The catalog stays — it describes the ELEMENT's features, and the + * filter panel still wants it. + * + * Runs once, on the probe's settle. A layer that is NOT tiling can re-request the + * preload on its next plan pass — an element read two ways pays for it once, rather + * than ping-ponging, because nothing evicts again. + */ + private releaseResidentBatch(key: string): void { + const entry = this.entries.get(key); + if (!entry || entry.preload.resolution.status === 'idle') { + return; + } + entry.preload.reset(); + entry.rowCodes.reset(); + entry.matching.reset(); + // Every memo derived from the row codes goes with them — they all key on + // `residentCodesSource`, so leaving one behind keeps a map alive for a batch that + // no longer exists, which is the opposite of the point of releasing it. + entry.residentCodes = undefined; + entry.residentCounts = undefined; + entry.residentCodesSource = undefined; + this.notify(); + } + // --- Retry ------------------------------------------------------------------ /** @@ -1246,7 +1563,7 @@ export class PointsResolver implements ResourceResolver { const entry = this.entries.get(key); if (!entry) return Promise.resolve(); - const pending = [entry.preload, entry.catalog, entry.rowCodes, entry.matching] + const pending = [entry.preload, entry.catalog, entry.rowCodes, entry.matching, entry.tiling] .filter((slot) => slot.isFailed) .map((slot) => slot.retry()) .filter((promise): promise is Promise => promise !== undefined); @@ -1265,6 +1582,7 @@ export class PointsResolver implements ResourceResolver; + try { + footer = parseParquetFileMetaData(metaBytes); + } catch { + return []; + } + for (const rowGroup of footer.rowGroups) { + const column = rowGroup.columns.find((col) => col.path === MORTON_CODE_2D_COLUMN); + const min = decodeUnsignedIntStat(column?.minValue, column?.physicalType ?? null); + const max = decodeUnsignedIntStat(column?.maxValue, column?.physicalType ?? null); + extents.push(min !== null && max !== null ? [min, max] : null); + } + } + return extents.length === expectedRowGroupCount ? extents : []; +} + /** Whether a row group's code range can contain any selected code. `null` extent * (missing stats) is treated as "might match" so it is scanned, not skipped. */ function extentMayContainSelectedCodes( @@ -208,15 +251,19 @@ function pointsScanChunkProgress( import { extractSentinelBoundingBox, - featureCodeAllowSet, filterPointsToBounds, isMortonSentinelValue, MORTON_CODE_2D_COLUMN, + type MortonRowGroupExtent, + mortonBoundsAgreeWithCodes, mortonIntervalsForBounds, + mortonRowGroupExtentsAreSorted, type PointsFeatureCatalog, type PointsInBoundsOptions, type PointsInBoundsResponse, type PointsTilingMetadata, + type SpatialBounds, + selectMortonRowGroups, } from '../pointsTiling.js'; import type { Axis } from '../schemas'; // import { normalizeAxes } from '@vitessce/spatial-utils'; @@ -2253,6 +2300,46 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { return featureCatalogFromCodeMap(featureKey, codeToName); } + /** + * Cross-check the sentinel bounding box against real rows before trusting it. + * + * Costs one row-group range read, once per element, cached with the metadata. That + * is roughly one step of the bisect a single viewport query already runs eight of — + * cheap next to deciding, wrongly, to read the whole artifact through a broken + * index. Sampled from the MIDDLE of the file: a truncated box can agree with the + * true one near the origin by coincidence, never in the interior. + * + * Only positive evidence of disagreement disables tiling. A file we cannot sample + * (one row group, an unreadable group) keeps today's behaviour rather than losing + * the feature to a read that failed for an unrelated reason. + */ + private async mortonBoundsMatchStoredCodes( + parquetPath: string, + bounds: SpatialBounds + ): Promise { + const dataset = await this.loadParquetDatasetMetadata(parquetPath); + const totalRowGroups = dataset?.totalNumRowGroups ?? 0; + if (totalRowGroups <= 1) { + return true; + } + const table = await this.loadParquetRowGroupByGroupIndex( + parquetPath, + Math.max(1, Math.floor(totalRowGroups / 2)), + { columns: ['x', 'y', MORTON_CODE_2D_COLUMN] } + ).catch(() => null); + if (!table || table.numRows === 0) { + return true; + } + const xs = table.getChild('x')?.toArray(); + const ys = table.getChild('y')?.toArray(); + const codes = table.getChild(MORTON_CODE_2D_COLUMN)?.toArray(); + if (!xs || !ys || !codes) { + return true; + } + const { checked, matched } = mortonBoundsAgreeWithCodes(xs, ys, codes, bounds); + return checked === 0 || matched * 2 > checked; + } + async getPointsTilingMetadata(elementPath: string): Promise { if (this.pointTilingMetadataCache.has(elementPath)) { return this.pointTilingMetadataCache.get(elementPath) ?? null; @@ -2294,6 +2381,23 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { } const canLoadRowGroups = await this.canLoadParquetRowGroups(); + // Having a morton_code_2d column does not make a file Morton-SORTED. A + // feature-primary artifact carries the same column with the same values, and the + // row-group bisect run over it lands arbitrarily — the visible result is a tile + // holding one or two features and missing the rest. Free to check: the footer + // statistics are already in `datasetMetadata.parts`. + const mortonExtents = datasetMetadata + ? rowGroupMortonExtents(datasetMetadata.parts, datasetMetadata.totalNumRowGroups) + : []; + const rowGroupsAreSorted = mortonRowGroupExtentsAreSorted(mortonExtents); + if (!rowGroupsAreSorted) { + console.warn( + `Morton tiling disabled for ${elementPath}: its morton_code_2d column is not ` + + 'sorted across row groups, so it is indexed by something else (a ' + + 'feature-primary artifact carries the same column unsorted). The row-group ' + + 'bisect needs a sorted index; falling back to the capped preload.' + ); + } const firstRowGroupRowCount = datasetMetadata?.rowGroupRows?.[0] ?? 0; const hasValidSentinelRowGroup = firstRowGroupRowCount >= 2 && firstRowGroupRowCount <= 4; const firstRowGroup = @@ -2303,9 +2407,29 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { limit: 4, }) : null; - const bounds = firstRowGroup + const claimedBounds = firstRowGroup ? (extractSentinelBoundingBox(firstRowGroup) ?? undefined) : undefined; + // Skip the sampling read when the sort has already ruled the file out — the answer + // cannot change the outcome, and this is the probe's only real I/O. The box then + // goes unreported rather than reported-unverified, which is the habit that let a + // stale sentinel box draw half a slide; nothing reads `bounds` without + // `supportsRowGroupRangeReads` in any case. + const bounds = + claimedBounds && + rowGroupsAreSorted && + (await this.mortonBoundsMatchStoredCodes(parquetPath, claimedBounds)) + ? claimedBounds + : undefined; + if (claimedBounds && !bounds && rowGroupsAreSorted) { + console.warn( + `Morton tiling disabled for ${elementPath}: its sentinel bounding box ` + + `(${claimedBounds.minX}, ${claimedBounds.minY})-(${claimedBounds.maxX}, ${claimedBounds.maxY}) ` + + 'is not the domain its morton_code_2d values were quantised against, so any ' + + 'viewport query against it would silently drop rows. Falling back to the ' + + 'capped preload; the artifact needs rewriting.' + ); + } const rowGroupSizes = datasetMetadata?.rowGroupRows ?? []; const metadata: PointsTilingMetadata = { @@ -2319,8 +2443,15 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { totalRowGroups: datasetMetadata?.totalNumRowGroups ?? 0, maxRowsPerGroup: rowGroupSizes.length ? Math.max(...rowGroupSizes) : 0, rowGroupRowCounts: datasetMetadata?.rowGroupRows, - supportsRowGroupRangeReads: Boolean(datasetMetadata && canLoadRowGroups && bounds), + supportsRowGroupRangeReads: Boolean( + datasetMetadata && canLoadRowGroups && bounds && rowGroupsAreSorted + ), bounds, + // Carried on the metadata so viewport queries select row groups from memory + // instead of bisecting the file. Empty means "no usable statistics" — the + // bisect stays as the fallback, so this stays an optimisation, not a + // requirement. + ...(mortonExtents.length > 0 ? { rowGroupMortonExtents: mortonExtents } : {}), }; return metadata; @@ -2461,6 +2592,46 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { }; } + /** + * Row groups a set of Morton intervals can touch. + * + * Prefers the in-memory index the probe read out of the footer: exact, and free. + * The bisect below is the fallback for an artifact whose statistics we could not + * read — it recovers the same two numbers per row group by range-reading and + * decoding the group's bytes, ~2MB a step on a real transcripts artifact, and a + * single viewport query walks `log2(rowGroups)` steps for each of a few hundred + * intervals. That is the whole reason viewport queries used to be able to pull the + * entire file down to answer a question the footer had already answered. + */ + private async selectRowGroupsForIntervals( + metadata: PointsTilingMetadata, + intervals: ReadonlyArray + ): Promise { + const extents = metadata.rowGroupMortonExtents; + if (extents && extents.length === metadata.totalRowGroups) { + return selectMortonRowGroups(extents, intervals); + } + const rowGroupSet = new Set(); + for (const [start, end] of intervals) { + const first = await this.bisectRowGroupsRight( + metadata.parquetPath, + metadata.totalRowGroups, + start + ); + const last = await this.bisectRowGroupsRight( + metadata.parquetPath, + metadata.totalRowGroups, + end + ); + for (let rowGroup = first; rowGroup <= last; rowGroup++) { + if (rowGroup >= 0 && rowGroup < metadata.totalRowGroups) { + rowGroupSet.add(rowGroup); + } + } + } + return [...rowGroupSet].sort((a, b) => a - b); + } + private async bisectRowGroupsRight( parquetPath: string, totalRowGroups: number, @@ -2494,27 +2665,8 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { return null; } checkAbort(options.signal); - const allowedFeatureCodes = featureCodeAllowSet(options.featureCodes); const intervals = mortonIntervalsForBounds(metadata.bounds, options.bounds); - const rowGroupSet = new Set(); - for (const [start, end] of intervals) { - const first = await this.bisectRowGroupsRight( - metadata.parquetPath, - metadata.totalRowGroups, - start - ); - const last = await this.bisectRowGroupsRight( - metadata.parquetPath, - metadata.totalRowGroups, - end - ); - for (let rowGroup = first; rowGroup <= last; rowGroup++) { - if (rowGroup >= 0 && rowGroup < metadata.totalRowGroups) { - rowGroupSet.add(rowGroup); - } - } - } - const rowGroups = [...rowGroupSet].sort((a, b) => a - b); + const rowGroups = await this.selectRowGroupsForIntervals(metadata, intervals); const totalRowsUpperBound = rowGroups.reduce( (sum, rowGroup) => sum + rowGroupCountForIndex(metadata, rowGroup), 0 @@ -2525,18 +2677,18 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { // Dynamic, like the call site below: keeps the worker scan module out of the // eager main-thread bundle. Hoisted above the loop so the buffers can be built. - const { Float32PointBuffer, scanMortonTableInBounds } = await import( + const { Float32PointBuffer, Int32PointBuffer, scanMortonTableInBounds } = await import( '../workers/pointsWorkerScan.js' ); const xs = new Float32PointBuffer(); const ys = new Float32PointBuffer(); const zs = new Float32PointBuffer(); const hasZ = metadata.axisNames.includes('z'); - const filterByFeature = allowedFeatureCodes !== null; - const featureCodeColumnName = - filterByFeature && metadata.featureCodeColumnName - ? metadata.featureCodeColumnName - : undefined; + // Project the code column whenever the artifact HAS one — not only when a filter + // is active. The codes ride back on the batch so a tiled layer can colour by + // feature, and the "all features" view (no filter) is exactly the case that used + // to arrive without them and render flat. + const featureCodeColumnName = metadata.featureCodeColumnName || undefined; ensurePointsWorker(); if (isPointsWorkerEnabled()) { @@ -2568,6 +2720,7 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { bounds: options.bounds, loadMode: 'row-groups', tiling: metadata, + ...(workerResult.featureCodes ? { featureCodes: workerResult.featureCodes } : {}), }; } } catch (error) { @@ -2586,6 +2739,7 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { metadata.mortonCodeColumnName, ...(featureCodeColumnName ? [featureCodeColumnName] : []), ]; + const codes = featureCodeColumnName ? new Int32PointBuffer() : undefined; for (const rowGroup of rowGroups) { checkAbort(options.signal); const table = await this.loadParquetRowGroupByGroupIndex(metadata.parquetPath, rowGroup, { @@ -2605,6 +2759,7 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { xs, ys, zs, + ...(codes ? { codes } : {}), }); } @@ -2612,12 +2767,17 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { return null; } + const pointCount = xs.length; + const outCodes = codes?.toArray(); return { data: hasZ ? [xs.toArray(), ys.toArray(), zs.toArray()] : [xs.toArray(), ys.toArray()], - shape: [hasZ ? 3 : 2, xs.length], + shape: [hasZ ? 3 : 2, pointCount], bounds: options.bounds, loadMode: 'row-groups', tiling: metadata, + // One code per point or none at all — a short array would leave the tail + // reading code 0, a valid feature, and mis-colour it with conviction. + ...(outCodes && outCodes.length === pointCount ? { featureCodes: outCodes } : {}), }; } } diff --git a/packages/core/src/models/VTableSource.ts b/packages/core/src/models/VTableSource.ts index efa18201..2e209e51 100644 --- a/packages/core/src/models/VTableSource.ts +++ b/packages/core/src/models/VTableSource.ts @@ -197,8 +197,22 @@ export default class SpatialDataTableSource extends AnnDataSource { * fallback used when a store has no range support, so the layout is still * resolved once rather than per call. */ parquetPartPathsCache: Map>; - /** Morton min/max per row group — avoids re-decoding row groups during bisect. */ - rowGroupColumnExtentCache: Map; + /** + * Morton min/max per row group — avoids re-decoding row groups during bisect. + * + * Holds the in-flight PROMISE, not the settled value. Caching only the result + * dedups nothing while the read is running, and this index is built under exactly + * that load: every viewport tile bisects concurrently over the same row groups, so + * each one used to start its own full row-group fetch for an entry the others were + * already fetching. + */ + rowGroupColumnExtentCache: Map< + string, + Promise<{ min: number | null; max: number | null } | null> + >; + /** First value of a column per row group — the boundary the extent is derived + * from, shared between neighbouring row groups so each is read once. */ + rowGroupColumnFirstValueCache: Map>; obsIndices: Record>; varIndices: Record>; varAliases: Record; @@ -222,6 +236,7 @@ export default class SpatialDataTableSource extends AnnDataSource { this.parquetDatasetMetadataCache = new Map(); this.parquetPartPathsCache = new Map(); this.rowGroupColumnExtentCache = new Map(); + this.rowGroupColumnFirstValueCache = new Map(); // Table-specific properties this.obsIndices = {}; @@ -976,6 +991,24 @@ export default class SpatialDataTableSource extends AnnDataSource { ); } + /** + * First and last value of a column within one row group — the sorted-order index + * the Morton row-group bisect searches. + * + * **This should be reading the row group's column statistics**, which parquet + * writers already put in the footer we have parsed: `min`/`max` are sitting there, + * exact, for nothing. Instead each call range-reads the row group's bytes (every + * column, ~2MB on a real transcripts artifact) and decodes it twice — once for the + * first row, once for the last. Building the index for a 245-row-group file that + * way can fetch the whole file to recover ~4KB of boundary values. + * + * It is written this way because the **vendored parquet-wasm build exposes no + * statistics accessor**: `RowGroupMetaData` offers only `numRows`/`fileOffset`/ + * `compressedSize`/`column`, and `ColumnChunkMetaData` offers no `statistics()`. + * Fixing it properly needs either a wasm rebuild that exposes statistics or a + * minimal Thrift read of the footer we already hold — see + * `docs/plans/points-morton-tiled-viewport-loading.md`. + */ async loadParquetRowGroupColumnExtent( parquetPath: string, columnName: string, @@ -986,38 +1019,105 @@ export default class SpatialDataTableSource extends AnnDataSource { if (cached) { return cached; } + const pending = this.readParquetRowGroupColumnExtent(parquetPath, columnName, rowGroupIndex); + this.rowGroupColumnExtentCache.set(cacheKey, pending); + // A failed probe must not be cached as a permanent "no extent" — that would + // strand the bisect on a transient network error for the life of the source. + pending + .then((extent) => { + if (extent === null && this.rowGroupColumnExtentCache.get(cacheKey) === pending) { + this.rowGroupColumnExtentCache.delete(cacheKey); + } + }) + .catch(() => { + if (this.rowGroupColumnExtentCache.get(cacheKey) === pending) { + this.rowGroupColumnExtentCache.delete(cacheKey); + } + }); + return pending; + } + + private async readParquetRowGroupColumnExtent( + parquetPath: string, + columnName: string, + rowGroupIndex: number + ): Promise<{ min: number | null; max: number | null } | null> { const dataset = await this.loadParquetDatasetMetadata(parquetPath); - const rowCount = dataset?.rowGroupRows?.[rowGroupIndex]; - if (!rowCount) { + const totalRowGroups = dataset?.totalNumRowGroups ?? 0; + if (!dataset?.rowGroupRows?.[rowGroupIndex]) { return null; } - const columnOptions: ParquetRowGroupReadOptions = { columns: [columnName] }; - const minTable = await this.loadParquetRowGroupByGroupIndex(parquetPath, rowGroupIndex, { - ...columnOptions, - limit: 1, - }); - const minColumn = minTable?.getChild(columnName); - if (!minColumn || minColumn.length === 0) { + const min = await this.readParquetRowGroupColumnFirstValue( + parquetPath, + columnName, + rowGroupIndex + ); + if (min === null) { return null; } - let maxValue: number | null = parquetColumnValueToNumber(minColumn.get(0)); - if (rowCount > 1) { - const maxTable = await this.loadParquetRowGroupByGroupIndex(parquetPath, rowGroupIndex, { - ...columnOptions, - offset: rowCount - 1, - limit: 1, - }); - const maxColumn = maxTable?.getChild(columnName); - if (maxColumn && maxColumn.length > 0) { - maxValue = parquetColumnValueToNumber(maxColumn.get(0)); - } + // The last row group has nothing after it, so its upper bound is open. `null` + // already means "unbounded" to the bisect, which treats it as "this group may + // contain the target". + const max = + rowGroupIndex + 1 < totalRowGroups + ? await this.readParquetRowGroupColumnFirstValue(parquetPath, columnName, rowGroupIndex + 1) + : null; + return { min, max }; + } + + /** + * First value of `columnName` in one row group — the only boundary value that can + * actually be read here, and the whole basis of {@link readParquetRowGroupColumnExtent}. + * + * **The last value cannot be read.** The obvious way to get it is + * `readParquetRowGroup(..., { offset: rowCount - 1, limit: 1 })`, and that is what + * this used to do — but the vendored parquet-wasm ignores `offset` on a row-group + * read and hands back the FIRST row again. Nothing failed; every row group simply + * reported `max === min`, i.e. that it spanned a single value. + * + * On a sorted column that is not a small error, it is a systematic one: the bisect + * asks "first row group whose max >= target", so an understated max moves the + * answer one group too far forward and the group actually CONTAINING the target is + * never read. On a Morton points artifact that is missing row groups per viewport + * query — holes in the render, in Z-order-shaped bands. + * + * The sort order gives the bound for free: the file is sorted on this column, so + * row group i's values all lie at or below row group i+1's first value. Using that + * as the upper bound is *conservative* — equal values spanning a boundary keep both + * groups in the range — and it costs one read per row group instead of two. + */ + private async readParquetRowGroupColumnFirstValue( + parquetPath: string, + columnName: string, + rowGroupIndex: number + ): Promise { + const cacheKey = `${parquetPath}::${rowGroupIndex}::${columnName}::first`; + const cached = this.rowGroupColumnFirstValueCache.get(cacheKey); + if (cached) { + return cached; } - const extent = { - min: parquetColumnValueToNumber(minColumn.get(0)), - max: maxValue, - }; - this.rowGroupColumnExtentCache.set(cacheKey, extent); - return extent; + const pending = (async () => { + const options: ParquetRowGroupReadOptions = { columns: [columnName], limit: 1 }; + const table = await this.loadParquetRowGroupByGroupIndex(parquetPath, rowGroupIndex, options); + const column = table?.getChild(columnName); + if (!column || column.length === 0) { + return null; + } + return parquetColumnValueToNumber(column.get(0)); + })(); + this.rowGroupColumnFirstValueCache.set(cacheKey, pending); + pending + .then((value) => { + if (value === null && this.rowGroupColumnFirstValueCache.get(cacheKey) === pending) { + this.rowGroupColumnFirstValueCache.delete(cacheKey); + } + }) + .catch(() => { + if (this.rowGroupColumnFirstValueCache.get(cacheKey) === pending) { + this.rowGroupColumnFirstValueCache.delete(cacheKey); + } + }); + return pending; } /** diff --git a/packages/core/src/parquetFooterStats.ts b/packages/core/src/parquetFooterStats.ts index 1cbb184b..a5866468 100644 --- a/packages/core/src/parquetFooterStats.ts +++ b/packages/core/src/parquetFooterStats.ts @@ -329,6 +329,29 @@ export function parseParquetFileMetaData(fileMetaDataBytes: Uint8Array): Parquet return new ThriftCompactReader(fileMetaDataBytes).readFileMetaData(); } +/** + * Decode a `Statistics` min/max for a column whose logical type is UNSIGNED. + * + * Parquet has no unsigned physical types: a `uint32` column is stored as INT32 with a + * UINT_32 logical annotation, so {@link decodeIntStat} would read the top half of the + * range as negative. `morton_code_2d` spans the full 32 bits by construction — the + * far corner of the slide is 0xFFFFFFFF — so the difference is not academic. + */ +export function decodeUnsignedIntStat( + bytes: Uint8Array | undefined, + physicalType: number | null +): number | null { + if (!bytes || bytes.length === 0) return null; + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); + if (physicalType === ParquetPhysicalType.INT32) { + return bytes.length >= 4 ? view.getUint32(0, true) : null; + } + if (physicalType === ParquetPhysicalType.INT64) { + return bytes.length >= 8 ? Number(view.getBigUint64(0, true)) : null; + } + return null; +} + /** Decode a `Statistics` min/max value for an integer physical type (little-endian). */ export function decodeIntStat( bytes: Uint8Array | undefined, diff --git a/packages/core/src/pointsLoadPlan.ts b/packages/core/src/pointsLoadPlan.ts new file mode 100644 index 00000000..1e981068 --- /dev/null +++ b/packages/core/src/pointsLoadPlan.ts @@ -0,0 +1,37 @@ +import type { PointsTilingMetadata } from './pointsTiling.js'; + +/** + * Which points loads to schedule at the start of a load pass. + * + * Moved here from `@spatialdata/layers` (D5 step 1): `PointsResolver.plan()` is now + * the caller that matters, and `core` cannot import from `layers`. The layers module + * re-exports it, so no consumer import moves. + */ +export interface PointsLoadPlanInput { + wantsOptimized: boolean; + metadataKnown: boolean; + tiledMetadata: PointsTilingMetadata | null | undefined; + hasPreloaded: boolean; + /** Known row count from parquet metadata, when available. */ + totalRows?: number; +} + +export interface PointsLoadPlan { + probeMetadata: boolean; + preloadFullTable: boolean; +} + +/** + * Decide which points loads to schedule at the start of a load pass. + * + * The two booleans are independent on purpose. Until the probe answers, we schedule + * NEITHER: preloading a table we are about to tile wastes the whole read, and probing + * a table we already hold resident answers a question nobody asked. + */ +export function planPointsLoads(input: PointsLoadPlanInput): PointsLoadPlan { + const { wantsOptimized, metadataKnown, tiledMetadata, hasPreloaded } = input; + const probeMetadata = wantsOptimized && !metadataKnown; + const preloadFullTable = + !hasPreloaded && (!wantsOptimized || (metadataKnown && tiledMetadata === null)); + return { probeMetadata, preloadFullTable }; +} diff --git a/packages/core/src/pointsLoader.ts b/packages/core/src/pointsLoader.ts index 144f5222..e77f94f1 100644 --- a/packages/core/src/pointsLoader.ts +++ b/packages/core/src/pointsLoader.ts @@ -20,6 +20,14 @@ export interface PointsLoaderCapabilities { bounds?: SpatialBounds; supportsViewportTiles: boolean; supportsFeatureCodes?: boolean; + /** Rows in the whole artifact, when the loader can know without reading it. */ + totalRows?: number; + /** + * Rows per row group — the granularity every viewport read is rounded up to. + * With {@link totalRows} and {@link bounds} it is what sizes the tile grid: a tile + * finer than one row group's footprint fetches the same bytes in more requests. + */ + maxRowsPerGroup?: number; } export interface ColumnarNdarrayPointsBatch { @@ -115,6 +123,8 @@ export function createMortonTiledPointsLoader( bounds: metadata.bounds, supportsViewportTiles: true, supportsFeatureCodes: Boolean(metadata.featureKey), + totalRows: metadata.totalRows, + maxRowsPerGroup: metadata.maxRowsPerGroup, }; return { diff --git a/packages/core/src/pointsTileGrid.ts b/packages/core/src/pointsTileGrid.ts new file mode 100644 index 00000000..90ec7b12 --- /dev/null +++ b/packages/core/src/pointsTileGrid.ts @@ -0,0 +1,139 @@ +import type { SpatialBounds } from './pointsTiling.js'; + +/** + * deck's tile size, in the units its traversal indexes with. Everything below is + * expressed against it: `getIdentityTileIndices` computes + * `scale = 2^z * 512 / tileSize`, so a tile spans `tileSize / 2^z` **element-local** + * units. Fixing it at 512 leaves `z` as the only free variable. + */ +export const POINTS_TILE_SIZE = 512; + +/** + * Rows a single tile may be expected to hold at the coarsest level. + * + * A tile is one request, decoded and uploaded whole, and deck shows nothing for it + * until it lands — so an over-large tile trades progressive filling for one long + * stall. 400k is roughly a tenth of the default resident cap, which keeps a coarse + * tile comparable to a chunk of preload rather than to the whole element. + */ +export const POINTS_TILE_TARGET_ROWS = 400_000; + +/** Tiles the deck cache may retain, when the artifact gives us nothing to derive from. */ +const FALLBACK_CACHE_TILES = 24; + +export interface PointsTileGrid { + tileSize: number; + minZoom: number; + maxZoom: number; + zoomOffset: number; + maxRequests: number; + maxCacheSize: number; + /** Rows a coarsest-level tile is expected to hold — the unit of the cache budget. */ + estimatedRowsPerTile: number; + /** Worst-case rows the tile cache can hold. Accounting, not a limit deck enforces. */ + cacheRowBudget: number; +} + +export interface PointsTileGridInput { + bounds: SpatialBounds; + totalRows: number; + /** Rows per row group — the granularity every read is rounded up to. */ + maxRowsPerGroup: number; + /** Uniform scale of the layer's model matrix: local units -> world units. */ + modelMatrixScale: number; + /** Row budget for the tile cache; defaults to the resident points memory cap. */ + cacheRowBudget?: number; +} + +/** `z` at which a tile spans `span` local units. */ +function zoomForSpan(span: number): number { + return Math.log2(POINTS_TILE_SIZE / span); +} + +/** + * Choose the tile grid for a Morton artifact. + * + * The grid used to be one fixed level (`minZoom: -1, maxZoom: -1`), so every tile was + * 1024 local units at every zoom: zooming in read a 1024-unit tile to look at 50 + * units of it, and the number 1024 came from deck's defaults rather than from the + * data. Both ends are now derived, and both ends are real constraints: + * + * - **Coarsest** — a tile should hold at most {@link POINTS_TILE_TARGET_ROWS} rows, so + * one request stays a fraction of the layer rather than most of it. + * - **Finest** — a tile should stay at least as large as one row group's footprint. + * Reads are rounded up to whole row groups, so below that size each tile still + * fetches a whole group while four tiles cover what one used to: the same bytes, + * more requests, more duplicate decoding. This is the same argument as + * `MORTON_ZCOVER_MAX_DEPTH` — resolution finer than the storage granularity is + * pure cost. + * + * Both come from one number, the point density `rows / area`: a row group's footprint + * is `sqrt(maxRowsPerGroup / density)` and the coarse limit `sqrt(target / density)`. + * On a 12.1M-point Xenium element (10871 x 3627 um, 50k-row groups) that is a floor of + * ~402 um and a ceiling of ~1139 um — a narrow range, and worth knowing: the fixed + * 1024 was accidentally near-optimal *for this artifact*, and would not be for one an + * order of magnitude smaller or denser. + * + * `zoomOffset` couples `z` to the viewport. deck picks `z = ceil(viewport.zoom + + * zoomOffset)` from a zoom expressed in WORLD units, while tile spans are in LOCAL + * units; the model matrix is the difference, so `log2(scale)` is exactly the term that + * makes a tile land at 256-512 screen pixels instead of at whatever the transform + * happened to imply. + */ +export function mortonTileGrid(input: PointsTileGridInput): PointsTileGrid { + const { bounds, totalRows, maxRowsPerGroup, modelMatrixScale } = input; + const width = Math.max(0, bounds.maxX - bounds.minX); + const height = Math.max(0, bounds.maxY - bounds.minY); + const area = width * height; + const density = area > 0 && totalRows > 0 ? totalRows / area : 0; + + // No density to reason from (an empty or degenerate artifact): keep the single + // level the grid had before, so this can only ever be an improvement. + if (density <= 0) { + return { + tileSize: POINTS_TILE_SIZE, + minZoom: -1, + maxZoom: -1, + zoomOffset: 0, + maxRequests: 6, + maxCacheSize: FALLBACK_CACHE_TILES, + estimatedRowsPerTile: 0, + cacheRowBudget: 0, + }; + } + + const rowGroupSpan = Math.sqrt(Math.max(1, maxRowsPerGroup) / density); + const coarseSpan = Math.sqrt(POINTS_TILE_TARGET_ROWS / density); + + // Smaller span => larger z. floor/ceil each round TOWARDS the allowed span. + const maxZoom = Math.floor(zoomForSpan(rowGroupSpan)); + let minZoom = Math.ceil(zoomForSpan(Math.max(coarseSpan, rowGroupSpan))); + if (minZoom > maxZoom) { + // A row group already covers more than the coarse budget (a sparse artifact, or + // very large row groups). One level, at the row-group footprint. + minZoom = maxZoom; + } + + const coarsestSpan = POINTS_TILE_SIZE / 2 ** minZoom; + const estimatedRowsPerTile = Math.round(density * coarsestSpan * coarsestSpan); + const budget = input.cacheRowBudget ?? 0; + const maxCacheSize = + budget > 0 && estimatedRowsPerTile > 0 + ? Math.min(512, Math.max(16, Math.round(budget / estimatedRowsPerTile))) + : FALLBACK_CACHE_TILES; + + return { + tileSize: POINTS_TILE_SIZE, + minZoom, + maxZoom, + // A non-finite or non-positive scale would poison every tile index. + zoomOffset: + Number.isFinite(modelMatrixScale) && modelMatrixScale > 0 ? Math.log2(modelMatrixScale) : 0, + // Each request is a row-group range read plus a decode; deck's default of 6 is a + // reasonable place to sit, but it is now a decision rather than an inheritance. + maxRequests: 6, + maxCacheSize, + estimatedRowsPerTile, + cacheRowBudget: maxCacheSize * estimatedRowsPerTile, + }; +} diff --git a/packages/core/src/pointsTiling.ts b/packages/core/src/pointsTiling.ts index 06551578..501ae428 100644 --- a/packages/core/src/pointsTiling.ts +++ b/packages/core/src/pointsTiling.ts @@ -8,6 +8,34 @@ export const MORTON_CODE_VALUE_MAX = 2 ** MORTON_CODE_BITS_PER_AXIS - 1; export type SpatialBounds = AxisAlignedBounds; +/** Whether a points layer probes for a Morton index before preloading (D5). */ +export type PointsTilingMode = 'auto' | 'off'; + +/** + * `'auto'` since D5 step 7. + * + * On a Morton element the tiled path is not merely an alternative, it is the better + * one: the preload it replaces keeps the first `cap` rows in FILE order, and file + * order on a Morton artifact is a prefix of the Z-curve — a spatially skewed chunk of + * the slide, not a sample of it. Tiles read what you are looking at instead. + * + * On anything else the probe answers `null` from the schema alone and costs nothing + * beyond footer metadata the preload path reads regardless; on a malformed Morton + * artifact the guards in `getPointsTilingMetadata` decline it, warn, and fall through + * to the preload. + */ +export const DEFAULT_POINTS_TILING: PointsTilingMode = 'auto'; + +/** + * Resolve the tiling mode, default included. Call this instead of comparing to + * `'auto'`: the default has to mean the same thing to the resolver deciding what to + * load, the hook deciding what to render, and the panel drawing the checkbox, and + * three literal comparisons is how those drift apart. + */ +export function pointsTilingEnabled(mode: PointsTilingMode | undefined): boolean { + return (mode ?? DEFAULT_POINTS_TILING) === 'auto'; +} + export interface PointsFeatureEntry { code: number; name: string; @@ -42,6 +70,13 @@ export interface PointsTilingMetadata { rowGroupRowCounts?: number[]; supportsRowGroupRangeReads: boolean; bounds?: SpatialBounds; + /** + * Per-row-group `[min, max]` of the Morton column, read from footer statistics + * during the probe. Present means viewport queries can select row groups from + * memory ({@link selectMortonRowGroups}); absent means falling back to the bisect, + * which pays ~2MB per step to recover the same two numbers. + */ + rowGroupMortonExtents?: MortonRowGroupExtent[]; } export type PointsInBoundsResponse = PointsColumnarData & { @@ -75,6 +110,161 @@ export function origCoordToNormCoord(x: number, y: number, bbox: SpatialBounds): ]; } +/** Spread the low 16 bits of `n` into the even bit positions of a 32-bit lane. */ +function spreadBits(n: number): number { + let x = n & 0xffff; + x = (x | (x << 8)) & 0x00ff00ff; + x = (x | (x << 4)) & 0x0f0f0f0f; + x = (x | (x << 2)) & 0x33333333; + x = (x | (x << 1)) & 0x55555555; + return x; +} + +/** + * Interleave a quantised (x, y) pair into the code stored in `morton_code_2d`: + * x in the even bits, y in the odd ones. + * + * `+ ... * 2` rather than `| ... << 1` on purpose — the y term reaches bit 31, and + * JS bitwise operators would hand back a negative int32 for the top of the domain. + * + * Must agree with {@link zcoverRectangle}'s quadrant order (child +1 is x-high, +2 is + * y-high) and with the writer's `morton_code_2d`; a disagreement here would make + * every interval query wrong, so {@link mortonBoundsAgreeWithCodes} checks it against + * real rows rather than trusting all three to stay in step. + */ +export function mortonCode2dFromNormCoord(nx: number, ny: number): number { + return spreadBits(nx) + spreadBits(ny) * 2; +} + +/** The code a point *should* carry if `bbox` is the quantisation domain. */ +export function mortonCode2dForPoint(x: number, y: number, bbox: SpatialBounds): number { + const [nx, ny] = origCoordToNormCoord(x, y, bbox); + return mortonCode2dFromNormCoord(nx, ny); +} + +/** + * Does `bounds` describe the domain the stored Morton codes were actually quantised + * against? Recomputes the code for a sample of real rows and counts agreement. + * + * The sentinel rows are a **claim the artifact makes about itself**, and nothing else + * in the file forces them to be true. When they are wrong every derived answer is + * wrong in the same silent, subtractive direction: the tile grid is clipped to the + * bogus box so whole regions are never even requested, and `mortonIntervalsForBounds` + * normalises viewports against it and selects the wrong row groups. Points are never + * misplaced — the reader re-filters to the query bounds — so the only visible symptom + * is that some of the map is missing. + * + * Measured on a real 12.1M-point Xenium artifact, the signal is not marginal: a sound + * element matched 320/320 sampled rows, one with a stale sentinel box matched 0/320. + * A handful of misses is normal — a coordinate landing exactly on a cell boundary can + * floor either way — hence a majority test rather than an exact one. + * + * Samples are spread evenly across the input: consecutive rows in a Morton-sorted row + * group share a long code prefix, so the first N would agree or disagree together. + */ +export function mortonBoundsAgreeWithCodes( + xs: ArrayLike, + ys: ArrayLike, + codes: ArrayLike, + bounds: SpatialBounds, + maxSamples = 64 +): { checked: number; matched: number } { + const rows = Math.min(xs.length, ys.length, codes.length); + if (rows === 0 || maxSamples <= 0) { + return { checked: 0, matched: 0 }; + } + const samples = Math.min(rows, maxSamples); + const stride = Math.max(1, Math.floor(rows / samples)); + let checked = 0; + let matched = 0; + for (let i = 0; i < rows && checked < samples; i += stride) { + const x = getNumericValue(xs[i]); + const y = getNumericValue(ys[i]); + const code = getNumericValue(codes[i]); + if (x === null || y === null || code === null) { + continue; + } + checked += 1; + if (mortonCode2dForPoint(x, y, bounds) === code) { + matched += 1; + } + } + return { checked, matched }; +} + +/** Inclusive `[min, max]` a row group's Morton column spans; `null` when unknown. */ +export type MortonRowGroupExtent = readonly [number, number] | null; + +/** + * Is the file actually Morton-**sorted**, row group by row group? + * + * The row-group bisect binary-searches this sequence, which is only meaningful if it + * is non-decreasing. A feature-primary artifact — sorted `(feature, morton)` — has a + * `morton_code_2d` column that restarts at every feature boundary, so each row group + * spans nearly the whole code range and the bisect lands somewhere arbitrary. What + * comes back is whichever feature blocks happened to live in the row groups it picked: + * a tile shows one or two genes and misses the rest. Measured on the permutations + * store, `transcripts_feature_then_morton` descends at 185 of its 244 row-group + * boundaries, while both morton-primary elements descend at none. + * + * Adjacent groups may share a boundary value, so the test is `min >= previous max`. + * A `null` extent is unknown rather than out of order: skip it and carry the last + * known maximum, so a column without statistics cannot fake a descent. + */ +export function mortonRowGroupExtentsAreSorted(extents: readonly MortonRowGroupExtent[]): boolean { + let previousMax: number | null = null; + for (const extent of extents) { + if (!extent) { + continue; + } + const [min, max] = extent; + if (previousMax !== null && min < previousMax) { + return false; + } + previousMax = previousMax === null ? max : Math.max(previousMax, max); + } + return true; +} + +/** + * Which row groups can hold a code inside any of `intervals`, from the in-memory + * index rather than a bisect over the file. + * + * The bisect this replaces reads the row group's BYTES to recover two boundary values + * — every column, ~2MB on a real transcripts artifact — and one viewport query runs + * `log2(rowGroups)` of them per interval. Building the index that way could fetch the + * whole 439MB file to recover ~4KB. The same numbers are in the parquet footer, so + * with {@link mortonRowGroupExtentsAreSorted}'s index in hand this costs nothing. + * + * It is also *stricter* than the bisect, which tested only `max` and assumed the + * groups tile the code space without gaps: this intersects both ends, so a row group + * whose range falls entirely between two intervals is skipped rather than swept in. + * + * A `null` extent means "no statistics for this group": include it, because the + * alternative is dropping rows for a reason that has nothing to do with the query. + */ +export function selectMortonRowGroups( + extents: readonly MortonRowGroupExtent[], + intervals: ReadonlyArray +): number[] { + const selected = new Set(); + for (let i = 0; i < extents.length; i++) { + const extent = extents[i]; + if (!extent) { + selected.add(i); + continue; + } + const [min, max] = extent; + for (const [start, end] of intervals) { + if (max >= start && min <= end) { + selected.add(i); + break; + } + } + } + return [...selected].sort((a, b) => a - b); +} + function intersects( ax0: number, ay0: number, @@ -126,14 +316,49 @@ export function mergeAdjacentIntervals( return merged; } +/** + * How far {@link zcoverRectangle} subdivides before emitting a whole cell. + * + * The cover exists to pick **row groups**, and a row group holds tens of thousands + * of points — so resolving the rectangle to individual quantised cells buys nothing + * and costs a great deal. Recursing to the full 16 bits produced **38,014 intervals** + * for one viewport-sized rectangle on a real 12.1M-point Xenium artifact (245 row + * groups), each interval driving two row-group bisects. + * + * Measured on that artifact, over a viewport tile, the whole slide, and a zoomed-in + * box — at this depth the selected row groups are **identical** to the full-depth + * cover (not merely a superset), while the interval count collapses: + * + * | depth | intervals (viewport tile) | row groups | + * |-------|---------------------------|------------| + * | 16 | 38,014 | 92 | + * | 10 | 521 | 92 | + * | 8 | 138 | 92 | + * + * 10 leaves headroom: the cover stays finer than the row-group granularity for files + * with up to ~1M row groups (4^10 cells), where 8 would start over-fetching. + */ +export const MORTON_ZCOVER_MAX_DEPTH = 10; + +/** + * Morton-code intervals covering a rectangle in quantised (x, y) space. + * + * Stopping early at {@link maxDepth} makes a cell **coarser than the rectangle** — + * the interval then covers some codes outside it. That is safe in both directions: + * the cover is still complete (no code inside the rectangle is dropped), and the + * extra codes only ever widen the row-group set, whose rows are filtered against the + * exact bounds after the read. + */ export function zcoverRectangle( rx0: number, ry0: number, rx1: number, ry1: number, - bits = MORTON_CODE_BITS_PER_AXIS + bits = MORTON_CODE_BITS_PER_AXIS, + maxDepth = MORTON_ZCOVER_MAX_DEPTH ): Array<[number, number]> { const maxCoord = 2 ** bits - 1; + const depthLimit = Math.max(0, Math.min(bits, maxDepth)); const x0 = Math.max(0, Math.min(maxCoord, Math.min(rx0, rx1))); const x1 = Math.max(0, Math.min(maxCoord, Math.max(rx0, rx1))); const y0 = Math.max(0, Math.min(maxCoord, Math.min(ry0, ry1))); @@ -153,7 +378,7 @@ export function zcoverRectangle( if (!intersects(xmin, ymin, xmax, ymax, x0, y0, x1, y1)) { continue; } - if (contained(xmin, ymin, xmax, ymax, x0, y0, x1, y1) || level === bits) { + if (contained(xmin, ymin, xmax, ymax, x0, y0, x1, y1) || level >= depthLimit) { intervals.push(cellRange(prefix, level, bits)); continue; } diff --git a/packages/core/src/spatialViewFit.ts b/packages/core/src/spatialViewFit.ts index b544698e..2bb7e5a9 100644 --- a/packages/core/src/spatialViewFit.ts +++ b/packages/core/src/spatialViewFit.ts @@ -187,6 +187,41 @@ export function boundsFromFlatPolygonPositions( return any ? { minX, minY, maxX, maxY } : null; } +/** + * World bounds for element-space bounds that are already known — the Morton tiling + * metadata's extent, where there is no geometry in memory to measure. + * + * All four corners are transformed, not just min/max: a rotating or shearing + * transform maps the min corner somewhere that is no longer the minimum, and taking + * two corners would silently frame the wrong box. + */ +export function transformAxisAlignedBounds( + bounds: AxisAlignedBounds, + modelMatrix: Matrix4 +): AxisAlignedBounds | null { + const corners: [number, number, number][] = [ + [bounds.minX, bounds.minY, 0], + [bounds.maxX, bounds.minY, 0], + [bounds.maxX, bounds.maxY, 0], + [bounds.minX, bounds.maxY, 0], + ]; + let minX = Number.POSITIVE_INFINITY; + let minY = Number.POSITIVE_INFINITY; + let maxX = Number.NEGATIVE_INFINITY; + let maxY = Number.NEGATIVE_INFINITY; + for (const corner of corners) { + const transformed = modelMatrix.transformAsPoint(corner); + if (!Number.isFinite(transformed[0]) || !Number.isFinite(transformed[1])) { + return null; + } + minX = Math.min(minX, transformed[0]); + minY = Math.min(minY, transformed[1]); + maxX = Math.max(maxX, transformed[0]); + maxY = Math.max(maxY, transformed[1]); + } + return { minX, minY, maxX, maxY }; +} + /** * Axis-aligned bounds for circle shapes (center + radius in store coordinates). */ diff --git a/packages/core/src/workers/points-worker.ts b/packages/core/src/workers/points-worker.ts index b65e8c5b..e30c5709 100644 --- a/packages/core/src/workers/points-worker.ts +++ b/packages/core/src/workers/points-worker.ts @@ -488,6 +488,11 @@ async function handleScanMortonRowGroupsInBounds( const xs = new Float32PointBuffer(); const ys = new Float32PointBuffer(); const zs = new Float32PointBuffer(); + // Collect per-point codes whenever the element has a code column — including when + // no filter is active, which is precisely the "all features" view that colouring + // needs. Gating this on `request.featureCodes` (the filter) would leave the + // default view flat. + const codes = request.featureCodeColumnName ? new Int32PointBuffer() : undefined; for (const chunk of request.rowGroups) { const table = tableFromIPC( parquetModule @@ -507,11 +512,13 @@ async function handleScanMortonRowGroupsInBounds( xs, ys, zs, + ...(codes ? { codes } : {}), }); } const outX = xs.toArray(); const outY = ys.toArray(); const outZ = hasZ ? zs.toArray() : undefined; + const outCodes = codes?.toArray(); const shape = outZ ? [3, outX.length] : [2, outX.length]; return { ok: true, @@ -521,6 +528,10 @@ async function handleScanMortonRowGroupsInBounds( xs: outX, ys: outY, ...(outZ ? { zs: outZ } : {}), + // Short codes are unusable, not partially usable: the remaining points would + // read code 0 — a VALID feature — and be confidently mis-coloured. Ship them + // only when there is exactly one per point. + ...(outCodes && outCodes.length === outX.length ? { featureCodes: outCodes } : {}), }, }; } diff --git a/packages/core/src/workers/pointsWorkerScan.ts b/packages/core/src/workers/pointsWorkerScan.ts index f945e9e1..3562338a 100644 --- a/packages/core/src/workers/pointsWorkerScan.ts +++ b/packages/core/src/workers/pointsWorkerScan.ts @@ -422,6 +422,18 @@ export function scanMortonTableInBounds(input: { xs: Float32PointBuffer; ys: Float32PointBuffer; zs: Float32PointBuffer; + /** + * Per-point feature codes for the matched rows, appended in lockstep with the + * geometry (D5 step 3). + * + * Optional, but supplying it is what lets a tiled layer colour by feature at all: + * the scan already READS this column to filter on it, and used to throw the value + * away, so a tile arrived as bare coordinates and the render fell back to a flat + * colour. Lockstep is the whole contract — index i of this buffer names the feature + * of point i in {@link xs}/{@link ys} — so every `continue` above a push must skip + * all four buffers together. + */ + codes?: Int32PointBuffer; }): void { const allowedFeatureCodes = featureCodeAllowSet(input.featureCodes); const filterByFeature = allowedFeatureCodes !== null; @@ -453,6 +465,11 @@ export function scanMortonTableInBounds(input: { if (filterByFeature && !featureCodeValues) { return; } + // Codes are collected whenever the caller asked for them AND the column is there. + // Not collecting silently is fine — the render falls back to a flat colour — but + // collecting a SHORT array would be worse than none: it would misalign against the + // geometry and confidently mis-colour every point after the first gap. + const collectCodes = input.codes !== undefined && featureCodeValues !== null; // Hoisted: `Table.numRows` is not a field but // `data.reduce((n, d) => n + d.length, 0)` — a closure allocation and a walk of // every chunk. As a loop CONDITION that ran per row. See `scanTableByFeatureCodes`. @@ -464,6 +481,9 @@ export function scanMortonTableInBounds(input: { if (zValues) { input.zs.reserve(numRows); } + if (input.codes) { + input.codes.reserve(numRows); + } for (let rowIndex = 0; rowIndex < numRows; rowIndex += 1) { // Sentinels only ever occupy the first rows of the first row group, so this // stays on the (rare) boxed read rather than materialising the whole column. @@ -502,6 +522,14 @@ export function scanMortonTableInBounds(input: { const z = zValues[rowIndex]; input.zs.push(Number.isFinite(z) ? z : 0); } + if (collectCodes) { + const code = (featureCodeValues as ArrayLike)[rowIndex]; + // A non-finite code is a real row with an unknown feature: keep the point and + // record -1, the same "no feature" sentinel the shader's `featureCode >= 0` + // guard already understands. Dropping the point instead would put a hole in + // the geometry to express a gap in the colour. + (input.codes as Int32PointBuffer).push(Number.isFinite(code) ? code : -1); + } } } diff --git a/packages/core/tests/mortonPointsTiling.spec.ts b/packages/core/tests/mortonPointsTiling.spec.ts index 684c8546..0e179a9f 100644 --- a/packages/core/tests/mortonPointsTiling.spec.ts +++ b/packages/core/tests/mortonPointsTiling.spec.ts @@ -106,6 +106,131 @@ PY`, ); } +/** + * A well-formed Morton artifact in every respect EXCEPT that its sentinel rows record + * a sub-box of the domain the codes were quantised against — the exact shape of the + * stale `index-permutations` fixture. The codes are internally consistent, the row + * groups are sorted, the sentinel prefix is the right size: nothing but recomputing a + * code from x/y can tell that the box is a lie. + */ +async function writeMismatchedSentinelMortonPointsZarr(root: string) { + const elementDir = join(root, 'points', 'transcripts'); + await mkdir(elementDir, { recursive: true }); + await writeFile(join(root, 'zarr.json'), JSON.stringify({ zarr_format: 3, node_type: 'group' })); + await writeFile( + join(elementDir, 'zarr.json'), + JSON.stringify({ + attributes: { + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }, + zarr_format: 3, + node_type: 'group', + }) + ); + + execSync( + `uv run python - <<'PY' +import numpy as np +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +from spatialdata_js_util.points import morton_code_2d, _norm_series_to_uint + +root = Path(${JSON.stringify(elementDir)}) +rows = 400 +rng = np.random.default_rng(0) +x = rng.uniform(0.0, 1000.0, rows) +y = rng.uniform(0.0, 1000.0, rows) +df = pd.DataFrame({"x": x, "y": y}) +# Codes quantised against the TRUE extent, as a real writer would. +df["morton_code_2d"] = morton_code_2d( + _norm_series_to_uint(df["x"], float(x.min()), float(x.max())), + _norm_series_to_uint(df["y"], float(y.min()), float(y.max())), +) +df["feature_name_codes"] = np.arange(rows) % 3 +df["feature_name"] = pd.Categorical(["gene_a", "gene_b", "gene_c"] * (rows // 3) + ["gene_a"]) +df = df.sort_values("morton_code_2d", kind="mergesort").reset_index(drop=True) + +# Sentinels claiming a sub-box: a quarter of the extent, in the middle. +sentinel = pd.DataFrame( + { + "x": [250.0, 750.0, 400.0, 600.0], + "y": [400.0, 600.0, 250.0, 750.0], + "morton_code_2d": np.zeros(4, dtype=np.uint32), + "feature_name_codes": np.zeros(4, dtype=np.int32), + "feature_name": pd.Categorical(["gene_a"] * 4, categories=["gene_a", "gene_b", "gene_c"]), + } +) +combined = pd.concat([sentinel, df], ignore_index=True) +table = pa.Table.from_pandas(combined, preserve_index=False) +writer = pq.ParquetWriter(root / "points.parquet", table.schema, compression="zstd") +try: + writer.write_table(table.slice(0, 4), row_group_size=4) + writer.write_table(table.slice(4), row_group_size=200) +finally: + writer.close() +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); +} + +/** + * A feature-primary artifact: sorted `(feature, morton)`, written by the real writer. + * Every column the probe looks for is present, the sentinel box is correct, the codes + * are correct — only the ORDER is wrong for a bisect, and nothing in the file says so. + */ +async function writeFeaturePrimaryMortonPointsZarr(root: string) { + const elementDir = join(root, 'points', 'transcripts'); + await mkdir(elementDir, { recursive: true }); + await writeFile(join(root, 'zarr.json'), JSON.stringify({ zarr_format: 3, node_type: 'group' })); + await writeFile( + join(elementDir, 'zarr.json'), + JSON.stringify({ + attributes: { + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { feature_key: 'feature_name', version: '0.2' }, + }, + zarr_format: 3, + node_type: 'group', + }) + ); + + execSync( + `uv run python - <<'PY' +import numpy as np +import pandas as pd + +from spatialdata_js_util.points import write_morton_points_parquet + +rng = np.random.default_rng(1) +rows = 900 +df = pd.DataFrame( + { + "x": rng.uniform(0.0, 1000.0, rows), + "y": rng.uniform(0.0, 1000.0, rows), + "feature_name": pd.Categorical(rng.choice(["gene_a", "gene_b", "gene_c"], rows)), + } +) +write_morton_points_parquet( + df, + ${JSON.stringify(join(elementDir, 'points.parquet'))}, + feature_key="feature_name", + sort_order=["feature_name_codes", "morton_code_2d"], + row_group_size=100, +) +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); +} + function createStore(files: Record) { let getRangeCalls = 0; let getCalls = 0; @@ -179,6 +304,137 @@ describe('Morton points tiling (canonical parquet)', () => { expect(mockStore.getRangeCalls()).toBeGreaterThan(0); }); + /** + * The bisect index must describe each row group's FULL span. + * + * `readParquetRowGroupColumnExtent` used to read the last value with + * `readParquetRowGroup(..., { offset: rowCount - 1, limit: 1 })`, which the vendored + * parquet-wasm ignores — it returned the first row again, so every row group + * reported `max === min`. Nothing failed; the bisect just answered "first row group + * whose max >= target" one group too late and never read the group that actually + * CONTAINED the target. On a real 12.1M-point artifact that dropped 11 of 92 row + * groups from a viewport query — 188k points, rendered as Z-order-shaped holes. + * + * So this asserts an exact count, not "more than zero": under-selection is silent + * by construction, and only a total can see it. + */ + it('returns every point inside the bounds, not just those in some row groups', async () => { + const full = await source.loadPoints('points/transcripts'); + const xs = full.data[0]; + const ys = full.data[1]; + const minX = Math.min(...xs); + const minY = Math.min(...ys); + // A rectangle deliberately spanning several row groups' Morton ranges. + const bounds = { minX: minX + 10, maxX: minX + 80, minY: minY + 10, maxY: minY + 80 }; + + let expected = 0; + for (let i = 0; i < xs.length; i += 1) { + if ( + xs[i] >= bounds.minX && + xs[i] <= bounds.maxX && + ys[i] >= bounds.minY && + ys[i] <= bounds.maxY + ) { + expected += 1; + } + } + expect(expected).toBeGreaterThan(0); + + const result = await source.loadPointsInBounds('points/transcripts', { bounds }); + + expect(result.loadMode).toBe('row-groups'); + expect(result.shape[1]).toBe(expected); + }); + + /** + * Per-point feature codes ride the tile batch (D5 step 3). + * + * Without them a tiled layer has nothing to colour by, so it drew flat while the + * preloaded path drew per-feature — the same element looking like two different + * datasets depending on a checkbox. The scan already READ this column to filter on + * it and threw the value away. + * + * Alignment is the contract worth pinning: index i of the codes names the feature + * of point i in the geometry, so the count must match exactly and every code must + * be one the catalog knows. + */ + it('returns a feature code per point, aligned with the geometry', async () => { + const full = await source.loadPoints('points/transcripts'); + const xs = full.data[0]; + const ys = full.data[1]; + const minX = Math.min(...xs); + const minY = Math.min(...ys); + const bounds = { minX: minX + 10, maxX: minX + 80, minY: minY + 10, maxY: minY + 80 }; + + const result = await source.loadPointsInBounds('points/transcripts', { bounds }); + const pointCount = result.shape[1] ?? 0; + expect(pointCount).toBeGreaterThan(0); + + expect(result.featureCodes).toBeDefined(); + expect(result.featureCodes?.length).toBe(pointCount); + + // The fixture has three genes, so every code is a real catalog entry — never the + // -1 "unknown feature" sentinel, and never a stray 0 left by a short buffer. + const catalog = await source.listPointsFeaturesWithCounts('points/transcripts'); + const known = new Set((catalog?.entries ?? []).map((entry) => entry.code)); + expect(known.size).toBeGreaterThan(0); + for (let i = 0; i < pointCount; i += 1) { + expect(known.has(result.featureCodes?.[i] as number)).toBe(true); + } + }); + + it('still returns codes when a feature filter is active, for the filtered rows only', async () => { + const catalog = await source.listPointsFeaturesWithCounts('points/transcripts'); + const wanted = catalog?.entries[0]?.code; + expect(wanted).toBeDefined(); + + const full = await source.loadPoints('points/transcripts'); + const minX = Math.min(...full.data[0]); + const minY = Math.min(...full.data[1]); + const bounds = { minX: minX + 10, maxX: minX + 80, minY: minY + 10, maxY: minY + 80 }; + + const result = await source.loadPointsInBounds('points/transcripts', { + bounds, + featureCodes: [wanted as number], + }); + const pointCount = result.shape[1] ?? 0; + + expect(result.featureCodes?.length).toBe(pointCount); + for (let i = 0; i < pointCount; i += 1) { + expect(result.featureCodes?.[i]).toBe(wanted); + } + }); + + // The bisect index is built lazily, under exactly the load that duplicates it: + // every viewport tile bisects concurrently over the same row groups. Caching only + // the settled value dedups nothing while a read is in flight, so each tile used to + // start its own full row-group fetch for an entry the others were already fetching. + it('dedups concurrent extent probes for the same row group', async () => { + const parquetPath = 'points/transcripts/points.parquet'; + const freshSource = () => + new SpatialDataPointsSource({ store: mockStore.store, fileType: '.zarr' }); + + const single = freshSource(); + mockStore.resetCalls(); + const expected = await single.loadParquetRowGroupColumnExtent(parquetPath, 'morton_code_2d', 1); + const singleCallCount = mockStore.getRangeCalls(); + expect(singleCallCount).toBeGreaterThan(0); + + const concurrent = freshSource(); + mockStore.resetCalls(); + const results = await Promise.all( + Array.from({ length: 8 }, () => + concurrent.loadParquetRowGroupColumnExtent(parquetPath, 'morton_code_2d', 1) + ) + ); + + // Eight callers, one read. + expect(mockStore.getRangeCalls()).toBe(singleCallCount); + for (const result of results) { + expect(result).toEqual(expected); + } + }); + it('loads a bounded viewport without returning the full table', async () => { const full = await source.loadPoints('points/transcripts'); const xs = full.data[0]; @@ -274,4 +530,75 @@ describe('Morton points tiling (canonical parquet)', () => { execSync(`rm -rf ${JSON.stringify(badFixtureRoot)}`, { stdio: 'pipe' }); } }); + + /** + * The sentinel box is a claim the artifact makes about itself. Believing a wrong one + * does not fail — it silently clips the tile grid and mis-maps every viewport to row + * groups, so parts of the map are never even requested. Refuse to tile instead. + */ + it('does not enable morton tiling when the sentinel bbox is not the code domain', async () => { + const badFixtureRoot = await mkdtemp(join(tmpdir(), 'mismatched-morton-points-')); + const warn = vi.spyOn(console, 'warn').mockImplementation(() => {}); + try { + await writeMismatchedSentinelMortonPointsZarr(badFixtureRoot); + const badStore = createStore({ + 'points/transcripts/points.parquet': new Uint8Array( + await readFile(join(badFixtureRoot, 'points/transcripts/points.parquet')) + ), + 'points/transcripts/zarr.json': new Uint8Array( + await readFile(join(badFixtureRoot, 'points/transcripts/zarr.json')) + ), + }); + const badSource = new SpatialDataPointsSource({ store: badStore.store, fileType: '.zarr' }); + + const metadata = await badSource.getPointsTilingMetadata('points/transcripts'); + + // Same degradation as every other unusable artifact: the resolver's probe gate + // reads this pair as "not tileable" and falls through to the capped preload. + expect(metadata?.supportsRowGroupRangeReads).toBe(false); + expect(metadata?.bounds).toBeUndefined(); + expect(warn).toHaveBeenCalledWith(expect.stringContaining('sentinel bounding box')); + } finally { + warn.mockRestore(); + execSync(`rm -rf ${JSON.stringify(badFixtureRoot)}`, { stdio: 'pipe' }); + } + }); + + /** + * The bisect binary-searches the row-group Morton index, which only means anything + * if it ascends. On a feature-primary file it does not, and the search lands + * arbitrarily: a tile comes back holding whichever feature blocks happened to be in + * the row groups it picked, and missing the rest. + */ + it('does not enable morton tiling on a feature-primary artifact', async () => { + const featureFirstRoot = await mkdtemp(join(tmpdir(), 'feature-primary-points-')); + const warn = vi.spyOn(console, 'warn').mockImplementation(() => {}); + try { + await writeFeaturePrimaryMortonPointsZarr(featureFirstRoot); + const featureFirstStore = createStore({ + 'points/transcripts/points.parquet': new Uint8Array( + await readFile(join(featureFirstRoot, 'points/transcripts/points.parquet')) + ), + 'points/transcripts/zarr.json': new Uint8Array( + await readFile(join(featureFirstRoot, 'points/transcripts/zarr.json')) + ), + }); + const source = new SpatialDataPointsSource({ + store: featureFirstStore.store, + fileType: '.zarr', + }); + + const metadata = await source.getPointsTilingMetadata('points/transcripts'); + + expect(metadata?.supportsRowGroupRangeReads).toBe(false); + expect(warn).toHaveBeenCalledWith(expect.stringContaining('not sorted across row groups')); + // No bounds either: the sort verdict short-circuits the sampling read that would + // verify the sentinel box, and an unverified box is exactly what we stopped + // reporting. Nothing consumes bounds without supportsRowGroupRangeReads anyway. + expect(metadata?.bounds).toBeUndefined(); + } finally { + warn.mockRestore(); + execSync(`rm -rf ${JSON.stringify(featureFirstRoot)}`, { stdio: 'pipe' }); + } + }); }); diff --git a/packages/core/tests/pointsFeatureTallySentinels.spec.ts b/packages/core/tests/pointsFeatureTallySentinels.spec.ts index e096ae9e..0c137556 100644 --- a/packages/core/tests/pointsFeatureTallySentinels.spec.ts +++ b/packages/core/tests/pointsFeatureTallySentinels.spec.ts @@ -38,7 +38,7 @@ function columns(dictionaryEncoded: boolean) { [MORTON_CODE_2D_COLUMN]: Int32Array.from(MORTON) as never, }); return { - name: table.getChild(FEATURE_KEY)as never, + name: table.getChild(FEATURE_KEY) as never, morton: table.getChild(MORTON_CODE_2D_COLUMN) as never, rows: table.numRows, }; diff --git a/packages/core/tests/pointsResolver.spec.ts b/packages/core/tests/pointsResolver.spec.ts index 985d989f..35fba8f4 100644 --- a/packages/core/tests/pointsResolver.spec.ts +++ b/packages/core/tests/pointsResolver.spec.ts @@ -9,6 +9,12 @@ import { } from '../src/engine/index.js'; import type { PointsElement } from '../src/models/index.js'; import type { PointsLoadResult } from '../src/pointsLoadOptions.js'; +import { + DEFAULT_POINTS_TILING, + MORTON_CODE_2D_COLUMN, + type PointsTilingMetadata, + pointsTilingEnabled, +} from '../src/pointsTiling.js'; /** * The points Resource Resolver, driven headless. @@ -48,6 +54,15 @@ function element(over: Partial> = {}) { } as unknown as PointsElement; } +/** + * Default config for these tests is tiling **off**. + * + * `pointsTiling` defaults to `'auto'` in production (D5 step 7), which makes a fresh + * entry plan the probe FIRST and defer the preload until it answers. Everything below + * is about preload / rowCodes / matching mechanics, which that deferral would push a + * pass later in every single case — so they pin it off and say so, and the default + * itself is pinned by its own tests rather than by 40 incidental assertions. + */ const ctx = ( el: PointsElement, config: PointsResolveConfig = {} @@ -56,7 +71,10 @@ const ctx = ( elementKey: 'transcripts', kind: 'points', element: el, - config, + // Merged, not defaulted: most callers pass a partial config, and a bare default + // parameter would hand tiling back to its 'auto' production default for every one + // of them. + config: { pointsTiling: 'off', ...config }, transform: new Matrix4(), }); @@ -391,11 +409,13 @@ describe('SpatialEntryStore — the reconcile loop', () => { it('does not block on a resource that is merely refining', async () => { // A catalog scan or a feature scan refines an already-drawable layer. Only the - // preload gates a first paint — and blockingResources says so as DATA, which is - // what today's isBlocking kind-switch collapses into. + // geometry gates a first paint — and blockingResources says so as DATA, which is + // what today's isBlocking kind-switch collapses into. `tiling` is here because + // until the probe answers we do not know which geometry path this entry is on; + // an entry whose snapshot omits either resource simply does not block on it. const resolver = new PointsResolver(); - expect(resolver.blockingResources).toEqual(['preload']); + expect(resolver.blockingResources).toEqual(['tiling', 'preload']); }); it('bumps its version when any resolver mutates', async () => { @@ -785,3 +805,630 @@ describe('getMatchingLoadState() — a failed scan is reportable', () => { expect(state?.matchedRows).toBe(2); }); }); + +describe('D5 step 1 — Morton tiling metadata probe', () => { + /** Renderable Morton metadata: range reads AND bounds, the two things the tile + * path cannot work without. */ + const tiling = (over: Partial = {}): PointsTilingMetadata => ({ + kind: 'morton-points', + parquetPath: 'points/transcripts/points.parquet', + axisNames: ['x', 'y'], + featureCodeColumnName: 'feature_name_codes', + mortonCodeColumnName: MORTON_CODE_2D_COLUMN, + totalRows: 12_000_000, + totalRowGroups: 96, + maxRowsPerGroup: 131_072, + supportsRowGroupRangeReads: true, + bounds: { minX: 0, minY: 0, maxX: 100, maxY: 100 }, + ...over, + }); + + const tiledElement = (metadata: PointsTilingMetadata | null, over = {}) => + element({ getPointsTilingMetadata: vi.fn(async () => metadata), ...over }); + + const store = (resolver: PointsResolver) => + new SpatialEntryStore({ + points: resolver, + shapes: resolver, + images: resolver, + labels: resolver, + }); + + // The step-1 acceptance criterion, as a test: with tiling off — the default — + // nothing about planning changes. Everything else in this file is the regression + // net for that claim; this is the direct statement of it. + it('is off when asked: no probe, and the preload is planned exactly as before', () => { + const el = tiledElement(tiling()); + const tasks = new PointsResolver().plan(ctx(el, { pointsTiling: 'off' })); + + expect(tasks.map((t) => t.resource)).toEqual(['preload', 'rowCodes']); + expect(el.getPointsTilingMetadata).not.toHaveBeenCalled(); + }); + + /** + * The step-7 default. Note what it is NOT: it does not plan a preload alongside the + * probe. Deferring is the entire point — planning both would do the full-table read + * we are about to discover we do not need, and settle it, so the waste would not + * even show up as a second load. + */ + it('is on by default: an unset config probes first and defers the preload', () => { + const el = tiledElement(tiling()); + const resolver = new PointsResolver(); + + // `config: {}` — no `pointsTiling` key at all, the shape a caller who has never + // heard of D5 passes. + const bare = { ...ctx(el), config: {} }; + expect(resolver.plan(bare).map((t) => t.resource)).toEqual(['tiling']); + expect(pointsTilingEnabled(undefined)).toBe(true); + expect(DEFAULT_POINTS_TILING).toBe('auto'); + }); + + it('falls through to the preload when the probe says the element is not tileable', async () => { + const el = element(); + el.getPointsTilingMetadata = vi.fn(async () => null); + const resolver = new PointsResolver(); + const bare = { ...ctx(el), config: {} }; + + await resolver.load(resolver.plan(bare)[0] as never, bare, new AbortController().signal); + + // The default costs one probe on an element that cannot use it, and then the + // preload path is exactly what it always was. + expect(resolver.plan(bare).map((t) => t.resource)).toEqual(['preload', 'rowCodes']); + }); + + it('plans a probe — and DEFERS the preload — when tiling is auto', () => { + const el = tiledElement(tiling()); + const tasks = new PointsResolver().plan(ctx(el, { pointsTiling: 'auto' })); + + // Preloading a table we are about to tile wastes the entire read, so until the + // probe answers we schedule neither. + expect(tasks.map((t) => t.resource)).toEqual(['tiling']); + // …and planning still starts nothing. + expect(el.getPointsTilingMetadata).not.toHaveBeenCalled(); + }); + + it('settles the metadata for a tileable element, and then plans no preload at all', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(tiling()); + + await resolver.ensureTilingMetadata({ key: 'transcripts', layerId: 'L', element: el }); + + expect(resolver.isTiled('transcripts')).toBe(true); + expect(resolver.getTilingMetadata('transcripts')?.totalRowGroups).toBe(96); + // The preload, its row codes and the matching scan are all resident-batch + // notions, and a tiled element has none of them. The catalog IS planned: it is + // the only way a name-based selection becomes the codes the tile scan filters on + // (step 4), and unlike the preloaded path nothing else builds one. + expect( + resolver.plan(ctx(el, { pointsTiling: 'auto', featureCodes: [0, 1] })).map((t) => t.resource) + ).toEqual(['catalog']); + }); + + it('settles null when the artifact cannot drive tiles, and falls back to the preload', async () => { + const resolver = new PointsResolver(); + // Morton metadata exists but the store cannot serve row-group range reads — the + // same renderability gate the render resolver applies, made once, here. + const el = tiledElement(tiling({ supportsRowGroupRangeReads: false })); + const config = { pointsTiling: 'auto' as const }; + + await store(resolver).reconcile([ctx(el, config)]); + + expect(resolver.getTilingMetadata('transcripts')).toBeNull(); + expect(resolver.isTiled('transcripts')).toBe(false); + expect(resolver.plan(ctx(el, config)).map((t) => t.resource)).toEqual(['preload', 'rowCodes']); + }); + + it('settles null when the element has no Morton artifact', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(null); + + await resolver.ensureTilingMetadata({ key: 'transcripts', layerId: 'L', element: el }); + + expect(resolver.getTilingMetadata('transcripts')).toBeNull(); + }); + + it('a failed probe falls through to the preload instead of stranding the layer', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(null, { + getPointsTilingMetadata: vi.fn(async () => { + throw new Error('footer read failed'); + }), + }); + const config = { pointsTiling: 'auto' as const }; + + await resolver.ensureTilingMetadata({ key: 'transcripts', layerId: 'L', element: el }); + + // The failure is a state — visible, and retryable… + const failed = resolver.snapshot(ctx(el, config)).resources.tiling; + expect(Resolution.isFailed(failed as never)).toBe(true); + if (failed.status === 'failed') expect(failed.error.retryable).toBe(true); + // …but it must not stop anything drawing: it reads as "cannot tile", so the next + // plan pass schedules the ordinary preload. + expect(resolver.getTilingMetadata('transcripts')).toBeNull(); + expect(resolver.plan(ctx(el, config)).map((t) => t.resource)).toEqual(['preload', 'rowCodes']); + }); + + it('does not re-probe a failed element on every reconcile', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(null, { + getPointsTilingMetadata: vi.fn(async () => { + throw new Error('footer read failed'); + }), + }); + const s = store(resolver); + const config = { pointsTiling: 'auto' as const }; + + await s.reconcile([ctx(el, config)]); + await s.reconcile([ctx(el, config)]); + + // A failure that re-planned itself would spin forever AND keep the preload it is + // standing in front of permanently unscheduled. + expect(el.getPointsTilingMetadata).toHaveBeenCalledTimes(1); + }); + + it('retry() re-runs a failed probe', async () => { + const resolver = new PointsResolver(); + let attempts = 0; + const el = tiledElement(null, { + getPointsTilingMetadata: vi.fn(async () => { + attempts += 1; + if (attempts === 1) throw new Error('footer read failed'); + return tiling(); + }), + }); + + await resolver.ensureTilingMetadata({ key: 'transcripts', layerId: 'L', element: el }); + expect(resolver.isTiled('transcripts')).toBe(false); + + await resolver.retry('transcripts'); + + expect(resolver.isTiled('transcripts')).toBe(true); + }); + + it('dedups the probe: one request per element, however many reconciles', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(tiling()); + const s = store(resolver); + const config = { pointsTiling: 'auto' as const }; + + await Promise.all([s.reconcile([ctx(el, config)]), s.reconcile([ctx(el, config)])]); + await s.reconcile([ctx(el, config)]); + + expect(el.getPointsTilingMetadata).toHaveBeenCalledTimes(1); + }); + + // The deferral only works because a settle notifies, the host replans, and the + // preload is scheduled on that second pass. Without it a non-tileable element would + // sit forever behind a probe that already answered. + it('schedules the deferred preload on the reconcile after the probe answers', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(null); + const s = store(resolver); + const config = { pointsTiling: 'auto' as const }; + + await s.reconcile([ctx(el, config)]); + expect(el.loadPoints).not.toHaveBeenCalled(); + + await s.reconcile([ctx(el, config)]); + expect(el.loadPoints).toHaveBeenCalledTimes(1); + }); + + it('a tileable element never preloads, however many reconciles run', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(tiling()); + const s = store(resolver); + const config = { pointsTiling: 'auto' as const }; + + await s.reconcile([ctx(el, config)]); + await s.reconcile([ctx(el, config)]); + + expect(el.loadPoints).not.toHaveBeenCalled(); + }); + + it('evict drops the probe answer with the rest of the entry', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(tiling()); + + await resolver.ensureTilingMetadata({ key: 'transcripts', layerId: 'L', element: el }); + resolver.evict('transcripts'); + + expect(resolver.getTilingMetadata('transcripts')).toBeUndefined(); + expect(resolver.isTilingSettled('transcripts')).toBe(false); + }); +}); + +describe('D5 step 2 — a tiled entry is drawable, framed and unblocked', () => { + const tiling = (over: Partial = {}): PointsTilingMetadata => ({ + kind: 'morton-points', + parquetPath: 'points/transcripts/points.parquet', + axisNames: ['x', 'y'], + featureCodeColumnName: 'feature_name_codes', + mortonCodeColumnName: MORTON_CODE_2D_COLUMN, + totalRows: 12_000_000, + totalRowGroups: 96, + maxRowsPerGroup: 131_072, + supportsRowGroupRangeReads: true, + bounds: { minX: 10, minY: 20, maxX: 110, maxY: 220 }, + ...over, + }); + + const tiledElement = (metadata: PointsTilingMetadata | null) => + element({ getPointsTilingMetadata: vi.fn(async () => metadata) }); + + const store = (resolver: PointsResolver) => + new SpatialEntryStore({ + points: resolver, + shapes: resolver, + images: resolver, + labels: resolver, + }); + + const auto = { pointsTiling: 'auto' as const }; + + // THE step-2 bug this guards. A tiled entry plans no preload, so a `preload` + // resolution left sitting at `idle` reads as blocking forever — and auto-fit rides + // the isBlocking true→false transition, so the layer never frames either. + it('stops blocking once the probe answers, without ever preloading', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(tiling()); + const s = store(resolver); + + // Blocked while the probe is open: we cannot draw what we cannot classify. + expect(s.isBlocking(ctx(el, auto))).toBe(true); + + await s.reconcile([ctx(el, auto)]); + + expect(s.isBlocking(ctx(el, auto))).toBe(false); + expect(el.loadPoints).not.toHaveBeenCalled(); + }); + + it('still blocks a non-tileable entry until its preload lands', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(null); + const s = store(resolver); + + await s.reconcile([ctx(el, auto)]); // probe settles null… + expect(s.isBlocking(ctx(el, auto))).toBe(true); // …and there is still nothing to draw + + await s.reconcile([ctx(el, auto)]); // …so the preload runs + + expect(s.isBlocking(ctx(el, auto))).toBe(false); + }); + + it('reports only the resources the entry actually has', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(tiling()); + + // Tiling off: the entry never asked the question, so it has no tiling resource. + expect(Object.keys(resolver.snapshot(ctx(el)).resources)).not.toContain('tiling'); + + await store(resolver).reconcile([ctx(el, auto)]); + + // Tiled: no resident preload exists — absent, not idle. `isBlocking` skips a + // resource that is not there, which is what the test above depends on. + const resources = resolver.snapshot(ctx(el, auto)).resources; + expect(Object.keys(resources)).toContain('tiling'); + expect(Object.keys(resources)).not.toContain('preload'); + }); + + it('frames from the artifact extent, through the element transform', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(tiling()); + await store(resolver).reconcile([ctx(el, auto)]); + + const shifted = { + ...ctx(el, auto), + transform: new Matrix4().translate([5, 7, 0]), + }; + expect(resolver.snapshot(shifted).bounds).toEqual({ + minX: 15, + minY: 27, + maxX: 115, + maxY: 227, + }); + }); + + it('returns identity-stable bounds — a fresh object per call is a deck teardown', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(tiling()); + await store(resolver).reconcile([ctx(el, auto)]); + + const base = ctx(el, auto); + const first = resolver.snapshot(base).bounds; + // A DIFFERENT snapshot object (another entry on the same element, so the snapshot + // memo misses) must still hand back the same bounds object: the memo below it + // keys on (metadata, transform), and both entries share the element's transform. + const again = resolver.snapshot({ ...base, entryId: 'other-layer' }).bounds; + + expect(first).not.toBeNull(); + expect(again).toBe(first); + }); + + it('reports geometry status for the tiled path, not silence', async () => { + const resolver = new PointsResolver(); + const statuses: Array<[string, string]> = []; + const withStatus = new PointsResolver({ + onStatus: (layerId, status) => statuses.push([layerId, status]), + }); + const el = tiledElement(tiling()); + void resolver; + + const pending = withStatus.ensureTilingMetadata({ + key: 'transcripts', + layerId: 'layer-p', + element: el, + }); + // Mid-probe the entry IS loading its geometry; reporting 'idle' would leave the + // host showing nothing-is-happening for the whole footer read. + expect(withStatus.getStatus('transcripts')).toBe('loading'); + await pending; + + // A tileable answer is terminal: there is no preload to wait for. + expect(withStatus.getStatus('transcripts')).toBe('ready'); + expect(statuses).toEqual([ + ['layer-p', 'loading'], + ['layer-p', 'ready'], + ]); + }); + + it('does not claim ready when the probe hands off to the preload', async () => { + const statuses: string[] = []; + const resolver = new PointsResolver({ + onStatus: (_layerId, status) => statuses.push(status), + }); + const el = tiledElement(null); + + await resolver.ensureTilingMetadata({ key: 'transcripts', layerId: 'L', element: el }); + + // 'ready' here would clear the spinner while the real geometry load had not even + // been planned yet. + expect(statuses).toEqual(['loading']); + expect(resolver.getStatus('transcripts')).toBe('idle'); + }); +}); + +describe('D5 — tiling is per entry, the probe answer is per element', () => { + const tiling = (): PointsTilingMetadata => ({ + kind: 'morton-points', + parquetPath: 'points/transcripts/points.parquet', + axisNames: ['x', 'y'], + featureCodeColumnName: 'feature_name_codes', + mortonCodeColumnName: MORTON_CODE_2D_COLUMN, + totalRows: 12_000_000, + totalRowGroups: 96, + maxRowsPerGroup: 131_072, + supportsRowGroupRangeReads: true, + bounds: { minX: 0, minY: 0, maxX: 100, maxY: 100 }, + }); + + const tiledElement = () => element({ getPointsTilingMetadata: vi.fn(async () => tiling()) }); + + /** + * Found in the browser, not by a test: switch tiling ON, then OFF, and the layer + * kept drawing tiles. The probe's answer is cached per ELEMENT and survives the + * config that asked for it, so anything keyed on `isTiled` alone ignores the switch. + * `plan()` meanwhile went back to preloading — so the app both preloaded AND drew + * tiles. + */ + it('goes back to the preloaded path when tiling is switched off', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(); + const on = ctx(el, { pointsTiling: 'auto' }); + const off = ctx(el, { pointsTiling: 'off' }); + + await resolver.ensureTilingMetadata({ key: 'transcripts', layerId: 'layer-p', element: el }); + + // The element fact does not change — the metadata is still cached and valid… + expect(resolver.isTiled('transcripts')).toBe(true); + // …but this entry no longer asked for it, so it plans a preload again… + expect(resolver.plan(off).map((t) => t.resource)).toEqual(['preload', 'rowCodes']); + // The tiled entry asks only for the catalog — no preload, no row codes, no scan. + expect(resolver.plan(on).map((t) => t.resource)).toEqual(['catalog']); + // …and its snapshot reports a preload resource (which gates first paint) and no + // tiling-derived bounds. + expect(Object.keys(resolver.snapshot(off).resources)).toContain('preload'); + expect(resolver.snapshot(off).bounds).toBeNull(); + expect(resolver.snapshot(on).bounds).not.toBeNull(); + }); + + it('lets two entries on one element disagree', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(); + const tiledEntry = { ...ctx(el, { pointsTiling: 'auto' }), entryId: 'tiled' }; + const preloadEntry = { ...ctx(el, { pointsTiling: 'off' }), entryId: 'preloaded' }; + + await resolver.ensureTilingMetadata({ key: 'transcripts', layerId: 'tiled', element: el }); + + expect(resolver.snapshot(tiledEntry).resources.preload).toBeUndefined(); + expect(resolver.snapshot(preloadEntry).resources.preload).toBeDefined(); + }); + + it('reports the preload status even for an element something else tiles', async () => { + // getStatus is per element; the preload is the load actually in flight for the + // entry that is not tiling, so it must not be masked by the probe's answer. + const resolver = new PointsResolver(); + const el = tiledElement(); + + await resolver.ensureTilingMetadata({ key: 'transcripts', layerId: 'tiled', element: el }); + expect(resolver.getStatus('transcripts')).toBe('ready'); + + await resolver.ensureLoaded({ key: 'transcripts', layerId: 'preloaded', element: el }); + expect(resolver.getStatus('transcripts')).toBe('ready'); + expect(el.loadPoints).toHaveBeenCalledTimes(1); + }); +}); + +describe('D5 — a tiled element releases the resident window', () => { + const tiling = (): PointsTilingMetadata => ({ + kind: 'morton-points', + parquetPath: 'points/transcripts/points.parquet', + axisNames: ['x', 'y'], + featureCodeColumnName: 'feature_name_codes', + mortonCodeColumnName: MORTON_CODE_2D_COLUMN, + totalRows: 12_000_000, + totalRowGroups: 96, + maxRowsPerGroup: 131_072, + supportsRowGroupRangeReads: true, + bounds: { minX: 0, minY: 0, maxX: 100, maxY: 100 }, + }); + + /** + * A layer switched to tiling mid-session has usually already preloaded. `plan()` + * stops ASKING for a preload, which is not the same as giving one back: the rows + * stayed resident, and the panel went on reporting "4M of 12.1M in memory — + * capped" over a render that has no cap. + */ + it('drops the preload, its row codes and its scan once the probe says tileable', async () => { + const resolver = new PointsResolver(); + const el = element({ getPointsTilingMetadata: vi.fn(async () => tiling()) }); + const target = { key: 'transcripts', layerId: 'L', element: el }; + + // The layer starts un-tiled and preloads. + await resolver.ensureLoaded(target); + await resolver.ensureRowFeatureCodes(target); + expect(resolver.hasData('transcripts')).toBe(true); + expect(resolver.hasRowFeatureCodes('transcripts')).toBe(true); + + await resolver.ensureTilingMetadata(target); + + expect(resolver.isTiled('transcripts')).toBe(true); + expect(resolver.hasData('transcripts')).toBe(false); + expect(resolver.getData('transcripts')).toBeUndefined(); + expect(resolver.hasRowFeatureCodes('transcripts')).toBe(false); + // …including everything memoised off those codes, or a map outlives the batch it + // describes — the opposite of releasing it. + expect(resolver.getResidentFeatureCodes('transcripts')).toBeUndefined(); + expect(resolver.getResidentFeatureCounts('transcripts')).toBeUndefined(); + // …so nothing is left to report a truncation over. + expect(resolver.getActiveTruncation('transcripts', undefined)).toBeUndefined(); + }); + + it('notifies, so a panel reading the resident batch repaints', async () => { + const resolver = new PointsResolver(); + const el = element({ getPointsTilingMetadata: vi.fn(async () => tiling()) }); + const target = { key: 'transcripts', layerId: 'L', element: el }; + await resolver.ensureLoaded(target); + + const before = resolver.getVersion(); + await resolver.ensureTilingMetadata(target); + + expect(resolver.getVersion()).toBeGreaterThan(before); + }); + + it('leaves the catalog alone — it describes the element, not the window', async () => { + const resolver = new PointsResolver(); + const el = element({ + getPointsTilingMetadata: vi.fn(async () => tiling()), + listFeaturesWithCounts: vi.fn(async () => ({ + featureKey: 'feature_name', + entries: [{ code: 0, name: 'GeneA' }], + })), + }); + const target = { key: 'transcripts', layerId: 'L', element: el }; + await resolver.ensureLoaded(target); + await resolver.ensureFeatureCatalog(target); + + await resolver.ensureTilingMetadata(target); + + expect(resolver.getFeatureCatalog('transcripts')).toEqual({ + featureKey: 'feature_name', + entries: [{ code: 0, name: 'GeneA' }], + }); + }); + + it('does not evict when the element cannot be tiled', async () => { + const resolver = new PointsResolver(); + const el = element({ getPointsTilingMetadata: vi.fn(async () => null) }); + const target = { key: 'transcripts', layerId: 'L', element: el }; + await resolver.ensureLoaded(target); + + await resolver.ensureTilingMetadata(target); + + expect(resolver.hasData('transcripts')).toBe(true); + }); +}); + +describe('D5 step 4 — a tiled entry asks for its own catalog', () => { + const tiling = (): PointsTilingMetadata => ({ + kind: 'morton-points', + parquetPath: 'points/transcripts/points.parquet', + axisNames: ['x', 'y'], + featureCodeColumnName: 'feature_name_codes', + mortonCodeColumnName: MORTON_CODE_2D_COLUMN, + totalRows: 12_000_000, + totalRowGroups: 96, + maxRowsPerGroup: 131_072, + supportsRowGroupRangeReads: true, + bounds: { minX: 0, minY: 0, maxX: 100, maxY: 100 }, + }); + + const catalog = { featureKey: 'feature_name', entries: [{ code: 0, name: 'GeneA' }] }; + const tiledElement = (over = {}) => + element({ + getPointsTilingMetadata: vi.fn(async () => tiling()), + listFeaturesWithCounts: vi.fn(async () => catalog), + ...over, + }); + + const store = (resolver: PointsResolver) => + new SpatialEntryStore({ + points: resolver, + shapes: resolver, + images: resolver, + labels: resolver, + }); + const auto = { pointsTiling: 'auto' as const }; + + /** + * On the preloaded path the catalog arrives free, as a preview off the geometry + * decode. A tiled entry never decodes a resident batch, so without planning it + * nothing builds one — and a selection stored as feature NAMES can never become the + * codes the tile scan filters on. + */ + it('builds a catalog with no preload in sight', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(); + const s = store(resolver); + + await s.reconcile([ctx(el, auto)]); // probe + await s.reconcile([ctx(el, auto)]); // catalog + + expect(resolver.getFeatureCatalog('transcripts')).toEqual(catalog); + expect(el.loadPoints).not.toHaveBeenCalled(); + }); + + it('asks once, not on every reconcile', async () => { + const resolver = new PointsResolver(); + const el = tiledElement(); + const s = store(resolver); + + await s.reconcile([ctx(el, auto)]); + await s.reconcile([ctx(el, auto)]); + await s.reconcile([ctx(el, auto)]); + await s.reconcile([ctx(el, auto)]); + + expect(el.listFeaturesWithCounts).toHaveBeenCalledTimes(1); + expect(resolver.plan(ctx(el, auto))).toEqual([]); + }); + + /** + * A failed catalog reports `undefined` exactly as one that never ran, so a gate on + * the value alone re-emits the task forever. `retry()` is the way back. + */ + it('does not re-plan a failed catalog on every reconcile', async () => { + const resolver = new PointsResolver(); + const el = tiledElement({ + listFeaturesWithCounts: vi.fn(async () => { + throw new Error('catalog scan failed'); + }), + }); + const s = store(resolver); + + await s.reconcile([ctx(el, auto)]); + await s.reconcile([ctx(el, auto)]); + await s.reconcile([ctx(el, auto)]); + + expect(el.listFeaturesWithCounts).toHaveBeenCalledTimes(1); + expect(resolver.plan(ctx(el, auto))).toEqual([]); + }); +}); diff --git a/packages/core/tests/pointsRowCodesCapAlignment.spec.ts b/packages/core/tests/pointsRowCodesCapAlignment.spec.ts index 37002d6e..92bcc220 100644 --- a/packages/core/tests/pointsRowCodesCapAlignment.spec.ts +++ b/packages/core/tests/pointsRowCodesCapAlignment.spec.ts @@ -54,6 +54,9 @@ function dictOnlyElement() { } as unknown as PointsElement; } +// Tiling pinned off: `pointsTiling` defaults to `'auto'`, which defers the preload +// behind the probe. These tests are about rowCodes staying aligned with the resident +// batch, which only exists on the preloaded path. const ctx = ( el: PointsElement, config: PointsResolveConfig = {} @@ -62,7 +65,10 @@ const ctx = ( elementKey: 'transcripts', kind: 'points', element: el, - config, + // Merged, not defaulted: most callers pass a partial config, and a bare default + // parameter would hand tiling back to its 'auto' production default for every one + // of them. + config: { pointsTiling: 'off', ...config }, transform: new Matrix4(), }); @@ -145,7 +151,9 @@ describe('row codes — cap alignment with the resident batch', () => { release(); await inFlight; - expect((el.loadRowFeatureCodes as ReturnType).mock.calls.length).toBe(readsBefore); + expect((el.loadRowFeatureCodes as ReturnType).mock.calls.length).toBe( + readsBefore + ); // It did NOT supply them (dict-only), so now the gate asks. expect(rowCodesTasks(resolver, el, LARGE_CAP)).toHaveLength(1); }); diff --git a/packages/core/tests/pointsTileGrid.spec.ts b/packages/core/tests/pointsTileGrid.spec.ts new file mode 100644 index 00000000..c0814b77 --- /dev/null +++ b/packages/core/tests/pointsTileGrid.spec.ts @@ -0,0 +1,103 @@ +import { describe, expect, it } from 'vitest'; +import { + mortonTileGrid, + POINTS_TILE_SIZE, + POINTS_TILE_TARGET_ROWS, +} from '../src/pointsTileGrid.js'; + +/** The real 12.1M-point Xenium transcripts element, for grounding. */ +const XENIUM = { + bounds: { minX: 3.42, minY: 2.45, maxX: 10874.72, maxY: 3629.29 }, + totalRows: 12_165_021, + maxRowsPerGroup: 50_000, + modelMatrixScale: 4.705882352941177, +}; + +/** Local units a tile spans at zoom level `z`. */ +const spanAt = (z: number) => POINTS_TILE_SIZE / 2 ** z; + +describe('morton tile grid', () => { + it('brackets the levels between a row group and the row budget', () => { + const grid = mortonTileGrid(XENIUM); + const area = + (XENIUM.bounds.maxX - XENIUM.bounds.minX) * (XENIUM.bounds.maxY - XENIUM.bounds.minY); + const density = XENIUM.totalRows / area; + + // Finest level still covers at least one row group's footprint... + expect(spanAt(grid.maxZoom) ** 2 * density).toBeGreaterThanOrEqual(XENIUM.maxRowsPerGroup); + // ...and the coarsest stays within the per-tile row budget. + expect(spanAt(grid.minZoom) ** 2 * density).toBeLessThanOrEqual(POINTS_TILE_TARGET_ROWS); + expect(grid.minZoom).toBeLessThanOrEqual(grid.maxZoom); + }); + + it('subdivides, where the old fixed grid never did', () => { + const grid = mortonTileGrid(XENIUM); + expect(grid.maxZoom).toBeGreaterThan(grid.minZoom); + }); + + it('offsets zoom by the model matrix, so a tile lands near tileSize on screen', () => { + const grid = mortonTileGrid(XENIUM); + // deck: z = ceil(viewport.zoom + zoomOffset); a tile is tileSize/2^z LOCAL units, + // which is scale x that in world units, which is 2^zoom x that in screen pixels. + for (const viewportZoom of [-4, -3.2, -2.32, -1, 0]) { + const z = Math.min( + grid.maxZoom, + Math.max(grid.minZoom, Math.ceil(viewportZoom + grid.zoomOffset)) + ); + const screenPx = spanAt(z) * XENIUM.modelMatrixScale * 2 ** viewportZoom; + // Only levels inside the range can hit the target; clamped ends legitimately + // over- or under-shoot, which is what min/maxZoom are FOR. + if (z > grid.minZoom && z < grid.maxZoom) { + expect(screenPx).toBeGreaterThan(POINTS_TILE_SIZE / 2); + expect(screenPx).toBeLessThanOrEqual(POINTS_TILE_SIZE * 1.01); + } + } + }); + + it('states the tile cache budget in rows instead of leaving it to deck', () => { + const grid = mortonTileGrid({ ...XENIUM, cacheRowBudget: 4_000_000 }); + expect(grid.maxCacheSize).toBeGreaterThan(0); + expect(grid.estimatedRowsPerTile).toBeGreaterThan(0); + expect(grid.cacheRowBudget).toBe(grid.maxCacheSize * grid.estimatedRowsPerTile); + // deck's default is 5 x the selected tile count, which on a coarse viewport of + // this element is ~220 tiles; whatever we choose has to be a stated number. + expect(grid.maxCacheSize).toBeLessThan(220); + }); + + it('collapses to one level when a row group already exceeds the row budget', () => { + // Huge row groups over a small extent: the floor is above the ceiling. + const grid = mortonTileGrid({ + bounds: { minX: 0, minY: 0, maxX: 100, maxY: 100 }, + totalRows: 1_000_000, + maxRowsPerGroup: 900_000, + modelMatrixScale: 1, + }); + expect(grid.minZoom).toBe(grid.maxZoom); + }); + + it('keeps the old single level when there is no density to reason from', () => { + const degenerate = mortonTileGrid({ + bounds: { minX: 0, minY: 0, maxX: 0, maxY: 0 }, + totalRows: 0, + maxRowsPerGroup: 0, + modelMatrixScale: 1, + }); + expect(degenerate).toMatchObject({ minZoom: -1, maxZoom: -1, zoomOffset: 0 }); + }); + + it('ignores a model matrix scale that would poison every tile index', () => { + for (const modelMatrixScale of [0, -3, Number.NaN, Number.POSITIVE_INFINITY]) { + expect(mortonTileGrid({ ...XENIUM, modelMatrixScale }).zoomOffset).toBe(0); + } + }); + + it('scales the grid to the artifact, not to a constant', () => { + // A tenth the extent at the same row count is 100x denser, so its tiles must be + // smaller in local units — the fixed 1024 could not express this. + const dense = mortonTileGrid({ + ...XENIUM, + bounds: { minX: 0, minY: 0, maxX: 1087, maxY: 363 }, + }); + expect(spanAt(dense.minZoom)).toBeLessThan(spanAt(mortonTileGrid(XENIUM).minZoom)); + }); +}); diff --git a/packages/core/tests/pointsTiling.spec.ts b/packages/core/tests/pointsTiling.spec.ts index 859ca789..9a916d6e 100644 --- a/packages/core/tests/pointsTiling.spec.ts +++ b/packages/core/tests/pointsTiling.spec.ts @@ -4,8 +4,12 @@ import { extractSentinelBoundingBox, filterColumnarByFeatureCodes, filterPointsToBounds, + MORTON_ZCOVER_MAX_DEPTH, mergeAdjacentIntervals, + mortonBoundsAgreeWithCodes, + mortonCode2dForPoint, mortonIntervalsForBounds, + mortonRowGroupExtentsAreSorted, zcoverRectangle, } from '../src/pointsTiling.js'; @@ -167,3 +171,175 @@ describe('points tiling helpers', () => { expect(filtered.featureCodes).toBe(sourceFeatureCodes); }); }); + +/** + * The cover picks ROW GROUPS, so resolving a rectangle down to individual quantised + * cells is pure cost. Full-depth recursion produced 38,014 intervals for one + * viewport-sized rectangle on a real 12.1M-point artifact — 76k row-group bisects to + * select the same 92 row groups a few hundred intervals select. + */ +describe('zcoverRectangle depth cap', () => { + /** Interleave two 16-bit coords into a Morton code, as the writer does. */ + const morton = (x: number, y: number): number => { + let code = 0; + for (let bit = 0; bit < 16; bit += 1) { + code += ((x >> bit) & 1) * 2 ** (2 * bit) + ((y >> bit) & 1) * 2 ** (2 * bit + 1); + } + return code; + }; + + const covers = (intervals: Array<[number, number]>, code: number) => + intervals.some(([lo, hi]) => lo <= code && code <= hi); + + it('collapses the interval count by orders of magnitude', () => { + const rect: [number, number, number, number] = [12345, 6789, 41000, 20000]; + + const full = zcoverRectangle(...rect, 16, 16); + const capped = zcoverRectangle(...rect); + + expect(full.length).toBeGreaterThan(10_000); + expect(capped.length).toBeLessThan(full.length / 20); + }); + + it('still covers every code inside the rectangle', () => { + // Completeness is the property that matters: a dropped code is a hole in the + // render. Coarser cells may cover MORE than the rectangle, which the exact + // bounds filter removes after the read. + const [x0, y0, x1, y1] = [1000, 2000, 3400, 2600]; + const intervals = zcoverRectangle(x0, y0, x1, y1); + + for (let x = x0; x <= x1; x += 37) { + for (let y = y0; y <= y1; y += 41) { + expect(covers(intervals, morton(x, y))).toBe(true); + } + } + }); + + it('never covers less than the exact cover', () => { + const rect: [number, number, number, number] = [700, 900, 5000, 3300]; + const full = zcoverRectangle(...rect, 16, 16); + const capped = zcoverRectangle(...rect); + + // Every code the full-depth cover claims must still be claimed. + for (const [lo, hi] of full) { + expect(covers(capped, lo)).toBe(true); + expect(covers(capped, hi)).toBe(true); + } + }); + + it('is unchanged for a rectangle that resolves above the cap', () => { + // A whole-space query is one cell at level 0 — the cap cannot affect it. + expect(zcoverRectangle(0, 0, 65535, 65535)).toEqual([[0, 4294967295]]); + }); + + it('honours an explicit depth, and clamps it to the coordinate bits', () => { + const rect: [number, number, number, number] = [10, 20, 5000, 6000]; + + expect(zcoverRectangle(...rect, 16, 4).length).toBeLessThan( + zcoverRectangle(...rect, 16, 8).length + ); + // Beyond `bits` there is nothing left to subdivide. + expect(zcoverRectangle(...rect, 16, 99)).toEqual(zcoverRectangle(...rect, 16, 16)); + expect(MORTON_ZCOVER_MAX_DEPTH).toBeLessThan(16); + }); + + it('applies the cap through mortonIntervalsForBounds', () => { + const bounds = { minX: 0, minY: 0, maxX: 1000, maxY: 1000 }; + const intervals = mortonIntervalsForBounds(bounds, { + minX: 137, + minY: 241, + maxX: 622, + maxY: 733, + }); + + expect(intervals.length).toBeLessThan(4_000); + }); +}); + +describe('morton bounds agreement', () => { + const bounds = { minX: 0, minY: 0, maxX: 1000, maxY: 1000 }; + const xs = Array.from({ length: 64 }, (_, i) => (i * 997) % 1000); + const ys = Array.from({ length: 64 }, (_, i) => (i * 613) % 1000); + const codes = xs.map((x, i) => mortonCode2dForPoint(x, ys[i], bounds)); + + it('agrees with the domain the codes came from', () => { + expect(mortonBoundsAgreeWithCodes(xs, ys, codes, bounds)).toEqual({ + checked: 64, + matched: 64, + }); + }); + + it('disagrees with a sub-box of that domain', () => { + // The stale-fixture shape: a box a quarter the size, offset into the middle. + const subBox = { minX: 250, minY: 250, maxX: 750, maxY: 750 }; + const { checked, matched } = mortonBoundsAgreeWithCodes(xs, ys, codes, subBox); + expect(checked).toBe(64); + expect(matched * 2).toBeLessThanOrEqual(checked); + }); + + it('stays inside the 32-bit code space at the top of the domain', () => { + // x and y both maxed puts the y term in bit 31 — a `<< 1` would go negative here. + const top = mortonCode2dForPoint(bounds.maxX, bounds.maxY, bounds); + expect(top).toBe(2 ** 32 - 1); + expect(top).toBeGreaterThan(0); + }); + + it('reports nothing to check rather than guessing, on empty or unusable input', () => { + expect(mortonBoundsAgreeWithCodes([], [], [], bounds)).toEqual({ checked: 0, matched: 0 }); + expect(mortonBoundsAgreeWithCodes([Number.NaN], [0], [0], bounds)).toEqual({ + checked: 0, + matched: 0, + }); + }); + + it('spreads its samples instead of taking a run of adjacent rows', () => { + // Adjacent rows of a Morton-sorted group share a code prefix, so a leading slice + // would agree or disagree together. Break only the tail and it must still be seen. + const broken = [...codes]; + for (let i = 32; i < broken.length; i++) { + broken[i] = 0; + } + const { checked, matched } = mortonBoundsAgreeWithCodes(xs, ys, broken, bounds, 8); + expect(checked).toBe(8); + expect(matched).toBeLessThan(checked); + }); +}); + +/** + * A morton_code_2d column does not make a file Morton-sorted. A feature-primary + * artifact carries the identical column with the identical values, unsorted, and the + * row-group bisect run over it lands arbitrarily. + */ +describe('morton row-group sort detection', () => { + it('accepts a monotonic sequence, including a shared boundary value', () => { + expect( + mortonRowGroupExtentsAreSorted([ + [0, 0], + [10, 40], + [40, 90], + [91, 120], + ]) + ).toBe(true); + }); + + it('rejects a sequence that restarts, as a feature-primary file does', () => { + // Real shape: each feature block spans nearly the whole code range. + expect( + mortonRowGroupExtentsAreSorted([ + [0, 0], + [450484663, 4193473654], + [443527237, 4288997289], + ]) + ).toBe(false); + }); + + it('treats a missing extent as unknown, not as a descent', () => { + expect(mortonRowGroupExtentsAreSorted([[10, 40], null, [50, 60]])).toBe(true); + // ...and does not lose the running maximum across the gap. + expect(mortonRowGroupExtentsAreSorted([[10, 40], null, [20, 60]])).toBe(false); + }); + + it('concludes nothing from an empty index', () => { + expect(mortonRowGroupExtentsAreSorted([])).toBe(true); + }); +}); diff --git a/packages/core/tests/vtableRangeProbeCache.spec.ts b/packages/core/tests/vtableRangeProbeCache.spec.ts index b653ac7d..26a30bec 100644 --- a/packages/core/tests/vtableRangeProbeCache.spec.ts +++ b/packages/core/tests/vtableRangeProbeCache.spec.ts @@ -99,7 +99,10 @@ describe('streaming range probe — cache policy', () => { }); it('treats a short body as a refusal — the reader would read the wrong window', async () => { - vi.stubGlobal('fetch', vi.fn(async () => partialResponse(4))); + vi.stubGlobal( + 'fetch', + vi.fn(async () => partialResponse(4)) + ); await expect(probeSource().serverSupportsStreamingRanges(freshUrl('short'))).resolves.toBe( false ); diff --git a/packages/layers/src/adapters/PointsRendererAdapter.ts b/packages/layers/src/adapters/PointsRendererAdapter.ts index 7492c9e2..e8033738 100644 --- a/packages/layers/src/adapters/PointsRendererAdapter.ts +++ b/packages/layers/src/adapters/PointsRendererAdapter.ts @@ -1,4 +1,4 @@ -import type { PointsElement, PointsLoadResult } from '@spatialdata/core'; +import type { PointsElement, PointsLoadResult, PointsTilingMetadata } from '@spatialdata/core'; import { columnarBatchFromPointData, type PointsLoader, @@ -77,14 +77,34 @@ interface GrowingPartial { revision: number; } +/** + * A tiled entry's render resource, held for the life of (element, metadata). + * + * Identity here is not a nicety: the resource's loader IS what `TileLayer` keys its + * `getTileData` on, so a fresh resource per `project()` tears the tile layer down and + * re-fetches every visible tile — turning a pan into a full reload. + */ +interface TiledMemo { + element: PointsElement; + metadata: PointsTilingMetadata; + resource: PointsRenderResource; +} + +/** Options for the preloaded path: tiling is decided by the resolver's probe, not + * re-derived here, so these resolves are always the non-tiled branch. */ const RESOLVE_OPTIONS = { experimentalOptimizations: 'off' as const }; +/** …and its counterpart for an entry the probe HAS declared tileable. */ +const TILED_RESOLVE_OPTIONS = { experimentalOptimizations: 'auto' as const }; + /** A batch with no points must not produce a resource — see the empty-lock guard below. */ const isEmpty = (batch: PointsLoadResult): boolean => (batch.shape[1] ?? 0) === 0; export class PointsRendererAdapter { private readonly memos = new Map(); private readonly growingPartials = new Map(); + /** The Morton-tiled resource per element — see {@link getTiledResource}. */ + private readonly tiled = new Map(); /** The base layer's stable resource per element — see {@link getBaseResource}. */ private readonly growingBases = new Map< string, @@ -188,6 +208,34 @@ export class PointsRendererAdapter { return growing.resource; } + /** + * The Morton-tiled entry's render resource (D5). + * + * Unlike every other memo here there is no batch to key on — a tiled entry holds no + * resident data at all. The key is the *metadata*, which the resolver replaces only + * when it re-probes, so this resource survives every pan, zoom and re-render in + * between. That is the whole requirement: `TileLayer` refetches its viewport when + * the loader identity changes. + */ + getTiledResource( + element: PointsElement, + key: string, + metadata: PointsTilingMetadata + ): PointsRenderResource | null { + const memo = this.tiled.get(key); + if (memo && memo.element === element && memo.metadata === metadata) { + return memo.resource; + } + const resource = resolvePointsRenderResource( + element, + { tilingMetadata: metadata, metadataKnown: true }, + TILED_RESOLVE_OPTIONS + ); + if (!resource) return null; + this.tiled.set(key, { element, metadata, resource }); + return resource; + } + /** The revision of the in-flight partial's growing buffer — a `PointsLayer` * `resourceRevision` prop, bumped each time the buffer grows so the composite * re-reads without a teardown. */ @@ -294,11 +342,13 @@ export class PointsRendererAdapter { this.memos.delete(key); this.growingPartials.delete(key); this.growingBases.delete(key); + this.tiled.delete(key); } dispose(): void { this.memos.clear(); this.growingPartials.clear(); this.growingBases.clear(); + this.tiled.clear(); } } diff --git a/packages/layers/src/engine/PointsDataEngine.ts b/packages/layers/src/engine/PointsDataEngine.ts index 3d4b3427..be14612a 100644 --- a/packages/layers/src/engine/PointsDataEngine.ts +++ b/packages/layers/src/engine/PointsDataEngine.ts @@ -160,6 +160,19 @@ export class PointsDataEngine { return this.adapter.getBaseRevision(key); } + /** + * The Morton-tiled render resource (D5), or null when this element is not tiled. + * + * The one resource that is NOT built from a batch: it reads the viewport through + * `loadInBounds`, so it needs only the probe's metadata. Callers gate on + * {@link isTiled}; this returns null rather than throwing if they do not. + */ + getTiledResource(element: PointsElement, key: string): PointsRenderResource | null { + const metadata = this.resolver.getTilingMetadata(key); + if (!metadata) return null; + return this.adapter.getTiledResource(element, key, metadata); + } + // --- Lifecycle (resolver-owned) --------------------------------------------- ensureLoaded(target: PointsLoadTarget, memoryCap?: number): Promise { @@ -186,12 +199,31 @@ export class PointsDataEngine { return this.resolver.ensureRowFeatureCodes(target); } + /** Probe for a Morton artifact (D5). Normally planned, not called — this is the + * facade's counterpart for hosts that drive the engine directly. */ + ensureTilingMetadata(target: PointsLoadTarget): Promise { + return this.resolver.ensureTilingMetadata(target); + } + // --- Reads (resolver-owned) ------------------------------------------------- hasData(key: string): boolean { return this.resolver.hasData(key); } + /** Whether this element renders through the Morton tile path (D5). Its geometry is + * the artifact plus the viewport, so it has no resident batch and {@link hasData} + * is false for it — hosts asking "is there anything to draw" must check both. */ + isTiled(key: string): boolean { + return this.resolver.isTiled(key); + } + + /** The element's tileable Morton metadata: the metadata, `null` when it cannot be + * tiled, `undefined` while the probe is still open. */ + getTilingMetadata(key: string) { + return this.resolver.getTilingMetadata(key); + } + getData(key: string): PointsLoadResult | undefined { return this.resolver.getData(key); } diff --git a/packages/layers/src/mortonTiledStrategy.ts b/packages/layers/src/mortonTiledStrategy.ts index 053a404e..bb0e9c39 100644 --- a/packages/layers/src/mortonTiledStrategy.ts +++ b/packages/layers/src/mortonTiledStrategy.ts @@ -1,4 +1,5 @@ import { COORDINATE_SYSTEM } from '@deck.gl/core'; +import { DEFAULT_POINTS_MEMORY_CAP, mortonTileGrid } from '@spatialdata/core'; import type { Layer, LayersList } from 'deck.gl'; import { PolygonLayer, TileLayer } from 'deck.gl'; import type { PointsLayer } from './PointsLayer.js'; @@ -12,7 +13,11 @@ import { import { featureCodesSignature } from './pointsFeatureCodes.js'; import type { ColumnarNdarrayPointsBatch } from './pointsLoader.js'; import type { PointsRenderStrategy } from './pointsRenderStrategies.js'; -import { DEFAULT_POINT_SIZE, renderColumnarScatterLayer } from './pointsScatterLayer.js'; +import { + DEFAULT_POINT_SIZE, + modelMatrixUniformScale, + renderColumnarScatterLayer, +} from './pointsScatterLayer.js'; import { POINTS_TILE_DEBUG_PICK_KIND, pointsTileDebugPolygonData, @@ -55,6 +60,10 @@ export const mortonTiledStrategy: PointsRenderStrategy = { pointRadiusMinPixels, pointRadiusMaxPixels, color = [255, 100, 100, 200], + colorByFeature, + featureCodeSpaceSize, + featureColorOverrides, + highlightFeatureCode, use3d, } = layer.props; @@ -63,7 +72,26 @@ export const mortonTiledStrategy: PointsRenderStrategy = { return null; } + // The grid comes from the artifact — point density and row-group size — rather + // than from deck's defaults; see mortonTileGrid for why both ends are bounded. + const capabilities = resource.loader.capabilities; + const grid = mortonTileGrid({ + bounds: localBounds, + totalRows: capabilities.totalRows ?? 0, + maxRowsPerGroup: capabilities.maxRowsPerGroup ?? 0, + modelMatrixScale: modelMatrixUniformScale(layer.props.modelMatrix), + // The tile cache is a second pool the resident memory cap cannot see (ADR 0005 + // — account first, manage after). Budgeting it against the same number is not + // the same as the cap applying: it bounds the cache in rows so the worst case + // is a stated quantity rather than deck's `5 x whatever is on screen`, which on + // a coarse viewport of this element was ~220 tiles. + cacheRowBudget: DEFAULT_POINTS_MEMORY_CAP, + }); + const debugHooks = createTiledPointsDebugHooks(layer.props.tileDebugStore); + // Colour rides the per-tile batch: the scan returns a feature code per point + // (D5 step 3), so a tile carries everything the colour extension needs and the + // tiled path stops being the odd one out that only ever drew flat. const scatterStyleProps = { color, pointSize, @@ -71,6 +99,10 @@ export const mortonTiledStrategy: PointsRenderStrategy = { pointRadiusMaxPixels, opacity, modelMatrix: layer.props.modelMatrix, + colorByFeature, + ...(featureCodeSpaceSize !== undefined ? { featureCodeSpaceSize } : {}), + ...(featureColorOverrides ? { featureColorOverrides } : {}), + ...(highlightFeatureCode !== undefined ? { highlightFeatureCode } : {}), use3d, }; @@ -83,9 +115,12 @@ export const mortonTiledStrategy: PointsRenderStrategy = { extent: [localBounds.minX, localBounds.minY, localBounds.maxX, localBounds.maxY], opacity, visible, - tileSize: 512, - minZoom: -1, - maxZoom: -1, + tileSize: grid.tileSize, + minZoom: grid.minZoom, + maxZoom: grid.maxZoom, + zoomOffset: grid.zoomOffset, + maxRequests: grid.maxRequests, + maxCacheSize: grid.maxCacheSize, refinementStrategy: 'best-available', updateTriggers: { getTileData: [resource.element.key, featureCodesSignature(featureCodes)], @@ -96,6 +131,10 @@ export const mortonTiledStrategy: PointsRenderStrategy = { color, opacity, layer.props.modelMatrix, + colorByFeature, + featureCodeSpaceSize, + featureColorOverrides, + highlightFeatureCode, use3d, ], }, @@ -196,7 +235,6 @@ export const mortonTiledStrategy: PointsRenderStrategy = { return renderColumnarScatterLayer(`${props.id}-scatter`, props.data, { ...scatterStyleProps, tileBounds: tileBbox ? scatterBoundsFromTileBbox(tileBbox) : undefined, - tileSubLayer: true, }); }, }) diff --git a/packages/layers/src/pointsLoadPlan.ts b/packages/layers/src/pointsLoadPlan.ts index a767b223..3fbfa5a9 100644 --- a/packages/layers/src/pointsLoadPlan.ts +++ b/packages/layers/src/pointsLoadPlan.ts @@ -1,4 +1,13 @@ -import { type PointsTilingMetadata, resolvePointsMemoryCap } from '@spatialdata/core'; +import { resolvePointsMemoryCap } from '@spatialdata/core'; + +// The load-plan decision itself moved to `core` (D5 step 1) so `PointsResolver.plan()` +// can call it — `core` cannot import from `layers`. Re-exported here so no consumer +// import moves; the cache-key helpers below stay, they are a `layers` concern. +export { + type PointsLoadPlan, + type PointsLoadPlanInput, + planPointsLoads, +} from '@spatialdata/core'; export interface PointsPreloadCacheKeyInput { pointsMemoryCap?: number; @@ -44,29 +53,6 @@ export function resolvePointsPreloadData( return cache.get(preloadCacheKey) ?? cache.get(elementKey); } -export interface PointsLoadPlanInput { - wantsOptimized: boolean; - metadataKnown: boolean; - tiledMetadata: PointsTilingMetadata | null | undefined; - hasPreloaded: boolean; - /** Known row count from parquet metadata, when available. */ - totalRows?: number; -} - -export interface PointsLoadPlan { - probeMetadata: boolean; - preloadFullTable: boolean; -} - -/** Decide which points loads to schedule at the start of a load pass. */ -export function planPointsLoads(input: PointsLoadPlanInput): PointsLoadPlan { - const { wantsOptimized, metadataKnown, tiledMetadata, hasPreloaded } = input; - const probeMetadata = wantsOptimized && !metadataKnown; - const preloadFullTable = - !hasPreloaded && (!wantsOptimized || (metadataKnown && tiledMetadata === null)); - return { probeMetadata, preloadFullTable }; -} - export interface ShouldPreloadAfterMetadataProbeInput { probeRan: boolean; renderableMetadata: boolean; diff --git a/packages/layers/src/pointsLoader.ts b/packages/layers/src/pointsLoader.ts index 4d88ef96..8440b468 100644 --- a/packages/layers/src/pointsLoader.ts +++ b/packages/layers/src/pointsLoader.ts @@ -14,6 +14,13 @@ export interface PointsLoaderCapabilities { bounds?: SpatialBounds; supportsViewportTiles: boolean; supportsFeatureCodes?: boolean; + /** Rows in the whole artifact, when the loader can know without reading it. */ + totalRows?: number; + /** + * Rows per row group — the granularity every viewport read is rounded up to. + * With {@link totalRows} and {@link bounds} it is what sizes the tile grid. + */ + maxRowsPerGroup?: number; } export interface ColumnarNdarrayPointsBatch { diff --git a/packages/layers/src/pointsScatterLayer.ts b/packages/layers/src/pointsScatterLayer.ts index be6ec697..880cd6ef 100644 --- a/packages/layers/src/pointsScatterLayer.ts +++ b/packages/layers/src/pointsScatterLayer.ts @@ -65,7 +65,6 @@ export interface PointsScatterStyleProps { modelMatrix: Matrix4; use3d?: boolean; tileBounds?: [number, number, number, number]; - tileSubLayer?: boolean; /** Colour points by their per-point feature code (requires batch codes). */ colorByFeature?: boolean; /** Number of feature codes the colour LUT must cover (catalog `maxCode + 1`). */ @@ -86,13 +85,18 @@ export function renderColumnarScatterLayer( batch: ColumnarNdarrayPointsBatch, props: PointsScatterStyleProps ) { - // Preloaded scatter sizes points in WORLD (common) units so the GPU scales - // them with zoom — points shrink when you zoom out, which is exactly where - // scatter overdraw is worst — while `radiusMinPixels`/`radiusMaxPixels` clamp - // the projected radius so points never vanish or bloat. The Morton tile path - // keeps fixed pixel sizing (tiles are already viewport-bounded). - const isTile = props.tileSubLayer === true; - const radiusUnits: 'common' | 'pixels' = isTile ? 'pixels' : 'common'; + // Points are sized in WORLD (common) units so the GPU scales them with zoom — + // they shrink when you zoom out, which is exactly where scatter overdraw is worst + // — while `radiusMinPixels`/`radiusMaxPixels` clamp the projected radius so points + // never vanish or bloat. + // + // This is deliberately the SAME on the Morton tile path. Tiles used to size in + // fixed pixels, on the reasoning that they are already viewport-bounded; the + // effect was that `pointSize` meant two different things depending on a checkbox, + // and that a zoomed-out tiled layer drew every one of its millions of points as a + // fixed screen dot. Density saturated into a flat mass and every tile seam and + // density edge hardened into an artefact, so a real acquisition boundary was + // indistinguishable from a rendering fault. const radiusMinPixels = props.pointRadiusMinPixels ?? DEFAULT_POINT_RADIUS_MIN_PIXELS; const radiusMaxPixels = props.pointRadiusMaxPixels ?? DEFAULT_POINT_RADIUS_MAX_PIXELS; // `pointSize` means "this many units of the ELEMENT's own coordinate space". @@ -100,8 +104,7 @@ export function renderColumnarScatterLayer( // alone, so without folding the matrix scale in here the same pointSize renders // wildly differently per element: an element with a 0.00012 mm affine drew points // ~8000x too large for its data, swamping the view and shredding fill rate. - // Pixel-unit tiles are already viewport-relative and must not be rescaled. - const transformScale = isTile ? 1 : modelMatrixUniformScale(props.modelMatrix); + const transformScale = modelMatrixUniformScale(props.modelMatrix); // Feed deck GPU-ready binary attributes (interleaved positions) instead of a // per-object `getPosition` closure. The buffer is memoized on the batch, so a @@ -143,7 +146,8 @@ export function renderColumnarScatterLayer( getFeatureCode: -1, // getRadius: props.pointSize, radiusScale: props.pointSize * transformScale, - radiusUnits, + // World units — see the note above; the tile path uses the same. + radiusUnits: 'common', radiusMinPixels, radiusMaxPixels, getFillColor: props.color, diff --git a/packages/layers/tests/pointsRenderStrategies.spec.ts b/packages/layers/tests/pointsRenderStrategies.spec.ts index b68afd25..17179708 100644 --- a/packages/layers/tests/pointsRenderStrategies.spec.ts +++ b/packages/layers/tests/pointsRenderStrategies.spec.ts @@ -1,4 +1,5 @@ import { describe, expect, it } from 'vitest'; +import { mortonTiledStrategy } from '../src/mortonTiledStrategy.js'; import type { PointsLayer } from '../src/PointsLayer.js'; import { filterBatchSignature } from '../src/pointsFeatureCodes.js'; import { resolvePointsRenderStrategy } from '../src/pointsRenderStrategies.js'; @@ -117,3 +118,85 @@ describe('preloadedScatterStrategy — never shows the previous selection while expect(drawnCount(Array.isArray(result) ? result[0] : result)).toBe(3); }); }); + +/** + * On the preloaded path a selection filters a batch already in memory. On the tiled + * path it is pushed down into the row-group scan, so a deselected feature's points + * are never READ — the selection narrows I/O, not just what is drawn. That is the + * whole reason the tiled path can serve an element far larger than the memory cap. + */ +describe('mortonTiledStrategy — the selection reaches the scan', () => { + const tiledLayer = (props: Record) => { + const calls: Array<{ bounds: unknown; featureCodes: unknown }> = []; + const resource = { + element: { key: 'transcripts' }, + loader: { + capabilities: { + kind: 'morton-tiled' as const, + batchFormat: 'columnar-ndarray' as const, + supportsViewportTiles: true, + bounds: { minX: 0, minY: 0, maxX: 100, maxY: 100 }, + }, + loadInBounds: async (options: { bounds: unknown; featureCodes?: unknown }) => { + calls.push({ bounds: options.bounds, featureCodes: options.featureCodes }); + return null; + }, + }, + }; + const layer = { + props: { id: 'points:transcripts', visible: true, resource, ...props }, + subLayerProps: (sub: Record) => ({ + ...sub, + id: `points:transcripts-${sub.id}`, + }), + } as unknown as PointsLayer; + return { layer, calls, resource }; + }; + + /** Pull the TileLayer out of the strategy's output. */ + const tileLayerOf = (layer: PointsLayer) => { + const result = mortonTiledStrategy.renderLayers(layer); + const layers = Array.isArray(result) ? result : [result]; + return layers[0] as unknown as { + props: { + getTileData: (t: unknown) => Promise; + updateTriggers: Record; + }; + }; + }; + + const tileProps = { + index: { x: 0, y: 0, z: -1 }, + id: '0-0--1', + bbox: { left: 0, top: 50, right: 50, bottom: 0 }, + }; + + it('passes the selected codes to loadInBounds', async () => { + const { layer, calls } = tiledLayer({ featureCodes: [3, 7] }); + + await tileLayerOf(layer).props.getTileData(tileProps); + + expect(calls).toHaveLength(1); + expect(calls[0]?.featureCodes).toEqual([3, 7]); + }); + + it('passes no codes at all for the unfiltered view', async () => { + // `undefined` means "no filter" to the scan; `[]` would mean "match nothing". + const { layer, calls } = tiledLayer({}); + + await tileLayerOf(layer).props.getTileData(tileProps); + + expect(calls[0]?.featureCodes).toBeUndefined(); + }); + + it('refetches tiles when the selection changes', () => { + // Without the selection in the trigger, deck keeps serving the cached tiles it + // fetched for the PREVIOUS selection — the filter would appear to do nothing. + const before = tileLayerOf(tiledLayer({ featureCodes: [3] }).layer); + const after = tileLayerOf(tiledLayer({ featureCodes: [3, 7] }).layer); + + expect(before.props.updateTriggers.getTileData).not.toEqual( + after.props.updateTriggers.getTileData + ); + }); +}); diff --git a/packages/layers/tests/pointsResourceIdentity.spec.ts b/packages/layers/tests/pointsResourceIdentity.spec.ts index 17d7e5b0..b9600916 100644 --- a/packages/layers/tests/pointsResourceIdentity.spec.ts +++ b/packages/layers/tests/pointsResourceIdentity.spec.ts @@ -343,3 +343,93 @@ describe('base render resource — getBaseResource (P2)', () => { expect(engine.getBaseRevision('pts')).toBe(0); }); }); + +describe('tiled render resource — getTiledResource (D5)', () => { + const tilingMetadata = (over: Record = {}) => + ({ + kind: 'morton-points', + parquetPath: 'points/pts/points.parquet', + axisNames: ['x', 'y'], + featureCodeColumnName: 'feature_name_codes', + mortonCodeColumnName: 'morton_code_2d', + totalRows: 1_000_000, + totalRowGroups: 16, + maxRowsPerGroup: 65_536, + supportsRowGroupRangeReads: true, + bounds: { minX: 0, minY: 0, maxX: 100, maxY: 100 }, + ...over, + }) as never; + + const tiledElement = (metadata: unknown) => + ({ + key: 'pts', + loadPoints: vi.fn(async () => batch(3)), + getPointsTilingMetadata: vi.fn(async () => metadata), + loadPointsInBounds: vi.fn(async () => ({ + shape: [2, 2], + data: [new Float32Array([1, 2]), new Float32Array([3, 4])], + bounds: { minX: 0, minY: 0, maxX: 10, maxY: 10 }, + loadMode: 'row-groups', + })), + }) as unknown as PointsElement; + + // The tile-path equivalent of the pan-flash guard, and a harder requirement: a new + // resource identity does not merely rebuild a batch, it makes `TileLayer` refetch + // EVERY visible tile. A pan would become a full reload. + it('is identity-stable across repeated reads', async () => { + const engine = new PointsDataEngine(); + const element = tiledElement(tilingMetadata()); + + await engine.ensureTilingMetadata({ key: 'pts', layerId: 'l', element }); + + const first = engine.getTiledResource(element, 'pts'); + expect(first).not.toBeNull(); + for (let i = 0; i < 10; i++) { + expect(engine.getTiledResource(element, 'pts')).toBe(first); + } + }); + + it('resolves to the morton-tiled encoding, reading the viewport rather than a batch', async () => { + const engine = new PointsDataEngine(); + const element = tiledElement(tilingMetadata()); + + await engine.ensureTilingMetadata({ key: 'pts', layerId: 'l', element }); + const resource = engine.getTiledResource(element, 'pts'); + + expect(resource?.loader.capabilities.kind).toBe('morton-tiled'); + expect(resource?.loader.capabilities.supportsViewportTiles).toBe(true); + expect(resource?.loader.capabilities.bounds).toEqual({ + minX: 0, + minY: 0, + maxX: 100, + maxY: 100, + }); + }); + + it('is null for an element the probe rejected, and for one never probed', async () => { + const engine = new PointsDataEngine(); + const element = tiledElement(tilingMetadata({ supportsRowGroupRangeReads: false })); + + // Never probed: no answer, so no resource. + expect(engine.getTiledResource(element, 'pts')).toBeNull(); + + await engine.ensureTilingMetadata({ key: 'pts', layerId: 'l', element }); + + expect(engine.isTiled('pts')).toBe(false); + expect(engine.getTiledResource(element, 'pts')).toBeNull(); + }); + + it('drops the resource on evict, so a reloaded element does not reuse a dead loader', async () => { + const engine = new PointsDataEngine(); + const element = tiledElement(tilingMetadata()); + + await engine.ensureTilingMetadata({ key: 'pts', layerId: 'l', element }); + const first = engine.getTiledResource(element, 'pts'); + engine.evict('pts'); + await engine.ensureTilingMetadata({ key: 'pts', layerId: 'l', element }); + + const second = engine.getTiledResource(element, 'pts'); + expect(second).not.toBeNull(); + expect(second).not.toBe(first); + }); +}); diff --git a/packages/layers/tests/pointsScatterSizing.spec.ts b/packages/layers/tests/pointsScatterSizing.spec.ts index 70b14ec5..094a4187 100644 --- a/packages/layers/tests/pointsScatterSizing.spec.ts +++ b/packages/layers/tests/pointsScatterSizing.spec.ts @@ -1,6 +1,6 @@ import { Matrix4 } from '@math.gl/core'; import { describe, expect, it } from 'vitest'; -import { modelMatrixUniformScale } from '../src/pointsScatterLayer.js'; +import { modelMatrixUniformScale, renderColumnarScatterLayer } from '../src/pointsScatterLayer.js'; /** * Point size is expressed in the ELEMENT's coordinate units. Deck applies @@ -46,3 +46,43 @@ describe('modelMatrixUniformScale', () => { expect(modelMatrixUniformScale(new Matrix4().scale([0, 0, 0]))).toBe(1); }); }); + +/** + * The Morton tile path used to size in fixed PIXELS (`radiusUnits: 'pixels'`, + * transform scale forced to 1) on the reasoning that tiles are already + * viewport-bounded. Two consequences, both user-visible: `pointSize` meant + * something different depending on a checkbox, and a zoomed-out tiled layer drew + * every one of its millions of points as a fixed screen dot — density saturated to + * a flat mass, and every tile seam and acquisition boundary hardened into what + * looked like a rendering fault. + */ +describe('columnar scatter sizing is one behaviour, not two', () => { + const batch = { + format: 'columnar-ndarray' as const, + shape: [2, 3], + data: [new Float32Array([0, 1, 2]), new Float32Array([0, 1, 2])], + pointCount: 3, + }; + + it('sizes in world units and folds in the model-matrix scale', () => { + const layer = renderColumnarScatterLayer('scatter', batch, { + pointSize: 2, + modelMatrix: new Matrix4().scale([4, 4, 1]), + }); + + expect(layer.props.radiusUnits).toBe('common'); + expect(layer.props.radiusScale).toBeCloseTo(8, 9); + }); + + it('does not change because the batch came from a tile', () => { + const common = { pointSize: 2, modelMatrix: new Matrix4().scale([4, 4, 1]) }; + const plain = renderColumnarScatterLayer('plain', batch, common); + const tiled = renderColumnarScatterLayer('tiled', batch, { + ...common, + tileBounds: [0, 0, 10, 10], + }); + + expect(tiled.props.radiusUnits).toBe(plain.props.radiusUnits); + expect(tiled.props.radiusScale).toBe(plain.props.radiusScale); + }); +}); diff --git a/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx b/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx index f6e537a4..71f6117a 100644 --- a/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx +++ b/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx @@ -1,4 +1,8 @@ -import { featureNamesForCodes, resolveFeatureSelectionCodes } from '@spatialdata/core'; +import { + featureNamesForCodes, + pointsTilingEnabled, + resolveFeatureSelectionCodes, +} from '@spatialdata/core'; import { featureCodeToRgb } from '@spatialdata/layers'; import type { CSSProperties } from 'react'; import { useEffect, useMemo, useRef, useState } from 'react'; @@ -178,6 +182,7 @@ export function PointsFeatureFilterPanel({ config }: PointsFeatureFilterPanelPro catalogLoading, catalogRefining, residentCodes, + tiled, loadedMatchingCodes, supportsOnDemandLoad, matchingLoadState, @@ -382,6 +387,9 @@ export function PointsFeatureFilterPanel({ config }: PointsFeatureFilterPanelPro // rendered — not the current scan's settled state — keeps already-loaded // features un-greyed while a newly added feature's scan is still in flight. const residentKnown = residentCodes !== undefined; + // Element fact AND this layer's config — the probe's answer is cached per element + // and outlives the config that asked for it. + const tiledLayer = tiled && pointsTilingEnabled(config.pointsTiling); const scanning = matchingLoadState?.loading ?? false; const rowInfo = (code: number) => { const resident = residentKnown && (residentCodes?.has(code) ?? false); @@ -396,6 +404,7 @@ export function PointsFeatureFilterPanel({ config }: PointsFeatureFilterPanelPro residentKnown, residentPointCount: residentFeatureCounts?.get(code), datasetPointCount: datasetCountByCode.get(code), + tiled: tiledLayer, }); return { resident, rendered, selected, state }; }; diff --git a/packages/vis/src/SpatialCanvas/PointsFeatureState.tsx b/packages/vis/src/SpatialCanvas/PointsFeatureState.tsx index 3cf8b447..3faa8641 100644 --- a/packages/vis/src/SpatialCanvas/PointsFeatureState.tsx +++ b/packages/vis/src/SpatialCanvas/PointsFeatureState.tsx @@ -123,6 +123,18 @@ export interface PointsFeatureState { /** Truncation of what's on screen for the selection passed to the hook (so the * UI can show when raising the memory cap would load more). */ truncation: ReturnType; + /** + * Whether the ELEMENT has usable Morton tiling metadata. Combine with the layer's + * own `pointsTiling` before concluding this layer draws tiles — the probe's answer + * is cached per element and outlives the config that asked for it. + * + * It belongs on this hook rather than a direct `engine.isTiled(...)` read because + * the probe settles ASYNCHRONOUSLY: a component reading the engine outside this + * subscription shows whatever was true when it last happened to render, which is + * how the tiling control kept saying "this element has no Morton index" about an + * element it was already tiling. + */ + tiled: boolean; /** Running per-feature counts over the resident window (`code → rows`), available * while the whole-dataset counts scan is still running. Partial by construction. */ residentFeatureCounts: ReturnType; @@ -148,6 +160,7 @@ const EMPTY_POINTS_FEATURE_STATE: Omit< supportsOnDemandLoad: false, matchingLoadState: undefined, truncation: undefined, + tiled: false, residentFeatureCounts: undefined, }; @@ -211,6 +224,7 @@ export function usePointsFeatureState( matchingLoadState: hasSelection && scannable ? engine.getMatchingLoadState(key, featureCodes) : undefined, truncation: engine.getActiveTruncation(key, featureCodes), + tiled: engine.isTiled(key), residentFeatureCounts: engine.getResidentFeatureCounts(key), requestCatalog, setHighlightedFeature, diff --git a/packages/vis/src/SpatialCanvas/PointsLayerPanel.tsx b/packages/vis/src/SpatialCanvas/PointsLayerPanel.tsx index 1032c470..9ce39c04 100644 --- a/packages/vis/src/SpatialCanvas/PointsLayerPanel.tsx +++ b/packages/vis/src/SpatialCanvas/PointsLayerPanel.tsx @@ -1,4 +1,4 @@ -import { DEFAULT_POINTS_MEMORY_CAP } from '@spatialdata/core'; +import { DEFAULT_POINTS_MEMORY_CAP, pointsTilingEnabled } from '@spatialdata/core'; import type { PointsDataEngine, PointsLoadTarget } from '@spatialdata/layers'; import { useSpatialCanvasActions } from './context'; import { PointsFeatureFilterPanel } from './PointsFeatureFilterPanel'; @@ -16,8 +16,17 @@ export interface PointsLayerPanelProps { } function PointsMemoryCap({ config }: { config: PointsLayerConfig }) { + // Opt out of the React Compiler — see PointsFeatureFilterPanel. The tiled read is + // engine-backed and settles asynchronously (the probe), so the compiler would + // memoize this JSX and leave a dead control on screen. + 'use no memo'; const actions = useSpatialCanvasActions(); + const { tiled } = usePointsFeatureState(config); const currentCap = config.pointsMemoryCap ?? DEFAULT_POINTS_MEMORY_CAP; + // A tiled layer holds no resident window, so the cap governs nothing. Leaving the + // control up would put "Max rows kept in memory" directly above "the memory cap + // does not apply" — each true, the pair nonsense. Same rule as ShowMatchingPoints. + if (tiled && pointsTilingEnabled(config.pointsTiling)) return null; // Discrete options (one reload per choice, vs. a free number // input that would reload on every keystroke). Include the // current value so a saved config off the preset list still @@ -73,7 +82,12 @@ function ShowMatchingPoints({ config }: { config: PointsLayerConfig }) { // read is engine-backed and updates on notify; the compiler would otherwise // memoize this line's JSX and never repaint it as the scan progresses. 'use no memo'; - const { truncation: t } = usePointsFeatureState(config); + const { truncation: t, tiled } = usePointsFeatureState(config); + // A tiled layer draws from the viewport, not from a resident window, so a + // truncation count is not a statement about what is on screen. It used to sit + // directly above "the memory cap does not apply", each true and the pair + // nonsense — and it survives eviction lag, since this renders before the release. + if (tiled && pointsTilingEnabled(config.pointsTiling)) return null; if (!t) return null; // Report the batch held in memory (always true), NOT a per-selection matched // count: t.loaded is the covered-batch size, which overstates the selection @@ -127,12 +141,72 @@ function PointSizeControl({ config }: { config: PointsLayerConfig }) { ); } +/** + * Morton viewport tiling (D5), and its tile-status overlay. + * + * **On by default** since step 7, for elements that have a usable Morton index. Tiles + * colour by feature, honour the feature filter (applied inside the row-group scan, so + * a filtered tile arrives small), and subdivide with zoom. Turning it off re-plans + * back to the capped preload — worth offering, because the preload keeps the first + * `cap` rows in FILE order, which on a Morton artifact is a prefix of the Z-curve: a + * skewed chunk of the slide rather than a sample of it. That comparison is exactly + * what the toggle is for. + */ +function PointsTilingControl({ config }: { config: PointsLayerConfig }) { + // Opt out of the React Compiler — see PointsFeatureFilterPanel. The tiling read is + // engine-backed and settles asynchronously (the probe), so the compiler would + // memoize this JSX and never show the layer switching over to tiles. + 'use no memo'; + const actions = useSpatialCanvasActions(); + const enabled = pointsTilingEnabled(config.pointsTiling); + // Through the hook, NOT `engine.isTiled(...)` directly: the hook carries the + // engine subscription, so this line updates when the probe settles instead of + // showing whatever was true at the last unrelated render. + const { tiled } = usePointsFeatureState(config); + return ( +
+ + {enabled && ( + + )} + + {enabled + ? tiled + ? 'Reading row groups for the viewport, at the zoom you are at — no memory cap applies. The feature filter is applied as tiles are read.' + : 'This element has no usable Morton index; using the capped preload.' + : 'Off: showing the first rows of the file up to the memory cap, whatever the viewport. Turn on to read only what you are looking at.'} + +
+ ); +} + export default function PointsLayerPanel({ config, engine, resolveTarget }: PointsLayerPanelProps) { return ( + ); diff --git a/packages/vis/src/SpatialCanvas/featureRowState.ts b/packages/vis/src/SpatialCanvas/featureRowState.ts index 6b4f2ba1..f19c9b08 100644 --- a/packages/vis/src/SpatialCanvas/featureRowState.ts +++ b/packages/vis/src/SpatialCanvas/featureRowState.ts @@ -11,6 +11,7 @@ export type FeatureRowTone = | 'resident' | 'partial' + | 'tiled' | 'loaded' | 'cached' | 'loading' @@ -52,6 +53,18 @@ export interface FeatureRowStateInput { */ residentPointCount?: number; datasetPointCount?: number; + /** + * The layer reads viewport tiles rather than a resident window (D5). + * + * It has to be said explicitly, because every OTHER signal here describes a + * resident batch that a tiled layer does not have: `resident` is false for every + * feature, `residentKnown` is false, and the fallback that produces —"the resident + * set is unknown, so treat everything as shown" — is accidentally the right + * *outcome* for the wrong *reason*. On a tiled layer coverage is not unknown: every + * feature in view is read on demand, and deselecting one drops its points inside + * the scan rather than filtering a batch afterwards. + */ + tiled?: boolean; } /** @@ -73,7 +86,19 @@ export function describeFeatureRowState({ residentKnown, residentPointCount, datasetPointCount, + tiled, }: FeatureRowStateInput): FeatureRowState { + // Ranked first: a tiled layer's coverage does not depend on any of the resident + // signals below, and answering from them would describe a batch it does not have. + if (tiled) { + return { + tone: 'tiled', + greyed: false, + label: 'in view', + reason: + 'Read from viewport tiles on demand — not limited by the memory cap. Deselecting it drops its points from the tiles before they are drawn.', + }; + } if (!residentKnown) { return { tone: 'loaded', diff --git a/packages/vis/src/SpatialCanvas/pointsTileProgress.ts b/packages/vis/src/SpatialCanvas/pointsTileProgress.ts new file mode 100644 index 00000000..fdee9283 --- /dev/null +++ b/packages/vis/src/SpatialCanvas/pointsTileProgress.ts @@ -0,0 +1,100 @@ +import { + type PointsTileLoadProgress, + type TileDebugStore, + type TiledPointsDebugState, + tileDebugEntriesSignature, +} from '@spatialdata/layers'; + +export type { PointsTileLoadProgress }; + +/** + * A `PointsLayer` `tileDebugSignature` prop: changes whenever the store's tile + * entries do, which is what makes the debug overlay redraw. The store itself is + * identity-stable per layer, so without this the overlay's `updateTriggers` would + * never fire. + */ +export function tileDebugSignature(store: TileDebugStore | undefined): string { + return tileDebugEntriesSignature(store?.getState().tileDebugEntries ?? []); +} + +export function emptyPointsTileLoadProgress(): PointsTileLoadProgress { + return { inFlight: 0, loaded: 0, loadedPoints: 0, viewportTotal: 0 }; +} + +export function pointsTileLoadProgressFromDebugState( + state: TiledPointsDebugState | undefined +): PointsTileLoadProgress { + if (!state) { + return emptyPointsTileLoadProgress(); + } + + const viewportTileIds = new Set((state.lastViewportTiles ?? []).map((tile) => tile.tileId)); + const loadingTileIds = new Set(state.loadingTileIds ?? []); + const completedTilesById = state.completedTilesById ?? {}; + + let inFlight = 0; + let loaded = 0; + let loadedPoints = 0; + for (const tileId of viewportTileIds) { + if (loadingTileIds.has(tileId)) { + inFlight += 1; + } + const completed = completedTilesById[tileId]; + if (completed?.status === 'loaded' || completed?.status === 'empty') { + loaded += 1; + loadedPoints += completed.pointCount ?? 0; + } + } + + return { + inFlight, + loaded, + loadedPoints, + viewportTotal: viewportTileIds.size, + }; +} + +export function pointsTileLoadProgressFromStore( + store: TileDebugStore | undefined +): PointsTileLoadProgress { + return pointsTileLoadProgressFromDebugState(store?.getState()); +} + +export function aggregatePointsTileLoadProgress( + progressByLayer: ReadonlyMap +): PointsTileLoadProgress { + let inFlight = 0; + let loaded = 0; + let loadedPoints = 0; + let viewportTotal = 0; + for (const progress of progressByLayer.values()) { + inFlight += progress.inFlight; + loaded += progress.loaded; + loadedPoints += progress.loadedPoints; + viewportTotal += progress.viewportTotal; + } + return { inFlight, loaded, loadedPoints, viewportTotal }; +} + +function formatLoadedPointCount(pointCount: number): string { + return pointCount.toLocaleString(); +} + +export function pointsTileLoadingMessage(progress: PointsTileLoadProgress): string | null { + const { inFlight, loaded, loadedPoints, viewportTotal } = progress; + if (inFlight <= 0) { + return null; + } + const pointsSuffix = loaded > 0 ? `, ${formatLoadedPointCount(loadedPoints)} points` : ''; + const message = + viewportTotal > 0 + ? `Loading points… (${loaded}/${viewportTotal} tiles${pointsSuffix})` + : inFlight > 0 + ? 'Loading points…' + : null; + return message; +} + +export function isPointsTileLoading(progress: PointsTileLoadProgress): boolean { + return pointsTileLoadingMessage(progress) !== null; +} diff --git a/packages/vis/src/SpatialCanvas/types.ts b/packages/vis/src/SpatialCanvas/types.ts index 19e0b229..a26fdff1 100644 --- a/packages/vis/src/SpatialCanvas/types.ts +++ b/packages/vis/src/SpatialCanvas/types.ts @@ -3,7 +3,7 @@ */ import type { Matrix4 } from '@math.gl/core'; -import type { SpatialElement } from '@spatialdata/core'; +import type { PointsTilingMode, SpatialElement } from '@spatialdata/core'; import type { FeatureCategoricalPaletteSpec, FeatureMissingValueOptions, @@ -202,6 +202,27 @@ export interface PointsLayerConfig extends BaseLayerConfig { * Serializable Stack-Entry state. */ featureColorOverrides?: Record; + /** + * Whether to use the Morton-tiled viewport path when the element supports it + * (D5). `'auto'` probes the element's parquet footer once and, if it is a Morton + * artifact whose store can serve row-group range reads, renders viewport tiles + * instead of a capped resident preload — so a dataset far larger than the memory + * cap can be explored at full detail. `'off'` always preloads. + * + * Defaults to `'auto'` (`DEFAULT_POINTS_TILING`) — resolve it with + * `pointsTilingEnabled` rather than comparing to `'auto'`, so `undefined` means the + * same thing everywhere. + * + * Serializable Stack-Entry state. An element that cannot be tiled falls back to + * the preload either way, so this is a preference, not a requirement. + */ + pointsTiling?: PointsTilingMode; + /** + * Draw the tile-status debug overlay on a tiled points layer (viewport tiles + * coloured by loading / loaded / empty / error). Development affordance; ignored + * on the preloaded path. + */ + showTileDebugOverlay?: boolean; } export interface LabelsLayerConfig extends BaseLayerConfig { diff --git a/packages/vis/src/SpatialCanvas/useLayerData.ts b/packages/vis/src/SpatialCanvas/useLayerData.ts index ef94b5a8..341a6b75 100644 --- a/packages/vis/src/SpatialCanvas/useLayerData.ts +++ b/packages/vis/src/SpatialCanvas/useLayerData.ts @@ -21,6 +21,7 @@ import { getTooltipSignature, type LabelsElement, type PointsElement, + pointsTilingEnabled, resolveFeatureSelectionCodes, resolvePointsMemoryCap, resolveTooltipItems, @@ -30,11 +31,13 @@ import { type SpatialData, SpatialEntryStore, type SpatialFeatureTooltipData, + transformAxisAlignedBounds, unionBoundsList, } from '@spatialdata/core'; import { buildShapeFillColorByFeatureId, buildShapesPrebuiltData, + createTileDebugStore, featureFilterAwaitingRowCodes, PointsDataEngine, PointsLayer, @@ -45,6 +48,7 @@ import { resolveShapeTooltipRowIndex, type ShapeFeatureRenderDatum, type ShapeFeatureStateRuntime, + type TileDebugStore, } from '@spatialdata/layers'; import type { Layer } from 'deck.gl'; import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; @@ -58,6 +62,13 @@ import { type LabelFillColorEntry, lastGoodLabelFillColorEntry, } from './labelsProjection'; +import { + aggregatePointsTileLoadProgress, + isPointsTileLoading, + type PointsTileLoadProgress, + pointsTileLoadProgressFromStore, + tileDebugSignature, +} from './pointsTileProgress'; import { renderLabelsLayer } from './renderers/labelsRenderer'; import { renderShapesLayer } from './renderers/shapesRenderer'; import { describeResolveInputs } from './resolveInputs'; @@ -78,6 +89,7 @@ import type { ElementsByType, LabelsLayerConfig, LayerConfig, + PointsLayerConfig, ShapesLayerConfig, } from './types'; import { useVivLoaderRegistry } from './VivLoaderRegistry'; @@ -356,6 +368,22 @@ export function getCachedWorldBounds( * 3. Caches loaded data * 4. Produces deck.gl Layer instances with the loaded data */ +/** + * Does this layer draw through the Morton tile path (D5)? + * + * BOTH halves are required. `isTiled` is a fact about the ELEMENT — the probe's + * answer, cached per element key — and it outlives the config that asked for it, so a + * layer that reads it alone keeps rendering tiles after the user turns tiling off (and + * two layers on one element cannot disagree). The config is the per-layer half. + */ +function usesTiledPath( + engine: PointsDataEngine, + elementKey: string, + config: PointsLayerConfig | undefined +): boolean { + return pointsTilingEnabled(config?.pointsTiling) && engine.isTiled(elementKey); +} + export function useLayerData( layers: Record, layerOrder: string[], @@ -514,6 +542,25 @@ export function useLayerData( }) ); + // Per-layer tile-status stores for the Morton path (D5). One per points layer, + // created on first render of a tiled layer and kept for its lifetime — the store is + // a `PointsLayer` prop, so a fresh one per render would churn the debug overlay. + // Its `update` no-ops when the state signature is unchanged, so the notify below + // fires on real tile transitions only, not on every deck frame. + const tileDebugStoresRef = useRef(new Map()); + const getTileDebugStore = useCallback( + (layerId: string): TileDebugStore => { + const stores = tileDebugStoresRef.current; + let store = stores.get(layerId); + if (!store) { + store = createTileDebugStore(notifyLoadedDataChanged); + stores.set(layerId, store); + } + return store; + }, + [notifyLoadedDataChanged] + ); + // Shapes / images / labels Resource Resolvers (ADR 0004). Shapes lives in `core`, // images/labels in `vis` (next to Viv/avivatorish) — the store below holds only // `ResourceResolver`s and cannot tell which package each came from. Each rebuilds @@ -701,6 +748,11 @@ export function useLayerData( return codes ? { featureCodes: codes } : {}; })(), ...(config.colorByFeature ? { colorByFeature: true } : {}), + // D5: opting in makes `plan()` probe for a Morton artifact before it + // commits to a full-table preload, and render viewport tiles when there + // is one. Off by default; an element that cannot be tiled preloads either + // way. + ...(config.pointsTiling ? { pointsTiling: config.pointsTiling } : {}), }, transform: elem.transform, }); @@ -929,7 +981,19 @@ export function useLayerData( return shapesResolver.getRenderData(elem.key) !== undefined; } if (elem.type === 'points') { - return pointsEngine.hasData(elem.key); + // A tiled element is renderable with no resident batch at all: its extent is + // known from the artifact and its geometry arrives per viewport tile. Reading + // only `hasData` would report "nothing to draw" forever and leave the + // blocking overlay up over a layer that is drawing fine. + const pointsConfig = layersRef.current[layerId]; + return ( + pointsEngine.hasData(elem.key) || + usesTiledPath( + pointsEngine, + elem.key, + pointsConfig?.type === 'points' ? pointsConfig : undefined + ) + ); } if (elem.type === 'image') { return imagesResolver.getLoadedData(elem.key) !== undefined; @@ -988,6 +1052,24 @@ export function useLayerData( ); } if (elem.type === 'points') { + // A tiled element is framed from the artifact's own extent — there is no + // resident geometry to measure, and waiting for one would mean never + // auto-fitting. Cached against the metadata's identity, like the preloaded + // path is against its batch. + const tilingMetadata = + config.type === 'points' && usesTiledPath(pointsEngine, elem.key, config) + ? pointsEngine.getTilingMetadata(elem.key) + : undefined; + if (tilingMetadata?.bounds) { + const elementBounds = tilingMetadata.bounds; + return getCachedWorldBounds( + loaded.worldBounds, + getWorldBoundsCacheKey(elem), + tilingMetadata, + elem.transform, + () => transformAxisAlignedBounds(elementBounds, elem.transform) + ); + } const pointData = pointsEngine.getData(elem.key); if (!pointData) return null; return getCachedWorldBounds( @@ -1118,6 +1200,73 @@ export function useLayerData( if (Array.isArray(layer)) deckLayers.push(...layer); else if (layer) deckLayers.push(layer); } + } else if (config.type === 'points' && usesTiledPath(pointsEngine, elem.key, config)) { + // --- Morton viewport tiles (D5 step 2) -------------------------------- + // + // A separate branch, not a variation of the preloaded one below: the tiled + // path has no resident batch, so none of the resident/matched/partial + // machinery below applies to it. Its geometry comes from `loadInBounds` per + // viewport tile, inside deck's own `TileLayer` lifecycle. + // + // Colour comes off the tile batch itself: the scan returns a feature code + // per point (step 3), so the same colour props the preloaded path uses + // apply here. + // + // The FILTER is pushed down into the row-group scan rather than applied to + // a batch already in memory, so a tile arrives holding only the selected + // features — 36x fewer points for one gene on a real transcripts element, + // never uploaded and never drawn. + // + // It does NOT narrow the fetch. Measured on that element: selecting one + // gene read the same 92 row groups and the same 158MB as the unfiltered + // view. Row groups are chosen SPATIALLY on a Morton artifact and a gene's + // points are spread across all of them, so only a feature-primary index + // could skip any (the open index-permutation question in ADR 0002/0003). + const element = elem.element as PointsElement; + // `undefined` means "no filter"; an empty array means "filter to nothing". + // Both reach `loadInBounds` unchanged and the scan honours the distinction. + const tiledFeatureCodes = resolveFeatureSelectionCodes( + config, + pointsEngine.getFeatureCatalog(elem.key) + ); + const tiledResource = pointsEngine.getTiledResource(element, elem.key); + if (tiledResource) { + const showTileDebugOverlay = config.showTileDebugOverlay === true; + const tileDebugStore = getTileDebugStore(layerId); + // The same three colour inputs the preloaded branch builds. They are + // element-scoped (catalog code space, name→rgb overrides, the runtime + // hover highlight), so a tiled layer reads them identically — only the + // per-point codes arrive by a different route. + const featureCodeSpaceSize = pointsEngine.getFeatureCodeSpaceSize(elem.key); + const featureColorOverrides = pointsEngine.getFeatureColorOverrideMap( + elem.key, + config.featureColorOverrides + ); + const highlightFeatureCode = pointsEngine.getHighlightedFeature(elem.key); + deckLayers.push( + new PointsLayer({ + id: layerId, + resource: tiledResource, + modelMatrix: elem.transform, + opacity: config.opacity, + visible: config.visible, + pointSize: config.pointSize ?? 1, + ...(config.color ? { color: config.color } : {}), + // Always pass the store — it is what the footer's tile progress + // reads, whether or not the overlay is drawn. + tileDebugStore, + showTileDebugOverlay, + ...(showTileDebugOverlay + ? { tileDebugSignature: tileDebugSignature(tileDebugStore) } + : {}), + ...(tiledFeatureCodes ? { featureCodes: tiledFeatureCodes } : {}), + ...(config.colorByFeature ? { colorByFeature: true } : {}), + featureCodeSpaceSize, + ...(featureColorOverrides ? { featureColorOverrides } : {}), + highlightFeatureCode, + }) + ); + } } else if (config.type === 'points') { const element = elem.element as PointsElement; const featureCodes = resolveFeatureSelectionCodes( @@ -1387,6 +1536,7 @@ export function useLayerData( layerOrder, getStableSelections, pointsEngine, + getTileDebugStore, getMergedShapeRenderData, getShapeFillColorEntry, getShapePrebuilt, @@ -1697,12 +1847,32 @@ export function useLayerData( return vivProps; }, [layers, layerOrder, getStableSelections, vivPassthrough, imagesResolver]); + /** + * Viewport-tile progress across every tiled points layer (D5). + * + * Tiles load inside deck's `TileLayer`, not through a resolver, so they never touch + * `layerLoadStates` — without this a tiled layer reports "nothing loading" while it + * is fetching row groups. Recomputed on `loadedDataRevision`, which the tile debug + * stores bump when a tile actually starts or finishes. + */ + const pointsTileProgress = useMemo((): PointsTileLoadProgress => { + // Bare reference: the stores are read through a ref (they must not re-create per + // render), so the revision is the only thing that CHANGES when a tile transitions. + void loadedDataRevision; + const byLayer = new Map(); + // eslint-disable-next-line react-hooks/refs -- intentional external-store read; see the isBlocking note below + for (const [layerId, store] of tileDebugStoresRef.current) { + byLayer.set(layerId, pointsTileLoadProgressFromStore(store)); + } + return aggregatePointsTileLoadProgress(byLayer); + }, [loadedDataRevision]); + const isLoading = useMemo( () => Object.values(layerLoadStates).some((state) => Object.values(state).some((status) => status === 'loading') - ), - [layerLoadStates] + ) || isPointsTileLoading(pointsTileProgress), + [layerLoadStates, pointsTileProgress] ); const isBlocking = useMemo( diff --git a/packages/vis/tests/pointsFeatureRowState.spec.ts b/packages/vis/tests/pointsFeatureRowState.spec.ts index 80c2fe06..5654a166 100644 --- a/packages/vis/tests/pointsFeatureRowState.spec.ts +++ b/packages/vis/tests/pointsFeatureRowState.spec.ts @@ -114,3 +114,56 @@ describe('describeFeatureRowState — partial residency', () => { ); }); }); + +describe('a tiled layer has no resident window to describe', () => { + const base = { + resident: false, + rendered: false, + selected: true, + scanning: false, + supportsOnDemandLoad: true, + residentKnown: false, + }; + + /** + * Every other signal here describes a resident batch a tiled layer does not have, + * so without saying so explicitly its rows fall through to "beyond the resident + * window; select it to fetch its points" — greyed, and wrong twice over: the points + * ARE available, and no feature-index scan is involved. + */ + it('reads as available rather than beyond the window', () => { + const tiled = describeFeatureRowState({ ...base, tiled: true }); + + expect(tiled.tone).toBe('tiled'); + expect(tiled.greyed).toBe(false); + expect(tiled.reason).toMatch(/viewport tiles/i); + }); + + it('is not greyed even when it is deselected and nothing is resident', () => { + const tiled = describeFeatureRowState({ ...base, selected: false, tiled: true }); + + expect(tiled.greyed).toBe(false); + }); + + it('outranks the resident-unknown fallback, which is right by accident', () => { + // Same inputs, no `tiled`: not greyed either, but for the wrong reason — it + // claims coverage is unknown, when for a tiled layer it is known and complete. + const untiled = describeFeatureRowState(base); + + expect(untiled.greyed).toBe(false); + expect(untiled.reason).toMatch(/unknown/i); + expect(untiled.tone).not.toBe('tiled'); + }); + + it('does not leak into a preloaded layer', () => { + const preloaded = describeFeatureRowState({ + ...base, + residentKnown: true, + resident: false, + selected: false, + }); + + expect(preloaded.tone).toBe('notLoaded'); + expect(preloaded.greyed).toBe(true); + }); +}); diff --git a/packages/vis/tests/pointsTileProgress.spec.ts b/packages/vis/tests/pointsTileProgress.spec.ts new file mode 100644 index 00000000..e2db2f25 --- /dev/null +++ b/packages/vis/tests/pointsTileProgress.spec.ts @@ -0,0 +1,130 @@ +import type { PointsTileHandle, TiledPointsDebugState } from '@spatialdata/layers'; +import { describe, expect, it } from 'vitest'; + +import { + aggregatePointsTileLoadProgress, + isPointsTileLoading, + pointsTileLoadingMessage, + pointsTileLoadProgressFromDebugState, +} from '../src/SpatialCanvas/pointsTileProgress.js'; + +const sampleTile: PointsTileHandle = { + tileId: '0-0--1', + index: { x: 0, y: 0, z: -1 }, + bbox: { left: 0, top: 512, right: 512, bottom: 0 }, +}; + +const otherTile: PointsTileHandle = { + tileId: '1-0--1', + index: { x: 1, y: 0, z: -1 }, + bbox: { left: 512, top: 512, right: 1024, bottom: 0 }, +}; + +function debugState(overrides: Partial): TiledPointsDebugState { + return { + tileDebugEntries: [], + completedTilesById: {}, + loadingTileIds: [], + tileHandlesById: {}, + ...overrides, + }; +} + +describe('pointsTileProgress', () => { + it('aggregates progress across layers', () => { + const aggregate = aggregatePointsTileLoadProgress( + new Map([ + ['a', { inFlight: 2, loaded: 1, loadedPoints: 100, viewportTotal: 4 }], + ['b', { inFlight: 1, loaded: 3, loadedPoints: 250, viewportTotal: 6 }], + ]) + ); + expect(aggregate).toEqual({ + inFlight: 3, + loaded: 4, + loadedPoints: 350, + viewportTotal: 10, + }); + }); + + it('reports loading while tiles are in flight', () => { + expect( + pointsTileLoadingMessage({ inFlight: 2, loaded: 1, loadedPoints: 42, viewportTotal: 6 }) + ).toBe('Loading points… (1/6 tiles, 42 points)'); + expect( + isPointsTileLoading({ inFlight: 2, loaded: 1, loadedPoints: 42, viewportTotal: 6 }) + ).toBe(true); + }); + + it('includes zero loaded points while later tiles are still loading', () => { + expect( + pointsTileLoadingMessage({ inFlight: 1, loaded: 1, loadedPoints: 0, viewportTotal: 2 }) + ).toBe('Loading points… (1/2 tiles, 0 points)'); + }); + + it('clears stale viewport messages when nothing is in flight', () => { + expect( + pointsTileLoadingMessage({ inFlight: 0, loaded: 0, loadedPoints: 0, viewportTotal: 4 }) + ).toBeNull(); + expect( + pointsTileLoadingMessage({ inFlight: 0, loaded: 4, loadedPoints: 900, viewportTotal: 4 }) + ).toBeNull(); + }); + + it('derives loaded and point totals from current viewport debug state', () => { + const progress = pointsTileLoadProgressFromDebugState( + debugState({ + lastViewportTiles: [sampleTile, otherTile], + loadingTileIds: [otherTile.tileId], + completedTilesById: { + [sampleTile.tileId]: { + status: 'loaded', + pointCount: 10, + clippedBounds: null, + completedAt: 10, + }, + }, + }) + ); + expect(progress).toEqual({ inFlight: 1, loaded: 1, loadedPoints: 10, viewportTotal: 2 }); + }); + + it('counts cached empty tiles as loaded after viewport refresh', () => { + const progress = pointsTileLoadProgressFromDebugState( + debugState({ + lastViewportTiles: [sampleTile], + completedTilesById: { + [sampleTile.tileId]: { + status: 'empty', + pointCount: 0, + clippedBounds: null, + completedAt: 10, + }, + }, + }) + ); + expect(progress).toEqual({ inFlight: 0, loaded: 1, loadedPoints: 0, viewportTotal: 1 }); + }); + + it('does not let stale completed tiles inflate the current viewport total', () => { + const progress = pointsTileLoadProgressFromDebugState( + debugState({ + lastViewportTiles: [sampleTile], + completedTilesById: { + [sampleTile.tileId]: { + status: 'loaded', + pointCount: 10, + clippedBounds: null, + completedAt: 10, + }, + [otherTile.tileId]: { + status: 'loaded', + pointCount: 20, + clippedBounds: null, + completedAt: 10, + }, + }, + }) + ); + expect(progress).toEqual({ inFlight: 0, loaded: 1, loadedPoints: 10, viewportTotal: 1 }); + }); +});