Points: Morton-tiled viewport loading, on by default (D5)

.changeset/morton-bisect-cost.md

@@ -0,0 +1,32 @@
+---
+"@spatialdata/core": patch
+---
+
+Points (Morton): stop the row-group bisect from doing orders of magnitude more work
+than the query needs.
+
+Two independent costs, both measured against a real 12.1M-point Xenium `transcripts`
+artifact (245 row groups) with a viewport-sized query rectangle:
+
+- **`zcoverRectangle` now stops at `MORTON_ZCOVER_MAX_DEPTH` (10).** It recursed to
+  the full 16 bits per axis, resolving the rectangle to individual quantised cells
+  when its only job is picking row groups — **38,014 intervals**, each driving two
+  bisects, to select 92 row groups. At the cap that is 521 intervals selecting the
+  **same 92 row groups**, verified over a viewport tile, the whole slide and a
+  zoomed-in box. A coarser cell can only widen the covered code range, and the rows
+  it brings in are filtered against the exact bounds after the read, so the cover
+  stays complete.
+- **Concurrent extent probes for the same row group are deduped.** The cache held the
+  settled value, so it dedups nothing while a read is in flight — and the index is
+  built under exactly that load, with every viewport tile bisecting over the same row
+  groups at once. Each tile started its own full row-group fetch for an entry the
+  others were already fetching (8 concurrent callers: 18 range reads, now 4). A
+  failed probe is evicted rather than cached, so a transient error does not strand
+  the bisect permanently.
+
+Not fixed, and now documented where it bites: the extent probe should read the row
+group's **column statistics**, which are already in the footer we parse, instead of
+range-reading and decoding the row group's bytes twice. The vendored parquet-wasm
+build exposes no statistics accessor (`RowGroupMetaData` offers only `numRows` /
+`fileOffset` / `compressedSize` / `column`), so this needs a wasm rebuild or a minimal
+Thrift read of the footer.

.changeset/morton-rowgroup-extent-max.md

@@ -0,0 +1,30 @@
+---
+"@spatialdata/core": patch
+---
+
+Points (Morton): fix viewport queries silently dropping row groups — the holes in the
+tiled render.
+
+`readParquetRowGroupColumnExtent` built the bisect index by reading a row group's
+first value and its last value, taking the last with
+`readParquetRowGroup(..., { offset: rowCount - 1, limit: 1 })`. The vendored
+parquet-wasm **ignores `offset` on a row-group read** and returned the first row
+again, so every row group reported `max === min` — claiming to span a single value.
+
+Nothing errored. The bisect asks "first row group whose max >= target", and an
+understated max moves that answer one group too far forward, so the row group that
+actually *contained* the interval start was never read. On a real 12.1M-point Xenium
+artifact one viewport query silently lost **11 of the 92 matching row groups —
+187,990 points (6%)**, which is what the Z-order-shaped holes in the tiled points
+render were.
+
+The upper bound now comes from the sort order instead: the file is sorted on this
+column, so a row group's values all lie at or below the next row group's first value.
+That is conservative (equal codes spanning a boundary keep both groups in range),
+needs only the one read that works, and halves the reads — each row group's first
+value is cached and shared with its neighbour. The last row group keeps an open
+bound.
+
+The regression test asserts an exact point count for a bounded query rather than
+"more than zero": this class of bug is silent by construction, and only a total sees
+it.

.changeset/points-morton-sentinel-guard.md

@@ -0,0 +1,30 @@
+---
+"@spatialdata/core": patch
+---
+
+Points: refuse to Morton-tile an artifact whose sentinel bounding box is not the domain
+its codes were quantised against.
+
+The sentinel rows are a claim a Parquet artifact makes about itself, and nothing else in
+the file forces them to be true. Believing a wrong one does not fail — it silently clips
+the tile grid to the bogus box, so whole regions are never even requested, and
+`mortonIntervalsForBounds` normalises viewports against it and selects the wrong row
+groups. Points are never misplaced (the reader re-filters to the query bounds), so the
+only symptom is that part of the map is missing.
+
+`getPointsTilingMetadata` now recomputes `morton_code_2d` from x/y for a sample of real
+rows and requires a majority to agree. On a real 12.1M-point element a sound artifact
+matches 320/320 sampled rows and one with a stale sentinel box matches 0/320, so the test
+is not marginal; a majority rather than an exact match tolerates coordinates landing on a
+cell boundary. A rejected element reports `supportsRowGroupRangeReads: false` with no
+`bounds` — the same pair the oversized-sentinel case already produced — so it degrades to
+the capped preload through the existing path, and warns rather than downgrading silently.
+
+Cost is one extra row-group read per element, cached with the metadata: the probe goes
+from 3 range reads / 0.37 MB / 414 ms to 4 / 2.16 MB / 523 ms on that element. The sample
+is taken from the middle of the file, because a truncated box can agree with the true one
+near the origin by coincidence but never in the interior.
+
+`mortonCode2dForPoint` / `mortonBoundsAgreeWithCodes` are exported for this, and pin the
+interleave convention (x in the even bits) that `zcoverRectangle` and the writer both
+already assumed without anything checking they stayed in step.

.changeset/points-morton-sort-guard.md

@@ -0,0 +1,30 @@
+---
+"@spatialdata/core": patch
+---
+
+Points: refuse to Morton-tile a file whose `morton_code_2d` column is not sorted.
+
+Having the column does not make a file Morton-ordered. A feature-primary artifact —
+sorted `(feature, morton)` — carries the identical column with identical, correct values,
+a correct sentinel box, and every field the tiling probe looks for. Only the order is
+wrong, and nothing in the file said so. The row-group bisect binary-searches that index
+assuming it ascends, so it landed arbitrarily and a tile came back holding whichever
+feature blocks happened to live in the row groups it picked: some tiles showed one or two
+genes, most showed none.
+
+The probe now reads the per-row-group `[min, max]` for the Morton column out of the
+parquet footer and requires it to be non-decreasing. On the permutations store,
+`transcripts_feature_then_morton` descends at 185 of its 244 row-group boundaries while
+both morton-primary elements descend at none — including `transcripts_morton_then_feature`,
+so a *secondary* feature key stays supported and the test is on the file rather than the
+element's name.
+
+The check is free: `datasetMetadata.parts` already carries the footer bytes when the probe
+runs, and the statistics are complete. Failing it also skips the sentinel sampling read,
+since the outcome can no longer change, so a rejected element now costs less than before.
+The element still loads through the capped preload, and its feature-code row-group index
+on that path — a separate mechanism — is unaffected.
+
+Adds `decodeUnsignedIntStat`: `morton_code_2d` is `uint32`, which parquet stores as INT32
+with a UINT_32 annotation, and Morton codes use the top bit for real, so `decodeIntStat`
+would read the far corner of a slide as negative.

.changeset/points-morton-tile-grid.md

@@ -0,0 +1,47 @@
+---
+"@spatialdata/core": patch
+"@spatialdata/layers": patch
+---
+
+Points: select Morton row groups from footer statistics, and size the tile grid from the
+artifact (D5 step 6).
+
+**Row-group selection no longer reads the file.** `selectMortonRowGroups` picks row
+groups from the per-row-group `[min, max]` the tiling probe already parsed out of the
+parquet footer. The bisect it replaces range-read the row group's BYTES — every column,
+~2MB on a real transcripts artifact — to recover two boundary values, `log2(rowGroups)`
+steps per Morton interval, for a few hundred intervals per query. Measured on one
+1024 um viewport tile of a 12.1M-point element, both returning the same 643,961 points:
+
+| row-group selection | range reads | bytes | wall |
+|---|---|---|---|
+| bisect | 97 | 175.12 MB | 2911 ms |
+| footer index | 32 | 57.83 MB | 1035 ms |
+
+The remaining 32 reads are the row-group data itself. The new path is also stricter: the
+bisect tested only `max` and assumed row groups tile the code space without gaps, while
+this intersects both ends. The bisect stays as the fallback when statistics will not
+parse, so this is an optimisation rather than a new requirement.
+
+**The tile grid is derived instead of hardcoded.** It was one fixed level
+(`minZoom/maxZoom: -1`), so every tile was 1024 local units at every zoom: zooming in
+read a 1024-unit tile to look at 50 units of it, and 1024 came from deck's defaults
+rather than from the data. `mortonTileGrid` now derives both ends from the point
+density: the finest level stays at least one row group's footprint (below that, four
+tiles fetch what one used to, for the same bytes and more requests), the coarsest holds
+at most 400k rows, and `zoomOffset = log2(modelMatrixScale)` couples deck's `z` — chosen
+from a world-space zoom — to tile spans expressed in local units.
+
+For the Xenium element that is two levels, 1024 um and 512 um, and the narrowness is the
+point: the row-group size is the floor, so 50k-row groups put it at ~402 um. The old
+fixed 1024 was accidentally near-optimal for that file and would not be for one an order
+of magnitude smaller or denser.
+
+**The tile cache is budgeted in rows.** `maxCacheSize` comes from a row budget rather
+than deck's default of `5 x the selected tile count`, which on a coarse viewport of this
+element could retain ~220 tiles / ~71M rows against a 4M resident cap. It is now 16 tiles
+/ ~5.2M rows, stated. `maxRequests` stays at 6, but as a decision rather than an
+inheritance. Accounting only — nothing evicts by bytes yet (ADR 0005).
+
+`PointsLoaderCapabilities` gains `totalRows` and `maxRowsPerGroup`, which is what the
+grid is derived from.

.changeset/points-morton-tiled-render.md

@@ -0,0 +1,37 @@
+---
+"@spatialdata/core": patch
+"@spatialdata/layers": patch
+"@spatialdata/vis": patch
+---
+
+Points: render Morton-tiled elements from viewport tiles (D5 step 2).
+
+A points layer with `pointsTiling: 'auto'` on a Morton artifact now draws through
+`mortonTiledStrategy` — deck's `TileLayer` reading row groups for the viewport —
+instead of a memory-capped resident preload. A 12.1M-point transcripts element can be
+explored at full detail, and the memory cap no longer applies to it.
+
+- **`@spatialdata/core`** — `PointsResolver` reports what a tiled entry actually has:
+  no `preload` resource (absent, not idle, so `isBlocking` skips it), world `bounds`
+  derived from the artifact's own extent so auto-fit can frame the layer before a
+  single tile loads, and geometry status driven by the probe. `blockingResources`
+  covers `tiling` as well as `preload`. New `transformAxisAlignedBounds` helper.
+- **`@spatialdata/layers`** — `PointsRendererAdapter.getTiledResource`, memoised on
+  (element, metadata): a new resource identity would make `TileLayer` refetch every
+  visible tile, so a pan would become a full reload. Exposed via `PointsDataEngine`
+  alongside `isTiled` / `getTilingMetadata` / `ensureTilingMetadata`.
+- **`@spatialdata/vis`** — the tiled branch in `getLayers`, tiled world bounds,
+  `hasRenderableLayerData` counting a tiled element as drawable, per-layer tile-debug
+  stores feeding viewport-tile progress into `isLoading`, and `pointsTiling` /
+  `showTileDebugOverlay` controls on the points layer panel.
+
+Tiling is per LAYER but the probe's answer is cached per ELEMENT, so every consumer
+combines the two (`usesTiledPath`, `isTiledFor`). Reading the probe alone left a layer
+rendering tiles after the user switched tiling off, while planning went back to
+preloading — both at once.
+
+Known gaps, tracked in `docs/plans/points-morton-tiled-viewport-loading.md`: a tiled
+layer draws flat-coloured and ignores the feature filter (the tile scan does not yet
+return per-point codes — step 3), switching a layer to tiling does not evict the
+preload it already did, and the panel's truncation notice still reports resident
+memory on a tiled layer.

.changeset/points-morton-tiling-default.md

@@ -0,0 +1,36 @@
+---
+"@spatialdata/core": minor
+"@spatialdata/vis": minor
+---
+
+Points: Morton viewport tiling is now on by default (D5 step 7, closes D5).
+
+`pointsTiling` defaults to `'auto'` (`DEFAULT_POINTS_TILING`), so every points element
+is probed once and takes the tiled path if — and only if — it can. Read the config
+through the new `pointsTilingEnabled(...)` rather than comparing to `'auto'`: the
+default has to mean the same thing to the resolver deciding what to load, the hook
+deciding what to render, and the panel drawing the checkbox.
+
+**Why on rather than opt-in.** On a Morton artifact the capped preload is not a neutral
+alternative: it keeps the first `cap` rows in FILE order, and file order there is a
+prefix of the Z-curve — a spatially skewed chunk of the slide rather than a sample of
+it. Tiles read what is actually in view, colour by feature, honour the feature filter
+inside the row-group scan, and subdivide with zoom.
+
+**What it costs, measured.** At the default zoomed-out framing of a 12.1M-point Xenium
+element, the tiled path loads all 44 tiles — 12,165,029 points / ~158 MB, the whole
+artifact — against a 4M-row prefix for the preload. So first paint on a fully
+zoomed-out view is ~3x the rows, in exchange for a correct picture that streams in 44
+pieces instead of blocking on one decode. Zooming OUT is the one direction viewport
+tiling does not help, because there is no coarser representation to read; that wants a
+multi-resolution points pyramid, not a finer index. `pointsTiling: 'off'` restores the
+previous behaviour per layer.
+
+An element that cannot be tiled is unaffected beyond one probe (4 range reads / ~2.16 MB
+on a 12.1M-point element, cached with the metadata; on a non-Morton element it is footer
+metadata the preload reads anyway). The three guards — no usable sentinel row group, a
+sentinel box that is not the code domain, an unsorted Morton column — decline loudly and
+fall through to the preload.
+
+The panel now hides the memory-cap control on a tiled layer: it governs nothing there,
+and it sat directly above a line saying the cap does not apply.

.changeset/points-tiled-coherence.md

@@ -0,0 +1,34 @@
+---
+"@spatialdata/core": patch
+"@spatialdata/layers": patch
+"@spatialdata/vis": patch
+---
+
+Points: make a tiled layer stop reporting — and looking like — a capped preload.
+
+Three things a tiled layer got wrong once it was actually drawing:
+
+- **The resident window is released.** A layer switched to tiling mid-session had
+  usually already preloaded, and nothing gave those rows back: `plan()` stops *asking*
+  for a preload, which is not the same as evicting one. The probe's settle now drops
+  the preload, its row-aligned codes and its feature-index scan (all defined against
+  that window); the catalog stays, since it describes the element rather than the
+  window.
+- **The truncation notice no longer contradicts itself.** It read "4,000,000 of
+  12,165,021 points in memory — capped; raise the cap for more" directly above "the
+  memory cap does not apply". A tiled layer draws from the viewport, so a resident
+  count is not a statement about what is on screen, and the panel now says nothing
+  rather than something true and misleading.
+- **The tiling status line is live.** It read `engine.isTiled(...)` directly, outside
+  the engine subscription, so it kept saying "this element has no Morton index" about
+  an element it was already tiling — the probe settles asynchronously and nothing
+  re-rendered it. `tiled` now comes through `usePointsFeatureState`, which carries the
+  subscription.
+
+Point sizing is also now **one behaviour instead of two**: the Morton tile path sized
+points in fixed pixels while the preloaded path used world units, so `pointSize` meant
+something different depending on a checkbox, and a zoomed-out tiled layer drew every
+one of its millions of points as a fixed screen dot. Density saturated into a flat
+mass and every tile seam and acquisition boundary hardened into what looked like a
+rendering fault. Both paths now size in world units with the model-matrix scale folded
+in, so points shrink as you zoom out and overdraw self-limits.

.changeset/points-tiled-feature-colours.md

@@ -0,0 +1,36 @@
+---
+"@spatialdata/core": patch
+"@spatialdata/layers": patch
+"@spatialdata/vis": patch
+---
+
+Points: colour a Morton-tiled layer by feature (D5 step 3).
+
+Viewport tiles now carry a feature code per point, so a tiled layer colours, takes
+per-feature overrides and responds to Feature Highlight exactly like the preloaded
+path. Until now it drew flat — the same element looked like two different datasets
+depending on a checkbox.
+
+The codes were already being read and thrown away: the tile scan consults the feature
+column to filter on it, then returned bare coordinates.
+
+- **`@spatialdata/core`** — `scanMortonTableInBounds` takes an optional
+  `Int32PointBuffer` and appends to it in lockstep with the geometry; the worker's
+  tile-scan handler builds one and returns it (the protocol already carried an
+  optional `featureCodes` and transferred its buffer, so the worker boundary needed no
+  change). `loadMortonPointsInBounds` now projects the code column whenever the
+  artifact **has** one rather than only when a filter is active — the no-filter "all
+  features" view was precisely the case that arrived without codes — and both its
+  worker and main-thread returns carry them.
+- **`@spatialdata/layers`** — `mortonTiledStrategy` forwards `colorByFeature`,
+  `featureCodeSpaceSize`, `featureColorOverrides` and `highlightFeatureCode` to its
+  per-tile scatter layers, and rebuilds them when those change.
+- **`@spatialdata/vis`** — the tiled branch stops forcing `colorByFeature: false` and
+  reads the same element-scoped colour inputs the preloaded branch does.
+
+A short codes array is dropped rather than padded, on both paths: the remaining points
+would read code 0 — a *valid* feature — and be confidently mis-coloured, which is
+worse than no colour at all.
+
+The feature **filter** still does not narrow tiles (step 4): a tiled layer draws every
+feature in the viewport regardless of the selection, and the panel now says so.

.changeset/points-tiled-feature-filter.md

@@ -0,0 +1,35 @@
+---
+"@spatialdata/core": patch
+"@spatialdata/vis": patch
+---
+
+Points: apply the feature filter to Morton-tiled layers (D5 step 4).
+
+A tiled layer used to draw every feature in the viewport whatever was selected. The
+selection now reaches `getTileData` — and its `updateTriggers`, so changing it
+refetches instead of serving the previous selection's cached tiles — and is applied
+**inside** the row-group scan, so a tile arrives holding only the selected features.
+On a real 12.1M-point transcripts element, selecting one gene takes a viewport tile
+from 3,128,988 points to 87,594: 36x fewer points uploaded and drawn.
+
+It does **not** reduce I/O, and the plan's original expectation that it would has been
+corrected. The same query read the same 92 row groups and the same 158MB either way:
+row groups are chosen *spatially* on a Morton artifact, and a gene's points are spread
+across all of them, so no feature filter can skip one. Narrowing the fetch by feature
+needs a feature-primary index — the open index-selection question in ADR 0002/0003.
+
+Two supporting fixes:
+
+- **The catalog is planned for a tiled entry.** On the preloaded path it arrives free
+  as a preview off the geometry decode; a tiled entry never decodes a resident batch,
+  so nothing built one — and a selection is stored as feature NAMES, which cannot
+  become the codes the scan filters on without it. A saved config with a selection
+  would draw every feature until someone happened to open the filter panel. A failed
+  catalog is not re-planned, or the task re-emits on every reconcile forever.
+- **Feature rows read honestly on a tiled layer.** Every other signal the panel uses
+  describes a resident batch a tiled layer does not have, so its rows fell through to
+  "beyond the resident window; select it to fetch its points" — greyed, and wrong
+  twice over: the points are available, and no feature-index scan is involved.
+
+`renderCap` stays unset on the tiled path: it is a resident-window notion, and a tile
+is already bounded by its viewport.