Points: make feature selection work on large elements, with a name-based selection API

.changeset/points-feature-scan-and-name-selection.md

@@ -0,0 +1,39 @@
+---
+"@spatialdata/core": patch
+"@spatialdata/layers": patch
+"@spatialdata/vis": patch
+---
+
+Points: feature selection now works on large elements, and persists by name.
+
+**Selections persist as feature names.** `PointsLayerConfig.featureNames` is the
+durable, serializable form and what the UI writes. Codes are app-assigned for a
+dictionary-only element (a Xenium `transcripts` has `feature_name` and no code
+column), so a stored code could silently come back meaning a different feature.
+`featureCodes` still works and still takes effect at runtime, but names win when
+both are present. `resolveFeatureSelectionCodes` / `featureNamesForCodes` are
+exported from `@spatialdata/core` for converting between the two.
+
+**The feature scan now completes on large elements.** Previously, selecting a
+feature on a multi-million-row element frequently never resolved — the scan
+plateaued part-way through and the layer sat there. It now reads through
+`ParquetFile.stream({ columns, rowGroups })`, which fetches per column chunk, so
+the projection reaches the network instead of pulling whole row groups — all 12
+columns of a Xenium `transcripts` to use three. The scan runs in the points
+worker, keeping the parquet decode off the main thread. Selecting one gene from a 12.1M-row element now
+settles in ~1.0s, with main-thread time roughly a third of what the pre-streaming
+path cost.
+
+Also fixed along the way: a full-dataset catalog scan being silently cancelled by
+the resident preview settling underneath it (leaving counts stuck and colours
+mismatched); row-group chunks handing out the cached footer buffer, which the
+worker transfer detached (`DataCloneError`, dropping the element onto whole-file
+reads); parquet part layout being re-probed on every call; a server that answers
+a directory path with 500 rather than 404 wedging part traversal; and point size
+not accounting for an element's transform scale.
+
+Known limitation: for a dictionary-only element the fallback catalog path cannot
+tally per-feature counts, and it settles *successfully* without them — so the
+retry path does not repair it and counts stay absent for the session. Names and
+selection are unaffected. Treat a missing count as unknown, not zero, and do not
+read the presence of counts as a signal that the scan completed.

.claude/launch.json

@@ -8,6 +8,12 @@
       "autoPort": true,
       "port": 5173
     },
+    {
+      "name": "vis-demo-alt",
+      "runtimeExecutable": "pnpm",
+      "runtimeArgs": ["--filter", "@spatialdata/vis", "dev:demo", "--port", "5180", "--strictPort"],
+      "port": 5180
+    },
     {
       "name": "docs",
       "runtimeExecutable": "pnpm",

docs/docs/vis/headless-viewer.mdx

@@ -253,6 +253,104 @@ Row alignment comes from `@spatialdata/core` (`createFeatureTableAlignment`);
 colour encoding from `@spatialdata/layers` (`buildShapeFillColorByFeatureId`).
 `SpatialCanvasViewer` wires these when `fillColorByColumn` is set.
 
+## Points: feature selection and colour
+
+Points styling is driven the same way as shapes — by updating the stack entry
+props, not by reaching into vis internals. The serializable fields on a points
+entry:
+
+```ts
+entry.props = {
+  ...entry.props,
+  // Which features are drawn. Omit for "all features".
+  featureNames: ['EPCAM', 'MALL'],
+  // Per-feature colour, keyed by feature name. Absent features keep the default
+  // categorical colour.
+  featureColorOverrides: { EPCAM: [220, 30, 30] },
+  // Categorical colour-by-feature. ON by default; pass false for a flat colour.
+  colorByFeature: true,
+  // Radius in the ELEMENT's own coordinate units (the layer folds in the
+  // element's transform scale, so the same value means the same apparent size
+  // across elements whose transforms differ).
+  pointSize: 0.1,
+  // Max rows retained in memory for the resident window. Raising it draws more
+  // points at the cost of memory and decode time.
+  pointsMemoryCap: 4_000_000,
+};
+```
+
+### Selections persist as names, not codes
+
+`featureNames` is the durable form and the one to serialize. There is also a
+`featureCodes: number[]`, which is retained for runtime use and for configs
+written before names existed — **do not persist it**.
+
+The reason is that a points element often has no feature-code column in the file.
+A Xenium `transcripts` carries `feature_name` and no codes; the same is true of a
+merfish `cell_type`. For those, codes are *assigned by the application* as a
+first-seen index while building the feature catalog, so the same gene is not
+guaranteed the same number between the instant resident-subset catalog and the
+full one, between the two catalog-building paths (which are chosen by row count),
+or between servers that differ in HTTP range support. A persisted code can
+therefore come back meaning a different feature, with nothing to signal it.
+
+Names are resolved to whatever codes the current catalog uses, at render time.
+Names the element does not have are dropped rather than coerced, so a config
+saved against one dataset can be applied to another without inventing features.
+`featureNames` takes precedence when both fields are present.
+
+If you need to convert in either direction yourself, `@spatialdata/core` exports
+`resolveFeatureSelectionCodes(selection, catalog)` and `featureNamesForCodes`.
+
+### Reading feature state headlessly
+
+To build your own feature UI, read the engine directly. `pointsEngine` and
+`resolvePointsTarget` come off the renderer-hook result; wrap a subtree in the
+provider and consume the hook:
+
+```tsx
+import { PointsFeatureStateProvider, usePointsFeatureState } from '@spatialdata/vis';
+
+const { pointsEngine, resolvePointsTarget } = useSpatialCanvasRenderer(/* … */);
+
+<PointsFeatureStateProvider engine={pointsEngine} target={resolvePointsTarget(layerId)}>
+  <MyFeatureList config={pointsConfig} />
+</PointsFeatureStateProvider>;
+
+function MyFeatureList({ config }) {
+  const {
+    catalog,             // { featureKey, entries: [{ code, name, count? }] } | null
+    catalogLoading,      // no catalog yet, one is on its way
+    catalogRefining,     // full scan running behind an instant preview
+    residentCodes,       // features present in the resident window
+    loadedMatchingCodes, // features currently on screen via the last scan
+    supportsOnDemandLoad,// a whole-dataset scan can reach beyond the window
+    matchingLoadState,   // progress of the scan for this selection
+    residentFeatureCounts,
+    requestCatalog,      // idempotent; upgrades the preview to the full list
+    setHighlightedFeature,
+  } = usePointsFeatureState(config);
+}
+```
+
+Pass the layer config (the hook resolves `featureNames` internally against the
+catalog it is already reading). An array of already-resolved codes is also
+accepted.
+
+Two things worth designing around:
+
+- The catalog arrives in two stages. An instant preview covering just the
+  resident window is published first, then the authoritative full-dataset list
+  supersedes it; `catalogRefining` distinguishes them. Counts may lag the names —
+  and for a dictionary-only element they may never arrive at all, because the
+  fallback that builds the catalog cannot tally them. That is a *successful*
+  settle, so nothing retries it. Treat counts as optional: sort and label from the
+  names, and do not read a missing count as zero or as "still loading".
+- Selecting a feature whose points are outside the resident window triggers a
+  whole-dataset scan when `supportsOnDemandLoad` is true. `matchingLoadState`
+  reports its progress so the UI can show that points are still arriving rather
+  than appearing to be complete.
+
 ## Auto-fit and view state
 
 - Pass `viewState={null}` on first render to let the viewer compute an initial
@@ -314,6 +412,7 @@ Before treating the API as stable for MDV:
 - [x] Controlled `coordinateSystem`, `renderStack`, `viewState`
 - [x] Host overlay descriptors with `hostLayerResolver`; `deckLayers` / `deckProps` passthrough remains for compatibility
 - [x] `renderTooltip={false}` for external tooltip ownership
+- [x] Points feature selection persisted by NAME (`featureNames`), with `featureColorOverrides` keyed by name
 - [x] `demo/headless` route with local `blobs.zarr` fixture
 - [ ] Additional `demo/headless-*` variants (Leva controls, custom deck layers)
 - [ ] Tooltip/pick row resolution fully on shared `FeatureTableAlignment` (in progress)

docs/docs/vis/mdv-release-checklist.mdx

@@ -63,7 +63,7 @@ geometry/images and maps **MDV’s render stack** to Viv/deck output.
 | Images | Yes (Viv) |
 | Shapes | Yes — MDV drives style/filter via `layers` state |
 | Labels | Render when configured; segmentation vs shapes experiment is not a release gate |
-| Points | v1.1 |
+| Points | v1.1 — feature selection and colour are config-driven and serializable; tiling/index strategy still in progress |
 
 ### Custom deck layers (additive, not a replacement for state)
 
@@ -95,6 +95,42 @@ Current local state:
 
 Optional later: MDV-only `PolygonLayer` built from exported geometry helpers; not required if config-driven styling is sufficient.
 
+### Points v1.1: feature selection and colour via `layers` state
+
+Points support the same config-driven model as shapes — MDV updates the layer
+config, vis owns loading and rendering:
+
+- [x] `PointsLayerConfig` accepts a serializable selection (`featureNames`), per-feature colour (`featureColorOverrides`), `colorByFeature`, `pointSize`, `pointsMemoryCap`
+- [x] Selections persist by feature NAME, not code, so a saved config survives catalog renumbering ([Headless viewer guide](./headless-viewer))
+- [x] Colour overrides keyed by feature name for the same reason
+- [x] Headless feature UI supported without vis internals: `PointsFeatureStateProvider` + `usePointsFeatureState`
+- [ ] Tiling / index strategy for very large elements (follow-up: benchmarking index strategies, Python writer utilities)
+
+The serialization detail that matters for saved MDV charts: a points element
+frequently has **no feature-code column in the file** — a Xenium `transcripts`
+carries `feature_name` only — so codes are assigned by the application while
+building the feature catalog and are not stable across catalog upgrades, catalog
+paths, or servers. `featureNames` is therefore the durable field; `featureCodes`
+exists for runtime and backwards compatibility and should not be persisted.
+
+Current local state:
+
+- `@spatialdata/core` exposes `resolveFeatureSelectionCodes()` and
+  `featureNamesForCodes()` for converting between the durable and runtime forms
+  against a given catalog.
+- The feature catalog is two-stage: an instant resident-subset preview, then the
+  authoritative full-dataset list. A selection made against either resolves to
+  the same features, which is what the name form buys.
+- Remaining release risk: a selection whose points fall outside the resident
+  window needs a whole-dataset scan, whose cost scales with element size — this
+  is what the tiling/index follow-up addresses. `matchingLoadState` exposes the
+  progress meanwhile.
+- Remaining release risk: for a **dictionary-only** element, the fallback catalog
+  path cannot tally counts, and it settles *successfully* without them — so the
+  retry path does not repair it and the counts stay absent for the session. Do not
+  treat count ordering or the presence of counts as a signal that the scan
+  finished; use the names, and render a missing count as unknown rather than zero.
+
 ### Bespoke layer extension path
 
 - Primary extension for **state-owned** SpatialData layers: MDV-controlled `renderStack.entries`

docs/plans/points-redesign-punchlist.md

@@ -42,6 +42,51 @@ state model, not by patching mutations here.
 
 ---
 
+## Known open — feature counts can settle permanently absent
+
+Observed intermittently on a 12.1M-row Xenium `transcripts`: the feature panel
+sticks on "sorted by count so far" and never reaches authoritative counts.
+**Remounting the panel does not clear it**, which distinguishes it from the
+preview-vs-full supersession bug fixed in `46d5b89`.
+
+Mechanism (read from code; not yet reproduced deterministically):
+
+1. `listPointsFeatures` tries `listPointsFeaturesByStreamingScan` first, which
+   tallies counts as it scans, and falls back on any failure.
+2. The fallback does not tally. `listPointsFeaturesWithCounts` then calls
+   `loadFeatureCounts`, which needs an integer code column — so for a
+   **dictionary-only** element (Xenium `transcripts`, merfish `cell_type`) it
+   returns an empty map and the catalog settles with no counts.
+3. `PointsResolver.ensureFeatureCatalog` short-circuits on
+   `slot.settledKey === 'full'`, so that countless catalog is never re-requested.
+   The failure is therefore permanent for the session, and remount-immune.
+
+The non-determinism plausibly comes from `serverSupportsStreamingRanges`, a live
+two-request probe memoised per ORIGIN in a static map: if it loses a race or is
+throttled once, the whole origin is marked unservable for the session and the
+countless path is taken. Suspected to have become more likely when `be64b65` put
+the feature scan on that same probe, so it now runs earlier and more often —
+**unverified**.
+
+**Update (`8d1a875`).** Review on #89 identified the concrete mechanism, and it
+is the one suspected above: the probe cached a *thrown* fetch as if it were the
+server's answer, so a single failed request demoted the origin for the life of
+the page. The probe now caches only definitive answers (a 416, or a 200 that
+ignored `Range`); a thrown fetch evicts. That removes the most likely trigger,
+but it does **not** close this item — it makes the countless path rarer without
+making it recoverable. The permanence below is untouched, and any other route to
+the fallback still produces the same stuck panel.
+
+Two independent fixes, either of which removes the permanence:
+
+- Do not settle `'full'` for a catalog missing counts it should have — settle it
+  under an upgradable phase so a retry is possible. Makes it self-healing.
+- Make the fallback path tally counts for dict-only elements, so falling back is
+  a performance difference rather than a correctness one.
+
+Deliberately not fixed blind: forcing `serverSupportsStreamingRanges` to false
+should give a deterministic reproduction to fix against first.
+
 ## Defer-to-redesign
 
 Each notes *why* it's coupled to the state-model / decode rework.
@@ -51,6 +96,30 @@ Each notes *why* it's coupled to the state-model / decode rework.
   about this ambient stateful thing` (onProgress), `// given ongoing problems with
   agent debugging, inclined to more purity. Might consider using
   Effect?`, `// there will be various mutating side-effects on entry…`.
+
+  **Effect remains an open question, not a closed one.** Where an ADR or plan
+  reads as having ruled it out, that is "not now", not "decided against" —
+  revisit it on its merits when this item is picked up.
+
+  Evidence from the #89 review, which is the argument for this item stated in
+  defects rather than in taste. Four independent findings, one shape: *state
+  arriving out of step with the thing it describes.*
+
+  | Finding | The step it fell out of |
+  | --- | --- |
+  | Range probe cached a thrown fetch (`8d1a875`) | a failure outliving the request that caused it |
+  | Worker `fromUrl` cache kept a rejection (`8d1a875`) | same, one layer down |
+  | Matched batch drawn unfiltered without row codes (`615c926`) | codes vs the batch they align to |
+  | `loadAll()` landing after its loader was replaced (`57f77fd`) | a read vs the resource it read from |
+  | `rowCodes` readiness gate ignored the cap (this entry's sibling) | codes vs the window they mask |
+
+  Each was individually cheap to fix and none were found by the type system,
+  because in every case the stale value is the *correct type* — the cache, slot
+  or state field simply has no way to say "this is no longer about the thing you
+  are asking about". That is the property a principled effect/resource model
+  makes structural instead of a per-site discipline, and it is why the fixes
+  above are guards rather than a design change: five guards is evidence for D1,
+  not a substitute for it.
 - **D2 — Break up `useLayerData`.** The monolith the engine threads through; also
   the reason for the `'use no memo'` hatches (`PointsFeatureFilterPanel`,
   `ShowMatchingPoints`). A properly reactive state layer retires the hatches.
@@ -70,15 +139,22 @@ Each notes *why* it's coupled to the state-model / decode rework.
   one worker; the engine keys by element and assumes single-demand-per-element.
   Multi-layer sharing / a work queue belongs with the engine redesign.
 - **D7 — GeoArrow encoding.** Unexplored; a decode-path spike, not this PR.
-- **D8 — Streaming cancellation semantics.** The generators have no `AbortSignal`
-  threaded to the worker, and an abandoned manual `.next()` loop won't clean up.
-  Fine while consumers drain; design it with the new state layer.
+- **D8 — Streaming cancellation semantics.** *Partly addressed on the Track A
+  branch:* an `AbortSignal` is threaded to the scan generator, so supersede/evict
+  abort it between chunks. What remains is the general case this entry was written
+  about — the signal does not reach the WORKER, and an abandoned manual `.next()`
+  loop still won't clean up. Design the rest with the new state layer.
 - **D9 — Remove `'use no memo'` hatches (stable-snapshot option).** Give the
   engine stable-identity snapshot accessors so `useSyncExternalStore` tracks the
   value directly and the compiler stops needing an opt-out. Part of D1/D2.
-- **D10 — Progressive-overlay visibility logic + flashing.** F1 fixed the
+- **D10 — Progressive-overlay visibility logic + flashing.** *The flashing is
+  fixed on the Track A branch* — a scan-stable partial resource plus
+  `resourceRevision` means the overlay updates in place instead of being torn down
+  per chunk. The rest of this entry stands: the stable-growing-GPU-buffer work
+  below is still the destination, and it is shared with D3. Historical description
+  of the flash follows. F1 fixed the
   deselected-feature-lingering slice, but *which* points show during a partial
-  load still has logic problems, and it **flashes badly**: every notify rebuilds
+  load still has logic problems, and it **flashed badly**: every notify rebuilt
   the partial buffer into a fresh `PointsRenderResource` (new identity each
   chunk), so deck tears down and recreates the `__partial` layer per step instead
   of updating it in place. The real fix is a stable growing GPU buffer (preallocate