Points: make feature selection work on large elements, with a name-based selection API

.changeset/points-feature-scan-and-name-selection.md

@@ -0,0 +1,31 @@
+---
+"@spatialdata/core": patch
+"@spatialdata/layers": patch
+"@spatialdata/vis": patch
+---
+
+Points: name-based feature selection, and a much faster feature scan.
+
+**Selections persist as feature names.** `PointsLayerConfig.featureNames` is the
+durable, serializable form and what the UI writes. Codes are app-assigned for a
+dictionary-only element (a Xenium `transcripts` has `feature_name` and no code
+column), so a stored code could silently come back meaning a different feature.
+`featureCodes` still works and still takes effect at runtime, but names win when
+both are present. `resolveFeatureSelectionCodes` / `featureNamesForCodes` are
+exported from `@spatialdata/core` for converting between the two.
+
+**Feature scan is 3-4x faster.** It now reads through
+`ParquetFile.stream({ columns, rowGroups })`, which fetches per column chunk, so
+the projection reaches the network instead of pulling whole row groups — all 12
+columns of a Xenium `transcripts` to use three. The scan runs in the points
+worker, keeping the parquet decode off the main thread. Selecting one gene from a
+12.1M-row element went from ~3.2-7.8s to ~1.0s, with main-thread time roughly a
+third of what the pre-streaming path cost.
+
+Also fixed along the way: a full-dataset catalog scan being silently cancelled by
+the resident preview settling underneath it (leaving counts stuck and colours
+mismatched); row-group chunks handing out the cached footer buffer, which the
+worker transfer detached (`DataCloneError`, dropping the element onto whole-file
+reads); parquet part layout being re-probed on every call; a server that answers
+a directory path with 500 rather than 404 wedging part traversal; and point size
+not accounting for an element's transform scale.

.claude/launch.json

@@ -8,6 +8,12 @@
       "autoPort": true,
       "port": 5173
     },
+    {
+      "name": "vis-demo-alt",
+      "runtimeExecutable": "pnpm",
+      "runtimeArgs": ["--filter", "@spatialdata/vis", "dev:demo", "--port", "5180", "--strictPort"],
+      "port": 5180
+    },
     {
       "name": "docs",
       "runtimeExecutable": "pnpm",

docs/docs/vis/headless-viewer.mdx

@@ -253,6 +253,100 @@ Row alignment comes from `@spatialdata/core` (`createFeatureTableAlignment`);
 colour encoding from `@spatialdata/layers` (`buildShapeFillColorByFeatureId`).
 `SpatialCanvasViewer` wires these when `fillColorByColumn` is set.
 
+## Points: feature selection and colour
+
+Points styling is driven the same way as shapes — by updating the stack entry
+props, not by reaching into vis internals. The serializable fields on a points
+entry:
+
+```ts
+entry.props = {
+  ...entry.props,
+  // Which features are drawn. Omit for "all features".
+  featureNames: ['EPCAM', 'MALL'],
+  // Per-feature colour, keyed by feature name. Absent features keep the default
+  // categorical colour.
+  featureColorOverrides: { EPCAM: [220, 30, 30] },
+  // Categorical colour-by-feature. ON by default; pass false for a flat colour.
+  colorByFeature: true,
+  // Radius in the ELEMENT's own coordinate units (the layer folds in the
+  // element's transform scale, so the same value means the same apparent size
+  // across elements whose transforms differ).
+  pointSize: 0.1,
+  // Max rows retained in memory for the resident window. Raising it draws more
+  // points at the cost of memory and decode time.
+  pointsMemoryCap: 4_000_000,
+};
+```
+
+### Selections persist as names, not codes
+
+`featureNames` is the durable form and the one to serialize. There is also a
+`featureCodes: number[]`, which is retained for runtime use and for configs
+written before names existed — **do not persist it**.
+
+The reason is that a points element often has no feature-code column in the file.
+A Xenium `transcripts` carries `feature_name` and no codes; the same is true of a
+merfish `cell_type`. For those, codes are *assigned by the application* as a
+first-seen index while building the feature catalog, so the same gene is not
+guaranteed the same number between the instant resident-subset catalog and the
+full one, between the two catalog-building paths (which are chosen by row count),
+or between servers that differ in HTTP range support. A persisted code can
+therefore come back meaning a different feature, with nothing to signal it.
+
+Names are resolved to whatever codes the current catalog uses, at render time.
+Names the element does not have are dropped rather than coerced, so a config
+saved against one dataset can be applied to another without inventing features.
+`featureNames` takes precedence when both fields are present.
+
+If you need to convert in either direction yourself, `@spatialdata/core` exports
+`resolveFeatureSelectionCodes(selection, catalog)` and `featureNamesForCodes`.
+
+### Reading feature state headlessly
+
+To build your own feature UI, read the engine directly. `pointsEngine` and
+`resolvePointsTarget` come off the renderer-hook result; wrap a subtree in the
+provider and consume the hook:
+
+```tsx
+import { PointsFeatureStateProvider, usePointsFeatureState } from '@spatialdata/vis';
+
+const { pointsEngine, resolvePointsTarget } = useSpatialCanvasRenderer(/* … */);
+
+<PointsFeatureStateProvider engine={pointsEngine} target={resolvePointsTarget(layerId)}>
+  <MyFeatureList config={pointsConfig} />
+</PointsFeatureStateProvider>;
+
+function MyFeatureList({ config }) {
+  const {
+    catalog,             // { featureKey, entries: [{ code, name, count? }] } | null
+    catalogLoading,      // no catalog yet, one is on its way
+    catalogRefining,     // full scan running behind an instant preview
+    residentCodes,       // features present in the resident window
+    loadedMatchingCodes, // features currently on screen via the last scan
+    supportsOnDemandLoad,// a whole-dataset scan can reach beyond the window
+    matchingLoadState,   // progress of the scan for this selection
+    residentFeatureCounts,
+    requestCatalog,      // idempotent; upgrades the preview to the full list
+    setHighlightedFeature,
+  } = usePointsFeatureState(config);
+}
+```
+
+Pass the layer config (the hook resolves `featureNames` internally against the
+catalog it is already reading). An array of already-resolved codes is also
+accepted.
+
+Two things worth designing around:
+
+- The catalog arrives in two stages. An instant preview covering just the
+  resident window is published first, then the authoritative full-dataset list
+  supersedes it; `catalogRefining` distinguishes them. Counts may lag the names.
+- Selecting a feature whose points are outside the resident window triggers a
+  whole-dataset scan when `supportsOnDemandLoad` is true. `matchingLoadState`
+  reports its progress so the UI can show that points are still arriving rather
+  than appearing to be complete.
+
 ## Auto-fit and view state
 
 - Pass `viewState={null}` on first render to let the viewer compute an initial
@@ -314,6 +408,7 @@ Before treating the API as stable for MDV:
 - [x] Controlled `coordinateSystem`, `renderStack`, `viewState`
 - [x] Host overlay descriptors with `hostLayerResolver`; `deckLayers` / `deckProps` passthrough remains for compatibility
 - [x] `renderTooltip={false}` for external tooltip ownership
+- [x] Points feature selection persisted by NAME (`featureNames`), with `featureColorOverrides` keyed by name
 - [x] `demo/headless` route with local `blobs.zarr` fixture
 - [ ] Additional `demo/headless-*` variants (Leva controls, custom deck layers)
 - [ ] Tooltip/pick row resolution fully on shared `FeatureTableAlignment` (in progress)

docs/docs/vis/mdv-release-checklist.mdx

@@ -63,7 +63,7 @@ geometry/images and maps **MDV’s render stack** to Viv/deck output.
 | Images | Yes (Viv) |
 | Shapes | Yes — MDV drives style/filter via `layers` state |
 | Labels | Render when configured; segmentation vs shapes experiment is not a release gate |
-| Points | v1.1 |
+| Points | v1.1 — feature selection and colour are config-driven and serializable; tiling/index strategy still in progress |
 
 ### Custom deck layers (additive, not a replacement for state)
 
@@ -95,6 +95,37 @@ Current local state:
 
 Optional later: MDV-only `PolygonLayer` built from exported geometry helpers; not required if config-driven styling is sufficient.
 
+### Points v1.1: feature selection and colour via `layers` state
+
+Points support the same config-driven model as shapes — MDV updates the layer
+config, vis owns loading and rendering:
+
+- [x] `PointsLayerConfig` accepts a serializable selection (`featureNames`), per-feature colour (`featureColorOverrides`), `colorByFeature`, `pointSize`, `pointsMemoryCap`
+- [x] Selections persist by feature NAME, not code, so a saved config survives catalog renumbering ([Headless viewer guide](./headless-viewer))
+- [x] Colour overrides keyed by feature name for the same reason
+- [x] Headless feature UI supported without vis internals: `PointsFeatureStateProvider` + `usePointsFeatureState`
+- [ ] Tiling / index strategy for very large elements (follow-up: benchmarking index strategies, Python writer utilities)
+
+The serialization detail that matters for saved MDV charts: a points element
+frequently has **no feature-code column in the file** — a Xenium `transcripts`
+carries `feature_name` only — so codes are assigned by the application while
+building the feature catalog and are not stable across catalog upgrades, catalog
+paths, or servers. `featureNames` is therefore the durable field; `featureCodes`
+exists for runtime and backwards compatibility and should not be persisted.
+
+Current local state:
+
+- `@spatialdata/core` exposes `resolveFeatureSelectionCodes()` and
+  `featureNamesForCodes()` for converting between the durable and runtime forms
+  against a given catalog.
+- The feature catalog is two-stage: an instant resident-subset preview, then the
+  authoritative full-dataset list. A selection made against either resolves to
+  the same features, which is what the name form buys.
+- Remaining release risk: a selection whose points fall outside the resident
+  window needs a whole-dataset scan, whose cost scales with element size — this
+  is what the tiling/index follow-up addresses. `matchingLoadState` exposes the
+  progress meanwhile.
+
 ### Bespoke layer extension path
 
 - Primary extension for **state-owned** SpatialData layers: MDV-controlled `renderStack.entries`

docs/plans/points-redesign-punchlist.md

@@ -42,6 +42,42 @@ state model, not by patching mutations here.
 
 ---
 
+## Known open — feature counts can settle permanently absent
+
+Observed intermittently on a 12.1M-row Xenium `transcripts`: the feature panel
+sticks on "sorted by count so far" and never reaches authoritative counts.
+**Remounting the panel does not clear it**, which distinguishes it from the
+preview-vs-full supersession bug fixed in `46d5b89`.
+
+Mechanism (read from code; not yet reproduced deterministically):
+
+1. `listPointsFeatures` tries `listPointsFeaturesByStreamingScan` first, which
+   tallies counts as it scans, and falls back on any failure.
+2. The fallback does not tally. `listPointsFeaturesWithCounts` then calls
+   `loadFeatureCounts`, which needs an integer code column — so for a
+   **dictionary-only** element (Xenium `transcripts`, merfish `cell_type`) it
+   returns an empty map and the catalog settles with no counts.
+3. `PointsResolver.ensureFeatureCatalog` short-circuits on
+   `slot.settledKey === 'full'`, so that countless catalog is never re-requested.
+   The failure is therefore permanent for the session, and remount-immune.
+
+The non-determinism plausibly comes from `serverSupportsStreamingRanges`, a live
+two-request probe memoised per ORIGIN in a static map: if it loses a race or is
+throttled once, the whole origin is marked unservable for the session and the
+countless path is taken. Suspected to have become more likely when `be64b65` put
+the feature scan on that same probe, so it now runs earlier and more often —
+**unverified**.
+
+Two independent fixes, either of which removes the permanence:
+
+- Do not settle `'full'` for a catalog missing counts it should have — settle it
+  under an upgradable phase so a retry is possible. Makes it self-healing.
+- Make the fallback path tally counts for dict-only elements, so falling back is
+  a performance difference rather than a correctness one.
+
+Deliberately not fixed blind: forcing `serverSupportsStreamingRanges` to false
+should give a deterministic reproduction to fix against first.
+
 ## Defer-to-redesign
 
 Each notes *why* it's coupled to the state-model / decode rework.

docs/plans/resource-resolver-handoff.md

@@ -1,7 +1,8 @@
 # Resource Resolver — implementation handoff
 
-**Status:** Step 0 + Step 1 **landed** — see [Progress](#progress-2026-07-15). Step 2
-(Tracks A / B / C) and Step 3 (Renderer Adapter cleanup) remain.
+**Status:** Step 0 + Step 1 **landed**; **Step 2 Track A (points state model) landed**
+— see [Progress](#progress-2026-07-15). Step 2 Tracks B / C and Step 3 (Renderer
+Adapter cleanup) remain.
 **Decisions:** [ADR 0004 — Resource Resolver Owned By Core](../adr/0004-resource-resolver-owned-by-core.md), [ADR 0005 — Memory Accounting Before Management](../adr/0005-memory-accounting-before-management.md)
 **Supersedes:** [layer-data-engine-decomposition.md](layer-data-engine-decomposition.md)
 **Vocabulary:** [CONTEXT.md](../../CONTEXT.md) — *Resource Resolver, Renderer Adapter, Spatial Entry, Resolution, Spatial Entry Error, Entry Notice, Encoded/Decoded Tier, Resource Ceiling*
@@ -29,10 +30,26 @@ Read the two ADRs first. This document is sequencing, not rationale.
     `pointsEngine.ensureMatchingFeaturesLoaded` / `ensureRowFeatureCodes` calls in
     `getLayers` stay put (they migrate into `plan()` under Track A), so the points
     reconcile context carries only the memory cap.
-- **Remaining:** Step 2 Tracks A (points state model / `RequestSlot` / races R1–R5),
-  B (shapes loader seam + tooltip ping-pong), C (memory, ADR 0005 rungs 1–3); Step 3
-  (Renderer Adapter `project()`/`render()`, `'use no memo'` removal, dead-surface
-  cleanup). None started.
+- **Step 2 Track A — points state model: landed** (branch
+  `claude/points-implementation-stages`, commits A1–A8). All four points resources —
+  `preload`, `rowCodes`, `catalog`, `matching` — are now `RequestSlot`s (one tested
+  dedup/supersede/settle primitive, keyed so everything a request depends on is in the
+  key). **Races R1/R2/R3/R5 closed** with fail-before/pass-after tests. Failures are
+  structured, **retryable** `SpatialEntryError`s with a `retry()` API (the stuck
+  full-catalog-scan fix). Cancellation is threaded to the scan generator (D8;
+  supersede/evict abort it between chunks). The render-phase engine kicks are
+  **migrated into `plan()`** — `getLayers` is now pure reads, driven by the reconcile
+  effect. The streaming-overlay **flash is fixed** (D10) via a scan-stable partial
+  resource + `resourceRevision`. The Effect-vs-plain spike ran and **plain won**
+  (recorded above). `PointsResolver`'s public surface is unchanged; the 855-line
+  `pointsDataEngine` regression net stays green (a few failure/cap-alignment cases
+  flipped to the new behaviour). 573 tests green repo-wide.
+  - **Pending browser verification:** the D10 no-flash behaviour is proven headlessly
+    (adapter resource-identity + revision) but its *visual* confirmation needs a
+    running app with a large streaming Xenium `transcripts` scan.
+- **Remaining:** Step 2 Tracks B (shapes loader seam + tooltip ping-pong), C (memory,
+  ADR 0005 rungs 1–3); Step 3 (Renderer Adapter `project()`/`render()`, `'use no memo'`
+  removal, dead-surface cleanup; retire the `PointsDataEngine` facade). Not started.
 
 ---
 
@@ -176,6 +193,27 @@ unless it wins on *all three* of — supersession correctness under two concurre
 scans; interruption that actually reaches the worker; fewer lines to set up a race in
 a test. A tie means the plain slot wins.
 
+> **Spike outcome (Track A step A8): plain wins. Effect not adopted.** Run
+> empirically — `effect@3.22` added as a `core` devDependency, both slots implemented
+> and driven through the same two-concurrent-scan supersession race
+> (`packages/core/tests/matchingSlotEffectSpike.spec.ts`), then both removed per
+> "delete the loser". Against the three criteria:
+> 1. **Supersession correctness — tie.** Both drop the superseded scan's result; the
+>    plain slot by record identity (`this.current !== record`), Effect by
+>    `Fiber.interrupt`.
+> 2. **Interruption reaches the worker — tie.** Both abort the scan's `AbortSignal`
+>    (the seam A5 threads to the generator). The plain slot aborts its own
+>    `AbortController` *synchronously*; Effect runs the `Effect.async` canceler via
+>    `runFork(Fiber.interrupt(...))`, needing a runtime tick to propagate.
+> 3. **Lines to set up a race — plain wins.** Plain: two synchronous `request()`
+>    calls, assert immediately. Effect: an `Effect.async` + canceler + `runFork` slot
+>    (~15 extra lines) *and* an awaited runtime tick before every assertion; plus a
+>    multi-second import cost on a package whose ethos is zero runtime deps.
+>
+> Effect ties two and loses one, so by the agreed rule the plain `RequestSlot` wins.
+> The `RequestSlot` seam is deliberately shaped so a future reconsideration (e.g. if
+> `tgpu-htj2k`'s dependency-free stance is renegotiated) is a swap, not a rewrite.
+
 ---
 
 #### Track B — Shapes