Add points/vector-loading ADRs and status plans

docs/adr/0002-spatially-aware-vector-loading.md

@@ -0,0 +1,148 @@
+# Spatially-Aware Vector Loading
+
+SpatialData points and shapes can be large enough that whole-element Parquet
+loads are not a viable browser default. We will treat viewport-bounded vector
+loading as a first-class source API and keep persisted optimization artifacts in
+Parquet/GeoParquet rather than inventing a deck.gl-specific storage format.
+
+## Decision
+
+- Points v1 follows current Vitessce practice: a SpatialData Points Parquet
+  element may be sorted by 2D Morton order with a `morton_code_2d` column, a
+  feature-code column, controlled row-group sizes, and 2-4 leading sentinel rows
+  whose `morton_code_2d` is `0` and whose coordinates encode the full point
+  extent.
+- `@spatialdata/core` exposes bounded point loading through
+  `PointsElement.loadPointsInBounds()`. When the Parquet module supports
+  Vitessce's row-group APIs (`readMetadata` and `readParquetRowGroup`) and the
+  store supports range reads, the loader may fetch selected row groups. Otherwise
+  it degrades to the existing full-table read followed by bounds filtering.
+- Render-time code uses ADR 0003's **Points Render Resource** (`{ element,
+  loader }`) and calls the `PointsLoader` facet. `PointsElement` remains source
+  identity and public source API, not the deck strategy contract.
+- `@spatialdata/vis` may render compatible points through a deck.gl `TileLayer`.
+  The tile layer owns async viewport loads and abort signals; ordinary
+  `ScatterplotLayer` rendering remains the fallback for preloaded point data.
+- `points.experimental/<key>` and `shapes.experimental/<key>` are reserved as
+  top-level Experimental Optimization Collections. They link back to the source
+  element by key and metadata rather than modifying canonical SpatialData
+  element semantics.
+- GeoParquet is the durable shape optimization target. GeoArrow is a runtime
+  columnar layout / deck adapter option, not a duplicate persisted artifact.
+
+## Experimental Optimization Collections
+
+Use `points.experimental/<key>/` and `shapes.experimental/<key>/` only for
+persisted layouts that **standard SpatialData / Vitessce readers cannot correctly
+consume** — not for every browser optimization.
+
+| Layout | Where it lives | Why |
+|--------|----------------|-----|
+| **Morton v1** (`morton_code_2d`, sentinels, `{feature_key}_codes`, row groups) | **Canonical** `points/<key>/points.parquet` | Follows Vitessce practice. Extra columns are additive; Python `spatialdata` full-table reads still work. |
+| **Feature-primary sort** (Morton not primary key) | `points.experimental/<key>/` | Breaks Morton row-group bisect; needs a new tiling `kind` |
+| **Padua multiscale** (`__spatial_index__`, levels in schema metadata) | `points.experimental/<key>/` | Non-standard vs morton-points v1 |
+| **GeoParquet shapes tiling** | `shapes.experimental/<key>/` | Future |
+
+`experimentalOptimizations` in `@spatialdata/vis` means use TileLayer / row-group
+reads when **canonical** parquet schema supports morton tiling — not “look in
+`points.experimental/`”.
+
+The experimental writer defaults to **in-place** Morton sorting on
+`points/<key>/points.parquet`. Use `--experimental` only when writing a layout
+that must not replace the canonical element.
+
+## Multi-part Parquet (reader)
+
+Wild-type SpatialData points may store `points/<key>/points.parquet` as a
+**directory** with `part.0.parquet`, `part.1.parquet`, … The logical path remains
+`points/<key>/points.parquet`. `@spatialdata/core` supports both single-file and
+multipart layouts for metadata, schema, and row-group range reads. The
+experimental writer outputs a **single-file** Morton artifact by design; row-group
+range reads fetch only the byte ranges needed per viewport.
+
+## Feature / gene filtering
+
+Transcript and other feature-bearing points declare `feature_key` in element
+`spatialdata_attrs` (for example `"feature_name"` on xenium transcripts). This is
+distinct from `instance_key` (for example `"cell_id"`), which identifies the
+object a point belongs to.
+
+The Morton writer adds `{feature_key}_codes` (for example `feature_name_codes`)
+as `int32` categorical codes alongside the string feature column. Sorting is
+**spatial** (Morton on x/y) by default; row groups are spatial chunks.
+
+**Core API** — extend bounded loading with optional feature codes:
+
+```typescript
+interface PointsInBoundsOptions {
+  bounds: SpatialBounds;
+  /** Integer codes matching `{feature_key}_codes` in the parquet artifact */
+  featureCodes?: readonly number[];
+  signal?: AbortSignal;
+}
+```
+
+**Vis API** — extend `PointsLayerConfig` with `featureCodes?: number[]` and
+wire through TileLayer `updateTriggers.getTileData`.
+
+v1 applies feature filtering as a **read-time row predicate** after spatial
+bounds filtering (and after row-group fetch on the Morton path). It does not
+skip row groups by gene. String-based `features?: string[]` and a codebook
+artifact are deferred.
+
+**Implementation status** (preload vs runtime filter, catalog, workers):
+[`docs/plans/points-preload-feature-filter-status.md`](../plans/points-preload-feature-filter-status.md).
+
+Feature filtering is separate from **feature-primary sort** experiments
+(`[feature_codes, morton_code_2d]`), which may require a new tiling `kind` if
+promoted. Use `write-index-permutations` on a derivative Zarr store to benchmark
+sort strategies; see the writer README.
+
+A hypothetical **per-gene density map** (2D histogram / KDE for one feature) is
+out of scope for the Morton tile path and may be an offline aggregation or
+dedicated viz mode later.
+
+## Sort strategy experiments
+
+Default Morton v1 sort is spatial on `morton_code_2d` (optionally `z` when
+low-cardinality). Multi-key sorts under evaluation include
+`[morton_code_2d, feature_name_codes]` and `[feature_name_codes, morton_code_2d]`.
+The reader's row-group bisect assumes Morton is the **primary** sort key; do not
+silently swap sort order under the existing `morton-points` format id.
+
+Generate comparable permutations with:
+
+```bash
+spatialdata-experimental-writer write-index-permutations SOURCE_ZARR DEST_ZARR
+```
+
+The derivative store includes sibling `points/<condition>/` elements and
+`index-manifest.json` for benchmark tooling.
+
+## Prior Art
+
+- scverse Padua hackathon points work:
+  <https://github.com/scverse/2026_04_hackathon_padua/issues/17> and
+  <https://github.com/scverse/2026_04_hackathon_padua/issues/24>.
+- Padua branch prototype:
+  <https://github.com/scverse/2026_04_hackathon_padua/tree/viz/point_chunking/visualization>.
+- Vitessce tiled SpatialData Points:
+  <https://github.com/vitessce/vitessce/pull/2286>.
+- Vitessce sentinel bbox update:
+  <https://github.com/vitessce/vitessce/issues/2419> and
+  <https://github.com/vitessce/vitessce/pull/2489>.
+- Vitessce shapes format `0.3` compatibility:
+  <https://github.com/vitessce/vitessce/pull/2495> and
+  <https://github.com/vitessce/vitessce/releases/tag/v3.9.11>.
+
+## Consequences
+
+- Source loaders must expose typed/columnar batches and remain independent of
+  deck.gl. Rendering packages decide whether to use TileLayer, ScatterplotLayer,
+  or a future GeoArrow-aware layer.
+- Whole-table point loading is still supported and is the compatibility fallback,
+  but render paths can opt into experimental optimizations with a single
+  `experimentalOptimizations` switch.
+- Shapes format `0.3` remains on the current modern Parquet-backed path in
+  `VShapesSource`; large-shape spatial tiling still needs a separate GeoParquet
+  artifact/writer slice.

docs/adr/0003-points-render-resource.md

@@ -0,0 +1,97 @@
+# Points Render Resource
+
+ADR 0002 describes persisted Morton Parquet artifacts and bounded loading APIs on
+`PointsElement`. This ADR describes the **render-time** boundary between store
+I/O, the Resource Resolver, and the deck.gl `PointsLayer` composite.
+
+## Decision
+
+- A points **Spatial Entry** (`PointsElement`) remains the canonical spatial
+  identity handle. Deck layers stay associated with that element for picks,
+  tooltips, and Render Stack `elementKey`.
+- The **Resource Resolver** (today `resolvePointsRenderResource()` in
+  `@spatialdata/vis`) probes once and returns a **Points Render Resource**
+  bundle `{ element, loader }` with **frozen** encoding capabilities.
+- **`PointsLoader`** is the loader facet only: encoding kind, batch format,
+  bounds, and fetch methods. Render strategies call `loader.loadInBounds()` —
+  not `element.loadPointsInBounds()` directly from `@spatialdata/layers`.
+- **`PointsLayer`** (`@spatialdata/layers` `CompositeLayer`) takes
+  `resource: PointsRenderResource` plus cosmetic props. It delegates to
+  encoding-specific render strategies selected by `loader.capabilities.kind`.
+- **Store I/O loader factories** live in `@spatialdata/core` and close over
+  `PointsElement`. **Render strategies** and tile-debug overlay logic live in
+  `@spatialdata/layers`. The vis resolver associates element + loader.
+
+## Encoding selection (v1)
+
+| Condition | Encoding kind | Strategy |
+|-----------|---------------|----------|
+| Full table preloaded in resolver cache | `preloaded-columnar` | `ScatterplotLayer` |
+| Morton metadata with row-group range reads + bounds | `morton-tiled` | `TileLayer` + per-tile scatter |
+| Future GeoArrow batch from core | `geoarrow-binary` | stub → `GeoArrowScatterplotLayer` |
+| Future tiled Arrow/Parquet deck path | `geoarrow-tiled` | stub |
+
+Resolver probing is **eager**: capabilities do not change mid-session unless
+the element or resolver cache inputs change.
+
+## GeoArrow boundary
+
+- **Core** may later expose deck-free Apache Arrow `RecordBatch` batches from
+  Parquet row groups (x/y/z columns or geometry).
+- **Layers** owns [deck.gl-geoarrow](https://github.com/geoarrow/deck.gl-geoarrow)
+  integration: GeoArrow geometry shaping and `GeoArrowScatterplotLayer` /
+  future tiled deck paths.
+- Core must not import deck.gl or `@geoarrow/deck.gl-geoarrow`.
+
+## Batch contract
+
+`PointsBatch` is a tagged union:
+
+- `columnar-ndarray` — v1 Morton and preloaded paths
+- `arrow-record-batch` — reserved for GeoArrow strategies
+
+## Tile debug overlay
+
+When `showTileDebugOverlay` is enabled on a tiled encoding, the morton strategy
+emits a pickable `PolygonLayer` sublayer with per-tile status (pending, loading,
+loaded, empty, error, aborted). This is cosmetic for tile fetching and must not
+appear in `TileLayer.updateTriggers.getTileData`.
+
+## Relationship to ADR 0002
+
+- ADR 0002: persisted artifacts and source-level
+  `PointsElement.loadPointsInBounds()` API.
+- ADR 0003: render-time bundle, strategy registry, and deck composite ownership.
+
+## Consequences
+
+- Swapping encodings or deck.gl parquet layers requires new loader factories
+  and/or strategies — not changes to `PointsLayer` public props.
+- `PointsElement` does not grow a mutable `renderResource` attachment; the
+  resolver cache holds stable bundle references per element key.
+- Image precedent: `ImageElement` + Viv loader built in vis; points precedent:
+  `PointsElement` + `PointsLoader` built in vis, rendered by `PointsLayer`.
+
+## Future performance investigations
+
+These are documented follow-ups — not part of the v1 render bundle.
+
+### CPU / compute hot paths
+
+The scan+compact loops in `filterColumnarByFeatureCodes` /
+`filterPointsToBounds` (`packages/core/src/pointsTiling.ts`) are hot paths for
+large preloaded datasets. Candidates include WASM SIMD and WebGPU compute (e.g.
+[typegpu](https://github.com/software-mansion/typegpu)) for parallel index
+selection and column compaction. **Worker offload** (`@spatialdata/core/workers`)
+is the near-term mitigation; GPU/WASM is a follow-up benchmark task.
+
+### FBO-based render caching
+
+For viewport-stable layers (tiled points, filtered preloaded batches, static
+image tiles), cache rasterized sublayer output in **framebuffer objects (FBOs)**
+so pan/zoom and cosmetic prop changes do not re-draw the full payload every
+frame. This should integrate with the broader **Render Stack compositing**
+story (`Group Entry`, Viv/deck stacking) via shared FBO cache utilities —
+invalidation keyed on structural `updateTriggers`, composition order with host
+overlays — rather than as a points-only hack. Detail deferred until compositing
+utils exist.

docs/plans/parquet-io-error-handling.md

@@ -0,0 +1,46 @@
+# Parquet I/O error handling — follow-up
+
+**Status:** deferred (exceptions kept for now)  
+**Last updated:** 2026-06-23  
+**Related:** [Error handling](../docs/core/error-handling.mdx), `VTableSource.ts`
+
+Example of limited `Result` adoption — see the error-handling doc for the
+general picture. This note is only about parquet I/O.
+
+## Current state
+
+Parquet loading mixes three patterns:
+
+| Pattern | Example | Semantics |
+|--------|---------|-----------|
+| `null` | `loadParquetFileBytesAtPath` | Missing or invalid bytes (store miss, non-parquet payload) |
+| `throw` | `readParquetDatasetBytesCapped`, `loadMultipartParquetTable` | Required bytes unavailable — fail the operation |
+| `continue` | `_loadParquetTableUncachedCapped`, `VPointsSource` feature-filter scans | Skip a part and try the rest |
+
+`partPaths` is built from dataset metadata (footer/range reads) or
+`discoverMultipartPartPaths` (full-byte probe). Metadata paths are **not**
+guaranteed loadable via `loadParquetFileBytesAtPath`; discovered paths were
+verified moments earlier in the same call.
+
+## Intentional strictness difference
+
+`readParquetDatasetBytesCapped` **throws** on the first missing part because it
+feeds worker decode paths that need reliable byte buffers. Sibling table loaders
+use **`continue`** so a later part can still contribute rows. That is a policy
+choice, not probing of paths known to be absent.
+
+## Follow-up (when revisiting)
+
+1. Decide whether to expand `Result` at all; if so, evaluate an established
+   library (e.g. `neverthrow`) rather than extending the in-house `zarrextra`
+   types.
+2. Introduce typed errors (e.g. missing part, invalid bytes, empty dataset).
+3. Move `loadParquetFileBytesAtPath` to `Result` first; keep a thin `null` shim
+   if needed during migration.
+4. Migrate protected helpers (`readParquetDatasetBytesCapped`, multipart
+   loaders) and unify skip-vs-fail policy per call site.
+5. Leave public APIs (`loadParquetTable`, `loadPoints`, …) throwing until vis /
+   layers need typed degradation; use `unwrap()` at boundaries meanwhile.
+
+See [Error handling](../docs/core/error-handling.mdx) for the current provisional
+`Result` API and adoption patterns (`getTransformation`).