diff --git a/docs/adr/0002-spatially-aware-vector-loading.md b/docs/adr/0002-spatially-aware-vector-loading.md new file mode 100644 index 00000000..669d204f --- /dev/null +++ b/docs/adr/0002-spatially-aware-vector-loading.md @@ -0,0 +1,148 @@ +# Spatially-Aware Vector Loading + +SpatialData points and shapes can be large enough that whole-element Parquet +loads are not a viable browser default. We will treat viewport-bounded vector +loading as a first-class source API and keep persisted optimization artifacts in +Parquet/GeoParquet rather than inventing a deck.gl-specific storage format. + +## Decision + +- Points v1 follows current Vitessce practice: a SpatialData Points Parquet + element may be sorted by 2D Morton order with a `morton_code_2d` column, a + feature-code column, controlled row-group sizes, and 2-4 leading sentinel rows + whose `morton_code_2d` is `0` and whose coordinates encode the full point + extent. +- `@spatialdata/core` exposes bounded point loading through + `PointsElement.loadPointsInBounds()`. When the Parquet module supports + Vitessce's row-group APIs (`readMetadata` and `readParquetRowGroup`) and the + store supports range reads, the loader may fetch selected row groups. Otherwise + it degrades to the existing full-table read followed by bounds filtering. +- Render-time code uses ADR 0003's **Points Render Resource** (`{ element, + loader }`) and calls the `PointsLoader` facet. `PointsElement` remains source + identity and public source API, not the deck strategy contract. +- `@spatialdata/vis` may render compatible points through a deck.gl `TileLayer`. + The tile layer owns async viewport loads and abort signals; ordinary + `ScatterplotLayer` rendering remains the fallback for preloaded point data. +- `points.experimental/` and `shapes.experimental/` are reserved as + top-level Experimental Optimization Collections. They link back to the source + element by key and metadata rather than modifying canonical SpatialData + element semantics. +- GeoParquet is the durable shape optimization target. GeoArrow is a runtime + columnar layout / deck adapter option, not a duplicate persisted artifact. + +## Experimental Optimization Collections + +Use `points.experimental//` and `shapes.experimental//` only for +persisted layouts that **standard SpatialData / Vitessce readers cannot correctly +consume** — not for every browser optimization. + +| Layout | Where it lives | Why | +|--------|----------------|-----| +| **Morton v1** (`morton_code_2d`, sentinels, `{feature_key}_codes`, row groups) | **Canonical** `points//points.parquet` | Follows Vitessce practice. Extra columns are additive; Python `spatialdata` full-table reads still work. | +| **Feature-primary sort** (Morton not primary key) | `points.experimental//` | Breaks Morton row-group bisect; needs a new tiling `kind` | +| **Padua multiscale** (`__spatial_index__`, levels in schema metadata) | `points.experimental//` | Non-standard vs morton-points v1 | +| **GeoParquet shapes tiling** | `shapes.experimental//` | Future | + +`experimentalOptimizations` in `@spatialdata/vis` means use TileLayer / row-group +reads when **canonical** parquet schema supports morton tiling — not “look in +`points.experimental/`”. + +The experimental writer defaults to **in-place** Morton sorting on +`points//points.parquet`. Use `--experimental` only when writing a layout +that must not replace the canonical element. + +## Multi-part Parquet (reader) + +Wild-type SpatialData points may store `points//points.parquet` as a +**directory** with `part.0.parquet`, `part.1.parquet`, … The logical path remains +`points//points.parquet`. `@spatialdata/core` supports both single-file and +multipart layouts for metadata, schema, and row-group range reads. The +experimental writer outputs a **single-file** Morton artifact by design; row-group +range reads fetch only the byte ranges needed per viewport. + +## Feature / gene filtering + +Transcript and other feature-bearing points declare `feature_key` in element +`spatialdata_attrs` (for example `"feature_name"` on xenium transcripts). This is +distinct from `instance_key` (for example `"cell_id"`), which identifies the +object a point belongs to. + +The Morton writer adds `{feature_key}_codes` (for example `feature_name_codes`) +as `int32` categorical codes alongside the string feature column. Sorting is +**spatial** (Morton on x/y) by default; row groups are spatial chunks. + +**Core API** — extend bounded loading with optional feature codes: + +```typescript +interface PointsInBoundsOptions { + bounds: SpatialBounds; + /** Integer codes matching `{feature_key}_codes` in the parquet artifact */ + featureCodes?: readonly number[]; + signal?: AbortSignal; +} +``` + +**Vis API** — extend `PointsLayerConfig` with `featureCodes?: number[]` and +wire through TileLayer `updateTriggers.getTileData`. + +v1 applies feature filtering as a **read-time row predicate** after spatial +bounds filtering (and after row-group fetch on the Morton path). It does not +skip row groups by gene. String-based `features?: string[]` and a codebook +artifact are deferred. + +**Implementation status** (preload vs runtime filter, catalog, workers): +[`docs/plans/points-preload-feature-filter-status.md`](../plans/points-preload-feature-filter-status.md). + +Feature filtering is separate from **feature-primary sort** experiments +(`[feature_codes, morton_code_2d]`), which may require a new tiling `kind` if +promoted. Use `write-index-permutations` on a derivative Zarr store to benchmark +sort strategies; see the writer README. + +A hypothetical **per-gene density map** (2D histogram / KDE for one feature) is +out of scope for the Morton tile path and may be an offline aggregation or +dedicated viz mode later. + +## Sort strategy experiments + +Default Morton v1 sort is spatial on `morton_code_2d` (optionally `z` when +low-cardinality). Multi-key sorts under evaluation include +`[morton_code_2d, feature_name_codes]` and `[feature_name_codes, morton_code_2d]`. +The reader's row-group bisect assumes Morton is the **primary** sort key; do not +silently swap sort order under the existing `morton-points` format id. + +Generate comparable permutations with: + +```bash +spatialdata-experimental-writer write-index-permutations SOURCE_ZARR DEST_ZARR +``` + +The derivative store includes sibling `points//` elements and +`index-manifest.json` for benchmark tooling. + +## Prior Art + +- scverse Padua hackathon points work: + and + . +- Padua branch prototype: + . +- Vitessce tiled SpatialData Points: + . +- Vitessce sentinel bbox update: + and + . +- Vitessce shapes format `0.3` compatibility: + and + . + +## Consequences + +- Source loaders must expose typed/columnar batches and remain independent of + deck.gl. Rendering packages decide whether to use TileLayer, ScatterplotLayer, + or a future GeoArrow-aware layer. +- Whole-table point loading is still supported and is the compatibility fallback, + but render paths can opt into experimental optimizations with a single + `experimentalOptimizations` switch. +- Shapes format `0.3` remains on the current modern Parquet-backed path in + `VShapesSource`; large-shape spatial tiling still needs a separate GeoParquet + artifact/writer slice. diff --git a/docs/adr/0003-points-render-resource.md b/docs/adr/0003-points-render-resource.md new file mode 100644 index 00000000..4ea7ff52 --- /dev/null +++ b/docs/adr/0003-points-render-resource.md @@ -0,0 +1,97 @@ +# Points Render Resource + +ADR 0002 describes persisted Morton Parquet artifacts and bounded loading APIs on +`PointsElement`. This ADR describes the **render-time** boundary between store +I/O, the Resource Resolver, and the deck.gl `PointsLayer` composite. + +## Decision + +- A points **Spatial Entry** (`PointsElement`) remains the canonical spatial + identity handle. Deck layers stay associated with that element for picks, + tooltips, and Render Stack `elementKey`. +- The **Resource Resolver** (today `resolvePointsRenderResource()` in + `@spatialdata/vis`) probes once and returns a **Points Render Resource** + bundle `{ element, loader }` with **frozen** encoding capabilities. +- **`PointsLoader`** is the loader facet only: encoding kind, batch format, + bounds, and fetch methods. Render strategies call `loader.loadInBounds()` — + not `element.loadPointsInBounds()` directly from `@spatialdata/layers`. +- **`PointsLayer`** (`@spatialdata/layers` `CompositeLayer`) takes + `resource: PointsRenderResource` plus cosmetic props. It delegates to + encoding-specific render strategies selected by `loader.capabilities.kind`. +- **Store I/O loader factories** live in `@spatialdata/core` and close over + `PointsElement`. **Render strategies** and tile-debug overlay logic live in + `@spatialdata/layers`. The vis resolver associates element + loader. + +## Encoding selection (v1) + +| Condition | Encoding kind | Strategy | +|-----------|---------------|----------| +| Full table preloaded in resolver cache | `preloaded-columnar` | `ScatterplotLayer` | +| Morton metadata with row-group range reads + bounds | `morton-tiled` | `TileLayer` + per-tile scatter | +| Future GeoArrow batch from core | `geoarrow-binary` | stub → `GeoArrowScatterplotLayer` | +| Future tiled Arrow/Parquet deck path | `geoarrow-tiled` | stub | + +Resolver probing is **eager**: capabilities do not change mid-session unless +the element or resolver cache inputs change. + +## GeoArrow boundary + +- **Core** may later expose deck-free Apache Arrow `RecordBatch` batches from + Parquet row groups (x/y/z columns or geometry). +- **Layers** owns [deck.gl-geoarrow](https://github.com/geoarrow/deck.gl-geoarrow) + integration: GeoArrow geometry shaping and `GeoArrowScatterplotLayer` / + future tiled deck paths. +- Core must not import deck.gl or `@geoarrow/deck.gl-geoarrow`. + +## Batch contract + +`PointsBatch` is a tagged union: + +- `columnar-ndarray` — v1 Morton and preloaded paths +- `arrow-record-batch` — reserved for GeoArrow strategies + +## Tile debug overlay + +When `showTileDebugOverlay` is enabled on a tiled encoding, the morton strategy +emits a pickable `PolygonLayer` sublayer with per-tile status (pending, loading, +loaded, empty, error, aborted). This is cosmetic for tile fetching and must not +appear in `TileLayer.updateTriggers.getTileData`. + +## Relationship to ADR 0002 + +- ADR 0002: persisted artifacts and source-level + `PointsElement.loadPointsInBounds()` API. +- ADR 0003: render-time bundle, strategy registry, and deck composite ownership. + +## Consequences + +- Swapping encodings or deck.gl parquet layers requires new loader factories + and/or strategies — not changes to `PointsLayer` public props. +- `PointsElement` does not grow a mutable `renderResource` attachment; the + resolver cache holds stable bundle references per element key. +- Image precedent: `ImageElement` + Viv loader built in vis; points precedent: + `PointsElement` + `PointsLoader` built in vis, rendered by `PointsLayer`. + +## Future performance investigations + +These are documented follow-ups — not part of the v1 render bundle. + +### CPU / compute hot paths + +The scan+compact loops in `filterColumnarByFeatureCodes` / +`filterPointsToBounds` (`packages/core/src/pointsTiling.ts`) are hot paths for +large preloaded datasets. Candidates include WASM SIMD and WebGPU compute (e.g. +[typegpu](https://github.com/software-mansion/typegpu)) for parallel index +selection and column compaction. **Worker offload** (`@spatialdata/core/workers`) +is the near-term mitigation; GPU/WASM is a follow-up benchmark task. + +### FBO-based render caching + +For viewport-stable layers (tiled points, filtered preloaded batches, static +image tiles), cache rasterized sublayer output in **framebuffer objects (FBOs)** +so pan/zoom and cosmetic prop changes do not re-draw the full payload every +frame. This should integrate with the broader **Render Stack compositing** +story (`Group Entry`, Viv/deck stacking) via shared FBO cache utilities — +invalidation keyed on structural `updateTriggers`, composition order with host +overlays — rather than as a points-only hack. Detail deferred until compositing +utils exist. diff --git a/docs/plans/parquet-io-error-handling.md b/docs/plans/parquet-io-error-handling.md new file mode 100644 index 00000000..e627b479 --- /dev/null +++ b/docs/plans/parquet-io-error-handling.md @@ -0,0 +1,46 @@ +# Parquet I/O error handling — follow-up + +**Status:** deferred (exceptions kept for now) +**Last updated:** 2026-06-23 +**Related:** [Error handling](../docs/core/error-handling.mdx), `VTableSource.ts` + +Example of limited `Result` adoption — see the error-handling doc for the +general picture. This note is only about parquet I/O. + +## Current state + +Parquet loading mixes three patterns: + +| Pattern | Example | Semantics | +|--------|---------|-----------| +| `null` | `loadParquetFileBytesAtPath` | Missing or invalid bytes (store miss, non-parquet payload) | +| `throw` | `readParquetDatasetBytesCapped`, `loadMultipartParquetTable` | Required bytes unavailable — fail the operation | +| `continue` | `_loadParquetTableUncachedCapped`, `VPointsSource` feature-filter scans | Skip a part and try the rest | + +`partPaths` is built from dataset metadata (footer/range reads) or +`discoverMultipartPartPaths` (full-byte probe). Metadata paths are **not** +guaranteed loadable via `loadParquetFileBytesAtPath`; discovered paths were +verified moments earlier in the same call. + +## Intentional strictness difference + +`readParquetDatasetBytesCapped` **throws** on the first missing part because it +feeds worker decode paths that need reliable byte buffers. Sibling table loaders +use **`continue`** so a later part can still contribute rows. That is a policy +choice, not probing of paths known to be absent. + +## Follow-up (when revisiting) + +1. Decide whether to expand `Result` at all; if so, evaluate an established + library (e.g. `neverthrow`) rather than extending the in-house `zarrextra` + types. +2. Introduce typed errors (e.g. missing part, invalid bytes, empty dataset). +3. Move `loadParquetFileBytesAtPath` to `Result` first; keep a thin `null` shim + if needed during migration. +4. Migrate protected helpers (`readParquetDatasetBytesCapped`, multipart + loaders) and unify skip-vs-fail policy per call site. +5. Leave public APIs (`loadParquetTable`, `loadPoints`, …) throwing until vis / + layers need typed degradation; use `unwrap()` at boundaries meanwhile. + +See [Error handling](../docs/core/error-handling.mdx) for the current provisional +`Result` API and adoption patterns (`getTransformation`). diff --git a/docs/plans/points-preload-feature-filter-status.md b/docs/plans/points-preload-feature-filter-status.md new file mode 100644 index 00000000..33f6f561 --- /dev/null +++ b/docs/plans/points-preload-feature-filter-status.md @@ -0,0 +1,368 @@ +# Points preload & feature filter — status and plan + +**Status:** work in progress (branch/worktree, not yet on `main`) +**Last updated:** 2026-06-20 +**Related:** [ADR 0002](../adr/0002-spatially-aware-vector-loading.md), [ADR 0003](../adr/0003-points-render-resource.md) + +This document captures what we built, what broke, what we fixed, and what still +needs cleanup — especially around **workers**, **parquet I/O**, and the **~30s +main-thread catalog load** on large Xenium `transcripts`. + +--- + +## Problem statement + +On ~12M-row Xenium `transcripts`: + +1. **Feature filter was unusably slow** — every checkbox toggle re-scanned the + full parquet dataset via `loadPoints({ featureCodes })`. +2. **Parquet was used incorrectly** for large reads — whole part files fetched + and decoded instead of row-group range reads with column projection. +3. **Feature catalog UI** failed or showed a single blank gene on the normal + `transcripts` element (dictionary-encoded `feature_name`, no separate codes + column). + +--- + +## Target architecture questions (current intent) + +The current implementation separates three concerns for the preloaded scatter +path: + +| Concern | When | Where | +|---------|------|--------| +| **Geometry preload** | Once per `(element, memoryCap)` | `loadPoints` → x/y only, capped (default 4M rows) | +| **Runtime feature filter** | Every checkbox toggle | `PointsLayer` → in-memory filter on preloaded batch | +| **Feature catalog** | Once per element (UI gene list) | `listFeatures` → feature columns only, not x/y | + +This is a useful strategy when a bounded preload fits comfortably in memory and +the user wants fast toggling across a moderate number of visible points. It is +not the general architecture for all points stores. + +The broader architecture should support multiple point loading strategies: + +| Strategy | Filter-change behavior | Runtime batch/layout | Good fit | +|----------|------------------------|----------------------|----------| +| **Preloaded scatter** | Does not reload geometry; filters an in-memory capped batch | `columnar-ndarray` today; possible Arrow/GeoArrow batch later | Moderate point counts, exploratory toggling, local responsiveness | +| **Spatial Morton tiles** | Reloads viewport tiles when filter props change | `columnar-ndarray` tile batches today; possible GeoArrow tile batches later | Spatial navigation where row groups are primarily spatial | +| **Feature-primary or compound index** | Intentionally loads new data for selected features | Same loader contract; likely benefits from Arrow/GeoArrow columnar batches | Looking at a few genes/features out of thousands without keeping all features in memory | + +The open design question is how the resolver chooses among these strategies and +how writers advertise their indexes. We should not assume that every feature +filter is a view over already-loaded data. + +### Runtime batch/layout direction + +Persisted optimized points remain Parquet-backed for now. GeoArrow is relevant +as a **runtime columnar layout** and deck.gl integration boundary, not as a +separate persisted copy of the same data. The `PointsLoader` / `PointsBatch` +contract should be able to return Arrow-ish or GeoArrow-compatible batches +later, while `@spatialdata/layers` owns deck.gl-geoarrow adaptation. This keeps +`@spatialdata/core` deck-free and lets each strategy evolve from current +`columnar-ndarray` batches toward GeoArrow where that proves faster or simpler. + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Vis (useLayerData) │ +│ ├─ loadPoints({ memoryCap }) once, key = element|m{cap} │ +│ ├─ loadRowFeatureCodes({ cap }) after preload, aligned rows │ +│ └─ listFeatures() catalog for filter panel │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ PointsLayer (preloaded scatter path) │ +│ ├─ preloadedBatch from render resource │ +│ ├─ preloadedFeatureCodes from useLayerData ref │ +│ └─ featureCodes prop from layer config (checkbox state) │ +│ → filterPreloadedBatch (worker when enabled) │ +└─────────────────────────────────────────────────────────────────┘ +``` + +Morton-tiled elements (`transcripts_morton`, etc.) use viewport tiles + +`featureCodes` in `loadPointsInBounds` per tile — no full-table preload. + +Feature-primary or compound-indexed stores are still experimental. They may make +filter changes part of the structural load key because the point of the index is +to fetch only selected features. + +--- + +## What we changed (summary) + +### Vis (`@spatialdata/vis`) + +- **`pointsPreloadCacheKey`** — for the current preloaded scatter path, memory + cap only; no `featureCodes` in key. +- **`loadPoints`** — no longer passes `featureCodes`; filter does not reload + geometry on this path. +- **`loadRowFeatureCodes`** — separate effect after preload; keyed by + `preloadCacheKey`; passed to `PointsLayer` as `preloadedFeatureCodes`. +- **Catalog cache** — retry when cached value is `null` (failed load), not when + a valid catalog exists. +- Removed filter-reload machinery (`beginPointsFilterReload`, + `resolveRenderablePointsPreload`, etc.). + +### Core (`@spatialdata/core`) + +- **`loadPoints`** — ignores `featureCodes` unless + `fullDatasetFeatureScan: true` (opt-in benchmark path, not used by vis). +- **`loadParquetTableCapped`** — prefers **row-group range reads** + + column projection when store supports `getRange`. +- **`listPointsFeatures`** (large datasets) — feature-column scan with + `readParquet` fallback when row-group decode is broken for dictionary columns. +- **Dictionary catalog helpers** — safe index extraction, merge across chunks, + `featureCatalogNeedsParquetFallback` (empty or all-blank names). + +### Layers (`@spatialdata/layers`) + +- **`PointsLayer`** — async filtered-batch cache in `updateState`; filter + signature includes `featureCodes`, `preloadedFeatureCodes`, `renderCap`. + +--- + +## Workers: what they do today + +The points worker is **enabled in the vis demo** via +`packages/vis/demo/src/enableDemoPointsWorker.ts`. + +| Operation | Worker? | Notes | +|-----------|---------|-------| +| Feature filter on preloaded batch | **Yes** | `filterColumnarByFeatureCodesInWorker` in `PointsLayer` | +| Geometry preload (`loadPoints`) | **Yes** | `decodeParquetGeometryCappedInWorker`; main-thread fallback | +| Row feature codes (`loadRowFeatureCodes`) | **Yes** | Worker decode via row-group bytes or part bytes; main-thread fallback | +| Feature catalog (large, dict-only) | **Yes** | `scanParquetFeatureCatalogInWorker`; dict fallback via full parts in worker | +| Feature counts | **Yes** | `scanParquetFeatureCountsInWorker` (row groups or parts); main-thread fallback | +| Morton viewport tiles | **Yes** | `scanMortonRowGroupsInBoundsInWorker`; main-thread fallback | +| Opt-in full-dataset filter scan | **Yes** | `scanParquetByFeatureCodesInWorker` (row groups or parts) | + +**Takeaway:** parquet decode and table scans run on the points worker when enabled; +main thread does metadata resolution and async byte-range I/O only. + +--- + +## Parquet I/O paths + +### Good (row-group + projection) + +- Morton viewport tiles: `loadParquetRowGroupByGroupIndex` + `store.getRange`. +- Geometry preload: `loadParquetTableCapped` → `_loadParquetTableRowGroupsCapped` + when range reads work. +- Catalog on **`transcripts_morton`** (has `feature_name_codes`): row-group + scan of feature columns only — fast (~hundreds of genes). + +### Bad / fallback (full-file or full-column decode) + +- **`loadParquetFileBytesAtPath`** still used in capped multipart fallback and + opt-in `loadPointsMatchingFeatureCodes` / `loadFeatureCounts` worker paths. +- **Catalog on plain `transcripts`**: row-group reads do not decode + dictionary-encoded `feature_name` correctly (empty names, single bogus + entry). Fallback is **`loadParquetTable(parquetPath, [feature_name])` over all + parts** — correct gene list, **~30s main-thread block** on 12M rows. + +### Debug evidence (Xenium) + +| Path | Catalog result | Mechanism | +|------|----------------|-----------| +| `transcripts` | ~30s, works after fallback | Dict-only; full feature-column read | +| `transcripts_morton` | Fast, ~541 genes | `feature_name_codes` + row-group scan | +| Row-group dict merge | 0 entries | `rowGroupsWithDictionary: 0` | +| Row-group scan (dict-only) | 1 blank entry | Indices without dictionary array | +| `RangeError: offset is out of bounds` | Catalog null | Fixed via safe `getDictionaryIndexAt` | + +--- + +## Feature catalog bug timeline (why it was confusing) + +1. Large datasets only built catalog from dictionary if a **1-row capped probe** + worked → Arrow slice drops dictionary values → `null` catalog. +2. Feature-column row-group scan for dict-only columns → **one entry, empty + name** (collapsed `nameToCode` map). +3. Dictionary row-group merge → **0 entries** (WASM row-group read not + dictionary-typed the way we expected). +4. **Working fix:** skip row-group scan when no `feature_name_codes`; if catalog + empty or all names blank → **`readParquet` full feature-column load**. + +This fixed the UI but introduced the main-thread stall. + +--- + +## Mental model: which path am I on? + +``` +transcripts (12M, dictionary feature_name, NO feature_name_codes) + ├─ preload: row-group x/y (capped 4M) main thread, moderate + ├─ row codes: feature cols via loadParquetTableCapped main thread, moderate + ├─ catalog: FULL readParquet [feature_name] main thread, ~30s ← pain point + ├─ counts: hidden until an explicit code/name mapping is available + └─ filter toggle: worker in-memory on preloaded batch fast, but capped + +transcripts_morton (feature_name_codes + morton_code_2d) + ├─ render: Morton TileLayer, viewport-bounded + ├─ catalog: row-group feature columns fast + └─ filter: per-tile featureCodes in getTileData + +future feature-primary / compound-indexed store + ├─ render: query selected features, possibly viewport-bounded + ├─ filter: changes load key and fetches new rows + └─ goal: avoid loading thousands of genes when viewing a few +``` + +--- + +## Current status + +### Working + +- Feature filter toggles are **instant** on the current preloaded scatter path + (no parquet rescan). +- Geometry preload uses row-group reads where the store supports them. +- Morton / coded elements get a reasonable catalog quickly. +- Plain `transcripts` catalog **populates** (after expensive fallback). +- Caps: memory cap (preload), render cap (draw), separate concerns. +- Tests: core 106, vis 65 (as of 2026-06-20). + +### Not ideal / known debt + +1. **Catalog for dict-only large datasets** — full-table feature-column read; + blocks main thread ~30s; not justified long-term. +2. **Worker policy inconsistent** — filter off-thread; preload/catalog on-thread. +3. **`loadFeatureCounts`** — counts are hidden unless code/name mapping is + explicit. Wrong counts are worse than missing counts. +4. **Strategy selection is unresolved** — preloaded in-memory filtering is one + useful path, but feature-primary or compound-indexed stores may intentionally + reload data when filters change. +5. **Legacy / dead-ish paths** — `loadPointsMatchingFeatureCodes`, + `decodeParquetPartsInWorker`, `fullDatasetFeatureScan` (opt-in, no UI). +6. **Row-group WASM + dictionary columns** — broken for catalog; we paper over + with full read; root cause not fixed in the reader layer. + +--- + +## Cleanup plan (prioritized) + +### P0 — Remove the 30s catalog stall (plain `transcripts`) + +Pick one or combine: + +| Approach | Effort | Notes | +|----------|--------|-------| +| **Dictionary from parquet metadata** | Medium | Read dictionary pages / schema without scanning 12M rows; ideal for Xenium | +| **Worker-backed catalog build** | Low–medium | Same bytes as today, off main thread; doesn't reduce total work | +| **Cache catalog per element** | Low | IndexedDB or in-memory; first visit still slow | +| **Writer: always emit `feature_name_codes`** | Medium | Aligns with morton path; Python writer change | +| **Sidecar gene list** | Low | Small JSON/parquet in element attrs (non-standard) | + +**Recommendation:** metadata/dictionary-page fast path first; worker offload as +a quick win if metadata path is hard in parquet-wasm. + +### P1 — Unify worker policy + +Document and implement one rule, e.g.: + +> All parquet decode and row scans run in the points worker; main thread only +> marshals Arrow IPC and deck props. + +Or explicitly drop worker for decode and accept main-thread decode with +chunking/`requestIdleCallback` — but be consistent. + +### P2 — Defer non-critical work + +- Show catalog from dictionary-only fast path **without counts** first. +- Load `loadFeatureCounts` only when user opens filter panel or on idle. +- Don't block first render on catalog (already partially true). + +### P3 — Trim legacy paths + +- Remove or gate `fullDatasetFeatureScan` unless needed for benchmarks. +- Audit `loadPointsMatchingFeatureCodes` vs runtime filter. The answer may be + different per strategy: preloaded scatter filters in memory, while + feature-indexed stores may use source-side feature queries. +- Remove unused worker decode entry points if nothing calls them. + +### P4 — Fix row-group dictionary decode (proper parquet) + +- Investigate parquet-wasm `readParquetRowGroup` + column projection for + dictionary columns on Xenium multipart layout. +- Goal: row-group catalog path works for dict-only `feature_name` without full + table read. + +### Future note — DuckDB / DuckDB-Wasm + +DuckDB is not part of the current render path. It may become useful later as a +correctness oracle for Parquet scans, an offline writer/benchmark validation +tool, or a worker-backed catalog/count query engine. Do not add it to browser +tile loading without a separate benchmark and bundle-size decision. + +--- + +## API contracts (for reference) + +### Vis preload cache key for preloaded scatter + +``` +{elementKey}|m{memoryCap} +``` + +Feature filter is **not** part of this key for the current preloaded scatter +path. A future feature-primary or compound-indexed strategy may include selected +features in its structural load key. + +### `featureCodes` semantics + +| Value | Meaning | +|-------|---------| +| `undefined` | All features | +| `[]` | No features | +| `[1, 2, 3]` | Subset | + +Feature codes must be in the same global code/name space as the feature +catalog. Explicit `{feature_key}_codes` columns are authoritative. For +dictionary-only feature columns, raw dictionary indices are local to a +chunk/row group/part and must not be treated as global feature codes; derive +filter row codes from decoded names using the catalog mapping instead. + +### Large-dataset catalog strategy (`listPointsFeatures`) + +1. If `feature_name_codes` (or `{feature_key}_codes`) present → row-group scan + of feature columns. +2. Else if catalog empty or all blank names after row groups → + `loadParquetTable` feature columns only (current fallback). +3. Small datasets → full `loadParquetTable` with feature columns (unchanged). + +--- + +## Files touched (main areas) + +| Area | Files | +|------|-------| +| Core load/filter | `packages/core/src/models/VPointsSource.ts`, `VTableSource.ts` | +| Feature catalog | `packages/core/src/pointsFeatures.ts` | +| Worker | `packages/core/src/workers/points-worker.ts`, `pointsWorkerClient.ts` | +| Vis preload/filter | `packages/vis/src/SpatialCanvas/useLayerData.ts`, `pointsLoadPlan.ts` | +| Layer filter cache | `packages/layers/src/PointsLayer.ts` | +| UI | `packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx` | + +--- + +## Open questions + +1. Is a one-time 30s catalog acceptable if cached for the session, or must + first open be sub-second? +2. Should we require Morton + `feature_name_codes` for large transcript datasets + in production, treating plain `transcripts` as legacy? +3. Should catalog/counts move entirely to the worker before further vis work? +4. What metadata should writers emit so the resolver can distinguish spatial + Morton, feature-primary, and compound spatial+feature indexes? +5. When should feature filter changes reload source data instead of filtering a + preloaded batch? + +--- + +## Changelog (this effort) + +- Runtime feature filter on preloaded scatter (no geometry reload on toggle). +- Row-group capped reads for geometry preload. +- Feature catalog fixes for dictionary-encoded large datasets (fallback read). +- Safe dictionary index extraction; catalog retry on `null` cache. +- Removed debug instrumentation (2026-06-20).