diff --git a/docs/parquet-wasm-limitations.md b/docs/parquet-wasm-limitations.md new file mode 100644 index 00000000..ee5c2ebf --- /dev/null +++ b/docs/parquet-wasm-limitations.md @@ -0,0 +1,119 @@ +# parquet-wasm limitations (and what we'd ideally have) + +**Status:** notes, 2026-07-07. Context for the points feature-filter perf work +(see [points MVP roadmap](plans/points-mvp-and-roadmap.md)). + +We vendor [`parquet-wasm`](https://github.com/kylebarron/parquet-wasm) for all +browser parquet decoding (`packages/core/src/parquetWasmLoader.ts`, +`packages/core/vendor/parquet-wasm/`). It works well, but its API shape forces a +specific trade-off for large transcripts `points.parquet` files, and it's worth +recording precisely what it can and cannot do so we can evaluate alternatives +(different bindings, a patched build, or hand-rolled footer parsing) later. + +## The capability we have + +The normalized surface we rely on (`ParquetModule` in `parquetWasmLoader.ts`): + +- `readParquet(fileBytes, { columns?, limit?, offset? })` — decode a **complete + parquet file** buffer, with column **projection during decode**. +- `readParquetRowGroup(schemaBytes, rowGroupBytes, rowGroupIndex, { columns? })` + — decode a **single row group** from its bytes + the file's schema bytes, again + projecting columns during decode. This is what makes per-row-group range reads + possible (`readParquetRowGroupBytesByGroupIndex` fetches + `[rowGroup.fileOffset(), rowGroup.compressedSize()]`). +- `readMetadata(footerBytes)` → row-group metadata exposing **only** + `numRows()`, `fileOffset()`, `compressedSize()` per row group + (`ParquetWasmRowGroupMetadata`). + +## The limitation that bites + +**There is no way to fetch or decode an individual column chunk.** Consequences: + +1. **No projected *fetch*.** Column projection (`{ columns }`) happens only + *during decode*; the bytes handed to `readParquet` / `readParquetRowGroup` + must be the **whole file** or the **whole row group** — i.e. *all* columns. + For a 14-column Xenium `transcripts` file, building the feature catalog or the + per-row feature codes (which need one string column) still downloads every + column's bytes. +2. **No column-chunk offsets in the metadata.** `ParquetWasmRowGroupMetadata` + does not expose per-`ColumnChunk` `file_offset` / `total_compressed_size` / + `data_page_offset` / `dictionary_page_offset`. Without those we cannot compute + the byte ranges of just the columns we want, so we cannot issue projected + range reads even manually. +3. **Row-group bytes are not relocatable.** Per + [kylebarron/parquet-wasm#804](https://github.com/kylebarron/parquet-wasm/issues/804) + ("How to read a single row group batch, given only the row group bytes and the + schema bytes"), the footer's byte offsets are absolute to the original file, + so you cannot hand-concatenate a subset of column chunks into a synthetic row + group and decode it — the offsets no longer line up. `readParquetRowGroup` + sidesteps this by taking `schemaBytes` separately and the *contiguous* row-group + bytes, but that means the whole (all-column) row group must be fetched. + +The practical upshot: on a large transcripts file the cost is dominated by +**decoding** the feature/geometry columns, and the only lever we have to keep the +UI responsive is to move that **decode** off the main thread — *not* to fetch +less. We still fetch whole row groups (all columns) via async range reads, but +the CPU-heavy decode runs in the points worker. See the off-thread +geometry+features decode in `VPointsSource.loadPoints` / +`decodeGeometryWithFeaturesFromPayload`. + +## Runtime probe (2026-07-07): what the Vitessce build actually exposes + +The `.d.ts` types `readMetadata` as `unknown` and our `ParquetModule` wrapper only +surfaces `numRows/fileOffset/compressedSize`, but the underlying wasm object +exposes **more** than the wrapper. Introspecting the live object: + +- `ParquetMetaData`: `fileMetadata()`, `numRowGroups()`, `rowGroup(i)`, `rowGroups()`. +- `RowGroupMetaData`: `numColumns()`, `column(j)`, `columns()`, `numRows()`, + `totalByteSize()`, `compressedSize()`, `fileOffset()`. +- `ColumnChunkMetaData`: `filePath()`, `fileOffset()`, `columnPath()`, + `encodings()`, `numValues()`, `compression()`, `compressedSize()`, + `uncompressedSize()`. +- **`ColumnChunkMetaData.statistics()` does NOT exist** — `col.statistics` is + `null`. So per-column-chunk **min/max are not reachable** even though the parquet + footer contains them (pyarrow reads them fine). + +So the situation is more nuanced than "no column info": + +- **Column-chunk *offsets* ARE available** (`column(j).fileOffset()` + + `compressedSize()` + `columnPath()`). A projected byte range per column is + computable. What still blocks a projected *fetch* is #804: `readParquetRowGroup` + needs the *contiguous* row-group bytes, and hand-concatenating a subset of column + chunks breaks the footer offsets — so we can compute the ranges but not feed a + sparse buffer back in for decode. +- **Column *statistics* are NOT available.** This blocks the **feature-primary + index** (skipping row groups whose feature range doesn't overlap the selected + genes): we'd need per-row-group `feature_name_codes` min/max to pick the ~3 of + 245 row groups a gene lives in, and the wasm won't give them. Reading them via + first/last-row reads (`loadParquetRowGroupColumnExtent`) fetches the *whole* + row group each time — fetching the entire 449 MB file just to build the index. + Getting stats efficiently needs one of: (a) a JS parse of the footer's Thrift + `FileMetaData` for `Statistics.min/max_value`; (b) an alternative metadata reader + (e.g. hyparquet, pure-JS, exposes row-group column stats); (c) extending the + vitessce/parquet-wasm build to surface `.statistics()`; or (d) a sidecar + per-row-group feature index emitted by the writer. + +## What we'd ideally have + +Roughly in priority order for our use case (points/transcripts): + +1. **Column-chunk offsets in the metadata** — expose `ColumnChunkMetaData` + (`file_offset`, `total_compressed_size`, `data_page_offset`, + `dictionary_page_offset`) so we can compute per-column byte ranges. This alone + unlocks projected fetching. +2. **A row-group decode that accepts a *sparse* set of column-chunk buffers** + (chunk bytes + which column + within-row-group offset), so we can fetch only + the columns we project and decode them without the whole row group. This is + exactly the ask in #804. +3. **Dictionary-page-only reads** — for categorical columns (e.g. `feature_name` + as `dictionary`), the distinct values live in per-row-group dictionary + pages. Reading just those would build a feature catalog from a few KB per row + group instead of decoding the full column. +4. **An async/range-read reader with random row-group access** that manages byte + sourcing itself (custom store), so we don't shuttle raw bytes across the + worker boundary. + +We're open to evaluating alternative bindings or extending/patching the vendored +build to get (1)–(3); (1) is the highest leverage and smallest change. Until +then, off-thread decode (fetch-all-columns, decode-projected-off-thread) is the +pragmatic ceiling. diff --git a/docs/plans/points-mvp-and-roadmap.md b/docs/plans/points-mvp-and-roadmap.md index f2b6fdb9..3e8a5d60 100644 --- a/docs/plans/points-mvp-and-roadmap.md +++ b/docs/plans/points-mvp-and-roadmap.md @@ -133,5 +133,19 @@ strategy — not a big-bang rewrite of the god-hook. escape hatch and "all on" is the least surprising.) 2. Worker-backed catalog: dictionary-from-metadata fast path vs. same-bytes off-thread decode. (Status doc P0 recommends metadata fast path first.) + **Finding (2026-07-06, live on a real Xenium `transcripts`):** the current + worker catalog path (`readParquetWorkerPayload` with `fullPartsForFallback` → + `scanParquetFeatureCatalogInWorker`) fetches the **entire** parquet file + before scanning, whereas the main-thread path does a *projected* single-column + range read of just the feature column. For a transcripts element with **no + `{feature_key}_codes` column** (so the cheap row-group *dictionary-page* scan + can't run), enabling the worker regressed catalog build from ~20s to >150s. + The request timeout in `pointsWorkerClient` (added this cycle) makes a silent + worker fall back safely, but does **not** fix this — the fetch is before the + worker call. **Next perf task:** give the worker a *projected/dictionary-only* + payload path (fetch only the feature column, or read dictionary pages) so + worker-offload is a win, not a regression — only then enable the worker in the + demo for catalog building. Until then the demo keeps the (blocking but faster) + main-thread path. 3. Engine submodule placement and one-object-vs-per-type facade — deferred to the decomposition plan's open questions. diff --git a/docs/plans/points-redesign-punchlist.md b/docs/plans/points-redesign-punchlist.md new file mode 100644 index 00000000..1450a5d5 --- /dev/null +++ b/docs/plans/points-redesign-punchlist.md @@ -0,0 +1,97 @@ +# Points — pre-merge punch-list & redesign backlog + +Purpose: draw a clean line under the `points-feature-filter` PR before a larger +redesign. Everything here is either **fix-before-merge** (cheap, durable, or +stops a regression / stops the UI lying) or **defer-to-redesign** (entangled with +the state model, so patching now is throwaway). + +## Root cause the redesign targets + +Most of the "wrong points / wrong stats" issues are one problem, not many: +`PointsEntry` (in `PointsDataEngine.ts`) is an **imperative mutable record** whose +fields are flipped with side effects mid-flight — `matchingLoading`, +`partialResult`, the atomic-swap on `ensureLoaded`, `reconcileRowCodes`, +`onProgress` mutating `loading` in place. That state is read through the +**monolithic `useLayerData`** and kept reactive only via `'use no memo'` escape +hatches. The decision of *which points to show* and *what the stats say* is spread +across those mutation sites, so it's ad-hoc and easy to get subtly wrong. + +The redesign — **break up `useLayerData`** and **spike Effect / TanStack Query** +scoped to this runtime — is what fixes the *class*. Individual selection/stats +bugs below marked "defer" are downstream of it: fix them there, with an explicit +state model, not by patching mutations here. + +--- + +## Fix-before-merge + +| # | Item | Where | Kind | Note | +|---|------|-------|------|------| +| F1 | **Deselected features reappear while a covering scan streams** | `useLayerData` getLayers partial overlay | render-breaking | The partial overlay draws the buffer with **no selection filter**, unlike the settled matched layer (which passes `featureCodes` + `preloadedFeatureCodes`). Deselect a feature whose scan is still in flight → engine keeps that scan ("covered"), its partial keeps the deselected rows, overlay shows them until settle. **Introduced by this PR.** Cheap fix: pass the same filter props to the overlay (the partial's own `featureCodes` are available). Or revert the overlay. | +| F2 | **Delete dead `pointsRenderer.ts`** | `vis/.../renderers/pointsRenderer.ts` | hygiene | `renderPointsLayer` + its interfaces have **zero importers** (superseded by `PointsLayer`). Safe delete; leaves a cleaner starting line. | +| F3 | **Stop the summary line lying** | `PointsLayerPanel.ShowMatchingPoints` (`t.loaded`) | cosmetic (wrong number) | `t.loaded` is the covered-batch size, not the count matching the current selection, so "Loaded all N …" is often wrong. *Proper* fix needs the engine to count selection-matched rows = redesign. For merge: make the line honest cheaply (show a number that's actually right, or drop the misleading clause). | +| F4 | **Resolve the working tree** | `PointsDataEngine`, `PointsFeatureFilterPanel`, `PointsLayerPanel`, `models/index`, `pointsRenderer` (all uncommitted) | hygiene | Includes dangling notes (`// how do I get the engine from the context?`). Commit-or-revert each so the branch is coherent. | + +**Undecided (cheap either way):** + +- **U1 — overlay compositing.** Today the partial is a *separate sub-layer on top + of* the base (resident / prior matched), so during a scan you see both. Your + call: keep base+partial, or show partial-only during a scan. Small render + change in getLayers; orthogonal to F1 (F1 is about *filtering* the partial, + this is about *whether the base also draws*). Fine to defer. + +--- + +## Defer-to-redesign + +Each notes *why* it's coupled to the state-model / decode rework. + +- **D1 — Mutable `PointsEntry` state model → Effect / TanStack Query.** The root + above. In-code smells already flagged: `PointsDataEngine.ts` `// I'm a bit iffy + about this ambient stateful thing` (onProgress), `// given ongoing problems with + agent debugging, inclined to more purity. Might consider using + Effect?`, `// there will be various mutating side-effects on entry…`. +- **D2 — Break up `useLayerData`.** The monolith the engine threads through; also + the reason for the `'use no memo'` hatches (`PointsFeatureFilterPanel`, + `ShowMatchingPoints`). A properly reactive state layer retires the hatches. +- **D3 — Progressive *initial* load (`loadPoints`).** Currently one-shot (bulk + fetch + single worker decode); making it progressive needs a per-part decode + loop **and** a general engine "partial resident" slot (the partial mechanism is + matching-specific today). The engine rework owns this. `pointsScanChunkProgress` + is already the reusable producer helper when we get there. +- **D4 — Progressive / active feature stats before the full catalog scan + completes.** Today stats only appear once the whole-dataset catalog settles; + there's real use in showing progressive/active counts. Tied to D3 (progressive + catalog build) and the stats state model (F3's proper fix). +- **D5 — Tiled (Morton) viewport-driven loading.** The tiled path isn't exercised; + viewport-driven load is a major feature and exactly the kind of demand-driven + state the new model should own (Morton tiling is still "dark" per the roadmap). +- **D6 — Worker contention with multiple layers.** Multiple point layers share + one worker; the engine keys by element and assumes single-demand-per-element. + Multi-layer sharing / a work queue belongs with the engine redesign. +- **D7 — GeoArrow encoding.** Unexplored; a decode-path spike, not this PR. +- **D8 — Streaming cancellation semantics.** The generators have no `AbortSignal` + threaded to the worker, and an abandoned manual `.next()` loop won't clean up. + Fine while consumers drain; design it with the new state layer. +- **D9 — Remove `'use no memo'` hatches (stable-snapshot option).** Give the + engine stable-identity snapshot accessors so `useSyncExternalStore` tracks the + value directly and the compiler stops needing an opt-out. Part of D1/D2. +- **D10 — Progressive-overlay visibility logic + flashing.** F1 fixed the + deselected-feature-lingering slice, but *which* points show during a partial + load still has logic problems, and it **flashes badly**: every notify rebuilds + the partial buffer into a fresh `PointsRenderResource` (new identity each + chunk), so deck tears down and recreates the `__partial` layer per step instead + of updating it in place. The real fix is a stable growing GPU buffer (preallocate + to cap, append, bump a draw count via `updateTriggers`) rather than a + rebuilt-per-chunk resource — which is the same append-buffer work noted for D3 + and the `pointsScanChunkProgress` O(chunks²) concat. Owned by the engine + + render redesign; the current overlay is a spike, not the destination. + +--- + +## Suggested merge line + +Do **F1–F4** (+ decide **U1**), confirm no regression vs `main`, tests + types +green. That yields a merged state that is *correct, honest, coherent, and +non-regressed* — without trying to make the selection logic *right*, which rides +the redesign (D1/D2). Everything in **Defer** stays untouched. diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index 92c09091..f05c98ef 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -10,7 +10,11 @@ export * from './store/index.js'; export * from './models/index.js'; export * from './spatialViewFit.js'; export * from './pointsTiling.js'; -export { mergeFeatureCountsIntoCatalog } from './pointsFeatures.js'; +export { + featureCodeMapFromCatalog, + mergeFeatureCountsIntoCatalog, + remapRowFeatureCodes, +} from './pointsFeatures.js'; export { POINTS_PRELOAD_MAX_ROWS, DEFAULT_POINTS_MEMORY_CAP, @@ -32,6 +36,7 @@ export { filterColumnarByFeatureCodesInWorker, isPointsWorkerEnabled, setPointsWorkerDefaultEnabled, + setPointsWorkerRequestTimeout, } from './workers/index.js'; export { createMortonTiledPointsLoader, diff --git a/packages/core/src/models/VPointsSource.ts b/packages/core/src/models/VPointsSource.ts index 968aeeb4..0d0fc7c0 100644 --- a/packages/core/src/models/VPointsSource.ts +++ b/packages/core/src/models/VPointsSource.ts @@ -6,6 +6,7 @@ import { resolveRowFeatureCodesFromTable, } from '../pointsFeatures.js'; import { + decodeGeometryWithFeaturesInWorker, decodeParquetGeometryCappedInWorker, decodeParquetRowFeatureCodesInWorker, ensurePointsWorker, @@ -16,6 +17,10 @@ import { scanParquetFeatureCountsInWorker, } from '../workers/pointsWorkerClient.js'; import { exceedsPointsPreloadLimit, resolvePointsMemoryCap } from '../pointsLimits.js'; +import { + decodeIntStat, + parseParquetFileMetaData, +} from '../parquetFooterStats.js'; import type { PointsLoadOptions, PointsLoadProgress, @@ -25,6 +30,69 @@ import type { interface ColumnarPointsChunk { shape: number[]; data: ArrayLike[]; + featureCodes?: ArrayLike; +} + +/** Inclusive `[min, max]` code range a row group's feature-code column spans. */ +interface FeatureCodeExtent { + min: number; + max: number; +} + +/** + * Per-row-group `[min, max]` for the feature-code column, parsed from each part's + * footer statistics and flattened into global row-group order. Powers the + * feature-primary index: a row group whose range can't contain any selected code + * is skipped without fetching it. Returns `[]` to signal "stats unavailable — + * scan everything" (footer parse failed, a column had no statistics, or the + * flattened count didn't match the dataset's row-group count). An entry is `null` + * when that specific row group lacks usable stats, so it is scanned rather than + * wrongly skipped. + */ +function rowGroupFeatureCodeExtents( + parts: readonly { schemaBytes: Uint8Array }[], + featureCodeColumnName: string, + expectedRowGroupCount: number +): Array { + const extents: Array = []; + for (const part of parts) { + // `schemaBytes` is the parquet footer: FileMetaData thrift + trailing 4-byte + // length + "PAR1". Strip the trailing 8 to get the FileMetaData for the parser. + if (part.schemaBytes.length <= 8) { + return []; + } + const metaBytes = part.schemaBytes.subarray(0, part.schemaBytes.length - 8); + let footer; + try { + footer = parseParquetFileMetaData(metaBytes); + } catch { + return []; + } + for (const rowGroup of footer.rowGroups) { + const column = rowGroup.columns.find((col) => col.path === featureCodeColumnName); + if (!column) { + extents.push(null); + continue; + } + const min = decodeIntStat(column.minValue, column.physicalType); + const max = decodeIntStat(column.maxValue, column.physicalType); + extents.push(min !== null && max !== null ? { min, max } : null); + } + } + return extents.length === expectedRowGroupCount ? extents : []; +} + +/** Whether a row group's code range can contain any selected code. `null` extent + * (missing stats) is treated as "might match" so it is scanned, not skipped. */ +function extentMayContainSelectedCodes( + extent: FeatureCodeExtent | null, + selectedMin: number, + selectedMax: number +): boolean { + if (!extent) { + return true; + } + return extent.max >= selectedMin && extent.min <= selectedMax; } function emptyFilteredPointsResult(axisNames: string[], totalRowCount: number): PointsLoadResult { @@ -41,13 +109,14 @@ function emptyFilteredPointsResult(axisNames: string[], totalRowCount: number): } function toColumnarPointsChunk( - data: { shape?: number[]; data: ArrayLike[] }, + data: { shape?: number[]; data: ArrayLike[]; featureCodes?: ArrayLike }, axisCount: number ): ColumnarPointsChunk { const rowCount = data.shape?.[1] ?? data.data[0]?.length ?? 0; return { shape: data.shape ?? [axisCount, rowCount], data: data.data, + ...(data.featureCodes ? { featureCodes: data.featureCodes } : {}), }; } @@ -75,7 +144,68 @@ function concatColumnarPointChunks(chunks: ColumnarPointsChunk[]): ColumnarPoint } return merged; }); - return { shape: [axisCount, totalRows], data }; + // Concatenate per-point feature codes in lockstep when every chunk carries + // them (they do when the source resolved a feature key). + let featureCodes: Int32Array | undefined; + if (chunks.every((chunk) => chunk.featureCodes)) { + featureCodes = new Int32Array(totalRows); + let offset = 0; + for (const chunk of chunks) { + const codes = chunk.featureCodes as ArrayLike; + const values = codes instanceof Int32Array ? codes : Int32Array.from(codes); + featureCodes.set(values, offset); + offset += values.length; + } + } + return { shape: [axisCount, totalRows], data, ...(featureCodes ? { featureCodes } : {}) }; +} + +/** + * Build the per-chunk streaming payload for a progressive points scan: the + * latest decoded chunk plus a `progress` whose `partialResult` is the GROWING + * buffer of everything matched so far (all `accumulatedChunks` concatenated), so + * a consumer can render points that accumulate rather than flash past. + * + * Shared by both scan branches (row-group / parts) here, and intended for reuse + * by other `VPointsSource` scans that want progressive display. The buffer is + * re-concatenated each chunk, so total copy work grows with the square of the + * *chunk count* — negligible in practice (a feature-indexed scan touches only a + * handful of row groups; the parts path is bounded by part count). Only worth + * replacing with a preallocated append buffer if a scan ever yields very many + * small chunks. + */ +function pointsScanChunkProgress( + accumulatedChunks: ColumnarPointsChunk[], + latest: ColumnarPointsChunk, + counts: { + scannedRows: number; + matchedRows: number; + totalRowCount: number; + memoryCap: number; + partIndex: number; + partCount: number; + } +): { chunk: ColumnarPointsChunk; progress: PointsLoadProgress } { + const buffer = concatColumnarPointChunks(accumulatedChunks); + const partialResult: PointsLoadResult = { + shape: buffer.shape, + data: buffer.data, + ...(buffer.featureCodes ? { featureCodes: buffer.featureCodes } : {}), + totalRowCount: counts.totalRowCount, + scannedRowCount: counts.scannedRows, + filterActive: true, + preloadTruncated: counts.matchedRows >= counts.memoryCap, + }; + return { + chunk: latest, + progress: { + scannedRows: counts.scannedRows, + matchedRows: counts.matchedRows, + partIndex: counts.partIndex, + partCount: counts.partCount, + partialResult, + }, + }; } import { MORTON_CODE_2D_COLUMN, @@ -254,6 +384,7 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { elementPath: string, options: PointsLoadOptions = {} ): Promise { + checkAbort(options.signal); const memoryCap = resolvePointsMemoryCap(options.memoryCap); if (options.featureCodes !== undefined && options.fullDatasetFeatureScan === true) { return this.loadPointsMatchingFeatureCodes(elementPath, { @@ -273,34 +404,100 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { const maxRows = truncatePreload ? memoryCap : rowCount; const columnNames = [...axisNames]; + // Optionally read the feature column(s) in the same projected, capped preload + // so the filter's catalog + per-row codes come from one decode — no separate + // blocking load at filter time (PointsLoadOptions.includeFeatureCodes). + const configuredFeatureKey = zattrs.spatialdata_attrs?.feature_key; + const wantFeatures = + options.includeFeatureCodes === true && + typeof configuredFeatureKey === 'string' && + configuredFeatureKey.length > 0; + const featureKey = wantFeatures ? (configuredFeatureKey as string) : undefined; + let featureCodeColumnName: string | undefined; + if (featureKey) { + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + const schemaTable = datasetMetadata ? null : await this.loadParquetSchemaTable(parquetPath); + const fields = datasetMetadata?.schema?.fields + ? datasetMetadata.schema.fields.flatMap((field) => + typeof field.name === 'string' ? [field.name] : [] + ) + : arrowSchemaFieldNames(schemaTable); + featureCodeColumnName = selectFeatureCodeColumn(fields, featureKey); + columnNames.push(featureKey); + if (featureCodeColumnName) { + columnNames.push(featureCodeColumnName); + } + } + ensurePointsWorker(); if (isPointsWorkerEnabled()) { try { - const payload = await this.readParquetWorkerPayload(parquetPath, { maxRows }); - const workerGeometry = await decodeParquetGeometryCappedInWorker( - { + if (featureKey) { + // Off-thread the codes-with-geometry decode: fetch whole row-group (or + // part) bytes via async range reads, then decode geometry + per-row + // codes + catalog in the worker so the CPU-heavy decode never blocks the + // main thread. parquet-wasm cannot fetch individual column chunks, so we + // still fetch all columns' bytes — see docs/parquet-wasm-limitations.md. + const payload = await this.fetchParquetPayloadCapped(parquetPath, maxRows); + const workerResult = payload + ? await decodeGeometryWithFeaturesInWorker({ + ...payload, + axisNames, + columns: columnNames, + maxRows, + featureKey, + featureCodeColumnName, + }) + : null; + if (workerResult) { + return { + shape: workerResult.shape as [number, number], + data: workerResult.data, + totalRowCount: rowCount, + preloadTruncated: truncatePreload, + hasFeatureCodeColumn: featureCodeColumnName !== undefined, + ...(workerResult.featureCodes ? { featureCodes: workerResult.featureCodes } : {}), + ...(workerResult.featureCatalog + ? { featureCatalog: workerResult.featureCatalog } + : {}), + }; + } + } else { + const payload = await this.readParquetWorkerPayload(parquetPath, { maxRows }); + const workerGeometry = await decodeParquetGeometryCappedInWorker({ parts: payload.parts, axisNames, columns: columnNames, maxRows, + }); + if (workerGeometry) { + return { + shape: workerGeometry.shape as [number, number], + data: workerGeometry.data, + totalRowCount: rowCount, + preloadTruncated: truncatePreload, + // No feature key requested/available on this branch → no code column. + hasFeatureCodeColumn: false, + }; } - ); - if (workerGeometry) { - return { - shape: workerGeometry.shape as [number, number], - data: workerGeometry.data, - totalRowCount: rowCount, - preloadTruncated: truncatePreload, - }; } } catch (error) { + // An abort is intentional — don't swallow it into the main-thread fallback. + if (error instanceof DOMException && error.name === 'AbortError') { + throw error; + } console.warn( - `Worker geometry preload failed for ${elementPath}; falling back to main thread.`, + `Worker points preload failed for ${elementPath}; falling back to main thread.`, error ); } } + // Guard the expensive main-thread fallback: if the load was superseded (e.g. + // the memory cap changed), bail here rather than decode a whole capped table + // on the main thread — the case that crashed the tab on large datasets. + checkAbort(options.signal); + const { table: arrowTable, totalRows, @@ -315,28 +512,82 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { return column.toArray(); }); + let featureCodes: ArrayLike | undefined; + let featureCatalog: PointsFeatureCatalog | undefined; + if (featureKey) { + const nameColumn = arrowTable.getChild(featureKey); + if (nameColumn) { + const codeColumn = featureCodeColumnName ? arrowTable.getChild(featureCodeColumnName) : null; + featureCatalog = buildFeatureCatalogFromColumns( + featureKey, + nameColumn, + codeColumn ?? null, + null, + arrowTable.numRows + ); + const featureCodeByName = featureCodeColumnName + ? undefined + : featureCodeMapFromCatalog(featureCatalog); + featureCodes = resolveRowFeatureCodesFromTable( + arrowTable, + featureKey, + featureCodeColumnName, + featureCodeByName + ); + } + } + return { shape: [axisColumnArrs.length, arrowTable.numRows], data: axisColumnArrs, totalRowCount: totalRows, preloadTruncated: truncated, + hasFeatureCodeColumn: featureCodeColumnName !== undefined, + ...(featureCodes ? { featureCodes } : {}), + ...(featureCatalog ? { featureCatalog } : {}), }; } - private async loadPointsMatchingFeatureCodes( + /** + * Fetch enough parquet bytes (via async range reads) to cover `maxRows` for a + * worker decode. Uses whole-part reads (decoded with `readParquet`) rather than + * per-row-group reads: `readParquetRowGroup` mis-decodes dictionary-encoded + * columns (e.g. `feature_name`) — the same reason `scanFeatureCatalogFromPayload` + * falls back to parts — which would corrupt the catalog + codes. The fetch is + * async I/O only; the CPU-heavy decode happens in the worker. Returns `null` if + * no bytes are available. + */ + private async fetchParquetPayloadCapped( + parquetPath: string, + maxRows: number + ): Promise<{ parts: Uint8Array[] } | null> { + const { parts } = await this.readParquetDatasetBytesCapped(parquetPath, maxRows); + return parts.length > 0 ? { parts } : null; + } + + async* loadPointsMatchingFeatureCodesByChunk( elementPath: string, options: { memoryCap: number; featureCodes: readonly number[]; - onProgress?: (progress: PointsLoadProgress) => void; - } - ): Promise { + // no onProgress side-effect here, it's part of what we yield + // we *do* want an AbortSignal, though. + abort?: AbortSignal; + /** Authoritative name→code map for dict-only elements (no `*_codes` + * column), letting the scan resolve each row's `feature_name` to the same + * code space the selection was made in. When absent for a dict-only + * element the scan cannot match by name and returns nothing. */ + featureCodeByName?: ReadonlyMap; + } + ) { ensurePointsWorker(); const parquetPath = getParquetPath(elementPath); const zattrs = await this.loadSpatialDataElementAttrs(elementPath); + // if (options.abort?.aborted) return; const { axes, spatialdata_attrs: spatialDataAttrs } = zattrs; const normAxes = normalizeAxes(axes); const axisNames = normAxes.map((axis: { name: string }) => axis.name); + const axisCount = axisNames.length; const featureKey = spatialDataAttrs?.feature_key; if (typeof featureKey !== 'string' || featureKey.length === 0) { throw new Error(`Points element "${elementPath}" is missing feature_key metadata.`); @@ -344,7 +595,9 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { const totalRowCount = await this.resolveParquetRowCount(parquetPath); if (options.featureCodes.length === 0) { - return emptyFilteredPointsResult(axisNames, totalRowCount); + // Nothing selected: yield no chunks and return the summary. The collector + // turns "no chunks matched" into an empty result via emptyFilteredPointsResult. + return { totalRowCount, axisNames, scannedRows: 0, matchedRows: 0 }; } if (!isPointsWorkerEnabled()) { @@ -357,8 +610,8 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { const schemaTable = datasetMetadata ? null : await this.loadParquetSchemaTable(parquetPath); const fields = datasetMetadata?.schema?.fields ? datasetMetadata.schema.fields.flatMap((field) => - typeof field.name === 'string' ? [field.name] : [] - ) + typeof field.name === 'string' ? [field.name] : [] + ) : arrowSchemaFieldNames(schemaTable); const featureCodeColumnName = selectFeatureCodeColumn(fields, featureKey); @@ -369,22 +622,69 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { columnNames.push(featureKey); } - const matchedChunks: ColumnarPointsChunk[] = []; + // Dict-only elements have no file-backed code column, so the worker resolves + // each row's `feature_name` against this authoritative map (from the caller's + // catalog) into the same code space the selection uses. A no-op for indexed + // elements (they match on `featureCodeColumnName`). + const featureCodeEntries = + !featureCodeColumnName && options.featureCodeByName + ? [...options.featureCodeByName].map(([name, code]) => ({ name, code })) + : undefined; + let matchedRows = 0; let scannedRows = 0; - - const canUseRowGroups = await this.canLoadParquetRowGroups(); + // Growing buffer of every matched chunk so far — `pointsScanChunkProgress` + // concatenates it into each `progress.partialResult` for progressive display. + const accumulatedChunks: ColumnarPointsChunk[] = []; + + // Row-group scanning only helps when a feature-code column lets footer stats + // skip row groups (feature-ordered index). Dict-only elements have no stats to + // skip on, so the row-group path would scan every group anyway — and its + // projected decode of the *dictionary* feature_name column is unreliable for + // multipart stores. Route dict-only scans through the parts path, which the + // catalog build already uses successfully. + const canUseRowGroups = + featureCodeColumnName !== undefined && (await this.canLoadParquetRowGroups()); const datasetRowGroups = datasetMetadata?.totalNumRowGroups ?? 0; if (canUseRowGroups && datasetRowGroups > 0) { + // Feature-primary index: skip row groups whose feature-code range cannot + // contain any selected code. For a feature-ordered file this leaves only + // the few row groups a gene actually lives in, so we fetch/decode almost + // nothing; unsorted files get `[]` (no stats) and fall back to a full scan. + const selectedMin = Math.min(...options.featureCodes); + const selectedMax = Math.max(...options.featureCodes); + const rowGroupExtents = + featureCodeColumnName && datasetMetadata + ? rowGroupFeatureCodeExtents( + datasetMetadata.parts, + featureCodeColumnName, + datasetRowGroups + ) + : []; + const canSkipRowGroups = rowGroupExtents.length === datasetRowGroups; + for (let rowGroupIndex = 0; rowGroupIndex < datasetRowGroups; rowGroupIndex += 1) { if (matchedRows >= options.memoryCap) { break; } + if ( + canSkipRowGroups && + !extentMayContainSelectedCodes( + rowGroupExtents[rowGroupIndex], + selectedMin, + selectedMax + ) + ) { + continue; + } const chunk = await this.readParquetRowGroupBytesByGroupIndex(parquetPath, rowGroupIndex); if (!chunk) { continue; } + // the memoryCap could work by the consumer choosing not to exhaust the stream + // (although that wouldn't help to pass last worker invocation a smaller chunk size) + // we should be passing abort to worker const partial = await scanParquetByFeatureCodesInWorker({ rowGroups: [chunk], axisNames, @@ -392,21 +692,25 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { featureCodeColumnName, featureCodes: options.featureCodes, memoryCap: options.memoryCap - matchedRows, + ...(featureCodeEntries ? { featureCodeEntries } : {}), }); if (!partial) { throw new Error('Feature-filtered points loading requires the points worker.'); } scannedRows += partial.scannedRows; if (partial.matchedRows > 0) { - matchedChunks.push(toColumnarPointsChunk(partial.data, axisNames.length)); matchedRows += partial.matchedRows; + const chunk = toColumnarPointsChunk(partial.data, axisCount); + accumulatedChunks.push(chunk); + yield pointsScanChunkProgress(accumulatedChunks, chunk, { + scannedRows, + matchedRows, + totalRowCount, + memoryCap: options.memoryCap, + partIndex: rowGroupIndex, + partCount: datasetRowGroups, + }); } - options.onProgress?.({ - scannedRows, - matchedRows, - partIndex: rowGroupIndex, - partCount: datasetRowGroups, - }); } } else { let partPaths: string[]; @@ -432,33 +736,65 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { featureCodeColumnName, featureCodes: options.featureCodes, memoryCap: options.memoryCap - matchedRows, + ...(featureCodeEntries ? { featureCodeEntries } : {}), }); if (!partial) { throw new Error('Feature-filtered points loading requires the points worker.'); } scannedRows += partial.scannedRows; if (partial.matchedRows > 0) { - matchedChunks.push(toColumnarPointsChunk(partial.data, axisNames.length)); matchedRows += partial.matchedRows; + const chunk = toColumnarPointsChunk(partial.data, axisCount); + accumulatedChunks.push(chunk); + yield pointsScanChunkProgress(accumulatedChunks, chunk, { + scannedRows, + matchedRows, + totalRowCount, + memoryCap: options.memoryCap, + partIndex, + partCount: partPaths.length, + }); } - options.onProgress?.({ - scannedRows, - matchedRows, - partIndex, - partCount: partPaths.length, - }); } } - - const data = concatColumnarPointChunks(matchedChunks); - return { - shape: data.shape, - data: data.data, - totalRowCount, - scannedRowCount: scannedRows, - filterActive: true, - preloadTruncated: matchedRows >= options.memoryCap, - }; + return { totalRowCount, axisNames, scannedRows, matchedRows }; + } + async loadPointsMatchingFeatureCodes( + elementPath: string, + options: { + memoryCap: number; + featureCodes: readonly number[]; + onProgress?: (progress: PointsLoadProgress) => void; + /** Authoritative name→code map for dict-only elements (no `*_codes` + * column), letting the scan resolve each row's `feature_name` to the same + * code space the selection was made in. When absent for a dict-only + * element the scan cannot match by name and returns nothing. */ + featureCodeByName?: ReadonlyMap; + } + ): Promise { + const chunkGenerator = this.loadPointsMatchingFeatureCodesByChunk(elementPath, options); + // Each `progress.partialResult` is already the full accumulated buffer, so the + // last one IS the whole matched batch — no need to re-accumulate/concat here. + // Final totals come from the generator's return value (authoritative: it also + // counts rows scanned after the last match, which the last partial can't see). + let latest: PointsLoadResult | undefined; + while (true) { + const next = await chunkGenerator.next(); + if (next.done) { + const { totalRowCount, scannedRows, matchedRows, axisNames } = next.value; + if (!latest) { + return emptyFilteredPointsResult(axisNames, totalRowCount); + } + return { + ...latest, + totalRowCount, + scannedRowCount: scannedRows, + preloadTruncated: matchedRows >= options.memoryCap, + }; + } + options.onProgress?.(next.value.progress); + latest = next.value.progress.partialResult; + } } private async resolveExplicitFeatureCodeColumn(elementPath: string): Promise<{ diff --git a/packages/core/src/models/VTableSource.ts b/packages/core/src/models/VTableSource.ts index d1aa43e2..3e3116f5 100644 --- a/packages/core/src/models/VTableSource.ts +++ b/packages/core/src/models/VTableSource.ts @@ -435,6 +435,7 @@ export default class SpatialDataTableSource extends AnnDataSource { } protected async resolveParquetRowCount(parquetPath: string): Promise { + // may be better to cache this? we get e.g. a lot of 404 requests for `points.parquet/points.4.parquet` const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); if (datasetMetadata?.totalNumRows) { return datasetMetadata.totalNumRows; diff --git a/packages/core/src/models/index.ts b/packages/core/src/models/index.ts index aa1ba2b0..915eba9b 100644 --- a/packages/core/src/models/index.ts +++ b/packages/core/src/models/index.ts @@ -12,7 +12,7 @@ import { shapesAttrsSchema, tableAttrsSchema, } from '../schemas'; -import type { PointsLoadOptions } from '../pointsLoadOptions.js'; +import type { PointsLoadOptions, PointsLoadProgress } from '../pointsLoadOptions.js'; import type { ShapesRenderData } from '../shapes'; import { isSpatialData, loadFeatureRowIndexByFeatureIndex } from '../tableAssociations'; import { type BaseTransformation, Identity, parseTransforms } from '../transformations'; @@ -545,6 +545,22 @@ export class PointsElement extends AbstractSpatialElement<'points', PointsAttrs> return this.vPoints.loadPointsRowFeatureCodes(`points/${this.key}`, options); } + /** + * Load only the points whose feature code is in `featureCodes`, scanning the + * whole dataset. For a feature-ordered file the footer-stats index skips every + * row group that can't match, so this touches only the few row groups the + * selected features live in; unsorted files fall back to a full scan. + */ + async loadPointsMatchingFeatureCodes(options: { + memoryCap: number; + featureCodes: readonly number[]; + onProgress?: (progress: PointsLoadProgress) => void; + featureCodeByName?: ReadonlyMap; + }) { + //todo generator version of this. + return this.vPoints.loadPointsMatchingFeatureCodes(`points/${this.key}`, options); + } + async loadFeatureCounts() { return this.vPoints.loadFeatureCounts(`points/${this.key}`); } diff --git a/packages/core/src/parquetFooterStats.ts b/packages/core/src/parquetFooterStats.ts new file mode 100644 index 00000000..481fb8e2 --- /dev/null +++ b/packages/core/src/parquetFooterStats.ts @@ -0,0 +1,344 @@ +/** + * Minimal parquet footer reader for per-row-group column statistics. + * + * The vendored parquet-wasm build exposes row-group + column-chunk metadata but + * NOT `ColumnChunkMetaData.statistics()` (see docs/parquet-wasm-limitations.md), + * so we cannot read a column's per-row-group min/max through it. Those values do + * live in the parquet footer's Thrift-encoded `FileMetaData`, so this module + * parses just enough of it (Thrift Compact Protocol) to recover, per row group, + * each column's `path_in_schema`, physical `type`, and `Statistics` min/max. + * + * This powers the feature-primary index: for a feature-ordered points file, the + * `feature_name_codes` min/max per row group lets us skip the row groups that + * cannot contain the selected features, reading only the few that do. + * + * Scope: read-only, and deliberately partial — it extracts the fields we need and + * skips everything else. Not a general Thrift/parquet implementation. + */ + +// Thrift Compact Protocol type ids (field/element types). +const T_STOP = 0; +const T_BOOL_TRUE = 1; +const T_BOOL_FALSE = 2; +const T_BYTE = 3; +const T_I16 = 4; +const T_I32 = 5; +const T_I64 = 6; +const T_DOUBLE = 7; +const T_BINARY = 8; +const T_LIST = 9; +const T_SET = 10; +const T_MAP = 11; +const T_STRUCT = 12; + +/** Parquet physical types (`Type` enum in parquet.thrift). */ +export const ParquetPhysicalType = { + BOOLEAN: 0, + INT32: 1, + INT64: 2, + INT96: 3, + FLOAT: 4, + DOUBLE: 5, + BYTE_ARRAY: 6, + FIXED_LEN_BYTE_ARRAY: 7, +} as const; + +export interface ParquetColumnStats { + /** Dotted `path_in_schema`, e.g. "feature_name_codes". */ + path: string; + /** Physical `Type` id, or null if absent. */ + physicalType: number | null; + /** Raw `Statistics.min_value` (preferred) or deprecated `min`. */ + minValue?: Uint8Array; + /** Raw `Statistics.max_value` (preferred) or deprecated `max`. */ + maxValue?: Uint8Array; +} + +export interface ParquetRowGroupStats { + numRows: number; + columns: ParquetColumnStats[]; +} + +export interface ParquetFooterStats { + numRows: number; + rowGroups: ParquetRowGroupStats[]; +} + +class ThriftCompactReader { + private pos = 0; + constructor(private readonly buf: Uint8Array) {} + + atEnd(): boolean { + return this.pos >= this.buf.length; + } + + private byte(): number { + if (this.pos >= this.buf.length) { + throw new Error('parquet footer: unexpected end of buffer'); + } + return this.buf[this.pos++]; + } + + /** Unsigned LEB128 varint as a JS number (values fit well under 2^53 here). */ + varint(): number { + let result = 0; + let shift = 0; + for (;;) { + const b = this.byte(); + result += (b & 0x7f) * 2 ** shift; + if ((b & 0x80) === 0) { + return result; + } + shift += 7; + if (shift > 63) { + throw new Error('parquet footer: varint too long'); + } + } + } + + /** Zigzag-decoded signed varint. */ + zigzag(): number { + const u = this.varint(); + return (u >>> 1) ^ -(u & 1); + } + + /** Length-prefixed bytes (binary/string). */ + binary(): Uint8Array { + const len = this.varint(); + const start = this.pos; + this.pos += len; + if (this.pos > this.buf.length) { + throw new Error('parquet footer: binary overruns buffer'); + } + return this.buf.subarray(start, start + len); + } + + /** + * Read a struct field header. Returns `{ type: T_STOP }` at the struct end. + * `prevId` carries the compact-protocol field-id delta state within a struct. + */ + fieldHeader(prevId: number): { type: number; id: number } { + const b = this.byte(); + if (b === 0) { + return { type: T_STOP, id: 0 }; + } + const delta = (b & 0xf0) >> 4; + const type = b & 0x0f; + const id = delta === 0 ? this.zigzag() : prevId + delta; + return { type, id }; + } + + /** Read a list/set header: `{ size, elemType }`. */ + listHeader(): { size: number; elemType: number } { + const b = this.byte(); + const elemType = b & 0x0f; + let size = (b & 0xf0) >> 4; + if (size === 0x0f) { + size = this.varint(); + } + return { size, elemType }; + } + + /** Skip a value of the given compact type (for fields we don't care about). */ + skip(type: number): void { + switch (type) { + case T_BOOL_TRUE: + case T_BOOL_FALSE: + return; + case T_BYTE: + this.byte(); + return; + case T_I16: + case T_I32: + case T_I64: + this.varint(); + return; + case T_DOUBLE: + this.pos += 8; + return; + case T_BINARY: + this.binary(); + return; + case T_LIST: + case T_SET: { + const { size, elemType } = this.listHeader(); + for (let i = 0; i < size; i += 1) { + this.skip(elemType); + } + return; + } + case T_MAP: { + const size = this.varint(); + if (size > 0) { + const kv = this.byte(); + const keyType = (kv & 0xf0) >> 4; + const valType = kv & 0x0f; + for (let i = 0; i < size; i += 1) { + this.skip(keyType); + this.skip(valType); + } + } + return; + } + case T_STRUCT: { + let prev = 0; + for (;;) { + const f = this.fieldHeader(prev); + if (f.type === T_STOP) { + return; + } + this.skip(f.type); + prev = f.id; + } + } + default: + throw new Error(`parquet footer: cannot skip thrift type ${type}`); + } + } + + // --- parquet.thrift structure readers ----------------------------------- + + private readStatistics(): { min?: Uint8Array; max?: Uint8Array } { + // Statistics { 1: max (deprecated), 2: min (deprecated), 5: max_value, 6: min_value } + let prev = 0; + let min: Uint8Array | undefined; + let max: Uint8Array | undefined; + let minValue: Uint8Array | undefined; + let maxValue: Uint8Array | undefined; + for (;;) { + const f = this.fieldHeader(prev); + if (f.type === T_STOP) break; + if (f.id === 1 && f.type === T_BINARY) max = this.binary(); + else if (f.id === 2 && f.type === T_BINARY) min = this.binary(); + else if (f.id === 5 && f.type === T_BINARY) maxValue = this.binary(); + else if (f.id === 6 && f.type === T_BINARY) minValue = this.binary(); + else this.skip(f.type); + prev = f.id; + } + return { min: minValue ?? min, max: maxValue ?? max }; + } + + private readColumnMetaData(): ParquetColumnStats { + // ColumnMetaData { 1: type, 3: path_in_schema (list), 12: statistics } + let prev = 0; + let physicalType: number | null = null; + const path: string[] = []; + let stats: { min?: Uint8Array; max?: Uint8Array } = {}; + const decoder = new TextDecoder(); + for (;;) { + const f = this.fieldHeader(prev); + if (f.type === T_STOP) break; + if (f.id === 1 && (f.type === T_I32 || f.type === T_I16)) { + physicalType = this.zigzag(); + } else if (f.id === 3 && f.type === T_LIST) { + const { size, elemType } = this.listHeader(); + for (let i = 0; i < size; i += 1) { + if (elemType === T_BINARY) path.push(decoder.decode(this.binary())); + else this.skip(elemType); + } + } else if (f.id === 12 && f.type === T_STRUCT) { + stats = this.readStatistics(); + } else { + this.skip(f.type); + } + prev = f.id; + } + return { + path: path.join('.'), + physicalType, + ...(stats.min ? { minValue: stats.min } : {}), + ...(stats.max ? { maxValue: stats.max } : {}), + }; + } + + private readColumnChunk(): ParquetColumnStats | null { + // ColumnChunk { 3: meta_data (ColumnMetaData) } + let prev = 0; + let column: ParquetColumnStats | null = null; + for (;;) { + const f = this.fieldHeader(prev); + if (f.type === T_STOP) break; + if (f.id === 3 && f.type === T_STRUCT) { + column = this.readColumnMetaData(); + } else { + this.skip(f.type); + } + prev = f.id; + } + return column; + } + + private readRowGroup(): ParquetRowGroupStats { + // RowGroup { 1: columns (list), 3: num_rows } + let prev = 0; + const columns: ParquetColumnStats[] = []; + let numRows = 0; + for (;;) { + const f = this.fieldHeader(prev); + if (f.type === T_STOP) break; + if (f.id === 1 && f.type === T_LIST) { + const { size, elemType } = this.listHeader(); + for (let i = 0; i < size; i += 1) { + if (elemType === T_STRUCT) { + const col = this.readColumnChunk(); + if (col) columns.push(col); + } else { + this.skip(elemType); + } + } + } else if (f.id === 3 && (f.type === T_I64 || f.type === T_I32)) { + numRows = this.zigzag(); + } else { + this.skip(f.type); + } + prev = f.id; + } + return { numRows, columns }; + } + + readFileMetaData(): ParquetFooterStats { + // FileMetaData { 3: num_rows, 4: row_groups (list) } + let prev = 0; + let numRows = 0; + const rowGroups: ParquetRowGroupStats[] = []; + for (;;) { + const f = this.fieldHeader(prev); + if (f.type === T_STOP) break; + if (f.id === 3 && (f.type === T_I64 || f.type === T_I32)) { + numRows = this.zigzag(); + } else if (f.id === 4 && f.type === T_LIST) { + const { size, elemType } = this.listHeader(); + for (let i = 0; i < size; i += 1) { + if (elemType === T_STRUCT) rowGroups.push(this.readRowGroup()); + else this.skip(elemType); + } + } else { + this.skip(f.type); + } + prev = f.id; + } + return { numRows, rowGroups }; + } +} + +/** + * Parse the Thrift-compact `FileMetaData` bytes (the footer, excluding the + * trailing 4-byte length + `PAR1` magic) into per-row-group column statistics. + */ +export function parseParquetFileMetaData(fileMetaDataBytes: Uint8Array): ParquetFooterStats { + return new ThriftCompactReader(fileMetaDataBytes).readFileMetaData(); +} + +/** Decode a `Statistics` min/max value for an integer physical type (little-endian). */ +export function decodeIntStat(bytes: Uint8Array | undefined, physicalType: number | null): number | null { + if (!bytes || bytes.length === 0) return null; + const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength); + if (physicalType === ParquetPhysicalType.INT32) { + return bytes.length >= 4 ? view.getInt32(0, true) : null; + } + if (physicalType === ParquetPhysicalType.INT64) { + // Feature codes fit comfortably in a JS number. + return bytes.length >= 8 ? Number(view.getBigInt64(0, true)) : null; + } + return null; +} diff --git a/packages/core/src/pointsFeatures.ts b/packages/core/src/pointsFeatures.ts index 4ffcd304..f555191e 100644 --- a/packages/core/src/pointsFeatures.ts +++ b/packages/core/src/pointsFeatures.ts @@ -102,6 +102,43 @@ export function featureCodeMapFromCatalog( return new Map(catalog.entries.map((entry) => [entry.name, entry.code])); } +/** + * Translate per-row feature codes from one catalog's code space into another, + * matching by feature name (`fromCode → name → toCode`). Rows whose code has no + * name in `fromCatalog`, or whose name is absent from `toCatalog`, become `-1`. + * + * For dictionary-only feature columns there is no file-backed code: each catalog + * build assigns codes by first-seen order, so the resident-preview catalog and + * the full-dataset catalog can give the same gene different codes. Re-deriving + * row codes against the authoritative (full) catalog with this helper keeps the + * render's per-row codes in the same space as the panel's selection and swatches. + * When both catalogs already agree (a real code column), every code maps to + * itself — a harmless identity pass. + */ +export function remapRowFeatureCodes( + rowCodes: ArrayLike, + fromCatalog: PointsFeatureCatalog, + toCatalog: PointsFeatureCatalog +): Int32Array { + const fromCodeToName = new Map( + fromCatalog.entries.map((entry) => [entry.code, entry.name]) + ); + const toNameToCode = new Map( + toCatalog.entries.map((entry) => [entry.name, entry.code]) + ); + // Translation is per distinct source code (a few hundred–thousand features), + // not per row: build the small code→code map once, then map the rows. + const codeRemap = new Map(); + for (const [fromCode, name] of fromCodeToName) { + codeRemap.set(fromCode, toNameToCode.get(name) ?? -1); + } + const out = new Int32Array(rowCodes.length); + for (let index = 0; index < rowCodes.length; index += 1) { + out[index] = codeRemap.get(rowCodes[index]) ?? -1; + } + return out; +} + function accumulateFeatureCatalogFromVectors( codeToName: Map, nameToCode: Map, diff --git a/packages/core/src/pointsLimits.ts b/packages/core/src/pointsLimits.ts index b9198f1d..91260e47 100644 --- a/packages/core/src/pointsLimits.ts +++ b/packages/core/src/pointsLimits.ts @@ -1,16 +1,31 @@ -/** Maximum rows allowed for full-table points preload (canonical scatter path). */ +/** + * Row count above which a dataset is treated as "large" for **catalog strategy** + * (route to the feature-column scan instead of a full-table decode). This is a + * fixed heuristic, deliberately separate from the configurable memory cap below. + */ export const POINTS_PRELOAD_MAX_ROWS = 4_000_000; -/** Default in-memory row cap for preloaded scatter (layer override via props panel). */ -export const DEFAULT_POINTS_MEMORY_CAP = POINTS_PRELOAD_MAX_ROWS; +/** + * Default in-memory row cap for the preloaded scatter (per-layer override via the + * props panel — `PointsLayerConfig.pointsMemoryCap`). Kept at 4M: on an + * UNINDEXED (dictionary-only, multipart) dataset the preload fetches WHOLE parts + * and only stops after accumulating this many rows, so a larger default pulls ~2× + * the bytes into the worker decode — which OOMs/hangs and falls back to a + * main-thread decode that crashes the tab. Higher caps are safe on indexed + * (row-group range-read) datasets and remain selectable in the panel for those. + */ +export const DEFAULT_POINTS_MEMORY_CAP = 4_000_000; -/** Default render row cap — points kept in memory may exceed this. */ -export const DEFAULT_POINTS_RENDER_CAP = POINTS_PRELOAD_MAX_ROWS; +/** Default render row cap — points kept in memory may exceed this. Matches the + * memory cap so, by default, everything loaded is drawn. */ +export const DEFAULT_POINTS_RENDER_CAP = DEFAULT_POINTS_MEMORY_CAP; export interface PointsColumnarLike { shape: number[]; data: ArrayLike[]; pointCount?: number; + /** Per-point feature code, aligned with {@link data}; truncated alongside it. */ + featureCodes?: ArrayLike; } export function resolvePointsMemoryCap(configured?: number): number { @@ -37,6 +52,24 @@ export function columnarPointCount(shape: number[], data: ArrayLike[]): return data[0]?.length ?? shape[0] ?? 0; } +/** Truncate a per-point feature-code array to `count`, preserving its element + * type via `subarray` for typed arrays (zero-copy) and `slice` otherwise. */ +function capFeatureCodes( + featureCodes: ArrayLike | undefined, + count: number +): ArrayLike | undefined { + if (!featureCodes || featureCodes.length <= count) { + return featureCodes; + } + if (ArrayBuffer.isView(featureCodes) && 'subarray' in featureCodes) { + return (featureCodes as { subarray(begin: number, end: number): ArrayLike }).subarray( + 0, + count + ); + } + return Array.prototype.slice.call(featureCodes, 0, count) as ArrayLike; +} + export function applyRenderCapToColumnar( batch: T, renderCap: number | undefined @@ -55,11 +88,13 @@ export function applyRenderCapToColumnar( } return Float32Array.from(column as ArrayLike).subarray(0, renderCap); }); + const nextFeatureCodes = capFeatureCodes(batch.featureCodes, renderCap); return { ...batch, data: nextData, shape: [axisCount, renderCap], pointCount: renderCap, + ...(nextFeatureCodes ? { featureCodes: nextFeatureCodes } : {}), }; } diff --git a/packages/core/src/pointsLoadOptions.ts b/packages/core/src/pointsLoadOptions.ts index 0602cdb9..43feeafb 100644 --- a/packages/core/src/pointsLoadOptions.ts +++ b/packages/core/src/pointsLoadOptions.ts @@ -1,8 +1,11 @@ +import type { PointsFeatureCatalog } from './pointsTiling.js'; + export interface PointsLoadProgress { scannedRows: number; matchedRows: number; partIndex: number; partCount: number; + partialResult: PointsLoadResult; } export interface PointsLoadOptions { @@ -17,12 +20,37 @@ export interface PointsLoadOptions { * Default UI uses in-memory runtime filtering instead. */ fullDatasetFeatureScan?: boolean; + /** + * Read the feature column alongside the geometry in the same (projected, + * capped) preload, and derive per-row feature codes + the feature catalog from + * that one decode. Lets the feature filter work with no separate, blocking + * catalog/row-code load at filter time. The catalog reflects the *resident* + * (preloaded) rows — i.e. the features present in the points actually drawn. + */ + includeFeatureCodes?: boolean; + /** + * Cancels a superseded load (e.g. the memory cap changed mid-load). Checked at + * the load boundaries — notably BEFORE the main-thread fallback decode — so an + * aborted load bails instead of running an expensive fallback to completion. + */ + signal?: AbortSignal; } export interface PointsLoadResult { shape: number[]; data: ArrayLike[]; featureCodes?: ArrayLike; + /** Catalog derived from the resident feature column (present when the load was + * requested with {@link PointsLoadOptions.includeFeatureCodes}). */ + featureCatalog?: PointsFeatureCatalog; + /** True when the element has a file-backed feature code column (e.g. + * `feature_name_codes`) — a real feature index whose codes are globally + * authoritative. False/absent for dictionary-only feature columns, where codes + * are assigned by the app and are only stable within a single catalog build. + * Present when the load was requested with + * {@link PointsLoadOptions.includeFeatureCodes}. Gates the whole-dataset + * feature-index scan (only worthwhile / correct when codes are authoritative). */ + hasFeatureCodeColumn?: boolean; totalRowCount?: number; preloadTruncated?: boolean; /** Rows scanned when loading with an active feature filter. */ diff --git a/packages/core/src/pointsLoader.ts b/packages/core/src/pointsLoader.ts index 67e6fc0f..4a3f5ef6 100644 --- a/packages/core/src/pointsLoader.ts +++ b/packages/core/src/pointsLoader.ts @@ -29,6 +29,14 @@ export interface ColumnarNdarrayPointsBatch { bounds?: SpatialBounds; loadMode?: PointsLoadMode; pointCount?: number; + /** + * Per-point feature code, aligned row-for-row with the geometry columns in + * {@link data}. Present when the source resolved a feature key; consumed by the + * render path to build a GPU `featureCode` attribute (colour-by-feature and + * per-code visibility). Any transform that reorders or truncates {@link data} + * (feature filter, render cap) must permute this in lockstep. + */ + featureCodes?: ArrayLike; } export type PointsBatch = ColumnarNdarrayPointsBatch; @@ -48,6 +56,8 @@ export interface CorePointsLoader { export interface PreloadedColumnarInput { shape: number[]; data: ArrayLike[]; + /** Optional per-point feature codes, carried onto the batch for colouring. */ + featureCodes?: ArrayLike; } export function resolvePointsEncoding( @@ -82,6 +92,7 @@ function toColumnarBatch( const shape = result.shape ?? []; const data = result.data; const pointCount = columnarPointCount(shape, data); + const featureCodes = 'featureCodes' in result ? result.featureCodes : undefined; return { format: 'columnar-ndarray', data, @@ -89,6 +100,7 @@ function toColumnarBatch( bounds: 'bounds' in result ? result.bounds : overrides?.bounds, loadMode: 'loadMode' in result ? result.loadMode : overrides?.loadMode, pointCount, + ...(featureCodes ? { featureCodes } : {}), ...overrides, }; } diff --git a/packages/core/src/pointsTiling.ts b/packages/core/src/pointsTiling.ts index 5d3a2cd5..06551578 100644 --- a/packages/core/src/pointsTiling.ts +++ b/packages/core/src/pointsTiling.ts @@ -265,13 +265,15 @@ export function filterColumnarByFeatureCodes( ): PointsColumnarData { const allowedFeatureCodes = featureCodeAllowSet(featureCodes); if (allowedFeatureCodes === null || !sourceFeatureCodes) { - return data; + // No filtering applied. Surface the aligned per-row codes when the source + // provided them, so callers can build a `featureCode` render attribute. + return sourceFeatureCodes ? { ...data, featureCodes: sourceFeatureCodes } : data; } if (allowedFeatureCodes.size === 0) { const axisCount = data.shape?.[0] ?? data.data.length; const empty = new Float32Array(0); const emptyData = axisCount >= 3 && data.data[2] ? [empty, empty, empty] : [empty, empty]; - return { shape: [axisCount, 0], data: emptyData }; + return { shape: [axisCount, 0], data: emptyData, featureCodes: new Int32Array(0) }; } const xs = data.data[0]; @@ -287,12 +289,13 @@ export function filterColumnarByFeatureCodes( } if (keep.length === n) { - return data; + return { ...data, featureCodes: sourceFeatureCodes }; } const outX = new Float32Array(keep.length); const outY = new Float32Array(keep.length); const outZ = zs ? new Float32Array(keep.length) : undefined; + const outCodes = new Int32Array(keep.length); for (let index = 0; index < keep.length; index += 1) { const sourceIndex = keep[index]; outX[index] = xs[sourceIndex]; @@ -300,11 +303,13 @@ export function filterColumnarByFeatureCodes( if (outZ) { outZ[index] = zs[sourceIndex] ?? 0; } + outCodes[index] = sourceFeatureCodes[sourceIndex]; } return { shape: [outZ ? 3 : 2, keep.length], data: outZ ? [outX, outY, outZ] : [outX, outY], + featureCodes: outCodes, }; } diff --git a/packages/core/src/spatialViewFit.ts b/packages/core/src/spatialViewFit.ts index b1ebc827..6919a599 100644 --- a/packages/core/src/spatialViewFit.ts +++ b/packages/core/src/spatialViewFit.ts @@ -24,6 +24,8 @@ export type OrthographicViewState2D = { export type PointsColumnarData = { data: ArrayLike[]; shape?: number[]; + /** Optional per-point feature code, aligned row-for-row with {@link data}. */ + featureCodes?: ArrayLike; }; /** Same default as Viv ImageView detail framing. */ diff --git a/packages/core/src/workers/index.ts b/packages/core/src/workers/index.ts index 796660f5..d0cc4f14 100644 --- a/packages/core/src/workers/index.ts +++ b/packages/core/src/workers/index.ts @@ -1,6 +1,7 @@ export { buildFeatureCatalogInWorker, countFeatureCodesInWorker, + decodeGeometryWithFeaturesInWorker, decodeParquetGeometryCappedInWorker, decodeParquetPartsInWorker, decodeParquetRowFeatureCodesInWorker, @@ -14,6 +15,7 @@ export { scanParquetFeatureCatalogInWorker, scanParquetFeatureCountsInWorker, setPointsWorkerDefaultEnabled, + setPointsWorkerRequestTimeout, transferablesForParquetPayload, } from './pointsWorkerClient.js'; diff --git a/packages/core/src/workers/points-worker.ts b/packages/core/src/workers/points-worker.ts index cf701a6a..a54f2807 100644 --- a/packages/core/src/workers/points-worker.ts +++ b/packages/core/src/workers/points-worker.ts @@ -9,6 +9,7 @@ import { getParquetModule, type ParquetModule } from '../parquetWasmLoader.js'; import type { PointsWorkerMessage, PointsWorkerRequest, PointsWorkerResponse } from './pointsWorkerProtocol.js'; import { countFeatureCodesFromArray, + decodeGeometryWithFeaturesFromPayload, decodeParquetPartsToTable, decodeParquetPayloadToTable, extractGeometryColumnar, @@ -27,6 +28,13 @@ function toFloat32Array(values: ArrayLike): Float32Array { return Float32Array.from(values); } +function toInt32Array(values: ArrayLike): Int32Array { + if (values instanceof Int32Array) { + return values; + } + return Int32Array.from(values); +} + function handleFilterColumnar(request: Extract) { const filtered = filterColumnarByFeatureCodes( { @@ -39,6 +47,7 @@ function handleFilterColumnar(request: Extract 0 ? filtered.shape @@ -53,6 +62,7 @@ function handleFilterColumnar(request: Extract +): Promise { + const parquetModule = await getParquetModule(); + const result = await decodeGeometryWithFeaturesFromPayload( + parquetModule.readParquet, + parquetModule.readParquetRowGroup, + request + ); + const [xs, ys, zs] = result.data; + return { + ok: true, + result: { + kind: 'geometryWithFeatures', + shape: result.shape, + xs, + ys, + ...(zs ? { zs } : {}), + ...(result.featureCodes ? { featureCodes: result.featureCodes } : {}), + ...(result.featureCatalog ? { featureCatalog: result.featureCatalog } : {}), + }, + }; +} + function handleCountFeatureCodes( request: Extract ): PointsWorkerResponse { @@ -225,6 +259,7 @@ async function scanPayloadByFeatureCodes( xs: number[]; ys: number[]; zs: number[]; + codes: number[]; scannedRows: number; } ): Promise<{ matchedRows: number; scannedRows: number }> { @@ -234,6 +269,9 @@ async function scanPayloadByFeatureCodes( request.featureKey, ...(request.featureCodeColumnName ? [request.featureCodeColumnName] : []), ]; + const featureCodeByName = request.featureCodeEntries + ? new Map(request.featureCodeEntries.map((entry) => [entry.name, entry.code])) + : undefined; if (request.rowGroups?.length && parquetModule.readParquetRowGroup) { for (const chunk of request.rowGroups) { @@ -260,6 +298,8 @@ async function scanPayloadByFeatureCodes( xs: input.xs, ys: input.ys, zs: input.zs, + codes: input.codes, + featureCodeByName, }); } return { matchedRows: input.matchedRows, scannedRows: input.scannedRows }; @@ -282,6 +322,8 @@ async function scanPayloadByFeatureCodes( xs: input.xs, ys: input.ys, zs: input.zs, + codes: input.codes, + featureCodeByName, }); } return { matchedRows: input.matchedRows, scannedRows: input.scannedRows }; @@ -295,16 +337,19 @@ async function handleScanParquetByFeatureCodes( const xs: number[] = []; const ys: number[] = []; const zs: number[] = []; + const codes: number[] = []; const { matchedRows, scannedRows } = await scanPayloadByFeatureCodes(parquetModule, request, { matchedRows: 0, xs, ys, zs, + codes, scannedRows: 0, }); const outX = Float32Array.from(xs); const outY = Float32Array.from(ys); const outZ = hasZ ? Float32Array.from(zs) : undefined; + const outCodes = codes.length > 0 ? Int32Array.from(codes) : undefined; const shape = outZ ? [3, outX.length] : [2, outX.length]; return { ok: true, @@ -314,6 +359,7 @@ async function handleScanParquetByFeatureCodes( xs: outX, ys: outY, ...(outZ ? { zs: outZ } : {}), + ...(outCodes ? { featureCodes: outCodes } : {}), matchedRows, scannedRows, }, @@ -413,6 +459,8 @@ async function handleRequest(request: PointsWorkerRequest): Promise) => { if (response.result.zs) { transferables.push(response.result.zs.buffer); } - if (response.result.kind === 'columnar' && response.result.featureCodes) { + if (response.result.featureCodes) { + transferables.push(response.result.featureCodes.buffer); + } + } else if (response.result.kind === 'geometryWithFeatures') { + transferables.push(response.result.xs.buffer, response.result.ys.buffer); + if (response.result.zs) { + transferables.push(response.result.zs.buffer); + } + if (response.result.featureCodes) { transferables.push(response.result.featureCodes.buffer); } } else if (response.result.kind === 'parquetTable') { diff --git a/packages/core/src/workers/pointsWorkerClient.ts b/packages/core/src/workers/pointsWorkerClient.ts index de5b221a..800b1c70 100644 --- a/packages/core/src/workers/pointsWorkerClient.ts +++ b/packages/core/src/workers/pointsWorkerClient.ts @@ -15,9 +15,40 @@ let worker: Worker | undefined; let nextRequestId = 0; const pending = new Map< number, - { resolve: (value: unknown) => void; reject: (error: Error) => void } + { + resolve: (value: unknown) => void; + reject: (error: Error) => void; + timeout?: ReturnType; + } >(); +// Safety net: if the worker was enabled but is not functionally wired (e.g. a +// host points enablePointsWorker() at a URL that loads but whose module never +// posts a response), a request would otherwise await forever. After this budget +// with no reply we reject the request so the caller falls back to the main +// thread (every *InWorker helper is wrapped in a try/catch fallback). Generous +// by default because a working worker legitimately spends many seconds decoding +// large parquet; the timeout is meant to catch a *silent* worker, not a slow one. +let requestTimeoutMs = 30_000; + +/** Override the per-request worker timeout (ms). Set to 0/Infinity to disable. */ +export function setPointsWorkerRequestTimeout(ms: number) { + requestTimeoutMs = ms; +} + +/** Remove a pending request, clearing its timeout, and return its callbacks. */ +function settlePending(id: number) { + const entry = pending.get(id); + if (!entry) { + return undefined; + } + if (entry.timeout !== undefined) { + clearTimeout(entry.timeout); + } + pending.delete(id); + return entry; +} + let enabled = false; // Points worker is opt-in: hosts call enablePointsWorker() (or // setPointsWorkerDefaultEnabled(true)) once they have wired the worker bundle. @@ -36,11 +67,10 @@ function ensureWorkerListener() { if (message.direction !== 'response') { return; } - const entry = pending.get(message.id); + const entry = settlePending(message.id); if (!entry) { return; } - pending.delete(message.id); if (message.response.ok) { entry.resolve(message.response.result); } else { @@ -48,10 +78,9 @@ function ensureWorkerListener() { } }; worker.onerror = (event) => { - for (const [, entry] of pending) { - entry.reject(new Error(event.message || 'Points worker error')); + for (const [id] of [...pending]) { + settlePending(id)?.reject(new Error(event.message || 'Points worker error')); } - pending.clear(); }; } @@ -62,7 +91,21 @@ function postRequest(request: PointsWorkerRequest, transferables: Transferabl } const id = ++nextRequestId; return new Promise((resolve, reject) => { - pending.set(id, { resolve: resolve as (value: unknown) => void, reject }); + const entry: { + resolve: (value: unknown) => void; + reject: (error: Error) => void; + timeout?: ReturnType; + } = { resolve: resolve as (value: unknown) => void, reject }; + if (requestTimeoutMs > 0 && Number.isFinite(requestTimeoutMs)) { + entry.timeout = setTimeout(() => { + settlePending(id)?.reject( + new Error( + `Points worker did not respond within ${requestTimeoutMs}ms; falling back to the main thread` + ) + ); + }, requestTimeoutMs); + } + pending.set(id, entry); const message: PointsWorkerMessage = { id, direction: 'request', request }; if (transferables.length > 0) { activeWorker.postMessage(message, transferables); @@ -95,6 +138,7 @@ function transferablesForRequest(request: PointsWorkerRequest): Transferable[] { case 'decodeParquetRowFeatureCodes': case 'scanParquetFeatureCounts': case 'decodeParquetGeometryCapped': + case 'decodeGeometryWithFeatures': case 'scanParquetByFeatureCodes': case 'scanParquetFeatureCatalog': return transferablesForParquetPayload(request.parts, request.rowGroups); @@ -135,10 +179,9 @@ export function disablePointsWorker() { worker.terminate(); worker = undefined; } - for (const [, entry] of pending) { - entry.reject(new Error('Points worker disabled')); + for (const [id] of [...pending]) { + settlePending(id)?.reject(new Error('Points worker disabled')); } - pending.clear(); } export function setPointsWorkerDefaultEnabled(value: boolean) { @@ -301,6 +344,59 @@ export async function decodeParquetGeometryCappedInWorker( }; } +export type DecodeGeometryWithFeaturesInput = ParquetWorkerPayload & { + axisNames: string[]; + columns: string[]; + maxRows?: number; + featureKey: string; + featureCodeColumnName?: string; +}; + +/** + * Off-thread codes-with-geometry preload: decode geometry + per-row feature + * codes + the feature catalog from one projected decode in the worker. The + * caller fetches whole row-group (or part) bytes via async range reads, so the + * CPU-heavy decode never blocks the main thread. Returns null when the worker is + * disabled or the payload is empty (caller falls back to the main-thread decode). + */ +export async function decodeGeometryWithFeaturesInWorker( + input: DecodeGeometryWithFeaturesInput +): Promise<{ + shape: number[]; + data: ArrayLike[]; + featureCodes?: Int32Array; + featureCatalog?: PointsFeatureCatalog; +} | null> { + ensurePointsWorker(); + if (!isPointsWorkerEnabled()) { + return null; + } + if (!input.parts?.length && !input.rowGroups?.length) { + return null; + } + if (input.parts?.length && input.rowGroups?.length) { + throw new Error('decodeGeometryWithFeaturesInWorker requires parts or rowGroups, not both'); + } + const request: Extract = { + type: 'decodeGeometryWithFeatures', + ...input, + }; + const result = await postRequest['result']>( + request, + transferablesForRequest(request) + ); + if (result.kind !== 'geometryWithFeatures') { + throw new Error('Unexpected points worker response for decodeGeometryWithFeatures'); + } + const data = result.zs ? [result.xs, result.ys, result.zs] : [result.xs, result.ys]; + return { + shape: result.shape, + data, + ...(result.featureCodes ? { featureCodes: result.featureCodes } : {}), + ...(result.featureCatalog ? { featureCatalog: result.featureCatalog } : {}), + }; +} + export async function countFeatureCodesInWorker( sourceFeatureCodes: ArrayLike ): Promise> { @@ -366,6 +462,8 @@ export type ScanParquetByFeatureCodesInput = ParquetWorkerPayload & { featureCodeColumnName?: string; featureCodes: readonly number[]; memoryCap: number; + /** Authoritative name→code entries for dict-only elements (no code column). */ + featureCodeEntries?: ReadonlyArray<{ name: string; code: number }>; }; export async function scanParquetByFeatureCodesInWorker( diff --git a/packages/core/src/workers/pointsWorkerProtocol.ts b/packages/core/src/workers/pointsWorkerProtocol.ts index 91959e71..db244a20 100644 --- a/packages/core/src/workers/pointsWorkerProtocol.ts +++ b/packages/core/src/workers/pointsWorkerProtocol.ts @@ -85,6 +85,17 @@ export type PointsWorkerRequest = featureCodeColumnName?: string; featureCodeEntries?: ReadonlyArray<{ name: string; code: number }>; } + | { + type: 'decodeGeometryWithFeatures'; + parts?: Uint8Array[]; + rowGroups?: ParquetRowGroupBytesChunk[]; + axisNames: string[]; + /** Projected columns: axes + feature key (+ code column when present). */ + columns: string[]; + maxRows?: number; + featureKey: string; + featureCodeColumnName?: string; + } | { type: 'scanParquetByFeatureCodes'; parts?: Uint8Array[]; @@ -94,6 +105,10 @@ export type PointsWorkerRequest = featureCodeColumnName?: string; featureCodes: readonly number[]; memoryCap: number; + /** Authoritative name→code map for dict-only elements (no *_codes column), + * so the scan resolves each row's feature_name to the same code space the + * selection was made in. Absent when a file-backed code column is present. */ + featureCodeEntries?: ReadonlyArray<{ name: string; code: number }>; } | { type: 'scanMortonRowGroupsInBounds'; @@ -114,7 +129,7 @@ export type PointsWorkerColumnarResult = { featureCodes?: Int32Array; }; -export type PointsWorkerScanResult = Omit & { +export type PointsWorkerScanResult = Omit & { kind: 'columnarScan'; matchedRows: number; scannedRows: number; @@ -126,6 +141,15 @@ export type PointsWorkerResponse = result: | PointsWorkerColumnarResult | PointsWorkerScanResult + | { + kind: 'geometryWithFeatures'; + shape: number[]; + xs: Float32Array; + ys: Float32Array; + zs?: Float32Array; + featureCodes?: Int32Array; + featureCatalog?: PointsFeatureCatalog; + } | { kind: 'parquetTable'; tableIpc: Uint8Array } | { kind: 'catalog'; catalog: PointsFeatureCatalog } | { kind: 'rowFeatureCodes'; codes: Int32Array; numRows: number } @@ -144,5 +168,6 @@ export function columnarDataFromWorkerResult( result: PointsWorkerColumnarResult | PointsWorkerScanResult ): PointsColumnarData { const data = result.zs ? [result.xs, result.ys, result.zs] : [result.xs, result.ys]; - return { shape: result.shape, data }; + const featureCodes = 'featureCodes' in result ? result.featureCodes : undefined; + return { shape: result.shape, data, ...(featureCodes ? { featureCodes } : {}) }; } diff --git a/packages/core/src/workers/pointsWorkerScan.ts b/packages/core/src/workers/pointsWorkerScan.ts index 323ab065..9b2edde4 100644 --- a/packages/core/src/workers/pointsWorkerScan.ts +++ b/packages/core/src/workers/pointsWorkerScan.ts @@ -1,9 +1,11 @@ import { tableFromIPC, type Table } from 'apache-arrow'; import { accumulateFeatureCatalogFromTable, + buildFeatureCatalogFromColumns, countFeatureCodesHistogram, featureCatalogFromCodeMap, featureCatalogNeedsParquetFallback, + featureCodeMapFromCatalog, resolveRowFeatureCodesFromTable, } from '../pointsFeatures.js'; import type { PointsFeatureCatalog } from '../pointsTiling.js'; @@ -151,6 +153,85 @@ export function extractGeometryColumnar( return { shape, xs, ys, ...(zs ? { zs } : {}) }; } +export type DecodeGeometryWithFeaturesInput = ParquetWorkerPayloadInput & { + axisNames: string[]; + /** Projected columns to decode: axes + feature key (+ code column if present). */ + columns: string[]; + featureKey: string; + featureCodeColumnName?: string; + maxRows?: number; +}; + +export type DecodeGeometryWithFeaturesResult = { + shape: number[]; + data: Float32Array[]; + featureCodes?: Int32Array; + featureCatalog?: PointsFeatureCatalog; +}; + +/** + * One projected decode → geometry + per-row feature codes + feature catalog. + * + * This is the off-thread half of the codes-with-geometry preload: the caller + * fetches whole row-group (or part) bytes via async range reads and hands them + * here (in the worker) so the CPU-heavy parquet decode never touches the main + * thread. Column projection still runs during decode, but the *bytes* are whole + * row groups (all columns) — parquet-wasm cannot fetch individual column chunks + * (see docs/parquet-wasm-limitations.md). Mirrors the main-thread derivation in + * `VPointsSource.loadPoints` so both paths produce identical codes + catalog. + */ +export async function decodeGeometryWithFeaturesFromPayload( + readParquet: ParquetModule['readParquet'], + readParquetRowGroup: ReadParquetRowGroup | undefined, + input: DecodeGeometryWithFeaturesInput +): Promise { + const table = await decodeParquetPayloadToTable( + readParquet, + readParquetRowGroup, + { rowGroups: input.rowGroups, parts: input.parts }, + input.columns, + input.maxRows + ); + + const geometry = extractGeometryColumnar(table, input.axisNames); + const data = geometry.zs ? [geometry.xs, geometry.ys, geometry.zs] : [geometry.xs, geometry.ys]; + + let featureCodes: Int32Array | undefined; + let featureCatalog: PointsFeatureCatalog | undefined; + const nameColumn = table.getChild(input.featureKey); + if (nameColumn) { + const codeColumn = input.featureCodeColumnName + ? table.getChild(input.featureCodeColumnName) + : null; + featureCatalog = buildFeatureCatalogFromColumns( + input.featureKey, + nameColumn, + codeColumn ?? null, + null, + table.numRows + ); + const featureCodeByName = input.featureCodeColumnName + ? undefined + : featureCodeMapFromCatalog(featureCatalog); + const codes = resolveRowFeatureCodesFromTable( + table, + input.featureKey, + input.featureCodeColumnName, + featureCodeByName + ); + if (codes) { + featureCodes = codes instanceof Int32Array ? codes : Int32Array.from(codes); + } + } + + return { + shape: geometry.shape, + data, + ...(featureCodes ? { featureCodes } : {}), + ...(featureCatalog ? { featureCatalog } : {}), + }; +} + export async function scanFeatureCatalogFromPayload( readParquet: ParquetModule['readParquet'], readParquetRowGroup: ReadParquetRowGroup | undefined, @@ -330,6 +411,11 @@ export function scanTableByFeatureCodes(input: { xs: number[]; ys: number[]; zs: number[]; + /** Optional per-matched-row feature codes, collected for colour-by-feature. */ + codes?: number[]; + /** Authoritative name→code map for dict-only elements (no code column), so a + * row's feature_name resolves to the same code space the selection uses. */ + featureCodeByName?: ReadonlyMap; }): number { const allowed = featureCodeAllowSet(input.featureCodes); if (allowed !== null && allowed.size === 0) { @@ -338,7 +424,8 @@ export function scanTableByFeatureCodes(input: { const rowCodes = extractRowFeatureCodesFromTable( input.table, input.featureKey, - input.featureCodeColumnName + input.featureCodeColumnName, + input.featureCodeByName ); const xColumn = input.axisNames.includes('x') ? input.table.getChild('x') : null; const yColumn = input.axisNames.includes('y') ? input.table.getChild('y') : null; @@ -365,6 +452,7 @@ export function scanTableByFeatureCodes(input: { const z = zColumn.get(rowIndex); input.zs.push(typeof z === 'number' ? z : 0); } + input.codes?.push(rowCodes[rowIndex] ?? -1); matchedRows += 1; } return matchedRows; diff --git a/packages/core/tests/parquetFooterStats.spec.ts b/packages/core/tests/parquetFooterStats.spec.ts new file mode 100644 index 00000000..1ce0ac76 --- /dev/null +++ b/packages/core/tests/parquetFooterStats.spec.ts @@ -0,0 +1,97 @@ +import { execSync } from 'node:child_process'; +import { mkdtemp, readFile, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { + decodeIntStat, + parseParquetFileMetaData, + ParquetPhysicalType, +} from '../src/parquetFooterStats.js'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const writerRoot = join(__dirname, '../../../python/spatialdata-experimental-writer'); + +/** Slice the Thrift `FileMetaData` bytes out of a full parquet file. */ +function footerMetaData(fileBytes: Uint8Array): Uint8Array { + const n = fileBytes.length; + expect(String.fromCharCode(...fileBytes.subarray(n - 4))).toBe('PAR1'); + const len = new DataView(fileBytes.buffer, fileBytes.byteOffset + n - 8, 4).getUint32(0, true); + return fileBytes.subarray(n - 8 - len, n - 8); +} + +describe('parseParquetFileMetaData', () => { + let root: string; + let fileBytes: Uint8Array; + + beforeAll(async () => { + root = await mkdtemp(join(tmpdir(), 'footer-stats-')); + // 8 rows, feature codes sorted, 2 rows per row group -> 4 row groups with + // code ranges [0,0], [1,1], [2,2], [3,3]. + execSync( + `uv run python - <<'PY' +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path +root = Path(${JSON.stringify(root)}) +table = pa.table({ + "x": pa.array([0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0], type=pa.float32()), + "y": pa.array([0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0], type=pa.float32()), + "feature_name": ["A","A","B","B","C","C","D","D"], + "feature_name_codes": pa.array([0,0,1,1,2,2,3,3], type=pa.int32()), +}) +pq.write_table(table, root / "sorted.parquet", row_group_size=2, write_statistics=True) +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); + fileBytes = new Uint8Array(await readFile(join(root, 'sorted.parquet'))); + }, 120_000); + + afterAll(async () => { + await rm(root, { recursive: true, force: true }); + }); + + it('recovers row-group count, num_rows, and column paths', () => { + const meta = parseParquetFileMetaData(footerMetaData(fileBytes)); + expect(meta.numRows).toBe(8); + expect(meta.rowGroups).toHaveLength(4); + for (const rg of meta.rowGroups) { + expect(rg.numRows).toBe(2); + expect(rg.columns.map((c) => c.path).sort()).toEqual([ + 'feature_name', + 'feature_name_codes', + 'x', + 'y', + ]); + } + }); + + it('recovers per-row-group feature_name_codes min/max from Statistics', () => { + const meta = parseParquetFileMetaData(footerMetaData(fileBytes)); + const ranges = meta.rowGroups.map((rg) => { + const col = rg.columns.find((c) => c.path === 'feature_name_codes'); + expect(col?.physicalType).toBe(ParquetPhysicalType.INT32); + return [ + decodeIntStat(col?.minValue, col?.physicalType ?? null), + decodeIntStat(col?.maxValue, col?.physicalType ?? null), + ]; + }); + // Sorted codes, 2 per group -> contiguous, non-overlapping ranges. + expect(ranges).toEqual([ + [0, 0], + [1, 1], + [2, 2], + [3, 3], + ]); + }); + + it('recovers string (feature_name) min/max bytes', () => { + const meta = parseParquetFileMetaData(footerMetaData(fileBytes)); + const decoder = new TextDecoder(); + const rg2 = meta.rowGroups[2].columns.find((c) => c.path === 'feature_name'); + expect(rg2?.physicalType).toBe(ParquetPhysicalType.BYTE_ARRAY); + expect(rg2?.minValue && decoder.decode(rg2.minValue)).toBe('C'); + expect(rg2?.maxValue && decoder.decode(rg2.maxValue)).toBe('C'); + }); +}); diff --git a/packages/core/tests/pointsFeatures.spec.ts b/packages/core/tests/pointsFeatures.spec.ts index df6298ca..61d7a2bb 100644 --- a/packages/core/tests/pointsFeatures.spec.ts +++ b/packages/core/tests/pointsFeatures.spec.ts @@ -193,6 +193,22 @@ PY`, expect(featureCodes?.length).toBe(5); }); + it('includeFeatureCodes: derives row codes + catalog from the geometry preload', async () => { + const points = await source.loadPoints('points/transcripts', { includeFeatureCodes: true }); + // geometry is still x/y for the 5 resident rows + expect(points.shape).toEqual([2, 5]); + // row-aligned codes and the catalog come from the one decode, no extra load + expect(points.featureCodes && Array.from(points.featureCodes)).toEqual([0, 1, 0, 2, 1]); + expect(points.featureCatalog).toEqual({ + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'gene_a' }, + { code: 1, name: 'gene_b' }, + { code: 2, name: 'gene_c' }, + ], + }); + }); + it('uses explicit feature code columns instead of dictionary indices', async () => { const elementDir = join(fixtureRoot, 'points', 'dict_with_codes'); await mkdir(elementDir, { recursive: true }); diff --git a/packages/core/tests/pointsLimits.spec.ts b/packages/core/tests/pointsLimits.spec.ts new file mode 100644 index 00000000..5797d43f --- /dev/null +++ b/packages/core/tests/pointsLimits.spec.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from 'vitest'; +import { applyRenderCapToColumnar } from '../src/pointsLimits.js'; + +describe('applyRenderCapToColumnar', () => { + it('returns the batch untouched when under the cap', () => { + const batch = { + shape: [2, 3], + data: [new Float32Array([0, 1, 2]), new Float32Array([0, 1, 2])], + featureCodes: new Int32Array([9, 8, 7]), + }; + expect(applyRenderCapToColumnar(batch, 10)).toBe(batch); + }); + + it('truncates feature codes in lockstep with geometry', () => { + const batch = { + shape: [2, 4], + data: [new Float32Array([0, 1, 2, 3]), new Float32Array([0, 10, 20, 30])], + featureCodes: new Int32Array([5, 6, 7, 8]), + }; + const capped = applyRenderCapToColumnar(batch, 2); + expect(capped.shape).toEqual([2, 2]); + expect(Array.from(capped.data[0])).toEqual([0, 1]); + expect(Array.from(capped.featureCodes ?? [])).toEqual([5, 6]); + }); + + it('leaves codes absent when the batch has none', () => { + const batch = { + shape: [2, 4], + data: [new Float32Array([0, 1, 2, 3]), new Float32Array([0, 10, 20, 30])], + }; + const capped = applyRenderCapToColumnar(batch, 2); + expect(capped.featureCodes).toBeUndefined(); + expect(capped.shape).toEqual([2, 2]); + }); +}); diff --git a/packages/core/tests/pointsTiling.spec.ts b/packages/core/tests/pointsTiling.spec.ts index ceeaafd6..859ca789 100644 --- a/packages/core/tests/pointsTiling.spec.ts +++ b/packages/core/tests/pointsTiling.spec.ts @@ -134,6 +134,8 @@ describe('points tiling helpers', () => { ); expect(Array.from(filtered.data[0])).toEqual([0, 2]); expect(filtered.shape).toEqual([2, 2]); + // Codes for the kept rows come back aligned with the filtered geometry. + expect(Array.from(filtered.featureCodes ?? [])).toEqual([0, 0]); }); it('returns no rows when feature filter is an empty selection', () => { @@ -147,5 +149,21 @@ describe('points tiling helpers', () => { ); expect(filtered.data[0].length).toBe(0); expect(filtered.shape).toEqual([2, 0]); + expect(filtered.featureCodes?.length).toBe(0); + }); + + it('surfaces aligned per-row codes when no filter is applied', () => { + const xs = new Float32Array([0, 1, 2]); + const ys = new Float32Array([0, 1, 2]); + const sourceFeatureCodes = new Int32Array([7, 3, 7]); + // `featureCodes: undefined` = "all features"; geometry is untouched but the + // aligned codes are surfaced so the render path can colour by feature. + const filtered = filterColumnarByFeatureCodes( + { data: [xs, ys], shape: [2, 3] }, + undefined, + sourceFeatureCodes + ); + expect(filtered.data[0]).toBe(xs); + expect(filtered.featureCodes).toBe(sourceFeatureCodes); }); }); diff --git a/packages/core/tests/pointsWorker.spec.ts b/packages/core/tests/pointsWorker.spec.ts index cea441d2..329cb4a5 100644 --- a/packages/core/tests/pointsWorker.spec.ts +++ b/packages/core/tests/pointsWorker.spec.ts @@ -1,10 +1,12 @@ -import { describe, expect, it, vi } from 'vitest'; +import { afterEach, describe, expect, it, vi } from 'vitest'; import { decodeParquetRowFeatureCodesInWorker, disablePointsWorker, + enablePointsWorker, filterColumnarByFeatureCodesInWorker, scanParquetFeatureCatalogInWorker, setPointsWorkerDefaultEnabled, + setPointsWorkerRequestTimeout, } from '../src/workers/pointsWorkerClient.js'; import { filterColumnarByFeatureCodes as filterSync } from '../src/pointsTiling.js'; @@ -47,4 +49,42 @@ describe('points worker client', () => { }); expect(result).toBeNull(); }); + + describe('timeout fallback for a silent worker', () => { + const originalWorker = (globalThis as { Worker?: unknown }).Worker; + + afterEach(() => { + disablePointsWorker(); + setPointsWorkerRequestTimeout(30_000); + setPointsWorkerDefaultEnabled(false); + (globalThis as { Worker?: unknown }).Worker = originalWorker; + }); + + it('rejects (so the caller can fall back) when an enabled worker never replies', async () => { + // A worker that loads but never posts a response — the exact hang the + // opt-in default guards against, here caught by the request timeout. + class SilentWorker { + onmessage: ((event: MessageEvent) => void) | null = null; + onerror: ((event: unknown) => void) | null = null; + postMessage() { + /* deliberately never reply */ + } + terminate() { + /* no-op */ + } + } + (globalThis as { Worker?: unknown }).Worker = SilentWorker; + + enablePointsWorker({ workerUrl: 'about:blank' }); + setPointsWorkerRequestTimeout(30); + + await expect( + scanParquetFeatureCatalogInWorker({ + parts: [new Uint8Array([1, 2, 3])], + columns: ['feature_name'], + featureKey: 'feature_name', + }) + ).rejects.toThrow(/did not respond within 30ms/); + }); + }); }); diff --git a/packages/core/tests/pointsWorkerScan.spec.ts b/packages/core/tests/pointsWorkerScan.spec.ts index d389553d..316dd794 100644 --- a/packages/core/tests/pointsWorkerScan.spec.ts +++ b/packages/core/tests/pointsWorkerScan.spec.ts @@ -1,12 +1,31 @@ import { tableFromArrays, tableToIPC } from 'apache-arrow'; import { describe, expect, it } from 'vitest'; import { + decodeGeometryWithFeaturesFromPayload, decodeParquetRowGroupsToTable, extractGeometryColumnar, extractRowFeatureCodesFromTable, scanFeatureCatalogFromPayload, + scanTableByFeatureCodes, } from '../src/workers/pointsWorkerScan.js'; +const throwingReadParquet = (() => { + throw new Error('readParquet should not be called on the rowGroup path'); +}) as unknown as (bytes: Uint8Array, options?: { columns?: string[] }) => { + intoIPCStream(): Uint8Array; +}; + +function singleRowGroup(columns: Record) { + const table = tableFromArrays(columns as never); + const read = () => ({ intoIPCStream: () => tableToIPC(table) }); + return { + read, + rowGroups: [ + { schemaBytes: new Uint8Array(0), rowGroupBytes: new Uint8Array(0), rowGroupIndex: 0 }, + ], + }; +} + function mockReadParquetRowGroup( chunks: Array> ): ( @@ -49,6 +68,55 @@ describe('decodeParquetRowGroupsToTable', () => { }); }); +describe('decodeGeometryWithFeaturesFromPayload', () => { + it('derives geometry, row codes, and catalog from one projected decode', async () => { + const { read, rowGroups } = singleRowGroup({ + x: Float32Array.from([0, 1, 2]), + y: Float32Array.from([0, 1, 2]), + feature_name: ['gene_a', 'gene_b', 'gene_a'], + feature_name_codes: Int32Array.from([0, 1, 0]), + }); + const result = await decodeGeometryWithFeaturesFromPayload(throwingReadParquet, read, { + rowGroups, + axisNames: ['x', 'y'], + columns: ['x', 'y', 'feature_name', 'feature_name_codes'], + featureKey: 'feature_name', + featureCodeColumnName: 'feature_name_codes', + }); + + expect(result.shape).toEqual([2, 3]); + expect(Array.from(result.data[0])).toEqual([0, 1, 2]); + expect(result.featureCodes && Array.from(result.featureCodes)).toEqual([0, 1, 0]); + expect(result.featureCatalog).toEqual({ + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'gene_a' }, + { code: 1, name: 'gene_b' }, + ], + }); + }); + + it('assigns codes by first-seen order for dict-only columns (no code column)', async () => { + const { read, rowGroups } = singleRowGroup({ + x: Float32Array.from([0, 1, 2]), + y: Float32Array.from([3, 4, 5]), + feature_name: ['B', 'A', 'B'], + }); + const result = await decodeGeometryWithFeaturesFromPayload(throwingReadParquet, read, { + rowGroups, + axisNames: ['x', 'y'], + columns: ['x', 'y', 'feature_name'], + featureKey: 'feature_name', + }); + + expect(result.featureCodes && Array.from(result.featureCodes)).toEqual([0, 1, 0]); + expect(result.featureCatalog?.entries).toEqual([ + { code: 0, name: 'B' }, + { code: 1, name: 'A' }, + ]); + }); +}); + describe('extractRowFeatureCodesFromTable with featureCodeByName', () => { it('maps dictionary feature names to catalog codes', () => { const names = ['gene_a', 'gene_b', 'gene_a']; @@ -69,6 +137,67 @@ describe('extractRowFeatureCodesFromTable with featureCodeByName', () => { }); }); +describe('scanTableByFeatureCodes with featureCodeByName (dict-only)', () => { + it('matches rows by feature_name against the catalog map and retains authoritative codes', () => { + // Dict-only: no code column. Rows for gene_c (code 2) live among others; the + // scan must resolve names via the map and keep only the selected code's rows. + const table = tableFromArrays({ + x: Float32Array.from([10, 11, 12, 13]), + y: Float32Array.from([20, 21, 22, 23]), + feature_name: ['gene_a', 'gene_c', 'gene_b', 'gene_c'], + }); + const featureCodeByName = new Map([ + ['gene_a', 0], + ['gene_b', 1], + ['gene_c', 2], + ]); + const xs: number[] = []; + const ys: number[] = []; + const codes: number[] = []; + const matched = scanTableByFeatureCodes({ + table, + axisNames: ['x', 'y'], + featureKey: 'feature_name', + featureCodeColumnName: undefined, + featureCodes: [2], + memoryCap: 1_000, + matchedRows: 0, + xs, + ys, + zs: [], + codes, + featureCodeByName, + }); + expect(matched).toBe(2); + expect(xs).toEqual([11, 13]); // the two gene_c rows + expect(ys).toEqual([21, 23]); + expect(codes).toEqual([2, 2]); // authoritative codes retained + }); + + it('matches nothing when no name→code map is supplied for dict-only data', () => { + const table = tableFromArrays({ + x: Float32Array.from([10, 11]), + y: Float32Array.from([20, 21]), + feature_name: ['gene_a', 'gene_c'], + }); + const xs: number[] = []; + const matched = scanTableByFeatureCodes({ + table, + axisNames: ['x', 'y'], + featureKey: 'feature_name', + featureCodeColumnName: undefined, + featureCodes: [2], + memoryCap: 1_000, + matchedRows: 0, + xs, + ys: [], + zs: [], + }); + expect(matched).toBe(0); + expect(xs).toEqual([]); + }); +}); + describe('extractGeometryColumnar', () => { it('returns float32 axis columns', () => { const table = tableFromArrays({ diff --git a/packages/core/tests/remapRowFeatureCodes.spec.ts b/packages/core/tests/remapRowFeatureCodes.spec.ts new file mode 100644 index 00000000..652a8471 --- /dev/null +++ b/packages/core/tests/remapRowFeatureCodes.spec.ts @@ -0,0 +1,60 @@ +import { describe, expect, it } from 'vitest'; + +import { remapRowFeatureCodes } from '../src/pointsFeatures.js'; +import type { PointsFeatureCatalog } from '../src/pointsTiling.js'; + +const catalog = (pairs: Array<[number, string]>): PointsFeatureCatalog => ({ + featureKey: 'feature_name', + entries: pairs.map(([code, name]) => ({ code, name })), +}); + +describe('remapRowFeatureCodes', () => { + it('translates codes across catalogs that assigned different codes to the same name', () => { + // Dictionary-only datasets assign codes by first-seen order, so the resident + // preview and the full-dataset scan can disagree. The resident batch here saw + // GeneB first (code 0), GeneA second (code 1); the full catalog is the reverse. + const resident = catalog([ + [0, 'GeneB'], + [1, 'GeneA'], + ]); + const full = catalog([ + [0, 'GeneA'], + [1, 'GeneB'], + ]); + // Rows: GeneB, GeneA, GeneB in resident codes. + const remapped = remapRowFeatureCodes(new Int32Array([0, 1, 0]), resident, full); + // Same genes, now in the full catalog's space: GeneB→1, GeneA→0. + expect(Array.from(remapped)).toEqual([1, 0, 1]); + }); + + it('is an identity pass when both catalogs agree (e.g. a real code column)', () => { + const shared = catalog([ + [0, 'GeneA'], + [1, 'GeneB'], + ]); + const remapped = remapRowFeatureCodes(new Int32Array([1, 0, 1]), shared, shared); + expect(Array.from(remapped)).toEqual([1, 0, 1]); + }); + + it('maps codes with no name in the source, or names absent from the target, to -1', () => { + const resident = catalog([ + [0, 'GeneA'], + [1, 'GeneB'], + ]); + // Target lacks GeneB entirely (and gained an unrelated gene). + const partial = catalog([ + [0, 'GeneA'], + [5, 'GeneC'], + ]); + // Row codes include an unknown source code (7) and GeneB (1, absent downstream). + const remapped = remapRowFeatureCodes(new Int32Array([0, 1, 7]), resident, partial); + expect(Array.from(remapped)).toEqual([0, -1, -1]); + }); + + it('returns an Int32Array of the same length', () => { + const c = catalog([[0, 'GeneA']]); + const remapped = remapRowFeatureCodes(new Int32Array([0, 0, 0, 0]), c, c); + expect(remapped).toBeInstanceOf(Int32Array); + expect(remapped).toHaveLength(4); + }); +}); diff --git a/packages/layers/src/PointsLayer.ts b/packages/layers/src/PointsLayer.ts index fbd5d5d6..1b58b658 100644 --- a/packages/layers/src/PointsLayer.ts +++ b/packages/layers/src/PointsLayer.ts @@ -27,6 +27,8 @@ export interface PointsLayerProps { pointMinSizeScale?: number; viewZoom?: number | null; color?: [number, number, number, number]; + /** Colour points by their per-point feature code instead of the flat color. */ + colorByFeature?: boolean; featureCodes?: readonly number[]; /** Source-side integer codes aligned with the preloaded table rows. */ preloadedFeatureCodes?: ArrayLike; @@ -55,6 +57,7 @@ function emptyFilteredBatch(batch: ColumnarNdarrayPointsBatch): ColumnarNdarrayP data: emptyData, shape: [axisCount, 0], pointCount: 0, + featureCodes: new Int32Array(0), }; } @@ -63,8 +66,13 @@ async function filterPreloadedBatch( featureCodes: readonly number[] | undefined, preloadedFeatureCodes: ArrayLike | undefined ): Promise { + // No feature filter: draw everything, but carry the row-aligned codes so the + // render path can colour by feature. `preloadedFeatureCodes` is aligned to the + // full preloaded batch, so it maps row-for-row onto the unfiltered geometry. if (featureCodes === undefined) { - return batch; + return hasPreloadedRowFeatureCodes(preloadedFeatureCodes) + ? { ...batch, featureCodes: preloadedFeatureCodes } + : batch; } if (featureCodes.length === 0) { return emptyFilteredBatch(batch); @@ -84,6 +92,7 @@ async function filterPreloadedBatch( data: filtered.data, shape: filteredShape, pointCount, + ...(filtered.featureCodes ? { featureCodes: filtered.featureCodes } : {}), }; } diff --git a/packages/layers/src/engine/PointsDataEngine.ts b/packages/layers/src/engine/PointsDataEngine.ts index cd48b83d..cd586930 100644 --- a/packages/layers/src/engine/PointsDataEngine.ts +++ b/packages/layers/src/engine/PointsDataEngine.ts @@ -1,4 +1,12 @@ -import type { PointsElement, PointsLoadResult } from '@spatialdata/core'; +import { + DEFAULT_POINTS_MEMORY_CAP, + featureCodeMapFromCatalog, + remapRowFeatureCodes, + type PointsElement, + type PointsFeatureCatalog, + type PointsLoadProgress, + type PointsLoadResult, +} from '@spatialdata/core'; import { pointsRenderResourceSignature, resolvePointsRenderResource } from '../resolvePointsRenderResource.js'; import type { PointsRenderResource } from '../pointsLoader.js'; @@ -15,10 +23,18 @@ import type { PointsRenderResource } from '../pointsLoader.js'; * - the stable render-resource memo (`stablePointsResourceRef`), * - the async preload orchestration (the load effect's points branch). * - * Parity scope: this mirrors the current branch's *preloaded flat scatter* path - * only. Metadata probing, Morton tiling, feature catalog/codes, and tile-debug - * state are deliberately NOT here yet — they are the dark capabilities that MVP - * steps 2–4 will wire *into this engine* (see docs/plans/points-mvp-and-roadmap). + * Scope: the *preloaded flat scatter* path plus the **feature catalog** and + * **row feature codes** that MVP step 2 (feature filter) needs. Metadata + * probing, Morton tiling, and tile-debug state are still dark — later MVP steps + * wire them *into this engine* (see docs/plans/points-mvp-and-roadmap). + * + * Alignment invariant (load-bearing): `getRowFeatureCodes(key)` is row-aligned + * with the resident batch from `ensureLoaded`. Both the geometry preload + * (`element.loadPoints()`) and the row codes (`element.loadRowFeatureCodes()`) + * read the first `min(rowCount, memoryCap)` rows in *file order* under the same + * default memory cap, so index i in the codes array names the feature of point i + * in the batch. If a configurable memory cap is ever threaded, it MUST go to + * both calls identically or the filter mask will be misaligned. */ export type PointsLoadStatus = 'idle' | 'loading' | 'ready' | 'error'; @@ -38,15 +54,89 @@ export interface PointsDataEngineCallbacks { interface PointsEntry { data?: PointsLoadResult; + /** Memory cap (max resident rows) the current `data`/`loading` was requested + * with. A change means the resident window must reload — see `ensureLoaded`. */ + memoryCap?: number; + /** Aborts the in-flight preload when it is superseded (a cap change), so a + * stale load doesn't run its expensive main-thread fallback to completion. */ + loadAbort?: AbortController; status: PointsLoadStatus; loading?: Promise; resource?: { signature: string; resource: PointsRenderResource }; + /** Feature catalog: `undefined` while unloaded, `null` once settled for an + * element with no `feature_key`, else the catalog. `catalogLoaded` disambiguates + * "not yet requested" from "settled as null". */ + catalog?: PointsFeatureCatalog | null; + catalogLoaded?: boolean; + catalogLoading?: Promise; + /** True once the full-dataset catalog scan (`listFeaturesWithCounts`) has + * replaced any resident-subset preview. Until then `catalog` may reflect only + * the resident batch, so the full scan is allowed to run and supersede it. */ + catalogComplete?: boolean; + /** Per-row feature codes aligned to the resident batch (see class doc). Value + * is `undefined` when the element exposes no feature codes; `rowCodesLoaded` + * marks the settled state. */ + rowCodes?: ArrayLike; + rowCodesLoaded?: boolean; + rowCodesLoading?: Promise; + /** The catalog whose code space {@link rowCodes} are expressed in. When the + * catalog is upgraded (resident preview → full-dataset), `reconcileRowCodes` + * remaps `rowCodes` into the new space and updates this — keeping the render's + * per-row codes aligned with the panel's selection codes for dictionary-only + * datasets, where codes are app-assigned and can differ between catalog builds. */ + rowCodesCatalog?: PointsFeatureCatalog; + /** True when the element has a file-backed feature code column (authoritative + * codes; a real feature index). Undefined until the resident batch loads; false + * for dictionary-only feature columns. Gates the whole-dataset feature-index + * scan — see {@link hasFeatureCodeColumn}. */ + featureCodeColumn?: boolean; + /** Memoized distinct codes in {@link rowCodes}, invalidated by identity via + * `residentCodesSource` (see `getResidentFeatureCodes`). */ + residentCodes?: ReadonlySet; + residentCodesSource?: ArrayLike; + /** Whole-dataset points for the active selection, loaded via the feature-index + * scan and keyed by the selected-codes `signature` so a selection change + * rebuilds it. `resource` is the stable render resource, built lazily. */ + matching?: { signature: string; result: PointsLoadResult; resource?: PointsRenderResource }; + /** In-flight feature-index scan, with progressive counts updated from the + * scan's `onProgress` so the panel can show partial stats as they accumulate. */ + matchingLoading?: { + signature: string; + promise: Promise; + matchedRows: number; + scannedRows: number; + partialResult?: PointsLoadResult; + /** SPIKE: render resource built from `partialResult`, cached on that chunk's + * identity so it only rebuilds when a new chunk arrives (not every pan). */ + partialResource?: { source: PointsLoadResult; resource: PointsRenderResource }; + }; +} + +/** Public snapshot of a selection's feature-index load, for the filter panel. */ +export interface PointsMatchingLoadState { + /** The scan for this exact selection is in flight. */ + loading: boolean; + /** Matched points so far (progressive while loading; final once settled). */ + matchedRows: number; + /** Rows examined so far (progressive while loading; final once settled). */ + scannedRows: number; + /** True once the scan for this selection has settled. */ + settled: boolean; + /** The selection is served by filtering a larger in-memory batch (a removal + * reused it — no scan ran). `matchedRows` is then the whole batch, not the + * drawn subset, so the panel words it as "served from memory". */ + covered?: boolean; } export class PointsDataEngine { private readonly entries = new Map(); private readonly listeners = new Set<() => void>(); private readonly callbacks: PointsDataEngineCallbacks; + /** Monotonic cache-mutation counter. Backs a `useSyncExternalStore` snapshot so + * React reliably re-renders on every settled load — including late async + * completions (e.g. the full-dataset catalog scan) that a plain subscribe → + * bump-a-counter → pull-during-render pattern was dropping. */ + private version = 0; constructor(callbacks: PointsDataEngineCallbacks = {}) { this.callbacks = callbacks; @@ -60,7 +150,13 @@ export class PointsDataEngine { }; } + /** Snapshot for `useSyncExternalStore`: changes on every {@link notify}. */ + getVersion(): number { + return this.version; + } + private notify(): void { + this.version += 1; for (const listener of this.listeners) { listener(); } @@ -103,39 +199,513 @@ export class PointsDataEngine { return resource; } + // --- Feature-index render scan (whole-dataset load of a selection) ---------- + + /** Order-independent cache key for a selected-codes set. */ + private static matchingSignature(featureCodes: readonly number[]): string { + return [...featureCodes].sort((left, right) => left - right).join(','); + } + + /** Feature codes a matched batch/scan covers, parsed from its signature + * (sorted-codes-joined; `''` → the empty selection). */ + private static coveredCodes(signature: string): Set { + if (signature === '') { + return new Set(); + } + return new Set(signature.split(',').map(Number)); + } + + /** + * Whether an already-loaded batch (resident preload OR matched scan) still + * satisfies a (possibly changed) memory cap. A COMPLETE batch (it captured all + * rows before hitting the cap) always does. A TRUNCATED batch (it filled up to + * its cap, more rows exist) only does while the new cap doesn't ask for more + * rows than it already holds — so lowering the cap never reloads/rescans, and + * raising it past a truncated batch does, to fetch the extra rows. + */ + private static batchAdequateForCap(result: PointsLoadResult, memoryCap: number): boolean { + if (!result.preloadTruncated) { + return true; + } + return (result.shape[1] ?? 0) >= memoryCap; + } + + /** + * Copy a resident batch keeping only its first `rows` points (file order), + * marked truncated. Used to shed rows when the memory cap is LOWERED below what + * is resident — so a 4M cap never keeps 8M rows around — without re-fetching. + * Columnar geometry + per-row codes are sliced in lockstep. + */ + private static sliceResidentBatch(data: PointsLoadResult, rows: number): PointsLoadResult { + const sliceArray = (array: ArrayLike): ArrayLike => { + const maybeSliceable = array as unknown as { + slice?: (start: number, end: number) => ArrayLike; + }; + return typeof maybeSliceable.slice === 'function' + ? maybeSliceable.slice(0, rows) + : Array.prototype.slice.call(array, 0, rows); + }; + const dims = data.shape[0] ?? data.data.length; + return { + ...data, + shape: [dims, rows], + data: data.data.map(sliceArray), + ...(data.featureCodes ? { featureCodes: sliceArray(data.featureCodes) } : {}), + preloadTruncated: true, + }; + } + + /** + * Ensure the selected features' points are available for rendering. The scan + * loads the whole dataset for a selection (footer stats skip non-matching row + * groups) and retains the per-row codes, so the render can **filter that batch + * in the layer**. That makes a selection that is a SUBSET of an already-loaded + * batch a free in-memory filter — removing a feature never re-scans (its rows + * are already in memory), symmetric with resident filtering. A scan runs only + * when the selection needs codes no loaded/in-flight batch covers. Settles → + * `notify()`. + */ + ensureMatchingFeaturesLoaded( + target: PointsLoadTarget, + featureCodes: readonly number[], + memoryCap: number = DEFAULT_POINTS_MEMORY_CAP + ): Promise { + const { key, element } = target; + // there will be various mutating side-effects on entry as we progress... + // so maybe that could include gradual accumulation of points, + // pending a less side-effect/mutation-ridden approach. + // we've been hitting a lot of general issues debugging this in general, + // (not necessarily this particular point in the code) and the behaviour is not right. + // I think I'm inclined to more purity. Might consider using Effect? + // would be a much bigger future change. + const entry = this.entries.get(key) ?? { status: 'idle' as PointsLoadStatus }; + this.entries.set(key, entry); + const signature = PointsDataEngine.matchingSignature(featureCodes); + const isCoveredBy = (sig: string): boolean => { + const covered = PointsDataEngine.coveredCodes(sig); + return featureCodes.every((code) => covered.has(code)); + }; + // A loaded batch already covers this selection AND still satisfies the memory + // cap → reuse it, the layer filters down to the current codes. No scan. This + // is both the removal fast path and the cap-lowering fast path: dropping the + // cap (or any cap change where the loaded rows already suffice) never rescans. + if ( + entry.matching && + isCoveredBy(entry.matching.signature) && + PointsDataEngine.batchAdequateForCap(entry.matching.result, memoryCap) + ) { + // Any in-flight scan for a different (now-unneeded) selection is superseded. + entry.matchingLoading = undefined; + return Promise.resolve(); + } + // An in-flight scan will cover this selection once it settles (e.g. a feature + // was removed mid-scan) → wait for it rather than starting another. + if (entry.matchingLoading && isCoveredBy(entry.matchingLoading.signature)) { + return entry.matchingLoading.promise; + } + + // Notify at most every `PROGRESS_NOTIFY_STEP` matched rows so the panel's + // partial stats update live without a re-render per scanned row group. + // (not sure how important this is, may prefer to see more granular update) + const PROGRESS_NOTIFY_STEP = 5_000; + let lastNotifiedMatched = 0; + const onProgress = (progress: PointsLoadProgress): void => { + // I'm a bit iffy about this ambient stateful thing + const loading = entry.matchingLoading; + if (!loading || loading.signature !== signature) { + return; + } + loading.matchedRows = progress.matchedRows; + loading.scannedRows = progress.scannedRows; + // we have a partialResult, which includes an accumulated buffer + // probably prefer to have AsyncGenerator throughout rather than this + // we're not doing the right thing yet, just seeing if we can push some data + // and render... at very least needs cleaning up resources, etc etc + loading.partialResult = progress.partialResult; + if (progress.matchedRows - lastNotifiedMatched >= PROGRESS_NOTIFY_STEP) { + lastNotifiedMatched = progress.matchedRows; + this.notify(); // runs during the async scan, not render — safe to notify sync + } + }; + + const promise = (async () => { + try { + // Dict-only elements have no file-backed code column, so the scan must + // resolve each row's feature_name against the same catalog the selection + // was made in. Pass that map; the core call ignores it for indexed + // elements (which match on their code column instead). + const featureCodeByName = + entry.featureCodeColumn === true + ? undefined + : featureCodeMapFromCatalog(entry.catalog); + //todo streamy version + const result = await element.loadPointsMatchingFeatureCodes({ + featureCodes, + memoryCap, + onProgress, + ...(featureCodeByName ? { featureCodeByName } : {}), + }); + // Apply only if this is still the latest requested scan — a newer + // selection may have superseded it while we were loading. Keeping the + // previous `matching` batch until the current one is ready is what lets + // the render keep showing the prior selection instead of blanking. + if (entry.matchingLoading?.signature === signature) { + entry.matching = { signature, result }; + } + } catch (error) { + console.error(`Failed feature-index scan for ${target.layerId}:`, error); + } finally { + if (entry.matchingLoading?.signature === signature) { + entry.matchingLoading = undefined; + } + this.notify(); + } + })(); + entry.matchingLoading = { signature, promise, matchedRows: 0, scannedRows: 0 }; + // Surface the loading transition, but DEFER it: this method is kicked from + // `getLayers` *during* render, so a synchronous notify would setState mid- + // render. A microtask runs after the current render commits. + queueMicrotask(() => this.notify()); + return promise; + } + + /** + * Load-state snapshot for a selection's feature-index scan: whether it is in + * flight, its progressive matched/scanned counts, and its final counts once + * settled. Drives the panel's "loading … / N points" indicator. Returns + * `undefined` when this selection has neither loaded nor started. + */ + getMatchingLoadState( + key: string, + featureCodes: readonly number[] + ): PointsMatchingLoadState | undefined { + const entry = this.entries.get(key); + const signature = PointsDataEngine.matchingSignature(featureCodes); + if (entry?.matchingLoading?.signature === signature) { + return { + loading: true, + matchedRows: entry.matchingLoading.matchedRows, + scannedRows: entry.matchingLoading.scannedRows, + settled: false, + }; + } + if (entry?.matching?.signature === signature) { + const result = entry.matching.result; + return { + loading: false, + matchedRows: result.shape[1] ?? 0, + scannedRows: result.scannedRowCount ?? 0, + settled: true, + }; + } + // The selection is a subset of an already-loaded batch (a removal reused it). + // It is settled — the layer just filters the batch — so report it as loaded + // rather than letting the indicator vanish. `covered` lets the panel word it + // as "served from memory" since the count is the whole in-memory batch. + if (entry?.matching && PointsDataEngine.coveredCodes(entry.matching.signature).size > 0) { + const covered = PointsDataEngine.coveredCodes(entry.matching.signature); + if (featureCodes.length > 0 && featureCodes.every((code) => covered.has(code))) { + const result = entry.matching.result; + return { + loading: false, + matchedRows: result.shape[1] ?? 0, + scannedRows: result.scannedRowCount ?? 0, + settled: true, + covered: true, + }; + } + } + return undefined; + } + + /** + * Feature codes of the **last completed** matched selection — i.e. the + * non-resident features whose points are actually on screen right now (see + * {@link getMatchingResource}, which keeps that batch during a new scan). + * + * The panel greys features that are neither resident nor rendered. Deriving + * "rendered" from this last-completed set (not the current scan's settled + * state) is what keeps already-loaded features un-greyed while a newly added + * feature's scan is still in flight. `undefined` when nothing has settled. + */ + getLoadedMatchingFeatureCodes(key: string): ReadonlySet | undefined { + const signature = this.entries.get(key)?.matching?.signature; + if (signature === undefined) { + return undefined; + } + return PointsDataEngine.coveredCodes(signature); + } + + /** + * Per-row feature codes of the last-completed matched batch, row-aligned with + * {@link getMatchingResource}'s geometry. The render passes these to the layer + * as `preloadedFeatureCodes` so it can filter the (possibly superset) matched + * batch down to the current selection in memory — no re-scan on a removal. + */ + getMatchingRowFeatureCodes(key: string): ArrayLike | undefined { + return this.entries.get(key)?.matching?.result.featureCodes; + } + + /** + * Stable render resource for the **last completed** matched selection, or null + * if no selection has ever settled. Deliberately NOT keyed to the current + * selection: while a new selection's scan is in flight, this keeps returning the + * previous selection's batch so the render shows those points instead of + * blanking for the (potentially multi-second) scan. + * + * The resource is cached on the matched batch and only changes identity when the + * batch does, so panning doesn't reset the composite. + * Pair with `getMatchingLoadState` (exact-signature) + * for the "is the current selection loaded" question. + */ + getMatchingResource(element: PointsElement, key: string): PointsRenderResource | null { + const entry = this.entries.get(key); + if (!entry?.matching) { + return null; + } + // Empty-lock guard: a scan that matched no rows must NOT supersede the resident + // preview — otherwise the render locks to an empty batch with no way to recover + // (the settled selection never re-scans). Returning null falls back to resident + // filtering. A legitimately empty selection can't reach here: an empty + // `featureCodes` selection short-circuits before any scan is kicked. + if ((entry.matching.result.shape[1] ?? 0) === 0) { + return null; + } + if (entry.matching.resource) { + return entry.matching.resource; + } + const cache = { preloaded: entry.matching.result, metadataKnown: false }; + const resource = resolvePointsRenderResource(element, cache, { + experimentalOptimizations: 'off' as const, + }); + if (resource) { + entry.matching.resource = resource; + } + return resource; + } + /** - * Idempotently preload an element's points. No-op if already loaded or a load - * is in flight; the returned promise resolves when the (possibly already - * running) load settles. Status transitions are reported via `onStatus`; the - * cache mutation notifies subscribers. + * Render resource for the in-flight scan's latest `partialResult`, which the + * producer builds as a GROWING buffer (every matched chunk accumulated so far), + * so points progressively fill in before the full scan settles. Cached on the + * partial's identity (rebuilds only when a new chunk grows the buffer, not per + * pan). `null` when no scan is in flight / nothing has decoded yet / empty. */ - ensureLoaded(target: PointsLoadTarget): Promise { + getMatchingPartialResource(element: PointsElement, key: string): PointsRenderResource | null { + const loading = this.entries.get(key)?.matchingLoading; + const partial = loading?.partialResult; + if (!loading || !partial || (partial.shape[1] ?? 0) === 0) { + return null; + } + if (loading.partialResource?.source === partial) { + return loading.partialResource.resource; + } + const resource = resolvePointsRenderResource( + element, + { preloaded: partial, metadataKnown: false }, + { experimentalOptimizations: 'off' as const } + ); + if (resource) { + loading.partialResource = { source: partial, resource }; + } + return resource; + } + + /** Per-row feature codes of the in-flight scan's partial buffer, row-aligned + * with {@link getMatchingPartialResource}. The render passes these as + * `preloadedFeatureCodes` so the partial overlay can filter to the *current* + * selection — otherwise a feature deselected mid-scan (whose scan is still + * running because the smaller selection is covered) keeps rendering until the + * scan settles. */ + getMatchingPartialRowFeatureCodes(key: string): ArrayLike | undefined { + return this.entries.get(key)?.matchingLoading?.partialResult?.featureCodes; + } + + /** Whether the feature-index scan for this exact selection is in flight. */ + isMatchingLoading(key: string, featureCodes: readonly number[]): boolean { + const entry = this.entries.get(key); + return entry?.matchingLoading?.signature === PointsDataEngine.matchingSignature(featureCodes); + } + + /** Whether the resident batch is in its final state for this cap — i.e. no + * resident work is needed. False (work needed) when it must GROW (a truncated + * batch and the cap was raised past it → reload) or SHRINK (it holds more rows + * than the cap → shed the excess). So raising past a truncated batch reloads, + * lowering below what's loaded sheds, and any cap a complete batch within the + * cap already covers is a no-op. */ + isLoadedWithCap(key: string, memoryCap: number): boolean { + const entry = this.entries.get(key); + if (entry?.data === undefined) { + return false; + } + return ( + PointsDataEngine.batchAdequateForCap(entry.data, memoryCap) && + (entry.data.shape[1] ?? 0) <= memoryCap + ); + } + + /** + * Truncation state of the resident preload — is it the whole dataset or only a + * capped window, and how many rows of how many. `undefined` until data loads. + * Surfaced so the user can see when raising the cap would show more points. + */ + getResidentTruncation( + key: string + ): { truncated: boolean; loaded: number; total?: number } | undefined { + const data = this.entries.get(key)?.data; + if (!data) { + return undefined; + } + return { + truncated: data.preloadTruncated === true, + loaded: data.shape[1] ?? 0, + ...(data.totalRowCount !== undefined ? { total: data.totalRowCount } : {}), + }; + } + + /** + * Truncation state of what is actually on screen. With an active selection that + * a scanned batch covers, that batch is the render — report ITS count and + * whether it hit the cap (`filtered`), not the resident preload's, so the panel + * doesn't keep saying "showing 4M" while a filtered subset is drawn. Otherwise + * falls back to the resident preload. `undefined` until something has loaded. + */ + getActiveTruncation( + key: string, + featureCodes: readonly number[] | undefined + ): { truncated: boolean; loaded: number; total?: number; filtered?: boolean } | undefined { + const entry = this.entries.get(key); + if (!entry) { + return undefined; + } + if (featureCodes && featureCodes.length > 0 && entry.matching) { + const covered = PointsDataEngine.coveredCodes(entry.matching.signature); + if (covered.size > 0 && featureCodes.every((code) => covered.has(code))) { + const result = entry.matching.result; + return { + truncated: result.preloadTruncated === true, + loaded: result.shape[1] ?? 0, + filtered: true, + }; + } + } + return this.getResidentTruncation(key); + } + + /** + * Idempotently preload an element's points at a given memory cap. A no-op when + * the resident data already satisfies the cap (see {@link isLoadedWithCap}) — + * so lowering the cap, or raising it when a complete batch already covers it, + * never reloads. Only RAISING the cap past a *truncated* batch reloads, and it + * keeps the previously-loaded data on screen until the larger batch settles + * (an atomic swap — no blank). The full-dataset catalog and the matched + * selection are preserved across the reload. Status via `onStatus`; notifies. + */ + ensureLoaded( + target: PointsLoadTarget, + memoryCap: number = DEFAULT_POINTS_MEMORY_CAP + ): Promise { const { key, layerId, element } = target; const existing = this.entries.get(key); - if (existing?.data !== undefined) { + // (1) Existing data covers this cap without a reload (it is complete, or a + // truncated batch the lowered cap doesn't outgrow). + if (existing?.data !== undefined && PointsDataEngine.batchAdequateForCap(existing.data, memoryCap)) { + // Cancel a now-unneeded in-flight load (e.g. the cap was raised then + // lowered back to what we already hold). + if (existing.loading) { + existing.loadAbort?.abort(); + existing.loading = undefined; + existing.loadAbort = undefined; + } + existing.memoryCap = memoryCap; + // Cap lowered below what's resident → shed the excess in memory (no + // re-fetch), so a 4M cap doesn't keep holding an 8M batch. Rebuild the + // render resource / resident-code memos from the sliced batch. + if ((existing.data.shape[1] ?? 0) > memoryCap) { + existing.data = PointsDataEngine.sliceResidentBatch(existing.data, memoryCap); + existing.resource = undefined; + existing.residentCodes = undefined; + existing.residentCodesSource = undefined; + if (existing.rowCodes && existing.rowCodes.length > memoryCap) { + existing.rowCodes = Array.prototype.slice.call(existing.rowCodes, 0, memoryCap); + } + this.notify(); + } return Promise.resolve(); } - if (existing?.loading) { + // (2) A load for this exact cap is already in flight → dedup. + if (existing?.loading && existing.memoryCap === memoryCap) { return existing.loading; } const entry: PointsEntry = existing ?? { status: 'idle' }; + // Reload needed (first load, or the cap was raised past a truncated batch). + // Abort any superseded in-flight load, but KEEP the old resident data / + // resource / row codes rendered — they are swapped atomically on completion, + // so the view keeps showing what it had while the larger batch loads. + entry.loadAbort?.abort(); + entry.memoryCap = memoryCap; + const abort = new AbortController(); + entry.loadAbort = abort; entry.status = 'loading'; this.entries.set(key, entry); this.callbacks.onStatus?.(layerId, 'loading'); const loading = (async () => { try { - const data = await element.loadPoints(); + // Read the feature column with the geometry so the filter's catalog and + // per-row codes come from this one decode — no separate blocking load at + // filter time. The catalog here reflects only the *resident* batch, so it + // is an instant preview (`catalogLoaded`, not `catalogComplete`): the + // full-dataset `ensureFeatureCatalog` scan is still allowed to run and + // supersede it (a feature-ordered file's first part holds only a slice of + // the features). Row codes are complete for the resident batch. + const data = await element.loadPoints({ + includeFeatureCodes: true, + memoryCap, + signal: abort.signal, + }); + // A newer cap may have superseded this load mid-flight; if so, drop it. + if (abort.signal.aborted || entry.memoryCap !== memoryCap) { + return; + } + // Atomic swap: replace the (possibly still-rendered) old batch and drop + // the resource/resident-codes memos derived from it so they rebuild. entry.data = data; + entry.resource = undefined; + entry.residentCodes = undefined; + entry.residentCodesSource = undefined; entry.status = 'ready'; + entry.featureCodeColumn = data.hasFeatureCodeColumn === true; + if (data.featureCatalog !== undefined && !entry.catalogComplete) { + entry.catalog = data.featureCatalog; + entry.catalogLoaded = true; + } + if (data.featureCodes !== undefined) { + entry.rowCodes = data.featureCodes; + entry.rowCodesLoaded = true; + // The preload derived these codes against its own catalog (the resident + // preview, unless a full-dataset catalog already superseded it). Record + // that space and reconcile to whatever catalog is current now. + entry.rowCodesCatalog = data.featureCatalog; + this.reconcileRowCodes(entry); + } this.callbacks.onStatus?.(layerId, 'ready'); } catch (error) { + // Aborted (cap changed) or superseded → not a real error; stay quiet. + if (abort.signal.aborted || entry.memoryCap !== memoryCap) { + return; + } entry.status = 'error'; this.callbacks.onStatus?.(layerId, 'error'); console.error(`Failed to load points for ${layerId}:`, error); } finally { - entry.loading = undefined; + // Only clear the in-flight markers if they are still ours (a superseding + // cap change installs its own `loading`/`loadAbort`). + if (entry.memoryCap === memoryCap) { + entry.loading = undefined; + entry.loadAbort = undefined; + } this.notify(); } })(); @@ -143,7 +713,217 @@ export class PointsDataEngine { return loading; } - /** Drop an element from the cache (on unload / dataset switch). */ + // --- Feature catalog (MVP step 2: feature filter) -------------------------- + + /** + * The feature catalog for an element: `undefined` until settled, then `null` + * for an element with no `feature_key`, else the catalog. Reactive via + * `subscribe` — a settled load calls `notify()`. + */ + getFeatureCatalog(key: string): PointsFeatureCatalog | null | undefined { + const entry = this.entries.get(key); + return entry?.catalogLoaded ? (entry.catalog ?? null) : undefined; + } + + isFeatureCatalogLoading(key: string): boolean { + const entry = this.entries.get(key); + if (!entry || entry.catalogLoaded) { + return false; + } + // The catalog rides the geometry preload (includeFeatureCodes), so a running + // geometry load counts as the catalog loading too — the panel shows a + // spinner rather than a premature "load feature list" prompt. + return entry.catalogLoading !== undefined || entry.loading !== undefined; + } + + /** + * True while the full-dataset catalog scan is still running behind an instant + * resident-subset preview. Lets the panel show a "loading the full feature + * list" hint without hiding the preview it already has. + */ + isFeatureCatalogRefining(key: string): boolean { + const entry = this.entries.get(key); + return ( + entry?.catalogLoaded === true && + entry.catalogComplete !== true && + entry.catalogLoading !== undefined + ); + } + + /** + * The distinct feature codes actually present in the resident batch (the + * preload cap means a feature-ordered file only loads a slice of its features). + * The panel greys features outside this set so selecting one that isn't loaded + * — which would render no points — is understandable rather than a glitch. + * Returns `undefined` when the row codes are not yet resident. Memoized against + * the row-codes identity so the O(rows) scan runs once per batch. + */ + /** + * True when the element has a file-backed feature code column — a real feature + * index whose codes are globally authoritative. False for dictionary-only + * feature columns (codes app-assigned, only stable within one catalog build) or + * an element with no feature codes. Undefined-safe: false until the resident + * batch has loaded. Gates the whole-dataset feature-index scan. + */ + hasFeatureCodeColumn(key: string): boolean { + return this.entries.get(key)?.featureCodeColumn === true; + } + + /** + * Whether a whole-dataset feature scan can run for this element — i.e. reach + * matching points beyond the resident preload window. True with a file-backed + * code column (footer stats skip row groups), AND for dictionary-only elements + * once a catalog is loaded: the scan resolves each row's `feature_name` against + * that catalog's code space (no row-group skipping, so it reads the whole file, + * but it retains every match up to the cap). False before any catalog loads — + * dict-only codes are only stable relative to a catalog, so there'd be nothing + * to match names against. Gates the render scan and the on-demand affordance. + */ + supportsFeatureScan(key: string): boolean { + const entry = this.entries.get(key); + if (!entry) { + return false; + } + return entry.featureCodeColumn === true || (entry.catalogLoaded === true && !!entry.catalog); + } + + /** + * Re-express {@link PointsEntry.rowCodes} in the current catalog's code space + * when it was derived against an older one (resident preview → full-dataset + * upgrade). No-op for authoritative file-backed codes (identical across builds) + * and when the source/target catalogs are the same object. See + * {@link remapRowFeatureCodes} for why dictionary-only codes need this. + */ + private reconcileRowCodes(entry: PointsEntry): void { + if (entry.featureCodeColumn === true) { + return; + } + const source = entry.rowCodesCatalog; + const target = entry.catalog; + if (!entry.rowCodes || !source || !target || source === target) { + return; + } + entry.rowCodes = remapRowFeatureCodes(entry.rowCodes, source, target); + entry.rowCodesCatalog = target; + // The distinct-codes memo was keyed to the old array identity — invalidate. + entry.residentCodes = undefined; + entry.residentCodesSource = undefined; + } + + getResidentFeatureCodes(key: string): ReadonlySet | undefined { + const entry = this.entries.get(key); + const rowCodes = entry?.rowCodes; + if (!entry || rowCodes === undefined) { + return undefined; + } + if (entry.residentCodes && entry.residentCodesSource === rowCodes) { + return entry.residentCodes; + } + const set = new Set(); + for (let i = 0; i < rowCodes.length; i += 1) { + set.add(rowCodes[i]); + } + entry.residentCodes = set; + entry.residentCodesSource = rowCodes; + return set; + } + + /** + * Idempotently build the *full-dataset* feature catalog (feature-column scan; + * worker-offloaded for oversized datasets). Uses `listFeaturesWithCounts` so the + * panel can show/sort by per-feature counts. Runs even when a resident-subset + * preview is already showing (`catalogLoaded` but not `catalogComplete`) and + * supersedes it; no-op once the full scan has settled (`catalogComplete`) or is + * in flight. + */ + ensureFeatureCatalog(target: PointsLoadTarget): Promise { + const { key, element } = target; + const entry = this.entries.get(key) ?? { status: 'idle' as PointsLoadStatus }; + this.entries.set(key, entry); + if (entry.catalogComplete) { + return Promise.resolve(); + } + if (entry.catalogLoading) { + return entry.catalogLoading; + } + + const loading = (async () => { + try { + const fullCatalog = await element.listFeaturesWithCounts(); + entry.catalog = fullCatalog; + // The full-dataset catalog is authoritative. Re-express any resident row + // codes (derived against the resident-preview catalog) in its code space + // so the render's per-row codes match the panel's selection + swatches. + this.reconcileRowCodes(entry); + } catch (error) { + // Keep any resident preview catalog on failure rather than blanking it. + if (!entry.catalogLoaded) entry.catalog = null; + console.error(`Failed to build points feature catalog for ${target.layerId}:`, error); + } finally { + entry.catalogLoaded = true; + entry.catalogComplete = true; + entry.catalogLoading = undefined; + this.notify(); + } + })(); + entry.catalogLoading = loading; + this.notify(); // surface the loading transition to the panel + return loading; + } + + // --- Row feature codes (the filter mask, aligned to the resident batch) ----- + + /** Per-row feature codes aligned to the resident batch, or `undefined` if the + * element exposes none / they are not yet loaded. See the class-doc alignment + * invariant. */ + getRowFeatureCodes(key: string): ArrayLike | undefined { + return this.entries.get(key)?.rowCodes; + } + + /** True once row codes have settled (even if the element has none). */ + hasRowFeatureCodes(key: string): boolean { + return this.entries.get(key)?.rowCodesLoaded === true; + } + + /** + * Idempotently load the row feature codes for the resident batch. Reuses the + * engine's catalog for name→code mapping when it is already built (else the + * core loader scans it internally). No-op once settled or in flight. + */ + ensureRowFeatureCodes(target: PointsLoadTarget): Promise { + const { key, element } = target; + const entry = this.entries.get(key) ?? { status: 'idle' as PointsLoadStatus }; + this.entries.set(key, entry); + if (entry.rowCodesLoaded) { + return Promise.resolve(); + } + if (entry.rowCodesLoading) { + return entry.rowCodesLoading; + } + + const loading = (async () => { + try { + const catalog = this.getFeatureCatalog(key); + entry.rowCodes = await element.loadRowFeatureCodes({ featureCatalog: catalog }); + // These codes were derived against `catalog`; record that space so a later + // catalog upgrade reconciles them (see reconcileRowCodes). + entry.rowCodesCatalog = catalog ?? undefined; + this.reconcileRowCodes(entry); + } catch (error) { + entry.rowCodes = undefined; + console.error(`Failed to load points row feature codes for ${target.layerId}:`, error); + } finally { + entry.rowCodesLoaded = true; + entry.rowCodesLoading = undefined; + this.notify(); + } + })(); + entry.rowCodesLoading = loading; + return loading; + } + + /** Drop an element from the cache (on unload / dataset switch). Catalog and row + * codes live in the same entry, so they are evicted together. */ evict(key: string): void { this.entries.delete(key); } diff --git a/packages/layers/src/index.ts b/packages/layers/src/index.ts index 08ae1a4b..ae59f2cd 100644 --- a/packages/layers/src/index.ts +++ b/packages/layers/src/index.ts @@ -96,6 +96,10 @@ export { POINT_SIZE_ZOOM_REFERENCE, zoomScaledPointSize, } from './pointsScatterLayer.js'; +export { PointsFeatureColorExtension } from './pointsFeatureColorExtension.js'; +export { featureCodeToRgb, featureCodeToCssColor } from './pointsFeatureColor.js'; +export { buildPointsAttributes } from './pointsRenderAttributes.js'; +export type { PointsRenderAttributes } from './pointsRenderAttributes.js'; export type { PointsTileHandle, PointsTileLoadResult } from './pointsTileLoadCallbacks.js'; export { createTileDebugStore, @@ -130,4 +134,5 @@ export { type PointsDataEngineCallbacks, type PointsLoadStatus, type PointsLoadTarget, + type PointsMatchingLoadState, } from './engine/PointsDataEngine.js'; diff --git a/packages/layers/src/pointsFeatureColor.ts b/packages/layers/src/pointsFeatureColor.ts new file mode 100644 index 00000000..dab2603c --- /dev/null +++ b/packages/layers/src/pointsFeatureColor.ts @@ -0,0 +1,68 @@ +/** + * Colour-by-feature palette constants — the SINGLE SOURCE for both this JS mirror + * (feature-list swatches) and the GPU shader. The shader + * (`./pointsFeatureColorExtension.ts`) interpolates these same values into its + * GLSL so the two can't drift; the OKLab→sRGB matrices and gamma below also match + * `pfc_oklab2rgb`. A negative code (the "no colour" sentinel) returns grey. + * + * Chroma note: sRGB can only hold OKLCh chroma up to ~0.32 (hue-dependent), so at + * this fixed C most hues are out of gamut and get hard-clamped to the sRGB + * boundary here and in the shader — vivid, at the cost of some hue accuracy and + * of the perceptual evenness the boundary erases. Lower C (~0.2–0.3) trades + * vividness for more in-gamut, even hues; raising C past ~0.32 changes little + * (already clamped). Kept in lockstep so a tweak here re-colours points too. + * + * TODO (revisit): CSS supports `oklch()`/`oklab()` directly, so the swatch could + * skip this CPU RGB conversion — but only once the shader gamut-maps the same way + * the browser does (today both hard-clamp here / the browser reduces chroma), or + * swatches and points would diverge at high C. See the library-wide colour story. + */ +export const PFC_GOLDEN_RATIO_CONJUGATE = 0.6180339887498949; +export const PFC_LIGHTNESS = 0.72; +export const PFC_CHROMA = 0.32; +const TWO_PI = 6.28318530717958648; + +function fract(x: number): number { + return x - Math.floor(x); +} + +function linearToSrgb(x: number): number { + return x <= 0.0031308 ? x * 12.92 : 1.055 * Math.pow(Math.max(x, 0), 1 / 2.4) - 0.055; +} + +function channel255(x: number): number { + return Math.round(Math.max(0, Math.min(1, x)) * 255); +} + +/** OKLab (L, a, b) → sRGB `[r, g, b]` in 0–255 — matches `pfc_oklab2rgb`. */ +function oklabToRgb255(L: number, a: number, b: number): [number, number, number] { + const l_ = L + 0.3963377774 * a + 0.2158037573 * b; + const m_ = L - 0.1055613458 * a - 0.0638541728 * b; + const s_ = L - 0.0894841775 * a - 1.291485548 * b; + const l = l_ * l_ * l_; + const m = m_ * m_ * m_; + const s = s_ * s_ * s_; + const r = 4.0767416621 * l - 3.3077115913 * m + 0.2309699292 * s; + const g = -1.2684380046 * l + 2.6097574011 * m - 0.3413193965 * s; + const bl = -0.0041960863 * l - 0.7034186147 * m + 1.707614701 * s; + return [ + channel255(linearToSrgb(r)), + channel255(linearToSrgb(g)), + channel255(linearToSrgb(bl)), + ]; +} + +/** Categorical colour for a feature code as `[r, g, b]` in 0–255. */ +export function featureCodeToRgb(code: number): [number, number, number] { + if (!(code >= 0)) { + return [128, 128, 128]; + } + const h = fract(code * PFC_GOLDEN_RATIO_CONJUGATE) * TWO_PI; + return oklabToRgb255(PFC_LIGHTNESS, PFC_CHROMA * Math.cos(h), PFC_CHROMA * Math.sin(h)); +} + +/** Same colour as a CSS `rgb(...)` string. */ +export function featureCodeToCssColor(code: number): string { + const [r, g, b] = featureCodeToRgb(code); + return `rgb(${r}, ${g}, ${b})`; +} diff --git a/packages/layers/src/pointsFeatureColorExtension.ts b/packages/layers/src/pointsFeatureColorExtension.ts new file mode 100644 index 00000000..693d8f32 --- /dev/null +++ b/packages/layers/src/pointsFeatureColorExtension.ts @@ -0,0 +1,148 @@ +import { LayerExtension } from '@deck.gl/core'; +import type { Layer } from '@deck.gl/core'; +import { PFC_CHROMA, PFC_GOLDEN_RATIO_CONJUGATE, PFC_LIGHTNESS } from './pointsFeatureColor.js'; + +/** Render a JS number as a GLSL float literal (always with a decimal point, so an + * integer-valued constant doesn't become an `int` in the shader). Lets the shader + * interpolate the SAME palette constants the JS swatch mirror uses. */ +function glslFloat(value: number): string { + const text = String(value); + return text.includes('.') || text.includes('e') ? text : `${text}.0`; +} + +/** + * Uniform block for the highlight. The stored value is `highlightCode + 1`, so + * the "no highlight" state is 0 — which is also what an unbound/zeroed UBO + * reads, making the default safe even if the binding ever fails (feature code 0 + * would otherwise be a valid, and wrongly-highlighted, value). + */ +const PFC_HIGHLIGHT_MODULE = { + name: 'pfcHighlight', + vs: /* glsl */ ` + layout(std140) uniform pfcHighlightUniforms { + float code; + } pfcHighlight; + `, + uniformTypes: { code: 'f32' as const }, +}; + +/** + * Colours scatter points by their per-point feature code, entirely on the GPU. + * + * The feature code rides along as an instance attribute (`featureCode`, supplied + * as the binary `getFeatureCode` attribute) and a small vertex-shader hook maps + * it to a categorical colour, overwriting `vFillColor`. The mapping is + * procedural (a golden-angle hue from the code) so there is no palette buffer to + * upload and no CPU colour pass; the code attribute is also the one a future + * per-code visibility mask will read. + * + * The extension is attached to EVERY scatter layer, not just when colour is on. + * This is load-bearing: deck only calls an extension's `initializeState` when + * the layer first mounts, so attaching it lazily (when colour is toggled on) + * would never register the `featureCode` attribute — the sublayer already + * exists and only updates. Colour is instead gated by the attribute value: with + * no `getFeatureCode` buffer the attribute reads its `-1` default and the shader + * leaves the flat fill colour untouched. + * + * Two more deck subtleties that cost a debugging session: + * - the `in float featureCode` declaration must be in `vs:#decl` (deck does NOT + * auto-declare it) and the main hook in a TOP-LEVEL `inject` (a module's own + * `inject` does not apply to the host layer here); + * - `defaultProps.getFeatureCode` must be declared or deck treats the attribute + * as constant and never reads the binary buffer (as DataFilterExtension does). + * + * Deliberately the smallest possible deck extension — one attribute, one shader + * hook — so it is a low-risk first candidate to port to a WebGPU shading model. + */ +export class PointsFeatureColorExtension extends LayerExtension { + static get componentName(): string { + return 'PointsFeatureColorExtension'; + } + + static defaultProps = { + getFeatureCode: { type: 'accessor', value: -1 }, + /** Emphasize one feature code: points of other codes are desaturated + dimmed + * while this is >= 0. -1 (default) highlights nothing. */ + highlightFeatureCode: { type: 'number', value: -1 }, + }; + + getShaders(this: Layer, extension: this) { + // The base returns null, and the module list may be absent — guard both. + const shaders = (super.getShaders(extension) ?? {}) as { modules?: unknown[] }; + return { + ...shaders, + modules: [...(shaders.modules ?? []), PFC_HIGHLIGHT_MODULE], + inject: { + 'vs:#decl': /* glsl */ ` + in float featureCode; + + // OKLab → linear sRGB → gamma sRGB. OKLab spaces hues perceptually + // evenly, so a golden-angle sweep of its hue gives adjacent codes + // colours that look as distinct as they are numerically (unlike HSV, + // where big hue arcs — the greens — read as one colour). Out-of-gamut + // (L,C) combinations are clamped rather than gamut-mapped; fine for + // categorical swatches at a fixed moderate chroma. + vec3 pfc_oklab2rgb(vec3 lab) { + float l_ = lab.x + 0.3963377774 * lab.y + 0.2158037573 * lab.z; + float m_ = lab.x - 0.1055613458 * lab.y - 0.0638541728 * lab.z; + float s_ = lab.x - 0.0894841775 * lab.y - 1.2914855480 * lab.z; + vec3 lms = vec3(l_ * l_ * l_, m_ * m_ * m_, s_ * s_ * s_); + vec3 rgb = vec3( + 4.0767416621 * lms.x - 3.3077115913 * lms.y + 0.2309699292 * lms.z, + -1.2684380046 * lms.x + 2.6097574011 * lms.y - 0.3413193965 * lms.z, + -0.0041960863 * lms.x - 0.7034186147 * lms.y + 1.7076147010 * lms.z + ); + vec3 low = rgb * 12.92; + vec3 high = 1.055 * pow(max(rgb, 0.0), vec3(1.0 / 2.4)) - 0.055; + return clamp(mix(high, low, step(rgb, vec3(0.0031308))), 0.0, 1.0); + } + + // Golden-angle hue in OKLCh at a fixed lightness/chroma. The lightness, + // chroma and golden-ratio constants come from pointsFeatureColor.ts so + // the swatches and the GPU points share one source (tweak them there). + vec3 pfc_codeToColor(float code) { + float h = fract(code * ${glslFloat(PFC_GOLDEN_RATIO_CONJUGATE)}) * 6.28318530717958648; + return pfc_oklab2rgb(vec3( + ${glslFloat(PFC_LIGHTNESS)}, + ${glslFloat(PFC_CHROMA)} * cos(h), + ${glslFloat(PFC_CHROMA)} * sin(h) + )); + } + `, + 'vs:#main-end': /* glsl */ ` + if (featureCode >= 0.0) { + vec3 pfcColor = pfc_codeToColor(featureCode); + // Highlight: uniform holds highlightCode + 1 (0 = off). Non-matching + // codes are desaturated toward their luminance and dimmed. + if (pfcHighlight.code > 0.5 && abs(featureCode - (pfcHighlight.code - 1.0)) > 0.5) { + float pfcLum = dot(pfcColor, vec3(0.2126, 0.7152, 0.0722)); + pfcColor = mix(vec3(pfcLum), pfcColor, 0.2) * 0.55; + } + vFillColor = vec4(pfcColor, vFillColor.a); + } + `, + }, + }; + } + + draw(this: Layer): void { + const highlight = (this.props as { highlightFeatureCode?: number }).highlightFeatureCode ?? -1; + // Store code + 1 so "no highlight" is 0 (safe default; see PFC_HIGHLIGHT_MODULE). + (this as unknown as { setShaderModuleProps(props: unknown): void }).setShaderModuleProps({ + pfcHighlight: { code: highlight >= 0 ? highlight + 1 : 0 }, + }); + } + + initializeState(this: Layer): void { + const attributeManager = this.getAttributeManager(); + attributeManager?.add({ + featureCode: { + size: 1, + type: 'float32', + stepMode: 'dynamic', + accessor: 'getFeatureCode', + defaultValue: -1, + }, + }); + } +} diff --git a/packages/layers/src/pointsLoader.ts b/packages/layers/src/pointsLoader.ts index 16b0edb9..85c9eb92 100644 --- a/packages/layers/src/pointsLoader.ts +++ b/packages/layers/src/pointsLoader.ts @@ -23,6 +23,10 @@ export interface ColumnarNdarrayPointsBatch { bounds?: SpatialBounds; loadMode?: PointsLoadMode; pointCount?: number; + /** Per-point feature code, aligned row-for-row with {@link data}. Carried + * through filtering/capping so the render path can build a GPU `featureCode` + * attribute (colour-by-feature, per-code visibility). */ + featureCodes?: ArrayLike; } /** Placeholder for future GeoArrow strategies. */ diff --git a/packages/layers/src/pointsRenderAttributes.ts b/packages/layers/src/pointsRenderAttributes.ts new file mode 100644 index 00000000..63525244 --- /dev/null +++ b/packages/layers/src/pointsRenderAttributes.ts @@ -0,0 +1,68 @@ +import type { ColumnarNdarrayPointsBatch } from './pointsLoader.js'; + +/** + * GPU-ready binary attributes for a points batch, replacing deck's per-object + * `getPosition`/`getFeatureCode` accessors (see deck.gl performance guide: + * https://deck.gl/docs/developer-guide/performance#optimize-accessors). + * + * `positions` is interleaved `[x, y, z, x, y, z, …]`; `featureCodes` is one + * float per point. Both are built once per batch and memoized on the batch + * identity, so a stable batch (e.g. the cached filtered result) hands deck the + * same buffer every render — no re-upload, no per-frame CPU pass. + * + * This is the seam where worker-emitted interleaved buffers will slot in when + * streaming lands: the worker produces these arrays directly and the batch + * carries them, making {@link buildPointsAttributes} a pass-through. + */ +export interface PointsRenderAttributes { + length: number; + positions: Float32Array; + /** Per-point feature code as float; `undefined` when the batch has no codes. */ + featureCodes?: Float32Array; +} + +interface CacheEntry extends PointsRenderAttributes { + use3d: boolean; +} + +const cache = new WeakMap(); + +function pointCountOf(batch: ColumnarNdarrayPointsBatch): number { + const fromShape = batch.pointCount ?? batch.shape[1]; + const fromData = batch.data[0]?.length ?? 0; + if (typeof fromShape === 'number' && Number.isFinite(fromShape)) { + return Math.min(fromShape, fromData); + } + return fromData; +} + +export function buildPointsAttributes( + batch: ColumnarNdarrayPointsBatch, + use3d: boolean +): PointsRenderAttributes { + const cached = cache.get(batch); + if (cached && cached.use3d === use3d) { + return cached; + } + + const length = pointCountOf(batch); + const xs = batch.data[0]; + const ys = batch.data[1]; + const zs = batch.data[2]; + const positions = new Float32Array(length * 3); + for (let i = 0; i < length; i += 1) { + positions[i * 3] = xs[i]; + positions[i * 3 + 1] = ys[i]; + positions[i * 3 + 2] = use3d && zs ? zs[i] || 0 : 0; + } + + let featureCodes: Float32Array | undefined; + const codes = batch.featureCodes; + if (codes && codes.length >= length) { + featureCodes = codes instanceof Float32Array ? codes : Float32Array.from(codes); + } + + const entry: CacheEntry = { length, positions, featureCodes, use3d }; + cache.set(batch, entry); + return entry; +} diff --git a/packages/layers/src/pointsScatterLayer.ts b/packages/layers/src/pointsScatterLayer.ts index a4008752..fcaa3320 100644 --- a/packages/layers/src/pointsScatterLayer.ts +++ b/packages/layers/src/pointsScatterLayer.ts @@ -2,7 +2,8 @@ import type { Matrix4 } from '@math.gl/core'; import { COORDINATE_SYSTEM } from '@deck.gl/core'; import { ScatterplotLayer } from 'deck.gl'; import type { ColumnarNdarrayPointsBatch } from './pointsLoader.js'; -import { pointDataFromColumnarBatch } from './pointsLoader.js'; +import { buildPointsAttributes } from './pointsRenderAttributes.js'; +import { PointsFeatureColorExtension } from './pointsFeatureColorExtension.js'; /** Orthographic zoom at which configured pointSize applies at full scale. */ export const POINT_SIZE_ZOOM_REFERENCE = 0; @@ -37,16 +38,20 @@ export interface PointsScatterStyleProps { use3d?: boolean; tileBounds?: [number, number, number, number]; tileSubLayer?: boolean; + /** Colour points by their per-point feature code (requires batch codes). */ + colorByFeature?: boolean; } +// One shared extension instance: it is stateless, so every scatter layer that +// opts into colour-by-feature can reuse it (deck keys shader compilation by the +// extension's identity + props). +const pointsFeatureColorExtension = new PointsFeatureColorExtension(); + export function renderColumnarScatterLayer( id: string, batch: ColumnarNdarrayPointsBatch, props: PointsScatterStyleProps ) { - const pointData = pointDataFromColumnarBatch(batch); - const d = pointData.data; - // Preloaded scatter sizes points in WORLD (common) units so the GPU scales // them with zoom — points shrink when you zoom out, which is exactly where // scatter overdraw is worst — while `radiusMinPixels`/`radiusMaxPixels` clamp @@ -57,18 +62,44 @@ export function renderColumnarScatterLayer( const radiusMinPixels = props.pointRadiusMinPixels ?? DEFAULT_POINT_RADIUS_MIN_PIXELS; const radiusMaxPixels = props.pointRadiusMaxPixels ?? DEFAULT_POINT_RADIUS_MAX_PIXELS; - const pointCount = batch.pointCount ?? batch.shape[1] ?? d[0]?.length ?? 0; + // Feed deck GPU-ready binary attributes (interleaved positions) instead of a + // per-object `getPosition` closure. The buffer is memoized on the batch, so a + // stable batch hands deck the same array every render (no re-upload). + const attributes = buildPointsAttributes(batch, props.use3d === true); + + // Colour-by-feature rides an extra `getFeatureCode` binary attribute consumed + // by the shader extension. Only wire it when codes are present; the extension + // gates on an `enabled` uniform so toggling colour is a uniform swap, not a + // re-batch. Kept off the Morton tile path (tiles carry no codes yet). + // The colour extension is attached to EVERY scatter layer (see the extension + // docs: deck only runs an extension's initializeState at first mount, so + // attaching lazily would never register the attribute). Colour is gated by the + // buffer: supply the per-point codes only when colour-by-feature is on and the + // batch has them; otherwise the attribute reads its -1 default and the shader + // leaves the flat fill colour alone. Buffer is keyed by the accessor name. + // Colour-by-feature is on by default (opt-out via colorByFeature: false) and + // applies whenever the batch carries codes. Future customisation (palettes, + // highlight modes) will hang off richer config. + const colorByFeature = props.colorByFeature !== false && attributes.featureCodes !== undefined; return new ScatterplotLayer({ id, - coordinateSystem: COORDINATE_SYSTEM.CARTESIAN, - data: d[0], + coordinateSystem: "cartesian", + data: { + length: attributes.length, + attributes: { + getPosition: { value: attributes.positions, size: 3 }, + ...(colorByFeature + ? { getFeatureCode: { value: attributes.featureCodes as Float32Array, size: 1 } } + : {}), + }, + }, ...(props.tileBounds ? { bounds: props.tileBounds } : {}), - getPosition: (_d, { index, target }) => [ - d[0][index], - d[1][index], - props.use3d ? d[2]?.[index] || 0 : 0, - ], + extensions: [pointsFeatureColorExtension], + // Constant default: the binary getFeatureCode attribute overrides it when + // colouring; when it is withdrawn (colour off), deck reverts to this -1, so + // the shader's `featureCode >= 0.0` guard falls through to the flat colour. + getFeatureCode: -1, getRadius: props.pointSize, radiusUnits, radiusMinPixels, @@ -80,7 +111,6 @@ export function renderColumnarScatterLayer( autoHighlight: true, highlightColor: [255, 255, 0, 200], updateTriggers: { - getPosition: [pointCount, d[0], d[1], d[2]], getRadius: [props.pointSize], }, }); diff --git a/packages/layers/src/preloadedScatterStrategy.ts b/packages/layers/src/preloadedScatterStrategy.ts index 475a8f81..2b473545 100644 --- a/packages/layers/src/preloadedScatterStrategy.ts +++ b/packages/layers/src/preloadedScatterStrategy.ts @@ -17,20 +17,48 @@ function resolveScatterBatch(layer: PointsLayer): ColumnarNdarrayPointsBatch | u filteredBatchSignature?: string; }; const signature = filterBatchSignature(featureCodes, preloadedFeatureCodes, renderCap); - const awaitingRowCodes = featureFilterAwaitingRowCodes(featureCodes, preloadedFeatureCodes); - if (awaitingRowCodes) { + const cappedPreloaded = (): ColumnarNdarrayPointsBatch | undefined => { if (!state.preloadedBatch) { return undefined; } - return applyRenderCapToColumnar(state.preloadedBatch, renderCap); + // Attach the row-aligned codes so this transient (pre-first-filter) fallback + // colours by feature too; applyRenderCapToColumnar truncates them in lockstep. + const withCodes = + preloadedFeatureCodes && preloadedFeatureCodes.length > 0 + ? { ...state.preloadedBatch, featureCodes: preloadedFeatureCodes } + : state.preloadedBatch; + return applyRenderCapToColumnar(withCodes, renderCap); + }; + + // Row codes not loaded yet: we cannot filter, so draw the full batch. This is + // only reachable on first load before the codes arrive (documented behavior). + if (featureFilterAwaitingRowCodes(featureCodes, preloadedFeatureCodes)) { + return cappedPreloaded(); } + + // Up-to-date filtered batch for the current selection. if (state.filteredBatch && state.filteredBatchSignature === signature) { return state.filteredBatch; } - if (!state.preloadedBatch) { - return undefined; + + // The selection changed and the new filtered batch is still computing off-thread. + // Keep showing the PREVIOUS filtered result rather than flashing the full + // unfiltered batch — but only when that previous result was itself a real + // selection, not the unfiltered "all features" batch (whose signature matches + // `featureCodes === undefined`). Reusing the unfiltered batch here is exactly + // the flash-of-all-points the filter is meant to avoid. + const unfilteredSignature = filterBatchSignature(undefined, preloadedFeatureCodes, renderCap); + if (state.filteredBatch && state.filteredBatchSignature !== unfilteredSignature) { + return state.filteredBatch; + } + + // No reusable filtered batch. Draw the full batch only when nothing is selected; + // while a selection's first filter is pending, draw nothing (a brief blank beats + // a misleading flash of every feature). + if (featureCodes === undefined) { + return cappedPreloaded(); } - return applyRenderCapToColumnar(state.preloadedBatch, renderCap); + return undefined; } export const preloadedScatterStrategy: PointsRenderStrategy = { @@ -46,6 +74,7 @@ export const preloadedScatterStrategy: PointsRenderStrategy = { viewZoom, color = [255, 100, 100, 200], use3d, + colorByFeature, } = layer.props; if (!visible) { @@ -71,6 +100,7 @@ export const preloadedScatterStrategy: PointsRenderStrategy = { opacity, modelMatrix: layer.props.modelMatrix, use3d, + colorByFeature, }); }, }; diff --git a/packages/layers/src/resolvePointsRenderResource.ts b/packages/layers/src/resolvePointsRenderResource.ts index 7066b33d..e0c46760 100644 --- a/packages/layers/src/resolvePointsRenderResource.ts +++ b/packages/layers/src/resolvePointsRenderResource.ts @@ -7,7 +7,7 @@ import { createPointsRenderResource } from './pointsLoaderAdapter.js'; import type { PointsRenderResource } from './pointsLoader.js'; export interface ResolvePointsRenderResourceCache { - preloaded?: { shape: number[]; data: ArrayLike[] } | null; + preloaded?: { shape: number[]; data: ArrayLike[]; featureCodes?: ArrayLike } | null; tilingMetadata?: PointsTilingMetadata | null; metadataKnown?: boolean; } diff --git a/packages/layers/tests/pointsDataEngine.spec.ts b/packages/layers/tests/pointsDataEngine.spec.ts index b9c15a43..a7e430cc 100644 --- a/packages/layers/tests/pointsDataEngine.spec.ts +++ b/packages/layers/tests/pointsDataEngine.spec.ts @@ -1,7 +1,12 @@ import { describe, expect, it, vi } from 'vitest'; import { PointsDataEngine } from '../src/engine/PointsDataEngine.js'; -import type { PointsElement, PointsLoadResult } from '@spatialdata/core'; +import { + DEFAULT_POINTS_MEMORY_CAP, + type PointsElement, + type PointsFeatureCatalog, + type PointsLoadResult, +} from '@spatialdata/core'; function makeBatch(): PointsLoadResult { return { @@ -17,6 +22,39 @@ function makeElement(key: string, batch = makeBatch()) { return { element: { key, loadPoints } as unknown as PointsElement, loadPoints }; } +const sampleCatalog: PointsFeatureCatalog = { + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'GeneA', count: 10 }, + { code: 1, name: 'GeneB', count: 5 }, + ], +}; + +/** PointsElement stub covering the feature-filter surface. `catalog`/`rowCodes` + * default to sensible values; pass `null`/`undefined` to model an element with + * no `feature_key` or no codes. */ +function makeFeatureElement( + key: string, + opts: { + catalog?: PointsFeatureCatalog | null; + rowCodes?: ArrayLike | undefined; + } = {} +) { + // Use `in` checks, not destructuring defaults: an explicit `rowCodes: undefined` + // (modeling an element with no codes) must NOT fall back to the sample array. + const catalog = 'catalog' in opts ? opts.catalog : sampleCatalog; + const rowCodes = 'rowCodes' in opts ? opts.rowCodes : new Int32Array([0, 1, 0]); + const listFeaturesWithCounts = vi.fn(async () => catalog); + const loadRowFeatureCodes = vi.fn(async () => rowCodes); + const element = { + key, + loadPoints: vi.fn(async () => makeBatch()), + listFeaturesWithCounts, + loadRowFeatureCodes, + } as unknown as PointsElement; + return { element, listFeaturesWithCounts, loadRowFeatureCodes }; +} + describe('PointsDataEngine', () => { it('loads once, reports status, and caches the batch', async () => { const statuses: Array<[string, string]> = []; @@ -39,6 +77,143 @@ describe('PointsDataEngine', () => { ]); }); + it('does not reload a COMPLETE resident batch when the cap changes', async () => { + const engine = new PointsDataEngine(); + // makeBatch() is complete (no preloadTruncated) → it holds the whole dataset. + const loadPoints = vi.fn(async () => makeBatch()); + const element = { key: 'pts:cap', loadPoints } as unknown as PointsElement; + + await engine.ensureLoaded({ key: 'pts:cap', layerId: 'l', element }, 4_000_000); + expect(loadPoints).toHaveBeenLastCalledWith( + expect.objectContaining({ includeFeatureCodes: true, memoryCap: 4_000_000 }) + ); + // A complete batch satisfies ANY cap — no reload raising or lowering. + expect(engine.isLoadedWithCap('pts:cap', 8_000_000)).toBe(true); + expect(engine.isLoadedWithCap('pts:cap', 2_000_000)).toBe(true); + await engine.ensureLoaded({ key: 'pts:cap', layerId: 'l', element }, 8_000_000); + await engine.ensureLoaded({ key: 'pts:cap', layerId: 'l', element }, 2_000_000); + expect(loadPoints).toHaveBeenCalledTimes(1); + }); + + it('sheds a resident batch in memory when the cap is lowered, reloads only when raised', async () => { + const engine = new PointsDataEngine(); + // Truncated batch: filled to its cap; more rows exist (total 12M). + const truncatedAt = (cap: number): PointsLoadResult => ({ + shape: [2, cap], + data: [new Float32Array(1), new Float32Array(1)], + preloadTruncated: true, + totalRowCount: 12_000_000, + }); + let call = 0; + let resolveRaise: (v: PointsLoadResult) => void = () => {}; + const loadPoints = vi.fn((opts: { memoryCap: number }) => { + call += 1; + if (call === 1) return Promise.resolve(truncatedAt(opts.memoryCap)); + return new Promise((resolve) => { + resolveRaise = resolve; + }); + }); + const element = { key: 'pts:trunc', loadPoints } as unknown as PointsElement; + const target = { key: 'pts:trunc', layerId: 'l', element }; + + await engine.ensureLoaded(target, 4_000_000); + expect(engine.getResidentTruncation('pts:trunc')).toMatchObject({ + truncated: true, + loaded: 4_000_000, + total: 12_000_000, + }); + // Lowering 4M → 2M: shed the excess IN MEMORY (no re-fetch) so a 2M cap does + // not keep holding 4M rows. isLoadedWithCap is false until the shed runs. + expect(engine.isLoadedWithCap('pts:trunc', 2_000_000)).toBe(false); + await engine.ensureLoaded(target, 2_000_000); + expect(loadPoints).toHaveBeenCalledTimes(1); // shed is in-memory, no reload + expect(engine.getResidentTruncation('pts:trunc')).toMatchObject({ loaded: 2_000_000 }); + expect(engine.isLoadedWithCap('pts:trunc', 2_000_000)).toBe(true); + const shed = engine.getData('pts:trunc'); + + // Raising past the truncated batch → reload, but the shed batch stays visible + // while the larger batch is in flight (no blank). + expect(engine.isLoadedWithCap('pts:trunc', 8_000_000)).toBe(false); + const raise = engine.ensureLoaded(target, 8_000_000); + expect(loadPoints).toHaveBeenCalledTimes(2); + expect(engine.getData('pts:trunc')).toBe(shed); // old batch still there + + resolveRaise(truncatedAt(8_000_000)); + await raise; + // Atomic swap to the larger batch. + expect(engine.getResidentTruncation('pts:trunc')).toMatchObject({ loaded: 8_000_000 }); + expect(engine.isLoadedWithCap('pts:trunc', 8_000_000)).toBe(true); + }); + + it('aborts the superseded preload when the memory cap changes', async () => { + const engine = new PointsDataEngine(); + const signals: Array = []; + const resolvers: Array<(v: PointsLoadResult) => void> = []; + const loadPoints = vi.fn( + (opts: { signal?: AbortSignal }) => + new Promise((resolve, reject) => { + signals.push(opts.signal); + resolvers.push(resolve); + opts.signal?.addEventListener('abort', () => + reject(new DOMException('The operation was aborted.', 'AbortError')) + ); + }) + ); + const element = { key: 'pts:abort', loadPoints } as unknown as PointsElement; + + const p1 = engine.ensureLoaded({ key: 'pts:abort', layerId: 'l', element }, 4_000_000); + expect(signals[0]?.aborted).toBe(false); + + // A cap change supersedes the in-flight 4M load → its signal aborts. + const p2 = engine.ensureLoaded({ key: 'pts:abort', layerId: 'l', element }, 8_000_000); + expect(signals[0]?.aborted).toBe(true); + + // The aborted load rejects with AbortError; the engine swallows it (no error). + await p1; + expect(engine.getStatus('pts:abort')).not.toBe('error'); + + // The current 8M load settles normally. + resolvers[1](makeBatch()); + await p2; + expect(engine.isLoadedWithCap('pts:abort', 8_000_000)).toBe(true); + }); + + it('defaults to DEFAULT_POINTS_MEMORY_CAP when no cap is given', async () => { + const engine = new PointsDataEngine(); + const { element } = makeElement('pts:defcap'); + await engine.ensureLoaded({ key: 'pts:defcap', layerId: 'l', element }); + expect(engine.isLoadedWithCap('pts:defcap', DEFAULT_POINTS_MEMORY_CAP)).toBe(true); + }); + + it('preserves the full-dataset catalog across a cap change (no re-scan)', async () => { + // A cap change reloads the resident geometry but must NOT drop the + // (cap-independent, expensive) full-dataset catalog. + const fullCatalog: PointsFeatureCatalog = { + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'GeneA', count: 10 }, + { code: 1, name: 'GeneB', count: 5 }, + { code: 2, name: 'GeneC', count: 1 }, + ], + }; + const listFeaturesWithCounts = vi.fn(async () => fullCatalog); + const element = { + key: 'pts:capcat', + loadPoints: vi.fn(async () => makeBatch()), + listFeaturesWithCounts, + } as unknown as PointsElement; + const engine = new PointsDataEngine(); + + await engine.ensureLoaded({ key: 'pts:capcat', layerId: 'l', element }, 4_000_000); + await engine.ensureFeatureCatalog({ key: 'pts:capcat', layerId: 'l', element }); + expect(engine.getFeatureCatalog('pts:capcat')).toEqual(fullCatalog); + + // Change the cap → catalog stays (never re-scans, cap-independent). + await engine.ensureLoaded({ key: 'pts:capcat', layerId: 'l', element }, 8_000_000); + expect(engine.getFeatureCatalog('pts:capcat')).toEqual(fullCatalog); + expect(listFeaturesWithCounts).toHaveBeenCalledTimes(1); + }); + it('is idempotent: concurrent loads trigger a single loadPoints', async () => { const engine = new PointsDataEngine(); const { element, loadPoints } = makeElement('pts:b'); @@ -114,3 +289,557 @@ describe('PointsDataEngine', () => { expect(listener).toHaveBeenCalledTimes(1); // no longer subscribed }); }); + +describe('PointsDataEngine — feature catalog', () => { + it('builds the catalog once, reactively, and reports loading', async () => { + const engine = new PointsDataEngine(); + const { element, listFeaturesWithCounts } = makeFeatureElement('pts:cat'); + const listener = vi.fn(); + engine.subscribe(listener); + + expect(engine.getFeatureCatalog('pts:cat')).toBeUndefined(); // not requested + expect(engine.isFeatureCatalogLoading('pts:cat')).toBe(false); + + const p = engine.ensureFeatureCatalog({ key: 'pts:cat', layerId: 'l', element }); + expect(engine.isFeatureCatalogLoading('pts:cat')).toBe(true); // in flight + await p; + + expect(engine.isFeatureCatalogLoading('pts:cat')).toBe(false); + expect(engine.getFeatureCatalog('pts:cat')).toEqual(sampleCatalog); + expect(listFeaturesWithCounts).toHaveBeenCalledTimes(1); + // one notify for the loading transition, one for settle + expect(listener).toHaveBeenCalledTimes(2); + }); + + it('is idempotent: concurrent + post-settle calls scan once', async () => { + const engine = new PointsDataEngine(); + const { element, listFeaturesWithCounts } = makeFeatureElement('pts:cat2'); + + await Promise.all([ + engine.ensureFeatureCatalog({ key: 'pts:cat2', layerId: 'l', element }), + engine.ensureFeatureCatalog({ key: 'pts:cat2', layerId: 'l', element }), + ]); + await engine.ensureFeatureCatalog({ key: 'pts:cat2', layerId: 'l', element }); + + expect(listFeaturesWithCounts).toHaveBeenCalledTimes(1); + }); + + it('settles to null for an element with no feature_key', async () => { + const engine = new PointsDataEngine(); + const { element } = makeFeatureElement('pts:nofk', { catalog: null }); + + await engine.ensureFeatureCatalog({ key: 'pts:nofk', layerId: 'l', element }); + + // null (settled), distinct from undefined (not requested) + expect(engine.getFeatureCatalog('pts:nofk')).toBeNull(); + expect(engine.getFeatureCatalog('pts:other')).toBeUndefined(); + }); + + it('records null and stays settled when the scan rejects', async () => { + const engine = new PointsDataEngine(); + const element = { + key: 'pts:boom', + listFeaturesWithCounts: vi.fn(async () => { + throw new Error('scan failed'); + }), + } as unknown as PointsElement; + const errSpy = vi.spyOn(console, 'error').mockImplementation(() => {}); + + await engine.ensureFeatureCatalog({ key: 'pts:boom', layerId: 'l', element }); + + expect(engine.getFeatureCatalog('pts:boom')).toBeNull(); + expect(engine.isFeatureCatalogLoading('pts:boom')).toBe(false); + errSpy.mockRestore(); + }); +}); + +describe('PointsDataEngine — row feature codes', () => { + it('loads codes aligned to the batch and passes the engine catalog', async () => { + const engine = new PointsDataEngine(); + const { element, loadRowFeatureCodes } = makeFeatureElement('pts:rc'); + + // Build the catalog first so the engine reuses it (no redundant core scan). + await engine.ensureFeatureCatalog({ key: 'pts:rc', layerId: 'l', element }); + await engine.ensureRowFeatureCodes({ key: 'pts:rc', layerId: 'l', element }); + + expect(engine.hasRowFeatureCodes('pts:rc')).toBe(true); + expect(Array.from(engine.getRowFeatureCodes('pts:rc')!)).toEqual([0, 1, 0]); + expect(loadRowFeatureCodes).toHaveBeenCalledWith({ featureCatalog: sampleCatalog }); + }); + + it('passes undefined catalog when none is built yet (core scans internally)', async () => { + const engine = new PointsDataEngine(); + const { element, loadRowFeatureCodes } = makeFeatureElement('pts:rc2'); + + await engine.ensureRowFeatureCodes({ key: 'pts:rc2', layerId: 'l', element }); + + expect(loadRowFeatureCodes).toHaveBeenCalledWith({ featureCatalog: undefined }); + }); + + it('is idempotent', async () => { + const engine = new PointsDataEngine(); + const { element, loadRowFeatureCodes } = makeFeatureElement('pts:rc3'); + + await Promise.all([ + engine.ensureRowFeatureCodes({ key: 'pts:rc3', layerId: 'l', element }), + engine.ensureRowFeatureCodes({ key: 'pts:rc3', layerId: 'l', element }), + ]); + await engine.ensureRowFeatureCodes({ key: 'pts:rc3', layerId: 'l', element }); + + expect(loadRowFeatureCodes).toHaveBeenCalledTimes(1); + }); + + it('settles even when the element exposes no codes', async () => { + const engine = new PointsDataEngine(); + const { element } = makeFeatureElement('pts:rc4', { rowCodes: undefined }); + + await engine.ensureRowFeatureCodes({ key: 'pts:rc4', layerId: 'l', element }); + + expect(engine.hasRowFeatureCodes('pts:rc4')).toBe(true); + expect(engine.getRowFeatureCodes('pts:rc4')).toBeUndefined(); + }); + + it('evict clears catalog and row codes with the batch', async () => { + const engine = new PointsDataEngine(); + const { element } = makeFeatureElement('pts:rc5'); + await engine.ensureLoaded({ key: 'pts:rc5', layerId: 'l', element }); + await engine.ensureFeatureCatalog({ key: 'pts:rc5', layerId: 'l', element }); + await engine.ensureRowFeatureCodes({ key: 'pts:rc5', layerId: 'l', element }); + + engine.evict('pts:rc5'); + + expect(engine.hasData('pts:rc5')).toBe(false); + expect(engine.getFeatureCatalog('pts:rc5')).toBeUndefined(); + expect(engine.hasRowFeatureCodes('pts:rc5')).toBe(false); + }); +}); + +describe('PointsDataEngine — codes with the geometry preload', () => { + it('shows an instant preview catalog, then supersedes it with the full-dataset scan', async () => { + // The geometry preload's catalog reflects only the resident batch (a + // feature-ordered file's first part holds a slice of the features), so it is + // an instant *preview*. The full-dataset scan supersedes it with the complete + // list + counts. Row codes are complete for the resident batch, so their lazy + // path stays a no-op. + const previewCatalog = sampleCatalog; + const fullCatalog = { + ...sampleCatalog, + entries: [ + ...sampleCatalog.entries, + { code: 99, name: 'LATE_PART_GENE', count: 7 }, + ], + }; + const loadPoints = vi.fn(async () => ({ + ...makeBatch(), + featureCatalog: previewCatalog, + featureCodes: new Int32Array([0, 1, 0]), + })); + const listFeaturesWithCounts = vi.fn(async () => fullCatalog); + const loadRowFeatureCodes = vi.fn(async () => new Int32Array([0, 1, 0])); + const element = { + key: 'pts:res', + loadPoints, + listFeaturesWithCounts, + loadRowFeatureCodes, + } as unknown as PointsElement; + const engine = new PointsDataEngine(); + + await engine.ensureLoaded({ key: 'pts:res', layerId: 'l', element }); + + // The preload requested the feature column: the preview catalog and the row + // codes are resident with no separate file loads, and the preview shows + // without a loading spinner. + expect(loadPoints).toHaveBeenCalledWith( + expect.objectContaining({ + includeFeatureCodes: true, + memoryCap: DEFAULT_POINTS_MEMORY_CAP, + }) + ); + expect(engine.getFeatureCatalog('pts:res')).toEqual(previewCatalog); + expect(engine.hasRowFeatureCodes('pts:res')).toBe(true); + expect(Array.from(engine.getRowFeatureCodes('pts:res')!)).toEqual([0, 1, 0]); + expect(engine.isFeatureCatalogLoading('pts:res')).toBe(false); + + // The full-dataset catalog scan runs (even with a preview present) and + // supersedes the preview; the row-code lazy path stays a no-op. + await engine.ensureFeatureCatalog({ key: 'pts:res', layerId: 'l', element }); + await engine.ensureRowFeatureCodes({ key: 'pts:res', layerId: 'l', element }); + expect(listFeaturesWithCounts).toHaveBeenCalledTimes(1); + expect(engine.getFeatureCatalog('pts:res')).toEqual(fullCatalog); + expect(loadRowFeatureCodes).not.toHaveBeenCalled(); + + // A second call is a no-op once the full scan has settled. + await engine.ensureFeatureCatalog({ key: 'pts:res', layerId: 'l', element }); + expect(listFeaturesWithCounts).toHaveBeenCalledTimes(1); + }); + + it('remaps resident row codes into the full catalog space on upgrade (dict-only)', async () => { + // Dictionary-only dataset: the resident preview saw GeneB first (code 0) and + // GeneA second (code 1); the full-dataset scan assigns the reverse. Without + // reconciliation, the render's per-row codes would be in the preview space + // while the panel selects in the full space — filtering/colouring the wrong + // genes. `hasFeatureCodeColumn: false` marks the codes as app-assigned. + const previewCatalog: PointsFeatureCatalog = { + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'GeneB' }, + { code: 1, name: 'GeneA' }, + ], + }; + const fullCatalog: PointsFeatureCatalog = { + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'GeneA' }, + { code: 1, name: 'GeneB' }, + ], + }; + const element = { + key: 'pts:remap', + loadPoints: vi.fn(async () => ({ + ...makeBatch(), + featureCatalog: previewCatalog, + featureCodes: new Int32Array([0, 1, 0]), // GeneB, GeneA, GeneB (preview space) + hasFeatureCodeColumn: false, + })), + listFeaturesWithCounts: vi.fn(async () => fullCatalog), + loadRowFeatureCodes: vi.fn(), + } as unknown as PointsElement; + const engine = new PointsDataEngine(); + + await engine.ensureLoaded({ key: 'pts:remap', layerId: 'l', element }); + // Before the upgrade, codes are in the preview space. + expect(Array.from(engine.getRowFeatureCodes('pts:remap')!)).toEqual([0, 1, 0]); + + await engine.ensureFeatureCatalog({ key: 'pts:remap', layerId: 'l', element }); + // After the upgrade, the same genes are re-expressed in the full space: + // GeneB→1, GeneA→0. + expect(Array.from(engine.getRowFeatureCodes('pts:remap')!)).toEqual([1, 0, 1]); + // The resident-codes memo reflects the remapped values. + expect([...engine.getResidentFeatureCodes('pts:remap')!].sort()).toEqual([0, 1]); + }); + + it('does not remap when codes are authoritative (a real feature-code column)', async () => { + // With a file-backed code column the codes are identical across catalog + // builds, so reconciliation is skipped — the row-codes array keeps its + // identity (no needless re-filter) and its values. + const previewCatalog: PointsFeatureCatalog = { + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'GeneA' }, + { code: 1, name: 'GeneB' }, + ], + }; + const fullCatalog: PointsFeatureCatalog = { + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'GeneA' }, + { code: 1, name: 'GeneB' }, + { code: 2, name: 'GeneC' }, + ], + }; + const residentCodes = new Int32Array([0, 1, 0]); + const element = { + key: 'pts:auth', + loadPoints: vi.fn(async () => ({ + ...makeBatch(), + featureCatalog: previewCatalog, + featureCodes: residentCodes, + hasFeatureCodeColumn: true, + })), + listFeaturesWithCounts: vi.fn(async () => fullCatalog), + loadRowFeatureCodes: vi.fn(), + } as unknown as PointsElement; + const engine = new PointsDataEngine(); + + await engine.ensureLoaded({ key: 'pts:auth', layerId: 'l', element }); + await engine.ensureFeatureCatalog({ key: 'pts:auth', layerId: 'l', element }); + + expect(engine.hasFeatureCodeColumn('pts:auth')).toBe(true); + // Same array identity: authoritative codes are never rewritten. + expect(engine.getRowFeatureCodes('pts:auth')).toBe(residentCodes); + }); + + it('reports the catalog as loading while the geometry preload is in flight', async () => { + let resolveLoad: (v: unknown) => void = () => {}; + const loadPoints = vi.fn( + () => + new Promise((resolve) => { + resolveLoad = resolve; + }) + ); + const element = { key: 'pts:inflight', loadPoints } as unknown as PointsElement; + const engine = new PointsDataEngine(); + + const p = engine.ensureLoaded({ key: 'pts:inflight', layerId: 'l', element }); + // Geometry (which carries the catalog) is loading → catalog counts as loading. + expect(engine.isFeatureCatalogLoading('pts:inflight')).toBe(true); + expect(engine.getFeatureCatalog('pts:inflight')).toBeUndefined(); + + resolveLoad({ ...makeBatch(), featureCatalog: sampleCatalog, featureCodes: new Int32Array([0]) }); + await p; + expect(engine.isFeatureCatalogLoading('pts:inflight')).toBe(false); + expect(engine.getFeatureCatalog('pts:inflight')).toEqual(sampleCatalog); + }); +}); + +describe('PointsDataEngine — hasFeatureCodeColumn', () => { + it('defaults to false and reflects the resident load flag', async () => { + const engine = new PointsDataEngine(); + expect(engine.hasFeatureCodeColumn('pts:unknown')).toBe(false); // never loaded + + const withColumn = { + key: 'pts:hascol', + loadPoints: vi.fn(async () => ({ ...makeBatch(), hasFeatureCodeColumn: true })), + } as unknown as PointsElement; + await engine.ensureLoaded({ key: 'pts:hascol', layerId: 'l', element: withColumn }); + expect(engine.hasFeatureCodeColumn('pts:hascol')).toBe(true); + + const dictOnly = { + key: 'pts:dict', + loadPoints: vi.fn(async () => ({ ...makeBatch(), hasFeatureCodeColumn: false })), + } as unknown as PointsElement; + await engine.ensureLoaded({ key: 'pts:dict', layerId: 'l', element: dictOnly }); + expect(engine.hasFeatureCodeColumn('pts:dict')).toBe(false); + }); +}); + +describe('PointsDataEngine — matching resource (empty-lock guard)', () => { + function matchingElement(key: string, result: PointsLoadResult) { + return { + key, + loadPoints: vi.fn(async () => makeBatch()), + loadPointsMatchingFeatureCodes: vi.fn(async () => result), + } as unknown as PointsElement; + } + + it('returns null for a scan that matched no rows, so the view is never locked empty', async () => { + const engine = new PointsDataEngine(); + // A degenerate scan settles with 0 matched rows (e.g. a selection whose codes + // matched nothing). It must NOT supersede the resident preview. + const element = matchingElement('pts:empty', { + shape: [2, 0], + data: [new Float32Array(0), new Float32Array(0)], + }); + await engine.ensureMatchingFeaturesLoaded({ key: 'pts:empty', layerId: 'l', element }, [7]); + + expect(engine.getMatchingResource(element, 'pts:empty')).toBeNull(); + }); + + it('returns a resource when the scan matched rows', async () => { + const engine = new PointsDataEngine(); + const element = matchingElement('pts:hit', { + shape: [2, 2], + data: [new Float32Array([0, 1]), new Float32Array([0, 1])], + }); + await engine.ensureMatchingFeaturesLoaded({ key: 'pts:hit', layerId: 'l', element }, [1]); + + expect(engine.getMatchingResource(element, 'pts:hit')).toBeTruthy(); + }); +}); + +describe('PointsDataEngine — matched-selection subset reuse', () => { + function scanElement(key: string) { + const loadPointsMatchingFeatureCodes = vi.fn(async (opts: { featureCodes: readonly number[] }) => ({ + shape: [2, opts.featureCodes.length], + data: [new Float32Array(opts.featureCodes.length), new Float32Array(opts.featureCodes.length)], + // Per-row codes the render uses to filter the batch in memory. + featureCodes: Int32Array.from(opts.featureCodes), + })); + const element = { + key, + loadPoints: vi.fn(async () => makeBatch()), + loadPointsMatchingFeatureCodes, + } as unknown as PointsElement; + return { element, loadPointsMatchingFeatureCodes }; + } + + it('reuses the batch (no re-scan) when a feature is removed — the removal fast path', async () => { + const engine = new PointsDataEngine(); + const { element, loadPointsMatchingFeatureCodes } = scanElement('pts:sub'); + const target = { key: 'pts:sub', layerId: 'l', element }; + + await engine.ensureMatchingFeaturesLoaded(target, [1, 2, 3]); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(1); + expect([...engine.getLoadedMatchingFeatureCodes('pts:sub')!].sort()).toEqual([1, 2, 3]); + + // Remove a feature → {1,2} ⊆ {1,2,3}: reuse the loaded batch, NO new scan. + await engine.ensureMatchingFeaturesLoaded(target, [1, 2]); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(1); + // The covered set stays {1,2,3}, so the removed feature is still in memory + // (re-adding it is a free filter, and the panel keeps it un-greyed). + expect([...engine.getLoadedMatchingFeatureCodes('pts:sub')!].sort()).toEqual([1, 2, 3]); + // Per-row codes are exposed for the layer to filter the batch in memory. + expect(engine.getMatchingRowFeatureCodes('pts:sub')).toBeInstanceOf(Int32Array); + // The load-state reports the subset as settled+covered (served from memory), + // so the panel indicator doesn't vanish on a removal. + const state = engine.getMatchingLoadState('pts:sub', [1, 2]); + expect(state).toMatchObject({ loading: false, settled: true, covered: true }); + }); + + it('re-scans when the selection adds a code no loaded batch covers', async () => { + const engine = new PointsDataEngine(); + const { element, loadPointsMatchingFeatureCodes } = scanElement('pts:add'); + const target = { key: 'pts:add', layerId: 'l', element }; + + await engine.ensureMatchingFeaturesLoaded(target, [1, 2]); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(1); + + // Removing back to {1} reuses (no scan)… + await engine.ensureMatchingFeaturesLoaded(target, [1]); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(1); + + // …but adding {3} (not covered) scans. + await engine.ensureMatchingFeaturesLoaded(target, [1, 3]); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(2); + expect([...engine.getLoadedMatchingFeatureCodes('pts:add')!].sort()).toEqual([1, 3]); + }); + + it('does not rescan when the cap is lowered and the loaded selection already fits', async () => { + const engine = new PointsDataEngine(); + // A COMPLETE batch: the scan found all matching rows before the cap. + const loadPointsMatchingFeatureCodes = vi.fn(async (opts: { featureCodes: readonly number[] }) => ({ + shape: [2, 500], + data: [new Float32Array(500), new Float32Array(500)], + featureCodes: Int32Array.from({ length: 500 }, () => opts.featureCodes[0]), + preloadTruncated: false, + })); + const element = { + key: 'pts:caplow', + loadPoints: vi.fn(async () => makeBatch()), + loadPointsMatchingFeatureCodes, + } as unknown as PointsElement; + const target = { key: 'pts:caplow', layerId: 'l', element }; + + await engine.ensureMatchingFeaturesLoaded(target, [1, 2], 4_000_000); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(1); + // Lower the cap 4M → 2M: the complete batch still covers the selection and + // fits — reuse, NO rescan (the user's case: the selection totals < 2M). + await engine.ensureMatchingFeaturesLoaded(target, [1, 2], 2_000_000); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(1); + }); + + it('rescans only when the cap is raised past a truncated batch', async () => { + const engine = new PointsDataEngine(); + // A TRUNCATED batch: the scan filled up to its cap (more rows exist). + const loadPointsMatchingFeatureCodes = vi.fn( + async (opts: { featureCodes: readonly number[]; memoryCap: number }) => ({ + shape: [2, opts.memoryCap], + data: [new Float32Array(1), new Float32Array(1)], + featureCodes: Int32Array.from([opts.featureCodes[0]]), + preloadTruncated: true, + }) + ); + const element = { + key: 'pts:capraise', + loadPoints: vi.fn(async () => makeBatch()), + loadPointsMatchingFeatureCodes, + } as unknown as PointsElement; + const target = { key: 'pts:capraise', layerId: 'l', element }; + + await engine.ensureMatchingFeaturesLoaded(target, [1, 2], 2_000_000); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(1); + // Lowering 2M → 1M: batch holds 2M ≥ 1M rows → reuse even though truncated. + await engine.ensureMatchingFeaturesLoaded(target, [1, 2], 1_000_000); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(1); + // Raising 1M → 4M: the batch was truncated at 2M < 4M → rescan for more. + await engine.ensureMatchingFeaturesLoaded(target, [1, 2], 4_000_000); + expect(loadPointsMatchingFeatureCodes).toHaveBeenCalledTimes(2); + }); +}); + +describe('PointsDataEngine — dict-only feature scan', () => { + const dictCatalog: PointsFeatureCatalog = { + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'GeneA', count: 3 }, + { code: 1, name: 'GeneB', count: 2 }, + { code: 2, name: 'GeneC', count: 4 }, + ], + }; + + function dictElement(key: string, hasCodeColumn: boolean) { + const loadPoints = vi.fn(async () => ({ + ...makeBatch(), + featureCatalog: dictCatalog, + featureCodes: new Int32Array([0, 1]), + hasFeatureCodeColumn: hasCodeColumn, + })); + const loadPointsMatchingFeatureCodes = vi.fn( + async (opts: { featureCodes: readonly number[] }) => ({ + shape: [2, opts.featureCodes.length], + data: [new Float32Array(opts.featureCodes.length), new Float32Array(opts.featureCodes.length)], + featureCodes: Int32Array.from(opts.featureCodes), + }) + ); + const element = { + key, + loadPoints, + loadPointsMatchingFeatureCodes, + } as unknown as PointsElement; + return { element, loadPointsMatchingFeatureCodes }; + } + + it('supports a scan once a catalog is loaded, even with no code column', async () => { + const engine = new PointsDataEngine(); + const { element } = dictElement('pts:dict', false); + const target = { key: 'pts:dict', layerId: 'l', element }; + expect(engine.supportsFeatureScan('pts:dict')).toBe(false); // nothing loaded yet + await engine.ensureLoaded(target, DEFAULT_POINTS_MEMORY_CAP); + expect(engine.hasFeatureCodeColumn('pts:dict')).toBe(false); + expect(engine.supportsFeatureScan('pts:dict')).toBe(true); // catalog present + }); + + it('passes the catalog name→code map to the scan for a dict-only element', async () => { + const engine = new PointsDataEngine(); + const { element, loadPointsMatchingFeatureCodes } = dictElement('pts:dictscan', false); + const target = { key: 'pts:dictscan', layerId: 'l', element }; + await engine.ensureLoaded(target, DEFAULT_POINTS_MEMORY_CAP); + await engine.ensureMatchingFeaturesLoaded(target, [2]); + const arg = loadPointsMatchingFeatureCodes.mock.calls[0][0] as { + featureCodeByName?: ReadonlyMap; + }; + expect(arg.featureCodeByName).toBeInstanceOf(Map); + expect(arg.featureCodeByName?.get('GeneC')).toBe(2); + }); + + it('omits the name→code map for an element with a file-backed code column', async () => { + const engine = new PointsDataEngine(); + const { element, loadPointsMatchingFeatureCodes } = dictElement('pts:indexed', true); + const target = { key: 'pts:indexed', layerId: 'l', element }; + await engine.ensureLoaded(target, DEFAULT_POINTS_MEMORY_CAP); + await engine.ensureMatchingFeaturesLoaded(target, [2]); + const arg = loadPointsMatchingFeatureCodes.mock.calls[0][0] as { + featureCodeByName?: ReadonlyMap; + }; + expect(arg.featureCodeByName).toBeUndefined(); + }); +}); + +describe('PointsDataEngine — shed complete batch on lower', () => { + it('slices a complete batch down to the cap in memory (no reload), marking it truncated', async () => { + const engine = new PointsDataEngine(); + // A COMPLETE batch of 5M rows (the whole dataset fits; not truncated). + const complete: PointsLoadResult = { + shape: [2, 5_000_000], + data: [new Float32Array([0, 1]), new Float32Array([0, 1])], + totalRowCount: 5_000_000, + }; + const loadPoints = vi.fn(async () => complete); + const element = { key: 'pts:shed', loadPoints } as unknown as PointsElement; + const target = { key: 'pts:shed', layerId: 'l', element }; + + await engine.ensureLoaded(target, 8_000_000); + expect(engine.getResidentTruncation('pts:shed')).toMatchObject({ + truncated: false, + loaded: 5_000_000, + }); + // Lower 8M → 4M below the 5M loaded → shed to 4M in memory, now truncated. + expect(engine.isLoadedWithCap('pts:shed', 4_000_000)).toBe(false); + await engine.ensureLoaded(target, 4_000_000); + expect(loadPoints).toHaveBeenCalledTimes(1); // no re-fetch + expect(engine.getResidentTruncation('pts:shed')).toMatchObject({ + truncated: true, + loaded: 4_000_000, + }); + }); +}); diff --git a/packages/layers/tests/pointsFeatureColor.spec.ts b/packages/layers/tests/pointsFeatureColor.spec.ts new file mode 100644 index 00000000..42f859d0 --- /dev/null +++ b/packages/layers/tests/pointsFeatureColor.spec.ts @@ -0,0 +1,26 @@ +import { describe, expect, it } from 'vitest'; +import { featureCodeToRgb, featureCodeToCssColor } from '../src/pointsFeatureColor.js'; + +describe('featureCodeToRgb', () => { + it('returns grey for the negative "no colour" sentinel', () => { + expect(featureCodeToRgb(-1)).toEqual([128, 128, 128]); + }); + + it('is deterministic and in range for a code', () => { + const rgb = featureCodeToRgb(231); + expect(rgb).toEqual(featureCodeToRgb(231)); + for (const channel of rgb) { + expect(channel).toBeGreaterThanOrEqual(0); + expect(channel).toBeLessThanOrEqual(255); + } + }); + + it('gives distinct colours to adjacent codes (golden-angle spread)', () => { + expect(featureCodeToRgb(10)).not.toEqual(featureCodeToRgb(11)); + }); + + it('formats a CSS rgb() string', () => { + const [r, g, b] = featureCodeToRgb(42); + expect(featureCodeToCssColor(42)).toBe(`rgb(${r}, ${g}, ${b})`); + }); +}); diff --git a/packages/layers/tests/pointsFeatureColorExtension.spec.ts b/packages/layers/tests/pointsFeatureColorExtension.spec.ts new file mode 100644 index 00000000..24250ea8 --- /dev/null +++ b/packages/layers/tests/pointsFeatureColorExtension.spec.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from 'vitest'; +import { PointsFeatureColorExtension } from '../src/pointsFeatureColorExtension.js'; + +// These assert the deck-specific invariants that were load-bearing and easy to +// get subtly wrong (each cost real debugging). They guard the shader wiring, not +// GPU output — the rendered colour is verified live in the demo. +describe('PointsFeatureColorExtension', () => { + const ext = new PointsFeatureColorExtension(); + // getShaders reads `this` only for super.getShaders(); a bare object with a + // no-op getShaders stands in for the host layer. + const shaders = ext.getShaders.call({ getShaders: () => ({}) } as never, ext); + + it('declares getFeatureCode as an accessor in defaultProps', () => { + // Without this, deck treats the attribute as constant and never reads the + // binary buffer supplied via data.attributes. + expect(PointsFeatureColorExtension.defaultProps.getFeatureCode).toEqual({ + type: 'accessor', + value: -1, + }); + }); + + it('declares `in float featureCode` in a top-level vs:#decl inject', () => { + // deck does not auto-declare the attribute; the declaration must be in + // vs:#decl (NOT a module) or the whole hook silently drops. + expect(shaders.inject['vs:#decl']).toContain('in float featureCode;'); + expect(shaders.inject).not.toHaveProperty('modules'); + }); + + it('recolours vFillColor only for a non-negative code in vs:#main-end', () => { + // The -1 default gates colour off (flat fill) without a uniform. + const mainEnd = shaders.inject['vs:#main-end']; + expect(mainEnd).toContain('featureCode >= 0.0'); + expect(mainEnd).toContain('vFillColor'); + }); +}); diff --git a/packages/layers/tests/pointsRenderAttributes.spec.ts b/packages/layers/tests/pointsRenderAttributes.spec.ts new file mode 100644 index 00000000..b34ac341 --- /dev/null +++ b/packages/layers/tests/pointsRenderAttributes.spec.ts @@ -0,0 +1,50 @@ +import { describe, expect, it } from 'vitest'; +import { buildPointsAttributes } from '../src/pointsRenderAttributes.js'; +import type { ColumnarNdarrayPointsBatch } from '../src/pointsLoader.js'; + +function batch(overrides: Partial): ColumnarNdarrayPointsBatch { + return { + format: 'columnar-ndarray', + data: [new Float32Array([0, 1, 2]), new Float32Array([10, 11, 12])], + shape: [2, 3], + pointCount: 3, + ...overrides, + }; +} + +describe('buildPointsAttributes', () => { + it('interleaves x/y into [x, y, 0] triples in 2D', () => { + const attrs = buildPointsAttributes(batch({}), false); + expect(attrs.length).toBe(3); + expect(Array.from(attrs.positions)).toEqual([0, 10, 0, 1, 11, 0, 2, 12, 0]); + }); + + it('includes z only when use3d and a z column exist', () => { + const b = batch({ + data: [new Float32Array([0, 1]), new Float32Array([10, 11]), new Float32Array([100, 101])], + shape: [3, 2], + pointCount: 2, + }); + expect(Array.from(buildPointsAttributes(b, true).positions)).toEqual([0, 10, 100, 1, 11, 101]); + // use3d=false flattens z to 0 even when a z column is present. + expect(Array.from(buildPointsAttributes(b, false).positions)).toEqual([0, 10, 0, 1, 11, 0]); + }); + + it('exposes feature codes as a float attribute aligned with points', () => { + const attrs = buildPointsAttributes(batch({ featureCodes: new Int32Array([7, 3, 7]) }), false); + expect(attrs.featureCodes).toBeInstanceOf(Float32Array); + expect(Array.from(attrs.featureCodes ?? [])).toEqual([7, 3, 7]); + }); + + it('omits feature codes when the batch has none', () => { + expect(buildPointsAttributes(batch({}), false).featureCodes).toBeUndefined(); + }); + + it('memoizes per batch identity so deck receives a stable buffer', () => { + const b = batch({}); + const first = buildPointsAttributes(b, false); + expect(buildPointsAttributes(b, false).positions).toBe(first.positions); + // A different use3d flag invalidates the cached entry. + expect(buildPointsAttributes(b, true).positions).not.toBe(first.positions); + }); +}); diff --git a/packages/layers/tsconfig.json b/packages/layers/tsconfig.json index 9e47b8e9..fbf41335 100644 --- a/packages/layers/tsconfig.json +++ b/packages/layers/tsconfig.json @@ -10,7 +10,12 @@ "skipLibCheck": true, "esModuleInterop": true, "resolveJsonModule": true, - "lib": ["ES2020", "DOM"] + "lib": ["ES2020", "DOM"], + "baseUrl": ".", + "paths": { + "@spatialdata/core": ["../core/src/index.ts"], + "@spatialdata/core/*": ["../core/src/*"] + } }, "include": ["src", "vite.config.ts", "tests"], "exclude": ["dist", "node_modules"] diff --git a/packages/vis/demo/src/main.tsx b/packages/vis/demo/src/main.tsx index 100820ec..b1c7caff 100644 --- a/packages/vis/demo/src/main.tsx +++ b/packages/vis/demo/src/main.tsx @@ -1,8 +1,22 @@ import React from 'react'; import ReactDOM from 'react-dom/client'; +import { enablePointsWorker, setPointsWorkerRequestTimeout } from '@spatialdata/core'; +// Vite bundles the core points worker and hands us a runtime URL. Enabling it +// moves the CPU-heavy work off the main thread so the UI stays responsive: +// - the codes-with-geometry preload decode (decodeGeometryWithFeatures) — the +// main thread only does async range-read fetches, the worker decodes; +// - the per-interaction batch filter (filterColumnarByFeatureCodes, transfers +// the resident batch — no file re-fetch). +// A silent/misconfigured worker still falls back to the main thread via the +// pointsWorkerClient request timeout. Large transcripts decodes can legitimately +// run tens of seconds in the worker, so widen the timeout accordingly. +import pointsWorkerUrl from '../../../core/src/workers/points-worker.ts?worker&url'; import App from './App'; import './index.css'; +enablePointsWorker({ workerUrl: pointsWorkerUrl }); +setPointsWorkerRequestTimeout(120_000); + const root = document.getElementById('root'); if (!root) { throw new Error('Root element not found'); diff --git a/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx b/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx new file mode 100644 index 00000000..6afb8034 --- /dev/null +++ b/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx @@ -0,0 +1,337 @@ +import { featureCodeToCssColor } from '@spatialdata/layers'; +import type { CSSProperties } from 'react'; +import { useEffect, useMemo, useState } from 'react'; +import { usePointsFeatureState } from './PointsFeatureState'; +import { useSpatialCanvasActions } from './context'; +import { describeFeatureRowState, featureRowOpacity } from './featureRowState'; +import type { PointsLayerConfig } from './types'; + +// we need a pass on how we manage styles +const swatchStyle: CSSProperties = { + width: 10, + height: 10, + borderRadius: 2, + flexShrink: 0, + border: '1px solid rgba(255, 255, 255, 0.25)', +}; + +const panelStyle: CSSProperties = { + display: 'flex', + flexDirection: 'column', + gap: 6, + color: '#ccc', + fontSize: '12px', +}; + +const listStyle: CSSProperties = { + display: 'flex', + flexDirection: 'column', + gap: 4, + maxHeight: 180, + overflowY: 'auto', + padding: '4px 0', +}; + +const checkboxLabelStyle: CSSProperties = { + display: 'flex', + alignItems: 'center', + gap: 6, +}; + +const helperStyle: CSSProperties = { + color: '#888', + fontSize: '11px', +}; + +const loadingStatStyle: CSSProperties = { + color: '#6cb6ff', + fontSize: '11px', +}; + +const countStyle: CSSProperties = { + color: '#888', + fontSize: '11px', + marginLeft: 'auto', + flexShrink: 0, +}; + +const searchStyle: CSSProperties = { + color: '#ccc', + fontSize: '12px', + padding: '4px 6px', + borderRadius: 4, + border: '1px solid #444', + background: '#1a1a1a', +}; + +const buttonStyle: CSSProperties = { + alignSelf: 'flex-start', + color: '#ddd', + fontSize: '12px', + padding: '4px 8px', + borderRadius: 4, + border: '1px solid #555', + background: '#2a2a2a', + cursor: 'pointer', +}; + +const FEATURE_LIST_SEARCH_THRESHOLD = 100; + +function formatFeatureCount(count: number | undefined): string { + if (count === undefined) { + return '—'; + } + return count.toLocaleString(); +} + +export interface PointsFeatureFilterPanelProps { + config: PointsLayerConfig; +} + +export function PointsFeatureFilterPanel({ config }: PointsFeatureFilterPanelProps) { + // Opt out of the React Compiler. The usePoints* hooks re-render this component + // on every engine `notify` (via useSyncExternalStore), but they read mutable + // engine state the compiler can't see as a dependency, so it would memoize the + // returned JSX and keep the pre-catalog "not loaded" branch on screen even + // after the component re-runs with the catalog present. Scoped to this leaf, + // this is far narrower than the old canvas-wide escape hatch. + 'use no memo'; + const layerId = config.id; + const { updateLayer } = useSpatialCanvasActions(); + // Reactive points state, read straight from the engine via the surrounding + // . + const { + catalog, + catalogLoading, + catalogRefining, + residentCodes, + loadedMatchingCodes, + supportsOnDemandLoad, + matchingLoadState, + requestCatalog, + } = usePointsFeatureState(config.featureCodes); + + const [searchQuery, setSearchQuery] = useState(''); + // Request the full-dataset catalog whenever this panel is shown for a layer. + // The engine dedupes (no-op once the full scan has settled), so this simply + // upgrades the instant resident-subset preview to the complete list + counts. + useEffect(() => { + requestCatalog(); + }, [requestCatalog]); + const entries = useMemo(() => catalog?.entries ?? [], [catalog?.entries]); + const hasCounts = entries.some((entry) => entry.count !== undefined); + const allSelected = config.featureCodes === undefined; + const noneSelected = config.featureCodes !== undefined && config.featureCodes.length === 0; + const selectedCodes = allSelected + ? new Set(entries.map((entry) => entry.code)) + : new Set(config.featureCodes ?? []); + + const sortedEntries = useMemo(() => { + const list = [...entries]; + if (hasCounts) { + list.sort((left, right) => { + const countDiff = (right.count ?? -1) - (left.count ?? -1); + if (countDiff !== 0) { + return countDiff; + } + return left.name.localeCompare(right.name); + }); + } else { + list.sort((left, right) => left.name.localeCompare(right.name)); + } + return list; + }, [entries, hasCounts]); + + const visibleEntries = useMemo(() => { + const query = searchQuery.trim().toLowerCase(); + if (!query) { + return sortedEntries; + } + return sortedEntries.filter((entry) => entry.name.toLowerCase().includes(query)); + }, [sortedEntries, searchQuery]); + + const setFeatureCodes = (nextCodes: number[] | undefined) => { + updateLayer(layerId, { featureCodes: nextCodes }); + }; + + const toggleFeature = (code: number, checked: boolean) => { + const current = new Set( + allSelected ? entries.map((entry) => entry.code) : (config.featureCodes ?? []) + ); + if (checked) { + current.add(code); + } else { + current.delete(code); + } + if (current.size === 0) { + setFeatureCodes([]); + return; + } + if (current.size === entries.length) { + setFeatureCodes(undefined); + return; + } + setFeatureCodes([...current].sort((left, right) => left - right)); + }; + + if (catalogLoading) { + return ( +
+
Loading features…
+
+ ); + } + + if (catalog === undefined) { + return ( +
+
Feature list not loaded.
+ +
+ ); + } + + if (!catalog || entries.length === 0) { + return ( +
+
+ {catalog === null + ? 'No feature catalog available for this points layer (missing feature_key or unsupported encoding for this dataset size).' + : 'No features found in the feature catalog.'} +
+
+ ); + } + + const selectedCount = noneSelected ? 0 : allSelected ? entries.length : selectedCodes.size; + const showSearch = entries.length > FEATURE_LIST_SEARCH_THRESHOLD; + // A feature's points are "loaded" (renderable now, not greyed) if it is in the + // instant resident preview OR its points are currently on screen via the + // last-completed feature-index scan (`loadedMatchingCodes`). Keying off what's + // rendered — not the current scan's settled state — keeps already-loaded + // features un-greyed while a newly added feature's scan is still in flight. + const residentKnown = residentCodes !== undefined; + const scanning = matchingLoadState?.loading ?? false; + const rowInfo = (code: number) => { + const resident = residentKnown && (residentCodes?.has(code) ?? false); + const rendered = loadedMatchingCodes?.has(code) ?? false; + const selected = !noneSelected && (allSelected || selectedCodes.has(code)); + const state = describeFeatureRowState({ + resident, + rendered, + selected, + scanning, + supportsOnDemandLoad, + residentKnown, + }); + return { resident, rendered, selected, state }; + }; + const notLoadedCount = residentKnown + ? entries.reduce((total, entry) => total + (rowInfo(entry.code).state.greyed ? 1 : 0), 0) + : 0; + + return ( +
+
+ Features ({catalog.featureKey}) + + {' '} + · {selectedCount}/{entries.length} selected + {hasCounts ? ' · sorted by count' : ''} + +
+ {catalogRefining ?
Loading the full feature list…
: null} + {notLoadedCount > 0 ? ( +
+ {notLoadedCount} of {entries.length} feature{entries.length === 1 ? '' : 's'}{' '} + {supportsOnDemandLoad + ? 'not loaded yet (greyed below) — selecting one loads it on demand.' + : "not in the loaded sample (greyed below) — this dataset has no feature index, so they can't be shown until the row cap is raised or it's rewritten with one."} +
+ ) : null} + {matchingLoadState ? ( +
+ {matchingLoadState.loading + ? `Loading selected features… ${matchingLoadState.matchedRows.toLocaleString()} points so far` + : matchingLoadState.covered + ? `Selection served from ${matchingLoadState.matchedRows.toLocaleString()} points in memory (no re-scan)` + : `${matchingLoadState.matchedRows.toLocaleString()} points loaded for this selection`} +
+ ) : null} + + + {showSearch ? ( + setSearchQuery(event.target.value)} + style={searchStyle} + /> + ) : null} +
+ {visibleEntries.map((entry) => { + const { resident, rendered, selected, state } = rowInfo(entry.code); + const countStr = + entry.count !== undefined ? ` · ${entry.count.toLocaleString()} pts` : ''; + // Multi-line diagnostic: the human state + reason, then the raw signals + // that drove the decision (what made this row grey / not grey). + const title = + `${entry.name} · code ${entry.code}${countStr}\n` + + `${state.label}: ${state.reason}\n` + + `[resident=${resident ? 'y' : 'n'} rendered=${rendered ? 'y' : 'n'} ` + + `selected=${selected ? 'y' : 'n'} scan=${scanning ? 'running' : 'idle'}]`; + return ( + + ); + })} + {showSearch && visibleEntries.length === 0 ? ( +
No features match your search.
+ ) : null} +
+
+ ); +} diff --git a/packages/vis/src/SpatialCanvas/PointsFeatureState.tsx b/packages/vis/src/SpatialCanvas/PointsFeatureState.tsx new file mode 100644 index 00000000..90ab286d --- /dev/null +++ b/packages/vis/src/SpatialCanvas/PointsFeatureState.tsx @@ -0,0 +1,171 @@ +/** + * Reactive points feature state for the properties panel. + * + * The `PointsDataEngine` is a mutable external store owned by `useLayerData` + * (it drives the render path). Rather than prop-drill a bundle of getters that + * read that mutable state, the panel subtree subscribes to the engine directly + * through `usePointsFeatureState`. + * + * One hook, one subscription: `usePointsFeatureState` runs a single + * `useSyncExternalStore` against the engine's monotonic `version` (a primitive + * snapshot — the engine's object-returning readers allocate fresh each call and + * would infinite-loop as a snapshot) and returns every derived read at once. + * + * The hook and its consumers carry `'use no memo'`. The reads take + * otherwise-stable inputs (the engine + resolved target), so the React Compiler + * would cache them as stable and never repaint — the escape hatch keeps the + * engine-backed reads live. It is scoped to this small data hook and the two + * leaf panels, far narrower than the old canvas-wide opt-out on SpatialCanvasInner. + * + * The context only carries stable handles (engine, resolved target, and bound + * subscribe/snapshot callbacks); config flows via props. Headless (panel-less) + * consumers read `pointsEngine` + `resolvePointsTarget` off the renderer-hook + * result, wrap a subtree in this provider, and consume `usePointsFeatureState`. + */ +import type { PointsDataEngine, PointsLoadTarget } from '@spatialdata/layers'; +import { + type ReactNode, + createContext, + useCallback, + useContext, + useMemo, + useSyncExternalStore, +} from 'react'; + +interface PointsFeatureStateContextValue { + engine: PointsDataEngine; + /** The resolved engine target for this layer, or `undefined` when the layer + * is not (yet) a resolvable points element. */ + target: PointsLoadTarget | undefined; + /** Stable `useSyncExternalStore` subscribe (bound to the engine). */ + subscribe: (onChange: () => void) => () => void; + /** Stable `useSyncExternalStore` snapshot: the engine's primitive version. */ + getVersion: () => number; +} + +const PointsFeatureStateContext = createContext(null); + +export interface PointsFeatureStateProviderProps { + engine: PointsDataEngine; + target: PointsLoadTarget | undefined; + children: ReactNode; +} + +export function PointsFeatureStateProvider({ + engine, + target: targetProp, + children, +}: PointsFeatureStateProviderProps) { + // The resolver allocates a fresh `{ key, layerId, element }` object every + // render, so pin the target's identity to its underlying fields. Left + // unpinned, the churn re-creates the request callback each render and re-fires + // its effect, looping ensureFeatureCatalog → notify → render → … Depending on + // the key/element fields (not `targetProp`) is the point — targetProp is a + // fresh object each render. + // biome-ignore lint/correctness/useExhaustiveDependencies: key+element ARE the identity + const target = useMemo( + () => + targetProp + ? { key: targetProp.key, layerId: targetProp.layerId, element: targetProp.element } + : undefined, + [targetProp?.key, targetProp?.layerId, targetProp?.element] + ); + // The engine methods are plain (unbound `this`), so wrap them in callbacks + // keyed on the engine. Stable identities keep useSyncExternalStore from + // re-subscribing every render. + const subscribe = useCallback((onChange: () => void) => engine.subscribe(onChange), [engine]); + const getVersion = useCallback(() => engine.getVersion(), [engine]); + const value = useMemo( + () => ({ engine, target, subscribe, getVersion }), + [engine, target, subscribe, getVersion] + ); + return ( + + {children} + + ); +} + +function usePointsFeatureContext(): PointsFeatureStateContextValue { + const value = useContext(PointsFeatureStateContext); + if (!value) { + throw new Error('usePointsFeatureState must be used within a .'); + } + return value; +} + +export interface PointsFeatureState { + /** The feature catalog: `undefined` until requested/settled, `null` when the + * element has no `feature_key`, else the catalog. */ + catalog: ReturnType; + /** Whether the feature catalog is currently being built. */ + catalogLoading: boolean; + /** Whether the full-dataset catalog scan is still refining an instant preview. */ + catalogRefining: boolean; + /** Distinct feature codes present in the resident batch, or `undefined` until + * the row codes are resident. */ + residentCodes: ReturnType; + /** Feature codes of the last-completed matched selection (non-resident + * features currently on screen), used to grey rows by what's rendered. */ + loadedMatchingCodes: ReturnType; + /** Whether a non-resident feature can be fetched on demand (feature-index scan). */ + supportsOnDemandLoad: boolean; + /** Progressive load state of the feature-index scan for the selection passed + * to the hook, or `undefined` when nothing is selected / it hasn't started. */ + matchingLoadState: ReturnType; + /** Truncation of what's on screen for the selection passed to the hook (so the + * UI can show when raising the memory cap would load more). */ + truncation: ReturnType; + /** Stable callback — trigger the full-dataset catalog build (idempotent). */ + requestCatalog: () => void; +} + +const EMPTY_POINTS_FEATURE_STATE: Omit = { + catalog: undefined, + catalogLoading: false, + catalogRefining: false, + residentCodes: undefined, + loadedMatchingCodes: undefined, + supportsOnDemandLoad: false, + matchingLoadState: undefined, + truncation: undefined, +}; + +/** + * Reactive snapshot of the points feature state for the surrounding + * ``. Subscribes the calling component to the engine + * (re-renders on every `notify`) and returns all derived reads for `featureCodes` + * (the active selection — pass `config.featureCodes`), plus a stable + * `requestCatalog`. + */ +export function usePointsFeatureState(featureCodes?: readonly number[]): PointsFeatureState { + 'use no memo'; + const { engine, target, subscribe, getVersion } = usePointsFeatureContext(); + // Reactivity: re-render this component on every engine mutation. The returned + // version is unused — the subscription is the point. + useSyncExternalStore(subscribe, getVersion, getVersion); + const requestCatalog = useCallback(() => { + if (target) void engine.ensureFeatureCatalog(target); + }, [engine, target]); + + if (!target) { + return { ...EMPTY_POINTS_FEATURE_STATE, requestCatalog }; + } + const key = target.key; + const scannable = engine.supportsFeatureScan(key); + const hasSelection = !!featureCodes && featureCodes.length > 0; + return { + catalog: engine.getFeatureCatalog(key), + catalogLoading: engine.isFeatureCatalogLoading(key), + catalogRefining: engine.isFeatureCatalogRefining(key), + residentCodes: engine.getResidentFeatureCodes(key), + loadedMatchingCodes: engine.getLoadedMatchingFeatureCodes(key), + supportsOnDemandLoad: scannable, + // Only a whole-dataset scan reports a load state; before a catalog loads a + // dict-only element can only filter the resident batch in memory (no scan). + matchingLoadState: + hasSelection && scannable ? engine.getMatchingLoadState(key, featureCodes) : undefined, + truncation: engine.getActiveTruncation(key, featureCodes), + requestCatalog, + }; +} diff --git a/packages/vis/src/SpatialCanvas/PointsLayerPanel.tsx b/packages/vis/src/SpatialCanvas/PointsLayerPanel.tsx new file mode 100644 index 00000000..e8a1d549 --- /dev/null +++ b/packages/vis/src/SpatialCanvas/PointsLayerPanel.tsx @@ -0,0 +1,109 @@ +import { DEFAULT_POINTS_MEMORY_CAP } from '@spatialdata/core'; +import type { PointsDataEngine, PointsLoadTarget } from '@spatialdata/layers'; +import { PointsFeatureFilterPanel } from './PointsFeatureFilterPanel'; +import { PointsFeatureStateProvider, usePointsFeatureState } from './PointsFeatureState'; +import { useSpatialCanvasActions } from './context'; +import type { PointsLayerConfig } from './types'; + +export interface PointsLayerPanelProps { + config: PointsLayerConfig; + /** The live engine (render path's owner) — the panel subscribes to it for + * reactive catalog / scan state instead of reading prop-drilled getters. */ + engine: PointsDataEngine; + /** Resolve a layer id to the engine's load target. Sourced from the renderer + * hook result so panel reads hit the same cache keys the render writes. */ + resolveTarget: (layerId: string) => PointsLoadTarget | undefined; +} + +function PointsMemoryCap({ config }: { config: PointsLayerConfig }) { + const actions = useSpatialCanvasActions(); + const currentCap = config.pointsMemoryCap ?? DEFAULT_POINTS_MEMORY_CAP; + // Discrete options (one reload per choice, vs. a free number + // input that would reload on every keystroke). Include the + // current value so a saved config off the preset list still + // shows correctly. + const capOptions = Array.from( + new Set([1, 2, 4, 8, 16].map((m) => m * 1_000_000).concat(currentCap)) + ).sort((a, b) => a - b); + return ( + + ); +} + +function ShowMatchingPoints({ config }: { config: PointsLayerConfig }) { + // Opt out of the React Compiler — see PointsFeatureFilterPanel. The truncation + // read is engine-backed and updates on notify; the compiler would otherwise + // memoize this line's JSX and never repaint it as the scan progresses. + 'use no memo'; + const { truncation: t } = usePointsFeatureState(config.featureCodes); + if (!t) return null; + // Report the batch held in memory (always true), NOT a per-selection matched + // count: t.loaded is the covered-batch size, which overstates the selection + // when it filters that batch in memory. A precise selection count needs the + // engine to track it — deferred to the redesign (punch-list F3/D4). + const message = t.truncated + ? `${t.loaded.toLocaleString()}${ + t.total !== undefined ? ` of ${t.total.toLocaleString()}` : '' + } points in memory — capped; raise the cap for more.` + : t.filtered + ? `${t.loaded.toLocaleString()} points in memory; view filtered to selection.` + : `All ${t.loaded.toLocaleString()} points loaded (not capped).`; + return ( + + {message} + + ); +} + +export default function PointsLayerPanel({ config, engine, resolveTarget }: PointsLayerPanelProps) { + return ( + + + + + + ); +} diff --git a/packages/vis/src/SpatialCanvas/featureRowState.ts b/packages/vis/src/SpatialCanvas/featureRowState.ts new file mode 100644 index 00000000..e067dc91 --- /dev/null +++ b/packages/vis/src/SpatialCanvas/featureRowState.ts @@ -0,0 +1,121 @@ +/** + * Feature-row state classification for the points feature filter panel. + * + * Kept out of `PointsFeatureFilterPanel.tsx` so that module exports only its + * component — a mixed component + plain-function export breaks Vite React Fast + * Refresh (full reload, dropped React state) for the whole file. + */ + +/** Why a feature row is (or isn't) greyed — drives both the dimming and the + * diagnostic tooltip so they can never disagree. */ +export type FeatureRowTone = 'resident' | 'loaded' | 'cached' | 'loading' | 'noIndex' | 'notLoaded'; + +export interface FeatureRowState { + tone: FeatureRowTone; + /** Whether the row is dimmed (its points are not on screen). */ + greyed: boolean; + /** Short state label, e.g. "loaded", "loading", "not loaded". */ + label: string; + /** One sentence explaining the state / why it is greyed. */ + reason: string; +} + +export interface FeatureRowStateInput { + /** In the preloaded (resident) window. */ + resident: boolean; + /** On screen now via the last-completed feature-index scan. */ + rendered: boolean; + /** In the current selection (checked). */ + selected: boolean; + /** A feature-index scan for the current selection is in flight. */ + scanning: boolean; + /** The element can fetch non-resident features on demand (has a feature index). */ + supportsOnDemandLoad: boolean; + /** The resident set is known (false → we can't distinguish, treat as shown). */ + residentKnown: boolean; +} + +/** + * Classify a feature's render state from the signals the panel already has. + * Precedence matters: `resident`/`rendered` (its points are in memory) win over + * selection/scan state. `rendered` here means "in the loaded matched batch", + * i.e. in memory — a deselected-but-loaded feature is `cached`, not dropped, + * because removing a feature filters the in-memory batch rather than re-scanning + * (re-adding it is instant). + * + * This is up for review. + */ +export function describeFeatureRowState({ + resident, + rendered, + selected, + scanning, + supportsOnDemandLoad, + residentKnown, +}: FeatureRowStateInput): FeatureRowState { + if (!residentKnown) { + return { + tone: 'loaded', + greyed: false, + label: 'shown', + reason: 'The resident set is unknown for this element, so every feature is treated as shown.', + }; + } + if (resident) { + return { + tone: 'resident', + greyed: false, + label: 'resident', + reason: + 'In the preloaded window — shown by filtering the in-memory batch (no dataset scan; a large batch can still take a moment to re-filter).', + }; + } + if (rendered) { + return selected + ? { + tone: 'loaded', + greyed: false, + label: 'loaded', + reason: 'On screen via the feature-index scan for the current selection.', + } + : { + tone: 'cached', + greyed: false, + label: 'in memory', + reason: + 'Loaded in the matched batch but hidden (deselected); re-adding it is instant, no scan.', + }; + } + if (selected && scanning) { + return { + tone: 'loading', + greyed: true, + label: 'loading', + reason: 'Selected — its feature-index scan is in progress.', + }; + } + if (!supportsOnDemandLoad) { + return { + tone: 'noIndex', + greyed: true, + label: 'not in sample', + reason: + 'Beyond the resident window, and this dataset has no feature index, so it can’t be fetched on demand. Raise the memory cap or rewrite the dataset with an index.', + }; + } + return { + tone: 'notLoaded', + greyed: true, + label: 'not loaded', + reason: 'Beyond the resident window; select it to fetch its points via the feature-index scan.', + }; +} + +/** Opacity for a row given its state: crisp when its points are on screen, + * mid-dim while loading, fully dim when not loaded. */ +export function featureRowOpacity(state: FeatureRowState): number { + if (!state.greyed) { + return 1; + } + return state.tone === 'loading' ? 0.6 : 0.4; +} diff --git a/packages/vis/src/SpatialCanvas/index.tsx b/packages/vis/src/SpatialCanvas/index.tsx index 6014b804..bb8a1620 100644 --- a/packages/vis/src/SpatialCanvas/index.tsx +++ b/packages/vis/src/SpatialCanvas/index.tsx @@ -52,6 +52,7 @@ import type { AvailableElement, ElementsByType, ViewState } from './types'; import type { ImageLayerConfig } from './useLayerData'; import { type ViewInteractionState, useViewInteractionGate } from './useViewInteractionGate'; import { generateLayerId, getAllCoordinateSystems } from './utils'; +import PointsLayerPanel from './PointsLayerPanel'; // ============================================ // Styles @@ -385,6 +386,11 @@ function SpatialCanvasInner({ renderTooltip, hoverTooltipMode = 'simple', }: SpatialCanvasInnerProps) { + // Points reactivity now lives in (the panel + // subscribes to the engine via useSyncExternalStore), so this component no + // longer reads mutable engine state through stable getters and doesn't need a + // React Compiler escape hatch. Deck-layer updates still flow through the + // renderer result (a fresh array on each rebuild — a tracked dependency). const { spatialData, loading: sdLoading } = useSpatialData(); const [tooltipMode, setTooltipMode] = useState(hoverTooltipMode); const [measureRef, { width, height }] = useMeasure(); @@ -425,6 +431,20 @@ function SpatialCanvasInner({ const { interacting, onInteractionStateChange } = useViewInteractionGate(); // Shapes are pickable unless tooltips are off or the camera is being moved. const pickingEnabled = tooltipMode !== 'off' && !interacting; + + // keeping a big bundle of these to pass into child components + // (may not be a good idea). + const rendererProps = useSpatialCanvasRendererFromLayerInputs({ + spatialData, + coordinateSystem, + layerInputs: { layers, layerOrder }, + // viewState and onViewStateChange are omitted: auto-fit and pan handling + // are managed entirely by ViewerSection so this hook never re-runs on pan. + width: vw, + height: vh, + pickingEnabled, + }); + const { availableElements, deckLayers, @@ -440,17 +460,10 @@ function SpatialCanvasInner({ hasRenderableLayerData, isBlocking, isLoading, + pointsEngine, + resolvePointsTarget, vivLayerProps, - } = useSpatialCanvasRendererFromLayerInputs({ - spatialData, - coordinateSystem, - layerInputs: { layers, layerOrder }, - // viewState and onViewStateChange are omitted: auto-fit and pan handling - // are managed entirely by ViewerSection so this hook never re-runs on pan. - width: vw, - height: vh, - pickingEnabled, - }); + } = rendererProps; const hoverPickLayerIds = useMemo(() => Array.from(enabledLayerIds), [enabledLayerIds]); useEffect(() => { @@ -849,6 +862,13 @@ function SpatialCanvasInner({ /> )} + {selectedConfig.type === 'points' && ( + + )} {selectedLayerLoadState && (
)} diff --git a/packages/vis/src/SpatialCanvas/public.ts b/packages/vis/src/SpatialCanvas/public.ts index fc2fd24b..4af00fdb 100644 --- a/packages/vis/src/SpatialCanvas/public.ts +++ b/packages/vis/src/SpatialCanvas/public.ts @@ -68,3 +68,9 @@ export type { ImageLoaderData, LayerLoadState, } from './useLayerData'; +// Reactive points feature state. Headless (panel-less) consumers read +// `pointsEngine` + `resolvePointsTarget` off the renderer-hook result, wrap a +// subtree in , and consume the usePoints* hooks. +export { PointsFeatureStateProvider, usePointsFeatureState } from './PointsFeatureState'; +export type { PointsFeatureState, PointsFeatureStateProviderProps } from './PointsFeatureState'; +export type { PointsDataEngine, PointsLoadTarget } from '@spatialdata/layers'; diff --git a/packages/vis/src/SpatialCanvas/renderers/index.ts b/packages/vis/src/SpatialCanvas/renderers/index.ts index 2f42cb12..8c25d239 100644 --- a/packages/vis/src/SpatialCanvas/renderers/index.ts +++ b/packages/vis/src/SpatialCanvas/renderers/index.ts @@ -7,5 +7,4 @@ export { renderImageLayer, type ImageLayerRenderConfig } from './imageRenderer'; export { renderShapesLayer, type ShapesLayerRenderConfig } from './shapesRenderer'; -export { renderPointsLayer, type PointsLayerRenderConfig } from './pointsRenderer'; export { renderLabelsLayer, type LabelsLayerRenderConfig } from './labelsRenderer'; diff --git a/packages/vis/src/SpatialCanvas/renderers/pointsRenderer.ts b/packages/vis/src/SpatialCanvas/renderers/pointsRenderer.ts deleted file mode 100644 index bd282b13..00000000 --- a/packages/vis/src/SpatialCanvas/renderers/pointsRenderer.ts +++ /dev/null @@ -1,100 +0,0 @@ -/** - * Points layer renderer using deck.gl ScatterplotLayer - * - * Renders point cloud data from SpatialData points elements. - */ - -import { ScatterplotLayer } from 'deck.gl'; -import type { Matrix4 } from '@math.gl/core'; -import type { PointsElement } from '@spatialdata/core'; -import type { Layer } from 'deck.gl'; - -export interface PointDataX { - position: [number, number] | [number, number, number]; - // Additional properties can be added for coloring, sizing, etc. - [key: string]: unknown; -} - -// this is ndarray and should be defined elsewhere -// not that we wouldn't also want to be able to have other data & accessors -export interface PointData { - shape: number[]; - // Columns may be plain arrays or TypedArrays; the core loader now returns - // ArrayLike columns (PointsLoadResult), so keep this widened to match. - data: ArrayLike[]; -} - -export interface PointsLayerRenderConfig { - /** The points element to render */ - element: PointsElement; - /** Unique layer ID */ - id: string; - /** Transformation matrix to target coordinate system */ - modelMatrix: Matrix4; - /** Layer opacity (0-1) */ - opacity: number; - /** Whether layer is visible */ - visible: boolean; - /** Point radius in pixels */ - pointSize?: number; - /** Point color [r, g, b, a] (0-255) */ - color?: [number, number, number, number]; - /** ndarray - if we want other data for properties like color/radius etc they will be handled differently */ - pointData?: PointData; - use3d?: boolean; -} - -/** - * Create a deck.gl ScatterplotLayer for points data. - * - * Note: This requires the point data to be pre-loaded since deck.gl layers - * are synchronous. The data loading should happen at a higher level. - */ -export function renderPointsLayer(config: PointsLayerRenderConfig): Layer | null { - const { - element, - id, - modelMatrix, - opacity, - visible, - pointSize = 1, - color = [255, 100, 100, 200], - pointData, - use3d, - } = config; - - if (!visible) return null; - - if (!pointData) { - // Data not loaded yet - console.debug( - `[PointsRenderer] No point data for layer "${id}" from ${element.url ?? element.path}` - ); - return null; - } - const d = pointData.data; - return new ScatterplotLayer({ - id, - data: d[0], //just for index really - // todo: more robust ndarray handling, be more efficient with target - // see https://deck.gl/docs/developer-guide/performance#supply-attributes-directly - // spatial data-structure (quad/oct-tree) vs pushing raw attributes. - // with ways of querying within view. - // also allow accessors for other props - getPosition: (_d, { index, target }) => [ - d[0][index], - d[1][index], - use3d ? d[2]?.[index] || 0 : 0, - ], - getRadius: pointSize, - radiusUnits: 'pixels', - getFillColor: color, - opacity, - // Apply coordinate transformation - modelMatrix, - // Picking - pickable: true, - autoHighlight: true, - highlightColor: [255, 255, 0, 200], - }); -} diff --git a/packages/vis/src/SpatialCanvas/types.ts b/packages/vis/src/SpatialCanvas/types.ts index fde3275c..275667d6 100644 --- a/packages/vis/src/SpatialCanvas/types.ts +++ b/packages/vis/src/SpatialCanvas/types.ts @@ -96,10 +96,29 @@ export interface ShapesLayerConfig extends BaseLayerConfig { export interface PointsLayerConfig extends BaseLayerConfig { type: 'points'; // Points-specific settings - // TODO: these should be accessors for getColor etc based on e.g. transcript type - // should be able to filter etc. Some kind of LOD... pointSize?: number; + /** + * Max rows retained in memory for the preloaded scatter (the "resident + * window"). `undefined` uses `DEFAULT_POINTS_MEMORY_CAP`. Raising it draws + * more points at the cost of memory + decode time; the feature-index scan for + * a selection retains up to this many matched rows too. Serializable + * Stack-Entry state; changing it reloads the resident window. + */ + pointsMemoryCap?: number; color?: [number, number, number, number]; + /** + * Colour each point by its feature code (categorical, GPU-shaded) instead of + * the flat {@link color}. Serializable Stack-Entry state. The runtime-only + * Feature Highlight (grey non-selected) is layered on top of this later. + */ + colorByFeature?: boolean; + /** + * Feature-filter selection by Feature Code. `undefined` means "all features + * shown" (no filter); an array restricts the drawn points to those codes. This + * is serializable Stack-Entry state (persists in a saved config), distinct from + * the runtime-only Feature Highlight added in MVP step 3. + */ + featureCodes?: number[]; } export interface LabelsLayerConfig extends BaseLayerConfig { diff --git a/packages/vis/src/SpatialCanvas/useLayerData.ts b/packages/vis/src/SpatialCanvas/useLayerData.ts index 675256fb..6c9cdb19 100644 --- a/packages/vis/src/SpatialCanvas/useLayerData.ts +++ b/packages/vis/src/SpatialCanvas/useLayerData.ts @@ -38,6 +38,7 @@ import { loadAssociatedTableFeatureRows, loadLabelsTooltipMetadata, loadShapesTooltipMetadata, + resolvePointsMemoryCap, resolveTooltipItems, unionBoundsList, } from '@spatialdata/core'; @@ -45,6 +46,8 @@ import { EMPTY_SHAPE_FEATURE_STATE_RUNTIME, PointsDataEngine, PointsLayer, + type PointsLoadTarget, + type PointsRenderResource, type ShapeFeatureRenderDatum, type ShapeFeatureStateRuntime, type ShapeFillColorMode, @@ -211,6 +214,15 @@ interface UseLayerDataResult { getLayerLoadState: (layerId?: string) => LayerLoadState | undefined; /** Whether a layer already has enough data to render. */ hasRenderableLayerData: (layerId: string) => boolean; + /** The live points data engine (the render path's single owner). Exposed so + * the feature panel can subscribe to it directly for reactive catalog / scan + * state via `PointsFeatureStateProvider`, instead of prop-drilling getters. */ + pointsEngine: PointsDataEngine; + /** Resolve a points layer to the engine's load target `{ key, layerId, + * element }`, or `undefined` when the layer isn't a resolvable points element. + * Reuses the same element resolution the load path uses, so panel hooks read + * the same cache keys the render writes. */ + resolvePointsTarget: (layerId: string) => PointsLoadTarget | undefined; /** Resolve a feature tooltip lazily from the picked row index. */ getFeatureTooltip: ( layerId: string, @@ -544,6 +556,7 @@ export function useLayerData( [] ); + //--- to be removed from here? // Points loading/caching/resolution engine (LayerDataEngine step 1b). This // framework-agnostic engine (in @spatialdata/layers) owns the points preload // cache, the stable render-resource memo, and the async load orchestration @@ -562,10 +575,15 @@ export function useLayerData( }) ); - useEffect(() => { - const unsubscribe = pointsEngine.subscribe(notifyLoadedDataChanged); - return unsubscribe; - }, [pointsEngine, notifyLoadedDataChanged]); + // Re-render on every points-engine cache mutation (async loads/scans settling). + // NOTE: the consuming component (SpatialCanvasInner) must opt out of the React + // Compiler (`'use no memo'`) — the compiler otherwise memoizes JSX built from + // these engine getters and never repaints on a late async settle, since the + // getters read mutable engine state with no compiler-tracked dependency. + useEffect( + () => pointsEngine.subscribe(notifyLoadedDataChanged), + [pointsEngine, notifyLoadedDataChanged] + ); // Load data for enabled layers that don't have data yet useEffect(() => { @@ -651,7 +669,12 @@ export function useLayerData( loadLabels, }); } - } else if (config.type === 'points' && !pointsEngine.hasData(elem.key)) { + } else if ( + config.type === 'points' && + // Reload when the resident window is missing OR was loaded at a + // different memory cap (the props-panel control changed it). + !pointsEngine.isLoadedWithCap(elem.key, resolvePointsMemoryCap(config.pointsMemoryCap)) + ) { toLoad.push({ layerId, element: elem, @@ -827,12 +850,20 @@ export function useLayerData( } } else if (element.type === 'points' && loadPoints) { // The engine owns loading/caching/status; it reports status back - // through the onStatus callback wired at construction. - await pointsEngine.ensureLoaded({ - key: element.key, - layerId, - element: element.element as PointsElement, - }); + // through the onStatus callback wired at construction. The resident + // window size is the layer's configured memory cap (props panel). + const pointsConfig = layers[layerId]; + const memoryCap = resolvePointsMemoryCap( + pointsConfig?.type === 'points' ? pointsConfig.pointsMemoryCap : undefined + ); + await pointsEngine.ensureLoaded( + { + key: element.key, + layerId, + element: element.element as PointsElement, + }, + memoryCap + ); } else if (element.type === 'image' && loadImage) { try { setLayerResourceStatus(layerId, 'image', 'loading'); @@ -1160,6 +1191,21 @@ export function useLayerData( return false; }, [pointsEngine]); + // --- Points feature state (filter panel) ----------------------------------- + // The panel no longer reads point state through prop-drilled getters. Instead + // it subscribes to `pointsEngine` directly (via `PointsFeatureStateProvider` + // + the `usePoints*` hooks), so its reactivity is self-contained and does not + // depend on this hook's re-render or a `'use no memo'` escape hatch. All this + // hook exposes is the engine and the element-key resolver the hooks need. + const resolvePointsTarget = useCallback( + (layerId: string): PointsLoadTarget | undefined => { + const elem = resolveLayerElement(layerId, layersRef.current[layerId], elementMap.current); + if (!elem || elem.type !== 'points') return undefined; + return { key: elem.key, layerId, element: elem.element as PointsElement }; + }, + [] + ); + const getWorldBoundsForLayer = useCallback( (layerId: string): AxisAlignedBounds | null => { try { @@ -1298,22 +1344,124 @@ export function useLayerData( if (layer) deckLayers.push(layer); } } else if (config.type === 'points') { - // The engine returns a STABLE render resource (memoized by signature), - // so re-running getLayers every pan/zoom frame reuses the same loader - // identity and the composite does not reset its batch (no flashing). - const resource = pointsEngine.getResource(elem.element as PointsElement, elem.key); - if (resource) { + const element = elem.element as PointsElement; + const featureCodes = config.featureCodes; + const selectionActive = featureCodes !== undefined && featureCodes.length > 0; + + // Feature-index render scan: when a selection is active, load the WHOLE + // dataset's matching points (footer stats skip the row groups a selected + // feature can't live in), so features outside the resident preload window + // still render. The scan is idempotent per selection; kicking it here is a + // no-op once resident/in-flight. On settle it notifies → re-render → the + // matched resource appears below. `getMatchingResource` returns the LAST + // completed matched batch, so a selection change keeps showing the prior + // selection's points until the new scan settles (no blank mid-scan). + // + // Gated on scan capability: an authoritative code column (footer stats + // skip row groups) OR a dictionary-only element with a catalog loaded — + // there the scan reads the whole file and matches each row's feature_name + // against the catalog's code space, so a selected gene's points render + // even when they fall outside the resident preload window. Before any + // catalog loads (no shared code space) there is nothing to match names + // against, so it falls through to resident in-memory filtering. + const canFeatureScan = pointsEngine.supportsFeatureScan(elem.key); + let matchingResource: PointsRenderResource | null = null; + let partialResource: PointsRenderResource | null = null; + if (selectionActive && canFeatureScan) { + void pointsEngine.ensureMatchingFeaturesLoaded( + { key: elem.key, layerId, element }, + featureCodes, + resolvePointsMemoryCap(config.pointsMemoryCap) + ); + matchingResource = pointsEngine.getMatchingResource(element, elem.key); + // The in-flight scan's growing buffer (all matched chunks so far), drawn + // as an extra overlay sub-layer below so the base (resident preview / + // prior matched batch) stays visible while points progressively fill in. + partialResource = pointsEngine.getMatchingPartialResource(element, elem.key); + } + + if (matchingResource) { + // The matched batch covers the selection (or a superset of it, when the + // selection just shrank). Pass the batch's per-row codes + the current + // selection so the layer filters IN MEMORY down to the selected codes — + // this is what makes removing a feature a free filter instead of a + // re-scan. When the selection equals what was scanned, skip the filter + // (render the batch whole); the batch's own codes still drive colour. + const matchedRowCodes = pointsEngine.getMatchingRowFeatureCodes(elem.key); + const coveredSize = pointsEngine.getLoadedMatchingFeatureCodes(elem.key)?.size ?? 0; + const filterMatched = featureCodes !== undefined && featureCodes.length < coveredSize; deckLayers.push( new PointsLayer({ id: layerId, - resource, + resource: matchingResource, + modelMatrix: elem.transform, + opacity: config.opacity, + visible: config.visible, + pointSize: config.pointSize ?? 1, + ...(filterMatched ? { featureCodes } : {}), + ...(matchedRowCodes ? { preloadedFeatureCodes: matchedRowCodes } : {}), + ...(config.color ? { color: config.color } : {}), + ...(config.colorByFeature ? { colorByFeature: true } : {}), + }) + ); + } else { + // Resident batch: the default view (no selection), and an instant preview + // of the resident subset while the feature-index scan is still running. + // The engine returns a STABLE render resource (memoized by signature), so + // re-running getLayers every pan/zoom frame reuses the same loader + // identity and the composite does not reset its batch (no flashing). + const resource = pointsEngine.getResource(element, elem.key); + if (resource) { + const filterActive = featureCodes !== undefined; + // Row codes are needed to filter by feature AND to colour by feature. + // Colour-by-feature applies even with no filter ("all features"), so + // load/pass the codes whenever either is on — not just when filtering. + const needsRowCodes = filterActive || config.colorByFeature === true; + if (needsRowCodes && !pointsEngine.hasRowFeatureCodes(elem.key)) { + void pointsEngine.ensureRowFeatureCodes({ key: elem.key, layerId, element }); + } + const preloadedFeatureCodes = needsRowCodes + ? pointsEngine.getRowFeatureCodes(elem.key) + : undefined; + deckLayers.push( + new PointsLayer({ + id: layerId, + resource, + modelMatrix: elem.transform, + opacity: config.opacity, + visible: config.visible, + // Legacy renderPointsLayer defaulted radius to 1px; preserve that + // for parity (the composite's own default is smaller). + pointSize: config.pointSize ?? 1, + ...(config.color ? { color: config.color } : {}), + ...(config.colorByFeature ? { colorByFeature: true } : {}), + ...(featureCodes ? { featureCodes } : {}), + ...(preloadedFeatureCodes ? { preloadedFeatureCodes } : {}), + }) + ); + } + } + + // Overlay the in-flight scan's growing buffer as a SEPARATE sub-layer on + // top of whichever base layer was pushed above, so the base doesn't blank + // while points progressively fill in. Distinct id so deck keeps them as two + // layers. Filter it to the CURRENT selection with the partial's own per-row + // codes — mirroring the settled matched layer — so a feature deselected + // mid-scan stops rendering immediately instead of lingering until settle. + if (partialResource) { + const partialRowCodes = pointsEngine.getMatchingPartialRowFeatureCodes(elem.key); + deckLayers.push( + new PointsLayer({ + id: `${layerId}__partial`, + resource: partialResource, modelMatrix: elem.transform, opacity: config.opacity, visible: config.visible, - // Legacy renderPointsLayer defaulted radius to 1px; preserve that - // for parity (the composite's own default is smaller). pointSize: config.pointSize ?? 1, + ...(featureCodes ? { featureCodes } : {}), + ...(partialRowCodes ? { preloadedFeatureCodes: partialRowCodes } : {}), ...(config.color ? { color: config.color } : {}), + ...(config.colorByFeature ? { colorByFeature: true } : {}), }) ); } @@ -1705,6 +1853,8 @@ export function useLayerData( getLabelsLayerLoadedData, getLayerLoadState, hasRenderableLayerData, + pointsEngine, + resolvePointsTarget, getFeatureTooltip, getFeaturePickEvent, getShapePickEvent, diff --git a/packages/vis/tests/pointsFeatureRowState.spec.ts b/packages/vis/tests/pointsFeatureRowState.spec.ts new file mode 100644 index 00000000..4c760948 --- /dev/null +++ b/packages/vis/tests/pointsFeatureRowState.spec.ts @@ -0,0 +1,62 @@ +import { describe, expect, it } from 'vitest'; + +import { + type FeatureRowStateInput, + describeFeatureRowState, +} from '../src/SpatialCanvas/featureRowState'; + +const base: FeatureRowStateInput = { + resident: false, + rendered: false, + selected: false, + scanning: false, + supportsOnDemandLoad: true, + residentKnown: true, +}; + +describe('describeFeatureRowState', () => { + it('resident features are never greyed', () => { + const s = describeFeatureRowState({ ...base, resident: true }); + expect(s).toMatchObject({ tone: 'resident', greyed: false }); + }); + + it('a rendered + selected feature is loaded (on screen)', () => { + const s = describeFeatureRowState({ ...base, rendered: true, selected: true }); + expect(s).toMatchObject({ tone: 'loaded', greyed: false }); + }); + + it('a rendered but deselected feature is cached (in memory, not dropped)', () => { + // The removal fast path: still in the matched batch, just hidden. NOT greyed — + // re-adding it is instant, which is the whole point of subset reuse. + const s = describeFeatureRowState({ ...base, rendered: true, selected: false }); + expect(s).toMatchObject({ tone: 'cached', greyed: false }); + expect(s.reason).toMatch(/re-adding it is instant/); + }); + + it('a selected feature whose scan is running is loading (greyed, distinct tone)', () => { + const s = describeFeatureRowState({ ...base, selected: true, scanning: true }); + expect(s).toMatchObject({ tone: 'loading', greyed: true }); + }); + + it('a non-resident, non-loaded feature on an indexed element is "not loaded"', () => { + const s = describeFeatureRowState({ ...base, supportsOnDemandLoad: true }); + expect(s).toMatchObject({ tone: 'notLoaded', greyed: true }); + expect(s.reason).toMatch(/select it to fetch/); + }); + + it('a non-resident feature on a dict-only element is "not in sample" (no on-demand)', () => { + const s = describeFeatureRowState({ ...base, supportsOnDemandLoad: false }); + expect(s).toMatchObject({ tone: 'noIndex', greyed: true }); + expect(s.reason).toMatch(/no feature index/); + }); + + it('treats everything as shown when the resident set is unknown', () => { + const s = describeFeatureRowState({ ...base, residentKnown: false }); + expect(s.greyed).toBe(false); + }); + + it('resident wins over an in-flight scan', () => { + const s = describeFeatureRowState({ ...base, resident: true, selected: true, scanning: true }); + expect(s.tone).toBe('resident'); + }); +});