diff --git a/.gitignore b/.gitignore index 611bca7d..18942b70 100644 --- a/.gitignore +++ b/.gitignore @@ -13,11 +13,10 @@ packages/zarrextra/src/*.d.ts.map coverage/ test-fixtures/ validation-results/ +**/.pytest_cache/ -# Python virtual environments (version-specific, completely separate) -python/v0.5.0/.venv/ -python/v0.6.1/.venv/ -python/v0.7.2/.venv/ +# Python virtual environments +python/*/.venv/ # Vendored from @cornerstonejs/codec-openjph (see scripts/vendor-openjph-for-python.mjs) python/spatialdata-codec-writer/src/spatialdata_codec_writer/vendor/openjph/ diff --git a/CONTEXT.md b/CONTEXT.md index 9f9f37ee..e12285e5 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -40,6 +40,22 @@ _Avoid_: serializable prop, stack entry prop A small MDV/control-layer UI area that edits observable state directly while passing plain values through renderer and third-party boundaries. _Avoid_: MobX renderer contract +**Points Render Resource**: +The **Resource Resolver** output for a points **Spatial Entry**: a bundle `{ element, loader }` pairing the canonical `PointsElement` with a frozen **`PointsLoader`** facet. +_Avoid_: treating `PointsLoader` alone as the full render resource, or storing the loader on/mutating the element + +**PointsElement**: +The source identity and public source API for SpatialData points. It owns element metadata and source-level methods such as full or bounded point loading; deck render strategies receive the resolved **Points Render Resource** instead of mutating or calling the element directly. +_Avoid_: render-time strategy object, mutable loader holder + +**PointsLoader**: +The loader facet of a **Points Render Resource**: encoding capabilities plus a fetch API (`loadInBounds`, optional `loadAll`). Built by `@spatialdata/core` store-I/O factories; consumed by `@spatialdata/layers` render strategies — not by calling `PointsElement` methods directly from deck code. +_Avoid_: conflating with Viv/image `loader` when discussing SpatialData element identity + +**Points Encoding**: +The render-time points layout selected after resolver probing, e.g. `preloaded-columnar`, `morton-tiled`, or future `geoarrow-*` kinds. Distinct from persisted Parquet layout described in ADR 0002. +_Avoid_: `experimentalOptimizations` as a synonym for encoding kind + ## Relationships - A **Render Stack** contains zero or more ordered **Stack Entries**. @@ -59,3 +75,12 @@ _Avoid_: MobX renderer contract - "Layer" was used for SpatialData elements, deck.gl layer instances, UI rows, and saved config entries. Resolved: use **Stack Entry** for saved/render order, **Spatial Entry** for SpatialData-backed entries, and deck.gl layer only for runtime renderer output. - "Snapshot" was used for both persisted config and temporary UI/render state. Resolved: persisted config is a **Render Stack**; live MDV direct-edit areas are **MobX Control Islands** and should not periodically snapshot the whole stack during interaction. - "Props" was used for both serialized renderer inputs and runtime callback objects. Resolved: `entry.props` must remain serializable renderer input; listeners, factories, portals, and raw deck integration points are **Runtime Attachments**. + +## In-progress work + +- **Points preload & feature filter** — runtime filter on preloaded scatter, row-group + geometry reads, catalog fallbacks for large dictionary-only transcripts. + Status and cleanup plan: + [`docs/plans/points-preload-feature-filter-status.md`](docs/plans/points-preload-feature-filter-status.md). + Related ADRs: [0002](docs/adr/0002-spatially-aware-vector-loading.md), + [0003](docs/adr/0003-points-render-resource.md). diff --git a/docs/adr/0002-spatially-aware-vector-loading.md b/docs/adr/0002-spatially-aware-vector-loading.md new file mode 100644 index 00000000..669d204f --- /dev/null +++ b/docs/adr/0002-spatially-aware-vector-loading.md @@ -0,0 +1,148 @@ +# Spatially-Aware Vector Loading + +SpatialData points and shapes can be large enough that whole-element Parquet +loads are not a viable browser default. We will treat viewport-bounded vector +loading as a first-class source API and keep persisted optimization artifacts in +Parquet/GeoParquet rather than inventing a deck.gl-specific storage format. + +## Decision + +- Points v1 follows current Vitessce practice: a SpatialData Points Parquet + element may be sorted by 2D Morton order with a `morton_code_2d` column, a + feature-code column, controlled row-group sizes, and 2-4 leading sentinel rows + whose `morton_code_2d` is `0` and whose coordinates encode the full point + extent. +- `@spatialdata/core` exposes bounded point loading through + `PointsElement.loadPointsInBounds()`. When the Parquet module supports + Vitessce's row-group APIs (`readMetadata` and `readParquetRowGroup`) and the + store supports range reads, the loader may fetch selected row groups. Otherwise + it degrades to the existing full-table read followed by bounds filtering. +- Render-time code uses ADR 0003's **Points Render Resource** (`{ element, + loader }`) and calls the `PointsLoader` facet. `PointsElement` remains source + identity and public source API, not the deck strategy contract. +- `@spatialdata/vis` may render compatible points through a deck.gl `TileLayer`. + The tile layer owns async viewport loads and abort signals; ordinary + `ScatterplotLayer` rendering remains the fallback for preloaded point data. +- `points.experimental/` and `shapes.experimental/` are reserved as + top-level Experimental Optimization Collections. They link back to the source + element by key and metadata rather than modifying canonical SpatialData + element semantics. +- GeoParquet is the durable shape optimization target. GeoArrow is a runtime + columnar layout / deck adapter option, not a duplicate persisted artifact. + +## Experimental Optimization Collections + +Use `points.experimental//` and `shapes.experimental//` only for +persisted layouts that **standard SpatialData / Vitessce readers cannot correctly +consume** — not for every browser optimization. + +| Layout | Where it lives | Why | +|--------|----------------|-----| +| **Morton v1** (`morton_code_2d`, sentinels, `{feature_key}_codes`, row groups) | **Canonical** `points//points.parquet` | Follows Vitessce practice. Extra columns are additive; Python `spatialdata` full-table reads still work. | +| **Feature-primary sort** (Morton not primary key) | `points.experimental//` | Breaks Morton row-group bisect; needs a new tiling `kind` | +| **Padua multiscale** (`__spatial_index__`, levels in schema metadata) | `points.experimental//` | Non-standard vs morton-points v1 | +| **GeoParquet shapes tiling** | `shapes.experimental//` | Future | + +`experimentalOptimizations` in `@spatialdata/vis` means use TileLayer / row-group +reads when **canonical** parquet schema supports morton tiling — not “look in +`points.experimental/`”. + +The experimental writer defaults to **in-place** Morton sorting on +`points//points.parquet`. Use `--experimental` only when writing a layout +that must not replace the canonical element. + +## Multi-part Parquet (reader) + +Wild-type SpatialData points may store `points//points.parquet` as a +**directory** with `part.0.parquet`, `part.1.parquet`, … The logical path remains +`points//points.parquet`. `@spatialdata/core` supports both single-file and +multipart layouts for metadata, schema, and row-group range reads. The +experimental writer outputs a **single-file** Morton artifact by design; row-group +range reads fetch only the byte ranges needed per viewport. + +## Feature / gene filtering + +Transcript and other feature-bearing points declare `feature_key` in element +`spatialdata_attrs` (for example `"feature_name"` on xenium transcripts). This is +distinct from `instance_key` (for example `"cell_id"`), which identifies the +object a point belongs to. + +The Morton writer adds `{feature_key}_codes` (for example `feature_name_codes`) +as `int32` categorical codes alongside the string feature column. Sorting is +**spatial** (Morton on x/y) by default; row groups are spatial chunks. + +**Core API** — extend bounded loading with optional feature codes: + +```typescript +interface PointsInBoundsOptions { + bounds: SpatialBounds; + /** Integer codes matching `{feature_key}_codes` in the parquet artifact */ + featureCodes?: readonly number[]; + signal?: AbortSignal; +} +``` + +**Vis API** — extend `PointsLayerConfig` with `featureCodes?: number[]` and +wire through TileLayer `updateTriggers.getTileData`. + +v1 applies feature filtering as a **read-time row predicate** after spatial +bounds filtering (and after row-group fetch on the Morton path). It does not +skip row groups by gene. String-based `features?: string[]` and a codebook +artifact are deferred. + +**Implementation status** (preload vs runtime filter, catalog, workers): +[`docs/plans/points-preload-feature-filter-status.md`](../plans/points-preload-feature-filter-status.md). + +Feature filtering is separate from **feature-primary sort** experiments +(`[feature_codes, morton_code_2d]`), which may require a new tiling `kind` if +promoted. Use `write-index-permutations` on a derivative Zarr store to benchmark +sort strategies; see the writer README. + +A hypothetical **per-gene density map** (2D histogram / KDE for one feature) is +out of scope for the Morton tile path and may be an offline aggregation or +dedicated viz mode later. + +## Sort strategy experiments + +Default Morton v1 sort is spatial on `morton_code_2d` (optionally `z` when +low-cardinality). Multi-key sorts under evaluation include +`[morton_code_2d, feature_name_codes]` and `[feature_name_codes, morton_code_2d]`. +The reader's row-group bisect assumes Morton is the **primary** sort key; do not +silently swap sort order under the existing `morton-points` format id. + +Generate comparable permutations with: + +```bash +spatialdata-experimental-writer write-index-permutations SOURCE_ZARR DEST_ZARR +``` + +The derivative store includes sibling `points//` elements and +`index-manifest.json` for benchmark tooling. + +## Prior Art + +- scverse Padua hackathon points work: + and + . +- Padua branch prototype: + . +- Vitessce tiled SpatialData Points: + . +- Vitessce sentinel bbox update: + and + . +- Vitessce shapes format `0.3` compatibility: + and + . + +## Consequences + +- Source loaders must expose typed/columnar batches and remain independent of + deck.gl. Rendering packages decide whether to use TileLayer, ScatterplotLayer, + or a future GeoArrow-aware layer. +- Whole-table point loading is still supported and is the compatibility fallback, + but render paths can opt into experimental optimizations with a single + `experimentalOptimizations` switch. +- Shapes format `0.3` remains on the current modern Parquet-backed path in + `VShapesSource`; large-shape spatial tiling still needs a separate GeoParquet + artifact/writer slice. diff --git a/docs/adr/0003-points-render-resource.md b/docs/adr/0003-points-render-resource.md new file mode 100644 index 00000000..4ea7ff52 --- /dev/null +++ b/docs/adr/0003-points-render-resource.md @@ -0,0 +1,97 @@ +# Points Render Resource + +ADR 0002 describes persisted Morton Parquet artifacts and bounded loading APIs on +`PointsElement`. This ADR describes the **render-time** boundary between store +I/O, the Resource Resolver, and the deck.gl `PointsLayer` composite. + +## Decision + +- A points **Spatial Entry** (`PointsElement`) remains the canonical spatial + identity handle. Deck layers stay associated with that element for picks, + tooltips, and Render Stack `elementKey`. +- The **Resource Resolver** (today `resolvePointsRenderResource()` in + `@spatialdata/vis`) probes once and returns a **Points Render Resource** + bundle `{ element, loader }` with **frozen** encoding capabilities. +- **`PointsLoader`** is the loader facet only: encoding kind, batch format, + bounds, and fetch methods. Render strategies call `loader.loadInBounds()` — + not `element.loadPointsInBounds()` directly from `@spatialdata/layers`. +- **`PointsLayer`** (`@spatialdata/layers` `CompositeLayer`) takes + `resource: PointsRenderResource` plus cosmetic props. It delegates to + encoding-specific render strategies selected by `loader.capabilities.kind`. +- **Store I/O loader factories** live in `@spatialdata/core` and close over + `PointsElement`. **Render strategies** and tile-debug overlay logic live in + `@spatialdata/layers`. The vis resolver associates element + loader. + +## Encoding selection (v1) + +| Condition | Encoding kind | Strategy | +|-----------|---------------|----------| +| Full table preloaded in resolver cache | `preloaded-columnar` | `ScatterplotLayer` | +| Morton metadata with row-group range reads + bounds | `morton-tiled` | `TileLayer` + per-tile scatter | +| Future GeoArrow batch from core | `geoarrow-binary` | stub → `GeoArrowScatterplotLayer` | +| Future tiled Arrow/Parquet deck path | `geoarrow-tiled` | stub | + +Resolver probing is **eager**: capabilities do not change mid-session unless +the element or resolver cache inputs change. + +## GeoArrow boundary + +- **Core** may later expose deck-free Apache Arrow `RecordBatch` batches from + Parquet row groups (x/y/z columns or geometry). +- **Layers** owns [deck.gl-geoarrow](https://github.com/geoarrow/deck.gl-geoarrow) + integration: GeoArrow geometry shaping and `GeoArrowScatterplotLayer` / + future tiled deck paths. +- Core must not import deck.gl or `@geoarrow/deck.gl-geoarrow`. + +## Batch contract + +`PointsBatch` is a tagged union: + +- `columnar-ndarray` — v1 Morton and preloaded paths +- `arrow-record-batch` — reserved for GeoArrow strategies + +## Tile debug overlay + +When `showTileDebugOverlay` is enabled on a tiled encoding, the morton strategy +emits a pickable `PolygonLayer` sublayer with per-tile status (pending, loading, +loaded, empty, error, aborted). This is cosmetic for tile fetching and must not +appear in `TileLayer.updateTriggers.getTileData`. + +## Relationship to ADR 0002 + +- ADR 0002: persisted artifacts and source-level + `PointsElement.loadPointsInBounds()` API. +- ADR 0003: render-time bundle, strategy registry, and deck composite ownership. + +## Consequences + +- Swapping encodings or deck.gl parquet layers requires new loader factories + and/or strategies — not changes to `PointsLayer` public props. +- `PointsElement` does not grow a mutable `renderResource` attachment; the + resolver cache holds stable bundle references per element key. +- Image precedent: `ImageElement` + Viv loader built in vis; points precedent: + `PointsElement` + `PointsLoader` built in vis, rendered by `PointsLayer`. + +## Future performance investigations + +These are documented follow-ups — not part of the v1 render bundle. + +### CPU / compute hot paths + +The scan+compact loops in `filterColumnarByFeatureCodes` / +`filterPointsToBounds` (`packages/core/src/pointsTiling.ts`) are hot paths for +large preloaded datasets. Candidates include WASM SIMD and WebGPU compute (e.g. +[typegpu](https://github.com/software-mansion/typegpu)) for parallel index +selection and column compaction. **Worker offload** (`@spatialdata/core/workers`) +is the near-term mitigation; GPU/WASM is a follow-up benchmark task. + +### FBO-based render caching + +For viewport-stable layers (tiled points, filtered preloaded batches, static +image tiles), cache rasterized sublayer output in **framebuffer objects (FBOs)** +so pan/zoom and cosmetic prop changes do not re-draw the full payload every +frame. This should integrate with the broader **Render Stack compositing** +story (`Group Entry`, Viv/deck stacking) via shared FBO cache utilities — +invalidation keyed on structural `updateTriggers`, composition order with host +overlays — rather than as a points-only hack. Detail deferred until compositing +utils exist. diff --git a/docs/docs/core/error-handling.mdx b/docs/docs/core/error-handling.mdx index c29b01cc..50a3868d 100644 --- a/docs/docs/core/error-handling.mdx +++ b/docs/docs/core/error-handling.mdx @@ -4,22 +4,41 @@ sidebar_position: 4 # Error Handling -The `@spatialdata/core` package uses a `Result` type pattern inspired by Rust for explicit error handling. This approach makes error cases visible in the type system and avoids unexpected exceptions. +`@spatialdata/core` exports a `Result` type (inspired by Rust) for operations +where failure is expected and should be handled explicitly. **Adoption is +currently very limited** — a handful of APIs (notably coordinate-system +lookups) return `Result`; most of the codebase still throws exceptions, returns +`null`, or uses other ad hoc patterns. -:::info Error types and safety +Because error handling is not uniform, you **cannot treat `Result` as a +system-wide contract**. When integrating with this library, assume exceptions +unless a specific API's signature returns `Result`. The sections below document +the type and patterns for the places that use it. -The use of this pattern in some parts of the code should not be taken as an assertion that all possible exceptions are handled in this way. +We find the `Result` approach appealing in principle, but wider use is an **open +design decision**. If we lean in more thoroughly, we would likely adopt an +**established library** (for example [`neverthrow`](https://github.com/supermacro/neverthrow)) rather than extend our small in-house definitions. -This aspect of the design may be subject to review, and community feedback is encouraged. +:::info Limited adoption — read this first -::: - -:::note Result implementation +The `Result` pattern is **not** adhered to consistently across packages. Do not +infer from this page that most failures are typed, composable, or exception-free. +Many code paths (including parquet I/O, element loading, and worker boundaries) +still throw or use nullable returns. -The `Result` type is implemented in `zarrextra` and re-exported from `@spatialdata/core` for convenience. This is currently a custom implementation for simplicity and to avoid dependencies. We may review using an existing Result library (such as `neverthrow`) in the future, but for now this provides a lightweight, dependency-free solution. +Community feedback on whether and how to expand `Result` use is welcome. ::: +:::note Current implementation (may change) + +Today the `Result` type lives in `zarrextra` and is re-exported from +`@spatialdata/core` — a minimal custom shape (`ok` / `value` / `error`) kept +dependency-free while adoption remains narrow. **Do not build heavily on these +definitions** as a long-term API promise; broader adoption would probably replace +or wrap them with a maintained library and standard combinators (`map`, `andThen`, +etc.). + ## The Result Type ```ts @@ -212,7 +231,10 @@ if (errors.length > 0) { } ``` -## Why Result Instead of Exceptions? +## Why use Result where we do? + +These are motivations for the pattern **where it is used today**, and for +possible wider adoption if we decide to lean in: 1. **Explicit error handling**: The type system shows you when operations can fail 2. **No hidden control flow**: Errors don't jump up the call stack unexpectedly @@ -220,5 +242,16 @@ if (errors.length > 0) { 4. **Composable**: Easy to chain, map, and aggregate results 5. **Performance**: Avoids the overhead of exception creation when errors are common -This is especially valuable for coordinate system lookups where it's common and expected that some systems may not be available for all elements. +This is especially valuable for coordinate system lookups, where it is common +and expected that some systems may not be available for all elements. + +## Parquet I/O (not yet migrated) + +Parquet loading in `VTableSource` / `VPointsSource` is a typical example of the +inconsistency above: exceptions, `null`, and skip-on-miss policies coexist. +Call sites disagree on whether a missing part should throw or be skipped. We are +keeping this as-is for now. See +[`docs/plans/parquet-io-error-handling.md`](../../plans/parquet-io-error-handling.md) +for current behaviour, rationale, and a suggested migration order if we expand +`Result` use. diff --git a/docs/docs/core/overview.mdx b/docs/docs/core/overview.mdx index 5d79c858..2c4f626a 100644 --- a/docs/docs/core/overview.mdx +++ b/docs/docs/core/overview.mdx @@ -4,7 +4,7 @@ sidebar_position: 1 # Core Package Overview -The `@spatialdata/core` package provides a TypeScript/JavaScript interface for reading and working with [SpatialData](https://spatialdata.scverse.org/en/stable/) Zarr stores. It mostly mirrors the Python library's API design while adapting to TypeScript idioms and the asynchronous nature of browser-based data loading. Hot-paths should make use of WASM with safe and ergonomic TypeScript interfaces where appropriate. +The `@spatialdata/core` package provides a TypeScript/JavaScript interface for reading and working with [SpatialData](https://spatialdata.scverse.org/en/stable/) Zarr stores. It mostly mirrors the Python library's API design while adapting to TypeScript idioms and the asynchronous nature of browser-based data loading. Hot-paths should make use of WASM with safe and ergonomic TypeScript interfaces where appropriate. For browser apps, enable the [points worker](../vis/browser-workers) at startup to keep parquet decode and scans off the main thread. The bundle should be tree-shakeable and avoid loading heavy dependencies before they are needed (please raise an issue if this is found not to be the case - it's not something that's been given much attention in the initial implementation). diff --git a/docs/docs/intro.mdx b/docs/docs/intro.mdx index 6fa587eb..e54abf06 100644 --- a/docs/docs/intro.mdx +++ b/docs/docs/intro.mdx @@ -149,6 +149,7 @@ that smoke test. - [SpatialCanvas status and roadmap](./vis/spatial-canvas-status) — what works, known gaps - [Visualization overview](./vis/overview) — package topology (`vis`, `layers`, `avivatorish`) - [MDV integration](./vis/mdv-integration) — phased MDV/Vitessce alignment +- [Browser workers](./vis/browser-workers) — worker setup and Vite bundler config for browser apps - [MDV release checklist](./vis/mdv-release-checklist) — branch shipping criteria ### Headless vis example diff --git a/docs/docs/vis/browser-workers.mdx b/docs/docs/vis/browser-workers.mdx new file mode 100644 index 00000000..36bd3b26 --- /dev/null +++ b/docs/docs/vis/browser-workers.mdx @@ -0,0 +1,111 @@ +--- +sidebar_position: 7 +--- + +# Browser workers + +SpatialData.js offloads CPU-heavy browser work to Web Workers so the main thread +stays responsive during parquet decode, zarr chunk decode, and related scans. +**Workers are generally recommended for browser apps**, even when you are not +using JP2K, HTJ2K, or other special image codecs. + +This page is the integration guide for MDV, Vitessce adapters, and custom Vite +apps. Package READMEs link here for bundler setup details. + +## When workers matter (and the tradeoff) + +Workers move expensive work off the UI thread. Without them, parquet decode, +feature catalog scans, Morton tiling, and routine zarr chunk reads can block +pan/zoom and other interaction. + +| Worker path | Enable API | Typical use | +|-------------|------------|-------------| +| **Points worker** | `enablePointsWorker()` from `@spatialdata/core` | Parquet geometry decode, feature filtering, catalog/count scans, Morton viewport tiles | +| **Chunk-decode pool** | `enableWorkerChunkDecode()` from `zarrextra/workers` | Zarr chunk decode via [fizarrita](https://www.npmjs.com/package/@fideus-labs/fizarrita); **required** for JP2K/HTJ2K in the browser, also helpful for other codecs on large or latency-sensitive datasets | + +**Cost:** worker scripts include WASM and codec dependencies, increasing bundle +and download size. Finer-grained opt-in worker bundles (for example lighter +workers without JP2K/HTJ2K) are planned for a future release. + +**Node / CI:** register codecs on the main thread (`registerJpeg2kCodec()`, +`registerExperimentalHtj2kCodec()` from `zarrextra`). Worker enable functions +are browser-oriented. + +## Call once at app startup + +Enable each worker path **once** from an app entry module (`main.tsx`, MDV +bootstrap, etc.) — not from render paths, data loaders, or unguarded +`useEffect` hooks. + +Bare `enablePointsWorker()` and `enableWorkerChunkDecode()` are **not** +idempotent. A second call tears down the existing worker(s), rejects in-flight +points-worker requests (`"Points worker disabled"`), and terminates the codec +pool before recreating it. + +Safe patterns: + +- Module-level `ensure*` wrapper with a flag (see the vis demo `ensureDemoPointsWorker`). +- `ensurePointsWorker()` from `@spatialdata/core` (no-op when already enabled). + +React StrictMode runs effects twice in development. If enable runs inside an +effect without a guard, workers may be torn down and recreated on mount. + +Example (vis demo style): + +```typescript +import { enablePointsWorker } from '@spatialdata/core'; +import { enableWorkerChunkDecode } from 'zarrextra/workers'; + +let workersReady = false; + +export function ensureBrowserWorkers() { + if (workersReady || typeof Worker === 'undefined') { + return; + } + enableWorkerChunkDecode(); + enablePointsWorker(); + workersReady = true; +} +``` + +Call `ensureBrowserWorkers()` once from your app entry module before loading +SpatialData. + +## Default worker URLs + +Call bare `enable*()` with **no `workerUrl` override** in production apps. +Packages ship compiled worker artifacts resolved by default: + +- `@spatialdata/core` → `dist/points-worker.js` (export: `@spatialdata/core/points-worker`) +- `zarrextra` → `dist/codec-worker.js` (export: `zarrextra/codec-worker`) + +Do **not** point `workerUrl` at monorepo source `.ts` paths. That works only in +local dev against unpublished sources and breaks production builds where `.ts` +files are not shipped. + +## Vite integration + +Vite apps should exclude `zarrextra/workers` from dependency pre-bundling: + +```ts +// vite.config.ts +export default defineConfig({ + optimizeDeps: { + exclude: ['zarrextra/workers'], + }, +}); +``` + +After upgrading `zarrextra` or `@spatialdata/core` when worker packaging changes, +delete `node_modules/.vite` to clear stale prebundles. + +The points worker default uses Vite's static `new Worker(new URL(..., import.meta.url))` +pattern when no override is passed. The codec worker pool passes a runtime URL to +fizarrita; a future `zarrextra/workers/vite` bundler entry may wrap +`?worker&url` resolution for Vite consumers. + +## Related docs + +- [Codec fixture guide](./codec-fixtures) — JP2K/HTJ2K fixtures and worker-backed decode in the vis demo +- [MDV integration](./mdv-integration) — embedding `SpatialCanvasViewer` +- [Core package overview](../core/overview) — WASM hot paths and element loaders diff --git a/docs/docs/vis/codec-fixtures.mdx b/docs/docs/vis/codec-fixtures.mdx index dcb34fa3..c1a0f104 100644 --- a/docs/docs/vis/codec-fixtures.mdx +++ b/docs/docs/vis/codec-fixtures.mdx @@ -30,7 +30,9 @@ pnpm --filter @spatialdata/vis dev Open [http://127.0.0.1:5173/codec](http://127.0.0.1:5173/codec). -The route calls `enableWorkerChunkDecode()` from `zarrextra/workers`, which uses +The route calls `enableWorkerChunkDecode()` from `zarrextra/workers` (see +[Browser workers](./browser-workers) for setup, Vite config, and when to enable +workers), which uses [`@fideus-labs/fizarrita`](https://www.npmjs.com/package/@fideus-labs/fizarrita) with a custom codec worker that registers JP2K and experimental HTJ2K support inside the worker before decode. Use the codec selector on the page to switch between `/test-fixtures/codecs/jpeg2k.zarr`, @@ -182,6 +184,7 @@ isolation, but normal user-facing `uv run` commands do not need it. - `registerJpeg2kCodec()` for the registered `imagecodecs_jpeg2k` id (Node/CI). - `enableWorkerChunkDecode()` from `zarrextra/workers` for browser apps (fizarrita worker pool + custom codec worker with JP2K and experimental HTJ2K registration). + See [Browser workers](./browser-workers) for Vite setup and integration guidance. - `registerExperimentalHtj2kCodec()` for OpenJPH HTJ2K decode (`experimental.openjph_htj2k` and legacy `experimental.imagecodecs_htj2k`) in Node/CI smoke tests. - `loadOmeZarrMultiscalesFromStore()` for loading multiscales from a diff --git a/docs/docs/vis/mdv-integration.mdx b/docs/docs/vis/mdv-integration.mdx index 7078d7fe..1c7e6669 100644 --- a/docs/docs/vis/mdv-integration.mdx +++ b/docs/docs/vis/mdv-integration.mdx @@ -70,6 +70,7 @@ Acceptance signal: demos should import the same public API MDV uses (`SpatialCan - [ ] Residual extension audit on both sides after MDV merge (shader-input paths, channel-count assumptions, extension prop passthrough through `deckProps` / `SpatialCanvasViewer`). - [ ] Run a clean `pnpm build` and pack the packages, then install them into `~/code/www/MDV`. - [ ] Add one minimal MDV smoke chart that imports the package and renders a known fixture before attempting a full chart replacement. +- [ ] Enable browser workers per [Browser workers](./browser-workers) (points worker + chunk-decode pool where JP2K/HTJ2K or heavy zarr decode is used). ## Phase 1: headless `SpatialCanvas` API diff --git a/docs/docs/vis/mdv-release-checklist.mdx b/docs/docs/vis/mdv-release-checklist.mdx index b1e45dbb..63d0cbe2 100644 --- a/docs/docs/vis/mdv-release-checklist.mdx +++ b/docs/docs/vis/mdv-release-checklist.mdx @@ -13,6 +13,9 @@ Scope constraints: - ensure MDV can compose its own deck layers and state cleanly alongside this library - avoid app-specific abstractions leaking into `@spatialdata/*` packages +Browser worker setup (points parquet offload, zarr chunk-decode pool, Vite +`optimizeDeps` contract): [Browser workers](./browser-workers). + ## What "ready for MDV" means A version is MDV-ready when MDV can embed `SpatialCanvasViewer` as a headless renderer while preserving existing chart behavior from: diff --git a/docs/package.json b/docs/package.json index 8fcfcf3f..d899453e 100644 --- a/docs/package.json +++ b/docs/package.json @@ -20,7 +20,6 @@ "@docusaurus/preset-classic": "3.9.1", "@mdx-js/react": "^3.0.0", "clsx": "^2.0.0", - "parquet-wasm": "catalog:", "prism-react-renderer": "^2.3.0", "react": "catalog:", "react-dom": "catalog:", diff --git a/docs/plans/parquet-io-error-handling.md b/docs/plans/parquet-io-error-handling.md new file mode 100644 index 00000000..e627b479 --- /dev/null +++ b/docs/plans/parquet-io-error-handling.md @@ -0,0 +1,46 @@ +# Parquet I/O error handling — follow-up + +**Status:** deferred (exceptions kept for now) +**Last updated:** 2026-06-23 +**Related:** [Error handling](../docs/core/error-handling.mdx), `VTableSource.ts` + +Example of limited `Result` adoption — see the error-handling doc for the +general picture. This note is only about parquet I/O. + +## Current state + +Parquet loading mixes three patterns: + +| Pattern | Example | Semantics | +|--------|---------|-----------| +| `null` | `loadParquetFileBytesAtPath` | Missing or invalid bytes (store miss, non-parquet payload) | +| `throw` | `readParquetDatasetBytesCapped`, `loadMultipartParquetTable` | Required bytes unavailable — fail the operation | +| `continue` | `_loadParquetTableUncachedCapped`, `VPointsSource` feature-filter scans | Skip a part and try the rest | + +`partPaths` is built from dataset metadata (footer/range reads) or +`discoverMultipartPartPaths` (full-byte probe). Metadata paths are **not** +guaranteed loadable via `loadParquetFileBytesAtPath`; discovered paths were +verified moments earlier in the same call. + +## Intentional strictness difference + +`readParquetDatasetBytesCapped` **throws** on the first missing part because it +feeds worker decode paths that need reliable byte buffers. Sibling table loaders +use **`continue`** so a later part can still contribute rows. That is a policy +choice, not probing of paths known to be absent. + +## Follow-up (when revisiting) + +1. Decide whether to expand `Result` at all; if so, evaluate an established + library (e.g. `neverthrow`) rather than extending the in-house `zarrextra` + types. +2. Introduce typed errors (e.g. missing part, invalid bytes, empty dataset). +3. Move `loadParquetFileBytesAtPath` to `Result` first; keep a thin `null` shim + if needed during migration. +4. Migrate protected helpers (`readParquetDatasetBytesCapped`, multipart + loaders) and unify skip-vs-fail policy per call site. +5. Leave public APIs (`loadParquetTable`, `loadPoints`, …) throwing until vis / + layers need typed degradation; use `unwrap()` at boundaries meanwhile. + +See [Error handling](../docs/core/error-handling.mdx) for the current provisional +`Result` API and adoption patterns (`getTransformation`). diff --git a/docs/plans/points-preload-feature-filter-status.md b/docs/plans/points-preload-feature-filter-status.md new file mode 100644 index 00000000..33f6f561 --- /dev/null +++ b/docs/plans/points-preload-feature-filter-status.md @@ -0,0 +1,368 @@ +# Points preload & feature filter — status and plan + +**Status:** work in progress (branch/worktree, not yet on `main`) +**Last updated:** 2026-06-20 +**Related:** [ADR 0002](../adr/0002-spatially-aware-vector-loading.md), [ADR 0003](../adr/0003-points-render-resource.md) + +This document captures what we built, what broke, what we fixed, and what still +needs cleanup — especially around **workers**, **parquet I/O**, and the **~30s +main-thread catalog load** on large Xenium `transcripts`. + +--- + +## Problem statement + +On ~12M-row Xenium `transcripts`: + +1. **Feature filter was unusably slow** — every checkbox toggle re-scanned the + full parquet dataset via `loadPoints({ featureCodes })`. +2. **Parquet was used incorrectly** for large reads — whole part files fetched + and decoded instead of row-group range reads with column projection. +3. **Feature catalog UI** failed or showed a single blank gene on the normal + `transcripts` element (dictionary-encoded `feature_name`, no separate codes + column). + +--- + +## Target architecture questions (current intent) + +The current implementation separates three concerns for the preloaded scatter +path: + +| Concern | When | Where | +|---------|------|--------| +| **Geometry preload** | Once per `(element, memoryCap)` | `loadPoints` → x/y only, capped (default 4M rows) | +| **Runtime feature filter** | Every checkbox toggle | `PointsLayer` → in-memory filter on preloaded batch | +| **Feature catalog** | Once per element (UI gene list) | `listFeatures` → feature columns only, not x/y | + +This is a useful strategy when a bounded preload fits comfortably in memory and +the user wants fast toggling across a moderate number of visible points. It is +not the general architecture for all points stores. + +The broader architecture should support multiple point loading strategies: + +| Strategy | Filter-change behavior | Runtime batch/layout | Good fit | +|----------|------------------------|----------------------|----------| +| **Preloaded scatter** | Does not reload geometry; filters an in-memory capped batch | `columnar-ndarray` today; possible Arrow/GeoArrow batch later | Moderate point counts, exploratory toggling, local responsiveness | +| **Spatial Morton tiles** | Reloads viewport tiles when filter props change | `columnar-ndarray` tile batches today; possible GeoArrow tile batches later | Spatial navigation where row groups are primarily spatial | +| **Feature-primary or compound index** | Intentionally loads new data for selected features | Same loader contract; likely benefits from Arrow/GeoArrow columnar batches | Looking at a few genes/features out of thousands without keeping all features in memory | + +The open design question is how the resolver chooses among these strategies and +how writers advertise their indexes. We should not assume that every feature +filter is a view over already-loaded data. + +### Runtime batch/layout direction + +Persisted optimized points remain Parquet-backed for now. GeoArrow is relevant +as a **runtime columnar layout** and deck.gl integration boundary, not as a +separate persisted copy of the same data. The `PointsLoader` / `PointsBatch` +contract should be able to return Arrow-ish or GeoArrow-compatible batches +later, while `@spatialdata/layers` owns deck.gl-geoarrow adaptation. This keeps +`@spatialdata/core` deck-free and lets each strategy evolve from current +`columnar-ndarray` batches toward GeoArrow where that proves faster or simpler. + +``` +┌─────────────────────────────────────────────────────────────────┐ +│ Vis (useLayerData) │ +│ ├─ loadPoints({ memoryCap }) once, key = element|m{cap} │ +│ ├─ loadRowFeatureCodes({ cap }) after preload, aligned rows │ +│ └─ listFeatures() catalog for filter panel │ +└─────────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────────┐ +│ PointsLayer (preloaded scatter path) │ +│ ├─ preloadedBatch from render resource │ +│ ├─ preloadedFeatureCodes from useLayerData ref │ +│ └─ featureCodes prop from layer config (checkbox state) │ +│ → filterPreloadedBatch (worker when enabled) │ +└─────────────────────────────────────────────────────────────────┘ +``` + +Morton-tiled elements (`transcripts_morton`, etc.) use viewport tiles + +`featureCodes` in `loadPointsInBounds` per tile — no full-table preload. + +Feature-primary or compound-indexed stores are still experimental. They may make +filter changes part of the structural load key because the point of the index is +to fetch only selected features. + +--- + +## What we changed (summary) + +### Vis (`@spatialdata/vis`) + +- **`pointsPreloadCacheKey`** — for the current preloaded scatter path, memory + cap only; no `featureCodes` in key. +- **`loadPoints`** — no longer passes `featureCodes`; filter does not reload + geometry on this path. +- **`loadRowFeatureCodes`** — separate effect after preload; keyed by + `preloadCacheKey`; passed to `PointsLayer` as `preloadedFeatureCodes`. +- **Catalog cache** — retry when cached value is `null` (failed load), not when + a valid catalog exists. +- Removed filter-reload machinery (`beginPointsFilterReload`, + `resolveRenderablePointsPreload`, etc.). + +### Core (`@spatialdata/core`) + +- **`loadPoints`** — ignores `featureCodes` unless + `fullDatasetFeatureScan: true` (opt-in benchmark path, not used by vis). +- **`loadParquetTableCapped`** — prefers **row-group range reads** + + column projection when store supports `getRange`. +- **`listPointsFeatures`** (large datasets) — feature-column scan with + `readParquet` fallback when row-group decode is broken for dictionary columns. +- **Dictionary catalog helpers** — safe index extraction, merge across chunks, + `featureCatalogNeedsParquetFallback` (empty or all-blank names). + +### Layers (`@spatialdata/layers`) + +- **`PointsLayer`** — async filtered-batch cache in `updateState`; filter + signature includes `featureCodes`, `preloadedFeatureCodes`, `renderCap`. + +--- + +## Workers: what they do today + +The points worker is **enabled in the vis demo** via +`packages/vis/demo/src/enableDemoPointsWorker.ts`. + +| Operation | Worker? | Notes | +|-----------|---------|-------| +| Feature filter on preloaded batch | **Yes** | `filterColumnarByFeatureCodesInWorker` in `PointsLayer` | +| Geometry preload (`loadPoints`) | **Yes** | `decodeParquetGeometryCappedInWorker`; main-thread fallback | +| Row feature codes (`loadRowFeatureCodes`) | **Yes** | Worker decode via row-group bytes or part bytes; main-thread fallback | +| Feature catalog (large, dict-only) | **Yes** | `scanParquetFeatureCatalogInWorker`; dict fallback via full parts in worker | +| Feature counts | **Yes** | `scanParquetFeatureCountsInWorker` (row groups or parts); main-thread fallback | +| Morton viewport tiles | **Yes** | `scanMortonRowGroupsInBoundsInWorker`; main-thread fallback | +| Opt-in full-dataset filter scan | **Yes** | `scanParquetByFeatureCodesInWorker` (row groups or parts) | + +**Takeaway:** parquet decode and table scans run on the points worker when enabled; +main thread does metadata resolution and async byte-range I/O only. + +--- + +## Parquet I/O paths + +### Good (row-group + projection) + +- Morton viewport tiles: `loadParquetRowGroupByGroupIndex` + `store.getRange`. +- Geometry preload: `loadParquetTableCapped` → `_loadParquetTableRowGroupsCapped` + when range reads work. +- Catalog on **`transcripts_morton`** (has `feature_name_codes`): row-group + scan of feature columns only — fast (~hundreds of genes). + +### Bad / fallback (full-file or full-column decode) + +- **`loadParquetFileBytesAtPath`** still used in capped multipart fallback and + opt-in `loadPointsMatchingFeatureCodes` / `loadFeatureCounts` worker paths. +- **Catalog on plain `transcripts`**: row-group reads do not decode + dictionary-encoded `feature_name` correctly (empty names, single bogus + entry). Fallback is **`loadParquetTable(parquetPath, [feature_name])` over all + parts** — correct gene list, **~30s main-thread block** on 12M rows. + +### Debug evidence (Xenium) + +| Path | Catalog result | Mechanism | +|------|----------------|-----------| +| `transcripts` | ~30s, works after fallback | Dict-only; full feature-column read | +| `transcripts_morton` | Fast, ~541 genes | `feature_name_codes` + row-group scan | +| Row-group dict merge | 0 entries | `rowGroupsWithDictionary: 0` | +| Row-group scan (dict-only) | 1 blank entry | Indices without dictionary array | +| `RangeError: offset is out of bounds` | Catalog null | Fixed via safe `getDictionaryIndexAt` | + +--- + +## Feature catalog bug timeline (why it was confusing) + +1. Large datasets only built catalog from dictionary if a **1-row capped probe** + worked → Arrow slice drops dictionary values → `null` catalog. +2. Feature-column row-group scan for dict-only columns → **one entry, empty + name** (collapsed `nameToCode` map). +3. Dictionary row-group merge → **0 entries** (WASM row-group read not + dictionary-typed the way we expected). +4. **Working fix:** skip row-group scan when no `feature_name_codes`; if catalog + empty or all names blank → **`readParquet` full feature-column load**. + +This fixed the UI but introduced the main-thread stall. + +--- + +## Mental model: which path am I on? + +``` +transcripts (12M, dictionary feature_name, NO feature_name_codes) + ├─ preload: row-group x/y (capped 4M) main thread, moderate + ├─ row codes: feature cols via loadParquetTableCapped main thread, moderate + ├─ catalog: FULL readParquet [feature_name] main thread, ~30s ← pain point + ├─ counts: hidden until an explicit code/name mapping is available + └─ filter toggle: worker in-memory on preloaded batch fast, but capped + +transcripts_morton (feature_name_codes + morton_code_2d) + ├─ render: Morton TileLayer, viewport-bounded + ├─ catalog: row-group feature columns fast + └─ filter: per-tile featureCodes in getTileData + +future feature-primary / compound-indexed store + ├─ render: query selected features, possibly viewport-bounded + ├─ filter: changes load key and fetches new rows + └─ goal: avoid loading thousands of genes when viewing a few +``` + +--- + +## Current status + +### Working + +- Feature filter toggles are **instant** on the current preloaded scatter path + (no parquet rescan). +- Geometry preload uses row-group reads where the store supports them. +- Morton / coded elements get a reasonable catalog quickly. +- Plain `transcripts` catalog **populates** (after expensive fallback). +- Caps: memory cap (preload), render cap (draw), separate concerns. +- Tests: core 106, vis 65 (as of 2026-06-20). + +### Not ideal / known debt + +1. **Catalog for dict-only large datasets** — full-table feature-column read; + blocks main thread ~30s; not justified long-term. +2. **Worker policy inconsistent** — filter off-thread; preload/catalog on-thread. +3. **`loadFeatureCounts`** — counts are hidden unless code/name mapping is + explicit. Wrong counts are worse than missing counts. +4. **Strategy selection is unresolved** — preloaded in-memory filtering is one + useful path, but feature-primary or compound-indexed stores may intentionally + reload data when filters change. +5. **Legacy / dead-ish paths** — `loadPointsMatchingFeatureCodes`, + `decodeParquetPartsInWorker`, `fullDatasetFeatureScan` (opt-in, no UI). +6. **Row-group WASM + dictionary columns** — broken for catalog; we paper over + with full read; root cause not fixed in the reader layer. + +--- + +## Cleanup plan (prioritized) + +### P0 — Remove the 30s catalog stall (plain `transcripts`) + +Pick one or combine: + +| Approach | Effort | Notes | +|----------|--------|-------| +| **Dictionary from parquet metadata** | Medium | Read dictionary pages / schema without scanning 12M rows; ideal for Xenium | +| **Worker-backed catalog build** | Low–medium | Same bytes as today, off main thread; doesn't reduce total work | +| **Cache catalog per element** | Low | IndexedDB or in-memory; first visit still slow | +| **Writer: always emit `feature_name_codes`** | Medium | Aligns with morton path; Python writer change | +| **Sidecar gene list** | Low | Small JSON/parquet in element attrs (non-standard) | + +**Recommendation:** metadata/dictionary-page fast path first; worker offload as +a quick win if metadata path is hard in parquet-wasm. + +### P1 — Unify worker policy + +Document and implement one rule, e.g.: + +> All parquet decode and row scans run in the points worker; main thread only +> marshals Arrow IPC and deck props. + +Or explicitly drop worker for decode and accept main-thread decode with +chunking/`requestIdleCallback` — but be consistent. + +### P2 — Defer non-critical work + +- Show catalog from dictionary-only fast path **without counts** first. +- Load `loadFeatureCounts` only when user opens filter panel or on idle. +- Don't block first render on catalog (already partially true). + +### P3 — Trim legacy paths + +- Remove or gate `fullDatasetFeatureScan` unless needed for benchmarks. +- Audit `loadPointsMatchingFeatureCodes` vs runtime filter. The answer may be + different per strategy: preloaded scatter filters in memory, while + feature-indexed stores may use source-side feature queries. +- Remove unused worker decode entry points if nothing calls them. + +### P4 — Fix row-group dictionary decode (proper parquet) + +- Investigate parquet-wasm `readParquetRowGroup` + column projection for + dictionary columns on Xenium multipart layout. +- Goal: row-group catalog path works for dict-only `feature_name` without full + table read. + +### Future note — DuckDB / DuckDB-Wasm + +DuckDB is not part of the current render path. It may become useful later as a +correctness oracle for Parquet scans, an offline writer/benchmark validation +tool, or a worker-backed catalog/count query engine. Do not add it to browser +tile loading without a separate benchmark and bundle-size decision. + +--- + +## API contracts (for reference) + +### Vis preload cache key for preloaded scatter + +``` +{elementKey}|m{memoryCap} +``` + +Feature filter is **not** part of this key for the current preloaded scatter +path. A future feature-primary or compound-indexed strategy may include selected +features in its structural load key. + +### `featureCodes` semantics + +| Value | Meaning | +|-------|---------| +| `undefined` | All features | +| `[]` | No features | +| `[1, 2, 3]` | Subset | + +Feature codes must be in the same global code/name space as the feature +catalog. Explicit `{feature_key}_codes` columns are authoritative. For +dictionary-only feature columns, raw dictionary indices are local to a +chunk/row group/part and must not be treated as global feature codes; derive +filter row codes from decoded names using the catalog mapping instead. + +### Large-dataset catalog strategy (`listPointsFeatures`) + +1. If `feature_name_codes` (or `{feature_key}_codes`) present → row-group scan + of feature columns. +2. Else if catalog empty or all blank names after row groups → + `loadParquetTable` feature columns only (current fallback). +3. Small datasets → full `loadParquetTable` with feature columns (unchanged). + +--- + +## Files touched (main areas) + +| Area | Files | +|------|-------| +| Core load/filter | `packages/core/src/models/VPointsSource.ts`, `VTableSource.ts` | +| Feature catalog | `packages/core/src/pointsFeatures.ts` | +| Worker | `packages/core/src/workers/points-worker.ts`, `pointsWorkerClient.ts` | +| Vis preload/filter | `packages/vis/src/SpatialCanvas/useLayerData.ts`, `pointsLoadPlan.ts` | +| Layer filter cache | `packages/layers/src/PointsLayer.ts` | +| UI | `packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx` | + +--- + +## Open questions + +1. Is a one-time 30s catalog acceptable if cached for the session, or must + first open be sub-second? +2. Should we require Morton + `feature_name_codes` for large transcript datasets + in production, treating plain `transcripts` as legacy? +3. Should catalog/counts move entirely to the worker before further vis work? +4. What metadata should writers emit so the resolver can distinguish spatial + Morton, feature-primary, and compound spatial+feature indexes? +5. When should feature filter changes reload source data instead of filtering a + preloaded batch? + +--- + +## Changelog (this effort) + +- Runtime feature filter on preloaded scatter (no geometry reload on toggle). +- Row-group capped reads for geometry preload. +- Feature catalog fixes for dictionary-encoded large datasets (fallback read). +- Safe dictionary index extraction; catalog retry on `null` cache. +- Removed debug instrumentation (2026-06-20). diff --git a/packages/core/package.json b/packages/core/package.json index a7ebf504..c99bd5df 100644 --- a/packages/core/package.json +++ b/packages/core/package.json @@ -9,10 +9,18 @@ ".": { "types": "./dist/index.d.ts", "import": "./dist/index.js" + }, + "./workers": { + "types": "./dist/workers/index.d.ts", + "import": "./dist/workers/index.js" + }, + "./points-worker": { + "import": "./dist/points-worker.js" } }, "files": [ - "dist" + "dist", + "vendor" ], "publishConfig": { "access": "public" @@ -31,7 +39,6 @@ "anndata.js": "catalog:", "apache-arrow": "catalog:", "ol": "^10.6.1", - "parquet-wasm": "catalog:", "zarrita": "catalog:", "zod": "catalog:" }, diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts index c5dd569e..92c09091 100644 --- a/packages/core/src/index.ts +++ b/packages/core/src/index.ts @@ -9,6 +9,44 @@ export * from './types.js'; export * from './store/index.js'; export * from './models/index.js'; export * from './spatialViewFit.js'; +export * from './pointsTiling.js'; +export { mergeFeatureCountsIntoCatalog } from './pointsFeatures.js'; +export { + POINTS_PRELOAD_MAX_ROWS, + DEFAULT_POINTS_MEMORY_CAP, + DEFAULT_POINTS_RENDER_CAP, + PointsPreloadTooLargeError, + applyRenderCapToColumnar, + exceedsPointsPreloadLimit, + pointsPreloadTruncatedMessage, + pointsFilteredMemoryCapMessage, + preloadedColumnarPointCount, + resolvePointsMemoryCap, + resolvePointsRenderCap, +} from './pointsLimits.js'; +export type { PointsLoadOptions, PointsLoadProgress, PointsLoadResult } from './pointsLoadOptions.js'; +export { + enablePointsWorker, + disablePointsWorker, + ensurePointsWorker, + filterColumnarByFeatureCodesInWorker, + isPointsWorkerEnabled, + setPointsWorkerDefaultEnabled, +} from './workers/index.js'; +export { + createMortonTiledPointsLoader, + createPointsLoaderForElement, + createPreloadedColumnarPointsLoader, + resolvePointsEncoding, + type CorePointsLoader, + type ColumnarNdarrayPointsBatch, + type PreloadedColumnarInput, + type PointsBatch, + type PointsBatchFormat, + type PointsEncodingKind, + type PointsLoadInBoundsOptions, + type PointsLoaderCapabilities, +} from './pointsLoader.js'; export * from './shapes.js'; export { inferShapesGeometryKindFromParquet, diff --git a/packages/core/src/models/VPointsSource.ts b/packages/core/src/models/VPointsSource.ts index 03d19743..968aeeb4 100644 --- a/packages/core/src/models/VPointsSource.ts +++ b/packages/core/src/models/VPointsSource.ts @@ -1,5 +1,94 @@ -import type { Axis } from '../schemas'; import { basename } from '../Vutils'; +import { + buildFeatureCatalogFromColumns, + featureCodeMapFromCatalog, + mergeFeatureCountsIntoCatalog, + resolveRowFeatureCodesFromTable, +} from '../pointsFeatures.js'; +import { + decodeParquetGeometryCappedInWorker, + decodeParquetRowFeatureCodesInWorker, + ensurePointsWorker, + isPointsWorkerEnabled, + scanMortonRowGroupsInBoundsInWorker, + scanParquetByFeatureCodesInWorker, + scanParquetFeatureCatalogInWorker, + scanParquetFeatureCountsInWorker, +} from '../workers/pointsWorkerClient.js'; +import { exceedsPointsPreloadLimit, resolvePointsMemoryCap } from '../pointsLimits.js'; +import type { + PointsLoadOptions, + PointsLoadProgress, + PointsLoadResult, +} from '../pointsLoadOptions.js'; + +interface ColumnarPointsChunk { + shape: number[]; + data: ArrayLike[]; +} + +function emptyFilteredPointsResult(axisNames: string[], totalRowCount: number): PointsLoadResult { + const hasZ = axisNames.includes('z'); + const empty = new Float32Array(0); + const data = hasZ ? [empty, empty, empty] : [empty, empty]; + return { + shape: [data.length, 0], + data, + totalRowCount, + scannedRowCount: 0, + filterActive: true, + }; +} + +function toColumnarPointsChunk( + data: { shape?: number[]; data: ArrayLike[] }, + axisCount: number +): ColumnarPointsChunk { + const rowCount = data.shape?.[1] ?? data.data[0]?.length ?? 0; + return { + shape: data.shape ?? [axisCount, rowCount], + data: data.data, + }; +} + +function concatColumnarPointChunks(chunks: ColumnarPointsChunk[]): ColumnarPointsChunk { + if (chunks.length === 0) { + const empty = new Float32Array(0); + return { shape: [2, 0], data: [empty, empty] }; + } + const axisCount = chunks[0].shape[0] ?? chunks[0].data.length; + const totalRows = chunks.reduce( + (sum, chunk) => sum + (chunk.shape[1] ?? chunk.data[0]?.length ?? 0), + 0 + ); + const data = Array.from({ length: axisCount }, (_, axisIndex) => { + const merged = new Float32Array(totalRows); + let offset = 0; + for (const chunk of chunks) { + const column = chunk.data[axisIndex]; + if (!column) { + continue; + } + const values = column instanceof Float32Array ? column : Float32Array.from(column); + merged.set(values, offset); + offset += values.length; + } + return merged; + }); + return { shape: [axisCount, totalRows], data }; +} +import { + MORTON_CODE_2D_COLUMN, + type PointsInBoundsOptions, + type PointsInBoundsResponse, + type PointsFeatureCatalog, + type PointsTilingMetadata, + extractSentinelBoundingBox, + featureCodeAllowSet, + filterPointsToBounds, + mortonIntervalsForBounds, +} from '../pointsTiling.js'; +import type { Axis } from '../schemas'; // import { normalizeAxes } from '@vitessce/spatial-utils'; import SpatialDataTableSource from './VTableSource'; @@ -68,7 +157,42 @@ function getParquetPath(arrPath?: string) { throw new Error(`Cannot determine parquet path for points array path: ${arrPath}`); } +function arrowSchemaFieldNames(table: { schema: { fields?: Array<{ name?: unknown }> } } | null) { + return ( + table?.schema.fields?.flatMap((field) => + typeof field.name === 'string' ? [field.name] : [] + ) ?? [] + ); +} + +function selectFeatureCodeColumn(fields: string[], featureKey: string | undefined) { + const candidates = [ + featureKey ? `${featureKey}_codes` : undefined, + 'feature_name_codes', + 'feature_index', + ].filter((value): value is string => typeof value === 'string'); + return candidates.find((candidate) => fields.includes(candidate)); +} + +function checkAbort(signal?: AbortSignal) { + if (signal?.aborted) { + throw new DOMException('The operation was aborted.', 'AbortError'); + } +} + +function rowGroupCountForIndex(metadata: PointsTilingMetadata, rowGroupIndex: number) { + if (rowGroupIndex < 0 || rowGroupIndex >= metadata.totalRowGroups) { + return 0; + } + return metadata.rowGroupRowCounts?.[rowGroupIndex] ?? metadata.maxRowsPerGroup; +} + export default class SpatialDataPointsSource extends SpatialDataTableSource { + private readonly pointTilingMetadataCache = new Map< + string, + Promise + >(); + /** * * @param path A path to within shapes. @@ -126,23 +250,779 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { * shape: [number, number], * }>} A promise for a zarr array containing the data. */ - async loadPoints(elementPath: string) { + async loadPoints( + elementPath: string, + options: PointsLoadOptions = {} + ): Promise { + const memoryCap = resolvePointsMemoryCap(options.memoryCap); + if (options.featureCodes !== undefined && options.fullDatasetFeatureScan === true) { + return this.loadPointsMatchingFeatureCodes(elementPath, { + memoryCap, + featureCodes: options.featureCodes, + onProgress: options.onProgress, + }); + } + const parquetPath = getParquetPath(elementPath); + const zattrs = await this.loadSpatialDataElementAttrs(elementPath); + const { axes } = zattrs; + const normAxes = normalizeAxes(axes); + const axisNames = normAxes.map((axis: { name: string }) => axis.name); + const rowCount = await this.resolveParquetRowCount(parquetPath); + const truncatePreload = rowCount > memoryCap; + const maxRows = truncatePreload ? memoryCap : rowCount; + const columnNames = [...axisNames]; + + ensurePointsWorker(); + if (isPointsWorkerEnabled()) { + try { + const payload = await this.readParquetWorkerPayload(parquetPath, { maxRows }); + const workerGeometry = await decodeParquetGeometryCappedInWorker( + { + parts: payload.parts, + axisNames, + columns: columnNames, + maxRows, + } + ); + if (workerGeometry) { + return { + shape: workerGeometry.shape as [number, number], + data: workerGeometry.data, + totalRowCount: rowCount, + preloadTruncated: truncatePreload, + }; + } + } catch (error) { + console.warn( + `Worker geometry preload failed for ${elementPath}; falling back to main thread.`, + error + ); + } + } + + const { + table: arrowTable, + totalRows, + truncated, + } = await this.loadParquetTableCapped(parquetPath, columnNames, maxRows); + + const axisColumnArrs = axisNames.map((name: string) => { + const column = arrowTable.getChild(name); + if (!column) { + throw new Error(`Column "${name}" not found in the arrow table.`); + } + return column.toArray(); + }); + return { + shape: [axisColumnArrs.length, arrowTable.numRows], + data: axisColumnArrs, + totalRowCount: totalRows, + preloadTruncated: truncated, + }; + } + + private async loadPointsMatchingFeatureCodes( + elementPath: string, + options: { + memoryCap: number; + featureCodes: readonly number[]; + onProgress?: (progress: PointsLoadProgress) => void; + } + ): Promise { + ensurePointsWorker(); + const parquetPath = getParquetPath(elementPath); const zattrs = await this.loadSpatialDataElementAttrs(elementPath); const { axes, spatialdata_attrs: spatialDataAttrs } = zattrs; const normAxes = normalizeAxes(axes); - // todo - use type from schema? const axisNames = normAxes.map((axis: { name: string }) => axis.name); + const featureKey = spatialDataAttrs?.feature_key; + if (typeof featureKey !== 'string' || featureKey.length === 0) { + throw new Error(`Points element "${elementPath}" is missing feature_key metadata.`); + } + + const totalRowCount = await this.resolveParquetRowCount(parquetPath); + if (options.featureCodes.length === 0) { + return emptyFilteredPointsResult(axisNames, totalRowCount); + } + + if (!isPointsWorkerEnabled()) { + throw new Error( + 'Feature-filtered points loading requires the points worker and parquet part bytes.' + ); + } + + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + const schemaTable = datasetMetadata ? null : await this.loadParquetSchemaTable(parquetPath); + const fields = datasetMetadata?.schema?.fields + ? datasetMetadata.schema.fields.flatMap((field) => + typeof field.name === 'string' ? [field.name] : [] + ) + : arrowSchemaFieldNames(schemaTable); + const featureCodeColumnName = selectFeatureCodeColumn(fields, featureKey); + + const columnNames = [...axisNames]; + if (featureCodeColumnName && !columnNames.includes(featureCodeColumnName)) { + columnNames.push(featureCodeColumnName); + } else if (!columnNames.includes(featureKey)) { + columnNames.push(featureKey); + } + + const matchedChunks: ColumnarPointsChunk[] = []; + let matchedRows = 0; + let scannedRows = 0; + + const canUseRowGroups = await this.canLoadParquetRowGroups(); + const datasetRowGroups = datasetMetadata?.totalNumRowGroups ?? 0; + + if (canUseRowGroups && datasetRowGroups > 0) { + for (let rowGroupIndex = 0; rowGroupIndex < datasetRowGroups; rowGroupIndex += 1) { + if (matchedRows >= options.memoryCap) { + break; + } + const chunk = await this.readParquetRowGroupBytesByGroupIndex(parquetPath, rowGroupIndex); + if (!chunk) { + continue; + } + const partial = await scanParquetByFeatureCodesInWorker({ + rowGroups: [chunk], + axisNames, + featureKey, + featureCodeColumnName, + featureCodes: options.featureCodes, + memoryCap: options.memoryCap - matchedRows, + }); + if (!partial) { + throw new Error('Feature-filtered points loading requires the points worker.'); + } + scannedRows += partial.scannedRows; + if (partial.matchedRows > 0) { + matchedChunks.push(toColumnarPointsChunk(partial.data, axisNames.length)); + matchedRows += partial.matchedRows; + } + options.onProgress?.({ + scannedRows, + matchedRows, + partIndex: rowGroupIndex, + partCount: datasetRowGroups, + }); + } + } else { + let partPaths: string[]; + if (datasetMetadata?.parts.length && datasetMetadata.parts.length > 0) { + partPaths = datasetMetadata.parts.map((part) => part.path); + } else { + partPaths = [parquetPath]; + } + + for (let partIndex = 0; partIndex < partPaths.length; partIndex += 1) { + const partPath = partPaths[partIndex]; + if (matchedRows >= options.memoryCap) { + break; + } + const bytes = await this.loadParquetFileBytesAtPath(partPath); + if (!bytes || bytes.length === 0) { + continue; + } + const partial = await scanParquetByFeatureCodesInWorker({ + parts: [bytes], + axisNames, + featureKey, + featureCodeColumnName, + featureCodes: options.featureCodes, + memoryCap: options.memoryCap - matchedRows, + }); + if (!partial) { + throw new Error('Feature-filtered points loading requires the points worker.'); + } + scannedRows += partial.scannedRows; + if (partial.matchedRows > 0) { + matchedChunks.push(toColumnarPointsChunk(partial.data, axisNames.length)); + matchedRows += partial.matchedRows; + } + options.onProgress?.({ + scannedRows, + matchedRows, + partIndex, + partCount: partPaths.length, + }); + } + } + + const data = concatColumnarPointChunks(matchedChunks); + return { + shape: data.shape, + data: data.data, + totalRowCount, + scannedRowCount: scannedRows, + filterActive: true, + preloadTruncated: matchedRows >= options.memoryCap, + }; + } + + private async resolveExplicitFeatureCodeColumn(elementPath: string): Promise<{ + featureKey: string; + featureCodeColumnName: string; + } | null> { + const parquetPath = getParquetPath(elementPath); + const zattrs = await this.loadSpatialDataElementAttrs(elementPath); + const featureKey = zattrs.spatialdata_attrs?.feature_key; + if (typeof featureKey !== 'string' || featureKey.length === 0) { + return null; + } + + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + const schemaTable = datasetMetadata ? null : await this.loadParquetSchemaTable(parquetPath); + const fields = datasetMetadata?.schema?.fields + ? datasetMetadata.schema.fields.flatMap((field) => + typeof field.name === 'string' ? [field.name] : [] + ) + : arrowSchemaFieldNames(schemaTable); + const featureCodeColumnName = selectFeatureCodeColumn(fields, featureKey); + if (!featureCodeColumnName) { + return null; + } + return { featureKey, featureCodeColumnName }; + } + + async loadFeatureCounts(elementPath: string): Promise> { + const parquetPath = getParquetPath(elementPath); + const resolvedFeatureColumn = await this.resolveExplicitFeatureCodeColumn(elementPath); + if (!resolvedFeatureColumn) { + return new Map(); + } + const { featureKey, featureCodeColumnName } = resolvedFeatureColumn; + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + const columnNames = [featureKey]; + if (!columnNames.includes(featureCodeColumnName)) { + columnNames.push(featureCodeColumnName); + } + + const canUseRowGroups = await this.canLoadParquetRowGroups(); + const datasetRowGroups = datasetMetadata?.totalNumRowGroups ?? 0; + + ensurePointsWorker(); + if (isPointsWorkerEnabled()) { + try { + const payload = await this.readParquetWorkerPayload(parquetPath, { + maxRows: Number.POSITIVE_INFINITY, + fullPartsForFallback: true, + includeRowGroups: true, + }); + const workerCounts = await scanParquetFeatureCountsInWorker( + canUseRowGroups && datasetRowGroups > 0 && payload.rowGroups.length > 0 + ? { + rowGroups: payload.rowGroups, + featureKey, + featureCodeColumnName, + } + : { + parts: payload.parts, + featureKey, + featureCodeColumnName, + } + ); + if (workerCounts) { + return workerCounts; + } + } catch (error) { + console.warn( + `Worker feature counts failed for ${elementPath}; falling back to main thread.`, + error + ); + } + } + + if (canUseRowGroups && datasetRowGroups > 0) { + const counts = new Map(); + for (let rowGroupIndex = 0; rowGroupIndex < datasetRowGroups; rowGroupIndex += 1) { + const table = await this.loadParquetRowGroupByGroupIndex(parquetPath, rowGroupIndex, { + columns: columnNames, + }); + if (table && table.numRows > 0) { + const { scanTableFeatureCounts } = await import('../workers/pointsWorkerScan.js'); + scanTableFeatureCounts(table, featureKey, featureCodeColumnName, counts); + } + } + return counts; + } + + const { parts } = await this.readParquetDatasetBytesCapped( + parquetPath, + Number.POSITIVE_INFINITY + ); + + if (parts.length > 0) { + const workerCounts = await scanParquetFeatureCountsInWorker({ + parts, + featureKey, + featureCodeColumnName, + }); + if (workerCounts) { + return workerCounts; + } + } + + const rowCodes = await this.loadPointsRowFeatureCodes(elementPath); + if (!rowCodes) { + return new Map(); + } + const { countFeatureCodesHistogram } = await import('../pointsFeatures.js'); + return countFeatureCodesHistogram(rowCodes); + } + + async listPointsFeaturesWithCounts(elementPath: string): Promise { + const catalog = await this.listPointsFeatures(elementPath); + if (!catalog) { + return null; + } + try { + const counts = await this.loadFeatureCounts(elementPath); + return mergeFeatureCountsIntoCatalog(catalog, counts); + } catch (error) { + console.warn(`Failed to load feature counts for ${elementPath}:`, error); + return catalog; + } + } + + /** + * Load per-row feature codes aligned with {@link loadPoints} rows. Deferred from + * geometry preload so large datasets do not block the first render. Parquet decode + * and code extraction run on the points worker when enabled; falls back to the + * main thread when the worker is unavailable. + */ + async loadPointsRowFeatureCodes( + elementPath: string, + options: { + memoryCap?: number; + featureCatalog?: PointsFeatureCatalog | null; + } = {} + ): Promise | undefined> { + const parquetPath = getParquetPath(elementPath); + const zattrs = await this.loadSpatialDataElementAttrs(elementPath); + const { spatialdata_attrs: spatialDataAttrs } = zattrs; + const featureKey = spatialDataAttrs?.feature_key; + if (typeof featureKey !== 'string' || featureKey.length === 0) { + return undefined; + } + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + const schemaTable = datasetMetadata ? null : await this.loadParquetSchemaTable(parquetPath); + const fields = datasetMetadata?.schema?.fields + ? datasetMetadata.schema.fields.flatMap((field) => + typeof field.name === 'string' ? [field.name] : [] + ) + : arrowSchemaFieldNames(schemaTable); + const featureCodeColumnName = selectFeatureCodeColumn(fields, featureKey); + const featureCodeByName = featureCodeColumnName + ? undefined + : featureCodeMapFromCatalog( + options.featureCatalog !== undefined + ? options.featureCatalog + : await this.listPointsFeatures(elementPath) + ); + + const rowCount = await this.resolveParquetRowCount(parquetPath); + const memoryCap = resolvePointsMemoryCap(options.memoryCap); + const maxRows = rowCount > memoryCap ? memoryCap : rowCount; + + const columnNames = [featureKey]; + if (featureCodeColumnName && !columnNames.includes(featureCodeColumnName)) { + columnNames.push(featureCodeColumnName); + } + + const featureCodeEntries = featureCodeByName + ? [...featureCodeByName.entries()].map(([name, code]) => ({ name, code })) + : undefined; + + ensurePointsWorker(); + if (isPointsWorkerEnabled()) { + try { + const payload = await this.readParquetWorkerPayload(parquetPath, { maxRows }); + const workerInput = { + columns: columnNames, + maxRows, + featureKey, + featureCodeColumnName, + featureCodeEntries, + }; + const workerCodes = await decodeParquetRowFeatureCodesInWorker( + payload.rowGroups.length > 0 + ? { ...workerInput, rowGroups: payload.rowGroups } + : { ...workerInput, parts: payload.parts } + ); + if (workerCodes) { + return workerCodes; + } + } catch (error) { + console.warn( + `Worker row feature codes failed for ${elementPath}; falling back to main thread.`, + error + ); + } + } + + const { table: arrowTable } = await this.loadParquetTableCapped( + parquetPath, + columnNames, + maxRows + ); + return resolveRowFeatureCodesFromTable( + arrowTable, + featureKey, + featureCodeColumnName, + featureCodeByName + ); + } + + async getPointsParquetRowCount(elementPath: string): Promise { + const parquetPath = getParquetPath(elementPath); + return this.resolveParquetRowCount(parquetPath); + } + + async listPointsFeatures(elementPath: string): Promise { + const zattrs = await this.loadSpatialDataElementAttrs(elementPath); + const featureKey = zattrs.spatialdata_attrs?.feature_key; + if (typeof featureKey !== 'string' || featureKey.length === 0) { + return null; + } + + const parquetPath = getParquetPath(elementPath); + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + const schemaTable = datasetMetadata ? null : await this.loadParquetSchemaTable(parquetPath); + const fields = datasetMetadata?.schema?.fields + ? datasetMetadata.schema.fields.flatMap((field) => + typeof field.name === 'string' ? [field.name] : [] + ) + : arrowSchemaFieldNames(schemaTable); + const featureCodeColumnName = selectFeatureCodeColumn(fields, featureKey); + const hasMortonColumn = fields.includes(MORTON_CODE_2D_COLUMN); + + const rowCount = await this.resolveParquetRowCount(parquetPath); + + const columns = [featureKey]; + if (featureCodeColumnName) { + columns.push(featureCodeColumnName); + } + if (hasMortonColumn) { + columns.push(MORTON_CODE_2D_COLUMN); + } + + if (rowCount > 0 && exceedsPointsPreloadLimit(rowCount)) { + return this.listPointsFeaturesByFeatureColumnScan( + parquetPath, + featureKey, + featureCodeColumnName, + hasMortonColumn + ); + } + + const arrowTable = await this.loadParquetTable(parquetPath, columns); + const nameColumn = arrowTable.getChild(featureKey); + const codeColumn = featureCodeColumnName ? arrowTable.getChild(featureCodeColumnName) : null; + const mortonColumn = hasMortonColumn ? arrowTable.getChild(MORTON_CODE_2D_COLUMN) : null; + + if (!nameColumn) { + return null; + } + + return buildFeatureCatalogFromColumns( + featureKey, + nameColumn, + codeColumn, + mortonColumn, + arrowTable.numRows + ); + } + + /** + * Build a feature catalog for oversized datasets by scanning only feature + * columns (row-group range reads when available), not x/y geometry. + */ + private async listPointsFeaturesByFeatureColumnScan( + parquetPath: string, + featureKey: string, + featureCodeColumnName: string | undefined, + hasMortonColumn: boolean + ): Promise { + const columnNames = [featureKey]; + if (featureCodeColumnName) { + columnNames.push(featureCodeColumnName); + } + if (hasMortonColumn) { + columnNames.push(MORTON_CODE_2D_COLUMN); + } + + ensurePointsWorker(); + if (isPointsWorkerEnabled()) { + try { + const payload = await this.readParquetWorkerPayload(parquetPath, { + maxRows: Number.POSITIVE_INFINITY, + fullPartsForFallback: true, + includeRowGroups: true, + }); + const catalog = await scanParquetFeatureCatalogInWorker({ + rowGroups: + featureCodeColumnName && payload.rowGroups.length > 0 + ? payload.rowGroups + : undefined, + parts: payload.parts, + columns: columnNames, + featureKey, + featureCodeColumnName, + skipMortonSentinels: hasMortonColumn, + }); + if (catalog) { + return catalog; + } + } catch (error) { + console.warn( + `Worker feature catalog scan failed for ${parquetPath}; falling back to main thread.`, + error + ); + } + } + + const { accumulateFeatureCatalogFromTable, featureCatalogFromCodeMap, featureCatalogNeedsParquetFallback } = + await import('../pointsFeatures.js'); + const codeToName = new Map(); + const nameToCode = new Map(); + const canUseRowGroups = await this.canLoadParquetRowGroups(); + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + + if ( + canUseRowGroups && + datasetMetadata && + datasetMetadata.totalNumRowGroups > 0 && + featureCodeColumnName + ) { + for ( + let rowGroupIndex = 0; + rowGroupIndex < datasetMetadata.totalNumRowGroups; + rowGroupIndex += 1 + ) { + const table = await this.loadParquetRowGroupByGroupIndex(parquetPath, rowGroupIndex, { + columns: columnNames, + }); + if (!table || table.numRows === 0) { + continue; + } + accumulateFeatureCatalogFromTable( + codeToName, + nameToCode, + table, + featureKey, + featureCodeColumnName, + { skipMortonSentinels: hasMortonColumn } + ); + } + } + + if (featureCatalogNeedsParquetFallback(codeToName)) { + codeToName.clear(); + nameToCode.clear(); + const arrowTable = await this.loadParquetTable(parquetPath, columnNames); + accumulateFeatureCatalogFromTable( + codeToName, + nameToCode, + arrowTable, + featureKey, + featureCodeColumnName, + { skipMortonSentinels: hasMortonColumn } + ); + } + + if (codeToName.size === 0) { + return null; + } + return featureCatalogFromCodeMap(featureKey, codeToName); + } + + async getPointsTilingMetadata(elementPath: string): Promise { + if (this.pointTilingMetadataCache.has(elementPath)) { + return this.pointTilingMetadataCache.get(elementPath) ?? null; + } + const promise = this.loadPointsTilingMetadataUncached(elementPath).catch(error => { + this.pointTilingMetadataCache.delete(elementPath); + throw error; + }); + this.pointTilingMetadataCache.set(elementPath, promise); + return promise; + } + + private async loadPointsTilingMetadataUncached( + elementPath: string + ): Promise { + const parquetPath = getParquetPath(elementPath); + const zattrs = await this.loadSpatialDataElementAttrs(elementPath); + const { axes, spatialdata_attrs: spatialDataAttrs } = zattrs; + const normAxes = normalizeAxes(axes); + const axisNames = normAxes.map((axis: { name: string }) => axis.name); const { feature_key: featureKey } = spatialDataAttrs; - const columnNames = [...axisNames, featureKey].filter(Boolean); - const arrowTable = await this.loadParquetTable(parquetPath, columnNames); + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + const schemaTable = datasetMetadata ? null : await this.loadParquetSchemaTable(parquetPath); + const fields = datasetMetadata?.schema?.fields + ? datasetMetadata.schema.fields.flatMap((field) => + typeof field.name === 'string' ? [field.name] : [] + ) + : arrowSchemaFieldNames(schemaTable); + + const featureCodeColumnName = selectFeatureCodeColumn(fields, featureKey); + if ( + !fields.includes('x') || + !fields.includes('y') || + !fields.includes(MORTON_CODE_2D_COLUMN) || + !featureCodeColumnName + ) { + return null; + } + + const canLoadRowGroups = await this.canLoadParquetRowGroups(); + const firstRowGroupRowCount = datasetMetadata?.rowGroupRows?.[0] ?? 0; + const hasValidSentinelRowGroup = firstRowGroupRowCount >= 2 && firstRowGroupRowCount <= 4; + const firstRowGroup = + datasetMetadata && canLoadRowGroups && hasValidSentinelRowGroup + ? await this.loadParquetRowGroupByGroupIndex(parquetPath, 0, { + columns: ['x', 'y', MORTON_CODE_2D_COLUMN], + limit: 4, + }) + : null; + const bounds = firstRowGroup + ? (extractSentinelBoundingBox(firstRowGroup) ?? undefined) + : undefined; + const rowGroupSizes = datasetMetadata?.rowGroupRows ?? []; + + const metadata: PointsTilingMetadata = { + kind: 'morton-points', + parquetPath, + axisNames, + featureKey, + featureCodeColumnName, + mortonCodeColumnName: MORTON_CODE_2D_COLUMN, + totalRows: datasetMetadata?.totalNumRows ?? 0, + totalRowGroups: datasetMetadata?.totalNumRowGroups ?? 0, + maxRowsPerGroup: rowGroupSizes.length ? Math.max(...rowGroupSizes) : 0, + rowGroupRowCounts: datasetMetadata?.rowGroupRows, + supportsRowGroupRangeReads: Boolean(datasetMetadata && canLoadRowGroups && bounds), + bounds, + }; + + return metadata; + } + + async loadPointsInBounds( + elementPath: string, + options: PointsInBoundsOptions + ): Promise { + checkAbort(options.signal); + const metadata = await this.getPointsTilingMetadata(elementPath); + if (metadata?.supportsRowGroupRangeReads && metadata.bounds) { + const rowGroupResult = await this.loadMortonPointsInBounds(elementPath, metadata, options); + if (rowGroupResult) { + return rowGroupResult; + } + } + checkAbort(options.signal); + const full = await this.loadPointsWithOptionalFeatureCodes(elementPath, metadata, options); + checkAbort(options.signal); + return filterPointsToBounds( + full.data, + options.bounds, + undefined, + options.featureCodes, + full.featureCodes + ); + } - // TODO: this table will also contain the index column, and potentially the featureKey column. - // Do something with these here, otherwise they will need to be loaded redundantly. + private async loadPointsWithOptionalFeatureCodes( + elementPath: string, + metadata: PointsTilingMetadata | null, + options: PointsInBoundsOptions + ) { + const parquetPath = getParquetPath(elementPath); + const zattrs = await this.loadSpatialDataElementAttrs(elementPath); + const { axes, spatialdata_attrs: spatialDataAttrs } = zattrs; + const normAxes = normalizeAxes(axes); + const axisNames = normAxes.map((axis: { name: string }) => axis.name); + const { feature_key: featureKey } = spatialDataAttrs; + let featureCodeColumnName = metadata?.featureCodeColumnName; + const needsFeatureFilter = options.featureCodes !== undefined; + if (!featureCodeColumnName && needsFeatureFilter) { + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + const schemaTable = datasetMetadata ? null : await this.loadParquetSchemaTable(parquetPath); + const fields = datasetMetadata?.schema?.fields + ? datasetMetadata.schema.fields.flatMap((field) => + typeof field.name === 'string' ? [field.name] : [] + ) + : arrowSchemaFieldNames(schemaTable); + featureCodeColumnName = selectFeatureCodeColumn(fields, featureKey); + } + const resolvedFeatureCodeColumn = + typeof featureCodeColumnName === 'string' ? featureCodeColumnName : undefined; + const featureCodeByName = resolvedFeatureCodeColumn + ? undefined + : featureCodeMapFromCatalog(await this.listPointsFeatures(elementPath)); + const columnNames = [...axisNames]; + if (needsFeatureFilter) { + if (resolvedFeatureCodeColumn) { + columnNames.push(resolvedFeatureCodeColumn); + } else if (typeof featureKey === 'string' && !columnNames.includes(featureKey)) { + columnNames.push(featureKey); + } + } + const featureCodeEntries = featureCodeByName + ? [...featureCodeByName.entries()].map(([name, code]) => ({ name, code })) + : undefined; + + ensurePointsWorker(); + if (isPointsWorkerEnabled()) { + try { + const payload = await this.readParquetWorkerPayload(parquetPath, { + maxRows: Number.POSITIVE_INFINITY, + fullPartsForFallback: true, + includeRowGroups: true, + }); + const workerResult = await decodeParquetGeometryCappedInWorker( + payload.rowGroups.length > 0 + ? { + rowGroups: payload.rowGroups, + axisNames, + columns: columnNames, + maxRows: Number.POSITIVE_INFINITY, + featureKey: needsFeatureFilter ? featureKey : undefined, + featureCodeColumnName: resolvedFeatureCodeColumn, + featureCodeEntries, + } + : { + parts: payload.parts, + axisNames, + columns: columnNames, + maxRows: Number.POSITIVE_INFINITY, + featureKey: needsFeatureFilter ? featureKey : undefined, + featureCodeColumnName: resolvedFeatureCodeColumn, + featureCodeEntries, + } + ); + if (workerResult) { + return { + data: { + shape: workerResult.shape as [number, number], + data: workerResult.data, + }, + featureCodes: workerResult.featureCodes, + }; + } + } catch (error) { + console.warn( + `Worker bounds geometry load failed for ${elementPath}; falling back to main thread.`, + error + ); + } + } + const arrowTable = await this.loadParquetTable(parquetPath, columnNames); const axisColumnArrs = axisNames.map((name: string) => { const column = arrowTable.getChild(name); if (!column) { @@ -150,10 +1030,178 @@ export default class SpatialDataPointsSource extends SpatialDataTableSource { } return column.toArray(); }); + const featureCodes = needsFeatureFilter + ? resolveRowFeatureCodesFromTable( + arrowTable, + featureKey, + resolvedFeatureCodeColumn, + featureCodeByName + ) + : undefined; + return { + data: { + shape: [axisColumnArrs.length, arrowTable.numRows], + data: axisColumnArrs, + }, + featureCodes, + }; + } + + private async bisectRowGroupsRight( + parquetPath: string, + totalRowGroups: number, + targetValue: number + ) { + let lo = 0; + let hi = totalRowGroups; + while (lo < hi) { + const mid = Math.floor((lo + hi) / 2); + const extent = await this.loadParquetRowGroupColumnExtent( + parquetPath, + MORTON_CODE_2D_COLUMN, + mid + ); + const max = extent?.max; + if (max === null || max === undefined || targetValue <= max) { + hi = mid; + } else { + lo = mid + 1; + } + } + return lo; + } + + private async loadMortonPointsInBounds( + elementPath: string, + metadata: PointsTilingMetadata, + options: PointsInBoundsOptions + ): Promise { + if (!metadata.bounds || metadata.totalRowGroups <= 0) { + return null; + } + checkAbort(options.signal); + const allowedFeatureCodes = featureCodeAllowSet(options.featureCodes); + const intervals = mortonIntervalsForBounds(metadata.bounds, options.bounds); + const rowGroupSet = new Set(); + for (const [start, end] of intervals) { + const first = await this.bisectRowGroupsRight( + metadata.parquetPath, + metadata.totalRowGroups, + start + ); + const last = await this.bisectRowGroupsRight( + metadata.parquetPath, + metadata.totalRowGroups, + end + ); + for (let rowGroup = first; rowGroup <= last; rowGroup++) { + if (rowGroup >= 0 && rowGroup < metadata.totalRowGroups) { + rowGroupSet.add(rowGroup); + } + } + } + const rowGroups = [...rowGroupSet].sort((a, b) => a - b); + const totalRowsUpperBound = rowGroups.reduce( + (sum, rowGroup) => sum + rowGroupCountForIndex(metadata, rowGroup), + 0 + ); + if (totalRowsUpperBound === 0) { + return null; + } + + const xs: number[] = []; + const ys: number[] = []; + const zs: number[] = []; + const hasZ = metadata.axisNames.includes('z'); + const filterByFeature = allowedFeatureCodes !== null; + const featureCodeColumnName = + filterByFeature && metadata.featureCodeColumnName + ? metadata.featureCodeColumnName + : undefined; + + ensurePointsWorker(); + if (isPointsWorkerEnabled()) { + const rowGroupChunks = []; + for (const rowGroup of rowGroups) { + checkAbort(options.signal); + const chunk = await this.readParquetRowGroupBytesByGroupIndex( + metadata.parquetPath, + rowGroup + ); + if (chunk) { + rowGroupChunks.push(chunk); + } + } + if (rowGroupChunks.length > 0) { + try { + const workerResult = await scanMortonRowGroupsInBoundsInWorker({ + rowGroups: rowGroupChunks, + bounds: options.bounds, + axisNames: metadata.axisNames, + mortonCodeColumnName: metadata.mortonCodeColumnName, + featureCodeColumnName, + featureCodes: options.featureCodes, + }); + if (workerResult) { + return { + data: workerResult.data, + shape: workerResult.shape as [number, number], + bounds: options.bounds, + loadMode: 'row-groups', + tiling: metadata, + }; + } + } catch (error) { + console.warn( + `Worker morton tile load failed for ${elementPath}; falling back to main thread.`, + error + ); + } + } + } + + const rowGroupColumns = [ + 'x', + 'y', + ...(hasZ ? ['z'] : []), + metadata.mortonCodeColumnName, + ...(featureCodeColumnName ? [featureCodeColumnName] : []), + ]; + for (const rowGroup of rowGroups) { + checkAbort(options.signal); + const table = await this.loadParquetRowGroupByGroupIndex(metadata.parquetPath, rowGroup, { + columns: rowGroupColumns, + }); + if (!table) { + continue; + } + const { scanMortonTableInBounds } = await import('../workers/pointsWorkerScan.js'); + scanMortonTableInBounds({ + table, + rowGroupIndex: rowGroup, + bounds: options.bounds, + axisNames: metadata.axisNames, + mortonCodeColumnName: metadata.mortonCodeColumnName, + featureCodeColumnName, + featureCodes: options.featureCodes, + xs, + ys, + zs, + }); + } + + if (xs.length === 0) { + return null; + } return { - shape: [axisColumnArrs.length, arrowTable.numRows], - data: axisColumnArrs, + data: hasZ + ? [new Float32Array(xs), new Float32Array(ys), new Float32Array(zs)] + : [new Float32Array(xs), new Float32Array(ys)], + shape: [hasZ ? 3 : 2, xs.length], + bounds: options.bounds, + loadMode: 'row-groups', + tiling: metadata, }; } } diff --git a/packages/core/src/models/VShapesSource.ts b/packages/core/src/models/VShapesSource.ts index 803a4fc1..e5bf115e 100644 --- a/packages/core/src/models/VShapesSource.ts +++ b/packages/core/src/models/VShapesSource.ts @@ -23,15 +23,28 @@ const log = console; // import SpatialDataTableSource from './SpatialDataTableSource.js'; -import type { TypedArray as ZarrTypedArray, Chunk, NumberDataType } from 'zarrita'; import type { Table as ArrowTable } from 'apache-arrow'; import type { Vector } from 'apache-arrow/vector'; -import SpatialDataTableSource from './VTableSource'; +import type { Chunk, NumberDataType, TypedArray as ZarrTypedArray } from 'zarrita'; +import type { SpatialBounds } from '../pointsTiling.js'; import type { ShapesGeometryKind, ShapesRenderData } from '../shapes'; +import SpatialDataTableSource from './VTableSource'; export type PolygonShape = Array>; //nb, not totally happy with this type. export type ZarrNumericArray = ZarrTypedArray | BigInt64Array | Array; +export interface ShapesInBoundsOptions { + bounds: SpatialBounds; + zoom?: number; + signal?: AbortSignal; + columns?: string[]; +} + +export type ShapesInBoundsResult = ShapesRenderData & { + bounds: SpatialBounds; + loadMode: 'full-filter'; +}; + // If the array path starts with table/something/rest // capture table/something. @@ -64,6 +77,12 @@ function getParquetPath(arrPath?: string) { throw new Error(`Cannot determine parquet path for shapes array path: ${arrPath}`); } +function checkAbort(signal?: AbortSignal) { + if (signal?.aborted) { + throw new DOMException('The operation was aborted.', 'AbortError'); + } +} + /** * Converts BigInt64Array or Float64Array to Float32Array if needed. * TODO: remove this and support BigInts/Float64s in downstream code. @@ -363,11 +382,10 @@ export default class SpatialDataShapesSource extends SpatialDataTableSource { // However this may complicate applying transformations, at least in the current way. // Reference: https://deck.gl/docs/api-reference/layers/polygon-layer#data-accessors return arr.map((geom: ArrayBuffer) => { - const coords = - wkb - .readGeometry(geom) - // @ts-expect-error - getCoordinates is not a method of Geometry, check this<<< - .getCoordinates(); + const coords = wkb + .readGeometry(geom) + // @ts-expect-error - getCoordinates is not a method of Geometry, check this<<< + .getCoordinates(); // Take first polygon (if multipolygon) return coords[0]; }); @@ -502,6 +520,20 @@ export default class SpatialDataShapesSource extends SpatialDataTableSource { }; } + async loadShapesInBounds( + elementPath: string, + options: ShapesInBoundsOptions + ): Promise { + checkAbort(options.signal); + const renderData = await this.loadShapesRenderData(elementPath); + checkAbort(options.signal); + return { + ...renderData, + bounds: options.bounds, + loadMode: 'full-filter', + }; + } + /** * * @param path diff --git a/packages/core/src/models/VTableSource.ts b/packages/core/src/models/VTableSource.ts index f303344d..d1aa43e2 100644 --- a/packages/core/src/models/VTableSource.ts +++ b/packages/core/src/models/VTableSource.ts @@ -1,59 +1,52 @@ // this is a direct copy of the Vitessce implementation, with changes mostly to make it more normal TypeScript. -import { tableFromIPC, type Table as ArrowTable } from 'apache-arrow'; +import { type Table as ArrowTable, tableFromIPC } from 'apache-arrow'; import type { DataSourceParams } from '../Vutils'; +import { + getParquetModule, + type ParquetModule, + type ParquetRowGroupReadOptions, + type ParquetWasmMetadata, +} from '../parquetWasmLoader.js'; import type { TableColumnData } from '../types'; import AnnDataSource from './VAnnDataSource'; +export type { ParquetRowGroupReadOptions }; + +function parquetColumnValueToNumber(value: unknown): number | null { + if (typeof value === 'number' && Number.isFinite(value)) { + return value; + } + if (typeof value === 'bigint') { + return Number(value); + } + return null; +} + +export interface ParquetPartMetadata { + path: string; + schema: ArrowTable['schema']; + schemaBytes: Uint8Array; + metadata: ParquetWasmMetadata; +} + +export interface ParquetDatasetMetadata { + totalNumRows: number; + totalNumRowGroups: number; + numRowsByPart: number[]; + numRowGroupsByPart: number[]; + numRowsPerGroupByPart: number[]; + rowGroupRows: number[]; + schema: ArrowTable['schema'] | null; + parts: ParquetPartMetadata[]; +} + // Note: This file also serves as the parent for // SpatialDataPointsSource and SpatialDataShapesSource, // because when a table annotates points and shapes, it can be helpful to // have all of the required functionality to load the // table data and the parquet data. -async function getParquetModule() { - // Dynamic import for code-splitting. parquet-wasm is a WebAssembly module - // that needs to be initialized before use in browser environments. - // In Node.js, the module loads WASM synchronously so no init is needed. - // - // TODO: Replace with a more civilised parquet module that's built in a way we can actually consume. - // - probably ultimately may be using geoarrow-wasm / investigate deck.gl arrow layer - // think about how that fits our 'core' (no deck deps) vs 'vis' structure etc. - - // Try local import first (works in Node.js, tests, and production builds) - try { - const module = await import('parquet-wasm'); - if (typeof module.default === 'function') { - await module.default(); - } - return { readParquet: module.readParquet, readSchema: module.readSchema }; - } catch (error) { - // Local import failed, try CDN fallback (needed in vite dev server) - // Reference: https://observablehq.com/@kylebarron/geoparquet-on-the-web - console.warn( - '[VTableSource] Local parquet-wasm import failed, falling back to CDN version. ' + - 'This is a temporary workaround pending a better parquet module solution.', - error - ); - - try { - const cdnModule = await import( - // @ts-expect-error - CDN import not recognized by TypeScript - 'https://cdn.vitessce.io/parquet-wasm@2c23652/esm/parquet_wasm.js' - ); - await cdnModule.default(); - return { readParquet: cdnModule.readParquet, readSchema: cdnModule.readSchema }; - } catch (cdnError) { - // Both imports failed, throw an error - const localErrorMsg = error instanceof Error ? error.message : String(error); - const cdnErrorMsg = cdnError instanceof Error ? cdnError.message : String(cdnError); - throw new Error( - `Failed to load parquet-wasm from both local package and CDN. Local error: ${localErrorMsg}. CDN error: ${cdnErrorMsg}` - ); - } - } -} - /** * Get the name of the index column from an Apache Arrow table. * In the future, this may not be needed if more metadata is included in the Zarr Attributes. @@ -162,6 +155,14 @@ function hasParquetTailMagic(bytes: Uint8Array) { return bytes.length >= 8 && hasParquetMagic(bytes, bytes.length - 4); } +function toSafeNumber(value: number | bigint, label: string) { + const n = typeof value === 'bigint' ? Number(value) : value; + if (!Number.isSafeInteger(n) || n < 0) { + throw new Error(`Invalid parquet ${label}: ${String(value)}`); + } + return n; +} + /** * This class is a parent class for tables, shapes, and points. * This is because these share functionality, for example: @@ -170,10 +171,7 @@ function hasParquetTailMagic(bytes: Uint8Array) { * - logic for manipulating spatialdata element paths is shared across all elements. */ export default class SpatialDataTableSource extends AnnDataSource { - static parquetModulePromise: Promise<{ - readParquet: (bytes: Uint8Array, options?: { columns?: string[] }) => any; - readSchema: (bytes: Uint8Array) => any; - }>; + static parquetModulePromise: Promise; rootAttrs: { softwareVersion: string; formatVersion: string } | null; // biome-ignore lint/suspicious/noExplicitAny: elementAttrs type should be a tree-ish thing elementAttrs: Record; @@ -186,6 +184,8 @@ export default class SpatialDataTableSource extends AnnDataSource { * `loadPolygonShapes` all target the same file). */ parquetTableCache: Record>; + /** Morton min/max per row group — avoids re-decoding row groups during bisect. */ + rowGroupColumnExtentCache: Map; obsIndices: Record>; varIndices: Record>; varAliases: Record; @@ -206,6 +206,7 @@ export default class SpatialDataTableSource extends AnnDataSource { // TODO: change to column-specific storage. this.parquetTableBytes = {}; this.parquetTableCache = {}; + this.rowGroupColumnExtentCache = new Map(); // Table-specific properties this.obsIndices = {}; @@ -322,44 +323,12 @@ export default class SpatialDataTableSource extends AnnDataSource { async loadParquetSchemaBytes(parquetPath: string) { const { store } = this.storeRoot; if (store.getRange) { - // Step 1: Fetch last 8 bytes to get footer length and magic number - const TAIL_LENGTH = 8; let lastError: Error | null = null; for (const candidatePath of getParquetCandidatePaths(parquetPath)) { try { - const tailBytes = await store.getRange(`/${candidatePath}`, { - suffixLength: TAIL_LENGTH, - }); - const normalizedTailBytes = toUint8Array(tailBytes); - if (!normalizedTailBytes || !hasParquetTailMagic(normalizedTailBytes)) { - continue; - } - - // Step 2: Extract footer length and magic number - // little-endian - const footerLength = new DataView( - normalizedTailBytes.buffer, - normalizedTailBytes.byteOffset, - normalizedTailBytes.byteLength - ).getInt32(0, true); - - // Step 3. Fetch the full footer bytes - const footerBytes = await store.getRange(`/${candidatePath}`, { - suffixLength: footerLength + TAIL_LENGTH, - }); - const normalizedFooterBytes = toUint8Array(footerBytes); - if ( - !normalizedFooterBytes || - normalizedFooterBytes.length !== footerLength + TAIL_LENGTH || - !hasParquetTailMagic(normalizedFooterBytes) - ) { - lastError = new Error(`Failed to load parquet footer bytes for ${parquetPath}`); - continue; - } - - // Step 4: Return the footer bytes - return normalizedFooterBytes; + const footerBytes = await this.loadParquetFooterBytesForPath(candidatePath); + if (footerBytes) return footerBytes; } catch (error) { lastError = error instanceof Error ? error : new Error(String(error)); } @@ -371,6 +340,394 @@ export default class SpatialDataTableSource extends AnnDataSource { return null; } + private async loadParquetFooterBytesForPath(path: string): Promise { + const { store } = this.storeRoot; + if (!store.getRange) { + return null; + } + const tailLength = 8; + const tailBytes = await store.getRange(`/${path}`, { + suffixLength: tailLength, + }); + const normalizedTailBytes = toUint8Array(tailBytes); + if (!normalizedTailBytes || !hasParquetTailMagic(normalizedTailBytes)) { + return null; + } + + const footerLength = new DataView( + normalizedTailBytes.buffer, + normalizedTailBytes.byteOffset, + normalizedTailBytes.byteLength + ).getInt32(0, true); + + const footerBytes = await store.getRange(`/${path}`, { + suffixLength: footerLength + tailLength, + }); + const normalizedFooterBytes = toUint8Array(footerBytes); + if ( + !normalizedFooterBytes || + normalizedFooterBytes.length !== footerLength + tailLength || + !hasParquetTailMagic(normalizedFooterBytes) + ) { + return null; + } + return normalizedFooterBytes; + } + + async loadParquetSchemaTable(parquetPath: string): Promise { + const schemaBytes = await this.loadParquetSchemaBytes(parquetPath); + if (!schemaBytes) { + return null; + } + const { readSchema } = await SpatialDataTableSource.parquetModulePromise; + const wasmSchema = readSchema(schemaBytes); + return tableFromIPC(wasmSchema.intoIPCStream()); + } + + private readParquetFooterBytesFromFileBytes(bytes: Uint8Array): Uint8Array | null { + if (bytes.length < 8) { + return null; + } + const footerLength = new DataView( + bytes.buffer, + bytes.byteOffset + bytes.length - 8, + 8 + ).getInt32(0, true); + const totalFooterSize = footerLength + 8; + if (totalFooterSize <= 0 || totalFooterSize > bytes.length) { + return null; + } + return bytes.subarray(bytes.length - totalFooterSize); + } + + private async loadParquetPartMetadataFromFullFile( + path: string + ): Promise { + const { readMetadata, readSchema } = await SpatialDataTableSource.parquetModulePromise; + if (!readMetadata) { + return null; + } + const fileBytes = await this.loadParquetFileBytesAtPath(path); + if (!fileBytes) { + return null; + } + const schemaBytes = this.readParquetFooterBytesFromFileBytes(fileBytes); + if (!schemaBytes) { + return null; + } + const schemaTable = await tableFromIPC(readSchema(schemaBytes).intoIPCStream()); + return { + path, + schema: schemaTable.schema, + schemaBytes, + metadata: readMetadata(schemaBytes), + }; + } + + private async countRowsFromFullParquetFile(path: string): Promise { + const fileBytes = await this.loadParquetFileBytesAtPath(path); + if (!fileBytes) { + return 0; + } + const { readParquet } = await SpatialDataTableSource.parquetModulePromise; + const table = await tableFromIPC(readParquet(fileBytes, { columns: ['x'] }).intoIPCStream()); + return table.numRows; + } + + protected async resolveParquetRowCount(parquetPath: string): Promise { + const datasetMetadata = await this.loadParquetDatasetMetadata(parquetPath); + if (datasetMetadata?.totalNumRows) { + return datasetMetadata.totalNumRows; + } + + const directPart = await this.loadParquetPartMetadataFromFullFile(parquetPath); + if (directPart) { + return directPart.metadata.fileMetadata().numRows(); + } + + let totalRows = 0; + let foundPart = false; + for (let partIndex = 0; ; partIndex += 1) { + const partPath = `${parquetPath}/part.${partIndex}.parquet`; + const part = await this.loadParquetPartMetadataFromFullFile(partPath); + if (part) { + foundPart = true; + totalRows += part.metadata.fileMetadata().numRows(); + continue; + } + const columnCount = await this.countRowsFromFullParquetFile(partPath); + if (columnCount > 0) { + foundPart = true; + totalRows += columnCount; + continue; + } + break; + } + if (foundPart) { + return totalRows; + } + + return this.countRowsFromFullParquetFile(parquetPath); + } + + private async loadParquetPartMetadata(path: string): Promise { + const { readMetadata, readSchema } = await SpatialDataTableSource.parquetModulePromise; + if (!readMetadata) { + return null; + } + const schemaBytes = await this.loadParquetFooterBytesForPath(path); + if (!schemaBytes) { + return null; + } + const schemaTable = await tableFromIPC(readSchema(schemaBytes).intoIPCStream()); + return { + path, + schema: schemaTable.schema, + schemaBytes, + metadata: readMetadata(schemaBytes), + }; + } + + async loadParquetDatasetMetadata(parquetPath: string): Promise { + const { readMetadata } = await SpatialDataTableSource.parquetModulePromise; + const { store } = this.storeRoot; + if (!readMetadata || !store.getRange) { + return null; + } + + const directPart = await this.loadParquetPartMetadata(parquetPath); + const parts: ParquetPartMetadata[] = []; + if (directPart) { + parts.push(directPart); + } else { + for (let partIndex = 0; ; partIndex++) { + const part = await this.loadParquetPartMetadata(`${parquetPath}/part.${partIndex}.parquet`); + if (!part) { + break; + } + parts.push(part); + } + } + + if (parts.length === 0) { + return null; + } + + const numRowsByPart = parts.map((part) => part.metadata.fileMetadata().numRows()); + const numRowGroupsByPart = parts.map((part) => part.metadata.numRowGroups()); + const numRowsPerGroupByPart = parts.map((part) => + part.metadata.numRowGroups() > 0 ? part.metadata.rowGroup(0).numRows() : 0 + ); + const rowGroupRows = parts.flatMap((part) => + Array.from({ length: part.metadata.numRowGroups() }, (_value, rowGroupIndex) => + part.metadata.rowGroup(rowGroupIndex).numRows() + ) + ); + return { + totalNumRows: numRowsByPart.reduce((acc, cur) => acc + cur, 0), + totalNumRowGroups: numRowGroupsByPart.reduce((acc, cur) => acc + cur, 0), + numRowsByPart, + numRowGroupsByPart, + numRowsPerGroupByPart, + rowGroupRows, + schema: parts[0]?.schema ?? null, + parts, + }; + } + + async canLoadParquetRowGroups(): Promise { + const module = await SpatialDataTableSource.parquetModulePromise; + return ( + typeof module.readMetadata === 'function' && typeof module.readParquetRowGroup === 'function' + ); + } + + /** + * Fetch compressed row-group bytes via range read (no parquet decode on the caller thread). + */ + protected async readParquetRowGroupBytesByGroupIndex( + parquetPath: string, + rowGroupIndex: number + ): Promise<{ + schemaBytes: Uint8Array; + rowGroupBytes: Uint8Array; + rowGroupIndex: number; + globalRowGroupIndex: number; + } | null> { + const { store } = this.storeRoot; + if (!store.getRange) { + return null; + } + const dataset = await this.loadParquetDatasetMetadata(parquetPath); + if (!dataset || rowGroupIndex < 0 || rowGroupIndex >= dataset.totalNumRowGroups) { + return null; + } + + let cumulativeRowGroups = 0; + for (const part of dataset.parts) { + const partRowGroupCount = part.metadata.numRowGroups(); + if (rowGroupIndex >= cumulativeRowGroups + partRowGroupCount) { + cumulativeRowGroups += partRowGroupCount; + continue; + } + const relativeRowGroupIndex = rowGroupIndex - cumulativeRowGroups; + const rowGroup = part.metadata.rowGroup(relativeRowGroupIndex); + const offset = toSafeNumber(rowGroup.fileOffset(), 'row-group file offset'); + const length = toSafeNumber(rowGroup.compressedSize(), 'row-group compressed size'); + const bytes = await store.getRange(`/${part.path}`, { offset, length }); + const rowGroupBytes = toUint8Array(bytes); + if (!rowGroupBytes) { + return null; + } + return { + schemaBytes: part.schemaBytes, + rowGroupBytes, + rowGroupIndex: relativeRowGroupIndex, + globalRowGroupIndex: rowGroupIndex, + }; + } + return null; + } + + protected async readParquetRowGroupsBytesCapped( + parquetPath: string, + maxRows: number + ): Promise< + Array<{ + schemaBytes: Uint8Array; + rowGroupBytes: Uint8Array; + rowGroupIndex: number; + }> + > { + const dataset = await this.loadParquetDatasetMetadata(parquetPath); + if (!dataset || dataset.totalNumRowGroups <= 0) { + return []; + } + + const chunks: Array<{ + schemaBytes: Uint8Array; + rowGroupBytes: Uint8Array; + rowGroupIndex: number; + }> = []; + let accumulated = 0; + for (let rowGroupIndex = 0; rowGroupIndex < dataset.totalNumRowGroups; rowGroupIndex += 1) { + if (accumulated >= maxRows) { + break; + } + const chunk = await this.readParquetRowGroupBytesByGroupIndex(parquetPath, rowGroupIndex); + if (!chunk) { + continue; + } + chunks.push(chunk); + const rowCount = dataset.rowGroupRows[rowGroupIndex]; + if (typeof rowCount === 'number' && Number.isFinite(rowCount)) { + accumulated += rowCount; + } else { + accumulated = maxRows; + } + } + return chunks; + } + + /** + * Row-group and part byte payloads for worker-side parquet decode. + */ + protected async readParquetWorkerPayload( + parquetPath: string, + options: { + maxRows: number; + fullPartsForFallback?: boolean; + /** When false (default), only part bytes are fetched for worker decode. */ + includeRowGroups?: boolean; + } + ): Promise<{ + rowGroups: Array<{ + schemaBytes: Uint8Array; + rowGroupBytes: Uint8Array; + rowGroupIndex: number; + }>; + parts: Uint8Array[]; + }> { + const includeRowGroups = options.includeRowGroups === true; + const canUseRowGroups = includeRowGroups && (await this.canLoadParquetRowGroups()); + const rowGroups = canUseRowGroups + ? await this.readParquetRowGroupsBytesCapped(parquetPath, options.maxRows) + : []; + const partsMaxRows = options.fullPartsForFallback + ? Number.POSITIVE_INFINITY + : options.maxRows; + const { parts } = await this.readParquetDatasetBytesCapped(parquetPath, partsMaxRows); + return { rowGroups, parts }; + } + + async loadParquetRowGroupByGroupIndex( + parquetPath: string, + rowGroupIndex: number, + readOptions?: ParquetRowGroupReadOptions + ): Promise { + const { readParquetRowGroup } = await SpatialDataTableSource.parquetModulePromise; + if (!readParquetRowGroup) { + return null; + } + const chunk = await this.readParquetRowGroupBytesByGroupIndex(parquetPath, rowGroupIndex); + if (!chunk) { + return null; + } + return tableFromIPC( + readParquetRowGroup( + chunk.schemaBytes, + chunk.rowGroupBytes, + chunk.rowGroupIndex, + readOptions + ).intoIPCStream() + ); + } + + async loadParquetRowGroupColumnExtent( + parquetPath: string, + columnName: string, + rowGroupIndex: number + ): Promise<{ min: number | null; max: number | null } | null> { + const cacheKey = `${parquetPath}::${rowGroupIndex}::${columnName}`; + const cached = this.rowGroupColumnExtentCache.get(cacheKey); + if (cached) { + return cached; + } + const dataset = await this.loadParquetDatasetMetadata(parquetPath); + const rowCount = dataset?.rowGroupRows?.[rowGroupIndex]; + if (!rowCount) { + return null; + } + const columnOptions: ParquetRowGroupReadOptions = { columns: [columnName] }; + const minTable = await this.loadParquetRowGroupByGroupIndex( + parquetPath, + rowGroupIndex, + { ...columnOptions, limit: 1 } + ); + const minColumn = minTable?.getChild(columnName); + if (!minColumn || minColumn.length === 0) { + return null; + } + let maxValue: number | null = parquetColumnValueToNumber(minColumn.get(0)); + if (rowCount > 1) { + const maxTable = await this.loadParquetRowGroupByGroupIndex(parquetPath, rowGroupIndex, { + ...columnOptions, + offset: rowCount - 1, + limit: 1, + }); + const maxColumn = maxTable?.getChild(columnName); + if (maxColumn && maxColumn.length > 0) { + maxValue = parquetColumnValueToNumber(maxColumn.get(0)); + } + } + const extent = { + min: parquetColumnValueToNumber(minColumn.get(0)), + max: maxValue, + }; + this.rowGroupColumnExtentCache.set(cacheKey, extent); + return extent; + } + /** * Get the index column from a parquet table. * @param parquetPath A path to a parquet file (or directory). @@ -409,75 +766,408 @@ export default class SpatialDataTableSource extends AnnDataSource { return tablePromise; } - private async _loadParquetTableUncached(parquetPath: string, columns?: string[]): Promise { + private async discoverMultipartPartPaths(parquetPath: string): Promise { + const partPaths: string[] = []; + for (let partIndex = 0; ; partIndex += 1) { + const partPath = `${parquetPath}/part.${partIndex}.parquet`; + const bytes = await this.loadParquetFileBytesAtPath(partPath); + if (!bytes) { + break; + } + partPaths.push(partPath); + } + return partPaths; + } + + private async loadMultipartParquetTableFromPartPaths( + parquetPath: string, + partPaths: string[], + columns: string[] | undefined, + readParquet: ParquetModule['readParquet'], + readSchema: ParquetModule['readSchema'] + ): Promise { + const tables: ArrowTable[] = []; + for (const partPath of partPaths) { + const parquetBytes = await this.loadParquetFileBytesAtPath(partPath); + if (!parquetBytes) { + throw new Error(`Failed to load parquet part at ${partPath}.`); + } + tables.push( + await this.readParquetTableFromFileBytes( + parquetBytes, + columns, + readParquet, + readSchema, + parquetPath + ) + ); + } + if (tables.length === 0) { + throw new Error(`Failed to load multipart parquet data from ${parquetPath}.`); + } + return tables.slice(1).reduce((merged, part) => merged.concat(part), tables[0]); + } + + protected async readParquetDatasetBytes(parquetPath: string): Promise { + const capped = await this.readParquetDatasetBytesCapped(parquetPath, Number.POSITIVE_INFINITY); + return capped.parts; + } + + /** + * Read parquet part bytes up to a row cap. Uses dataset metadata to avoid + * loading parts beyond the cap when row counts per part are known. + */ + protected async readParquetDatasetBytesCapped( + parquetPath: string, + maxRows: number + ): Promise<{ parts: Uint8Array[]; totalRows: number; truncated: boolean }> { + const totalRows = await this.resolveParquetRowCount(parquetPath); + if (totalRows <= maxRows) { + const dataset = await this.loadParquetDatasetMetadata(parquetPath); + if (dataset?.parts.length) { + const parts: Uint8Array[] = []; + for (const part of dataset.parts) { + const bytes = await this.loadParquetFileBytesAtPath(part.path); + if (!bytes) { + // Strict fail — see docs/plans/parquet-io-error-handling.md + throw new Error(`Missing parquet part bytes at ${part.path}`); + } + parts.push(bytes); + } + return { parts, totalRows, truncated: false }; + } + const bytes = await this.loadParquetFileBytesAtPath(parquetPath); + return { parts: bytes ? [bytes] : [], totalRows, truncated: false }; + } + + const dataset = await this.loadParquetDatasetMetadata(parquetPath); + let partPaths: string[] = []; + if (dataset?.parts.length) { + partPaths = dataset.parts.map((part) => part.path); + } else { + const discovered = await this.discoverMultipartPartPaths(parquetPath); + partPaths = discovered.length > 0 ? discovered : [parquetPath]; + } + + const numRowsByPart = dataset?.numRowsByPart ?? []; + const parts: Uint8Array[] = []; + let accumulated = 0; + for (let partIndex = 0; partIndex < partPaths.length; partIndex += 1) { + const remaining = maxRows - accumulated; + if (remaining <= 0) { + break; + } + const partPath = partPaths[partIndex]; + const bytes = await this.loadParquetFileBytesAtPath(partPath); + if (!bytes) { + // Strict fail (sibling capped table loader uses continue) — docs/plans/parquet-io-error-handling.md + throw new Error(`Missing parquet bytes at ${partPath}`); + } + parts.push(bytes); + const partRows = numRowsByPart[partIndex]; + if (typeof partRows === 'number' && Number.isFinite(partRows)) { + accumulated += partRows; + } else { + accumulated = maxRows; + } + if (accumulated >= maxRows) { + break; + } + } + + return { parts, totalRows, truncated: true }; + } + + async loadParquetTableCapped( + parquetPath: string, + columns: string[] | undefined, + maxRows: number, + options: { useRowGroupReads?: boolean } = {} + ): Promise<{ table: ArrowTable; totalRows: number; truncated: boolean }> { + const totalRows = await this.resolveParquetRowCount(parquetPath); + const truncated = totalRows > maxRows; + const targetRows = truncated ? maxRows : totalRows; + + if (options.useRowGroupReads === true && (await this.canLoadParquetRowGroups())) { + try { + const table = await this._loadParquetTableRowGroupsCapped( + parquetPath, + columns, + targetRows + ); + return { table, totalRows, truncated }; + } catch (error) { + console.warn( + `Row-group parquet read failed for ${parquetPath}; falling back to full-file decode.`, + error + ); + } + } + + if (!truncated) { + const table = await this.loadParquetTable(parquetPath, columns); + return { table, totalRows, truncated: false }; + } + const table = await this._loadParquetTableUncachedCapped(parquetPath, columns, maxRows); + return { table, totalRows, truncated: true }; + } + + /** + * Load up to {@link maxRows} via per-row-group range reads and optional column + * projection. Avoids fetching entire parquet part files when the store supports + * byte-range reads. + */ + private async _loadParquetTableRowGroupsCapped( + parquetPath: string, + columns: string[] | undefined, + maxRows: number + ): Promise { + const dataset = await this.loadParquetDatasetMetadata(parquetPath); + if (!dataset || dataset.totalNumRowGroups <= 0) { + throw new Error(`No row groups available for ${parquetPath}.`); + } + + const { readSchema } = await SpatialDataTableSource.parquetModulePromise; + const resolvedColumns = columns?.length + ? await this.resolveParquetTableColumns( + parquetPath, + columns, + readSchema, + dataset.parts[0]?.schemaBytes + ) + : undefined; + const readOptions: ParquetRowGroupReadOptions | undefined = resolvedColumns?.length + ? { columns: resolvedColumns } + : undefined; + + const tables: ArrowTable[] = []; + let accumulated = 0; + for (let rowGroupIndex = 0; rowGroupIndex < dataset.totalNumRowGroups; rowGroupIndex += 1) { + if (accumulated >= maxRows) { + break; + } + let table = await this.loadParquetRowGroupByGroupIndex( + parquetPath, + rowGroupIndex, + readOptions + ); + if (!table || table.numRows === 0) { + continue; + } + const remaining = maxRows - accumulated; + if (table.numRows > remaining) { + table = table.slice(0, remaining); + } + tables.push(table); + accumulated += table.numRows; + } + + if (tables.length === 0) { + throw new Error(`Failed to load row-group capped parquet data from ${parquetPath}.`); + } + return tables.slice(1).reduce((merged, part) => merged.concat(part), tables[0]); + } + + private async _loadParquetTableUncachedCapped( + parquetPath: string, + columns: string[] | undefined, + maxRows: number + ): Promise { const { readParquet, readSchema } = await SpatialDataTableSource.parquetModulePromise; - const options = { - columns, - }; + const dataset = await this.loadParquetDatasetMetadata(parquetPath); + let partPaths: string[] = []; + if (dataset?.parts.length) { + partPaths = dataset.parts.map((part) => part.path); + } else { + const discovered = await this.discoverMultipartPartPaths(parquetPath); + partPaths = discovered.length > 0 ? discovered : [parquetPath]; + } + + const tables: ArrowTable[] = []; + let accumulated = 0; + for (let partIndex = 0; partIndex < partPaths.length; partIndex += 1) { + const remaining = maxRows - accumulated; + if (remaining <= 0) { + break; + } + const partPath = partPaths[partIndex]; + const parquetBytes = await this.loadParquetFileBytesAtPath(partPath); + if (!parquetBytes) { + continue; + } + const partTable = await this.readParquetTableFromFileBytes( + parquetBytes, + columns, + readParquet, + readSchema, + parquetPath + ); + if (partTable.numRows <= remaining) { + tables.push(partTable); + accumulated += partTable.numRows; + } else { + tables.push(partTable.slice(0, remaining)); + break; + } + } + + if (tables.length === 0) { + throw new Error(`Failed to load capped parquet data from ${parquetPath}.`); + } + return tables.slice(1).reduce((merged, part) => merged.concat(part), tables[0]); + } + + protected async loadParquetFileBytesAtPath(path: string): Promise { + try { + const parquetBytes = await this.storeRoot.store.get(`/${path}`); + const normalizedBytes = toUint8Array(parquetBytes); + if (!normalizedBytes || !isParquetFileBytes(normalizedBytes)) { + return null; + } + return normalizedBytes; + } catch { + return null; + } + } + + private async resolveParquetTableColumns( + parquetPath: string, + columns: string[] | undefined, + readSchema: ParquetModule['readSchema'], + schemaBytesFromPath?: Uint8Array | null + ): Promise { + if (!columns?.length) { + return undefined; + } let indexColumnName: string | undefined; + try { + const schemaBytes = schemaBytesFromPath ?? (await this.loadParquetSchemaBytes(parquetPath)); + if (schemaBytes) { + const wasmSchema = readSchema(schemaBytes); + const arrowTableForSchema = await tableFromIPC(wasmSchema.intoIPCStream()); + indexColumnName = tableToIndexColumnName(arrowTableForSchema); + } + } catch (e: unknown) { + //@ts-expect-error e.message not a property of e: unknown + console.warn(`Failed to load parquet schema bytes for ${parquetPath}: ${e.message}`); + } + if (indexColumnName && !columns.includes(indexColumnName)) { + return [...columns, indexColumnName]; + } + return columns; + } + + private async readParquetTableFromFileBytes( + parquetBytes: Uint8Array, + columns: string[] | undefined, + readParquet: ParquetModule['readParquet'], + readSchema: ParquetModule['readSchema'], + parquetPath: string + ): Promise { + let normalizedBytes = parquetBytes; + if (!ArrayBuffer.isView(normalizedBytes)) { + normalizedBytes = new Uint8Array(normalizedBytes); + } + + let resolvedColumns = columns; if (columns?.length) { - // If columns are specified, we also want to ensure that the index column is included. - // Otherwise, the user wants the full table anyway. - - // We first try to load the schema bytes to determine the index column name. - // Perhaps in the future SpatialData can store the index column name - // in the .zattrs so that we do not need to load the schema first, - // since only certain stores such as FetchStores support getRange. - // Reference: https://github.com/scverse/spatialdata/issues/958 - try { - const schemaBytes = await this.loadParquetSchemaBytes(parquetPath); - if (schemaBytes) { - const wasmSchema = readSchema(schemaBytes); - const arrowTableForSchema = await tableFromIPC(wasmSchema.intoIPCStream()); - indexColumnName = tableToIndexColumnName(arrowTableForSchema); - } - } catch (e: unknown) { - // If we fail to load the schema bytes, we can proceed to try to load the full table bytes, - // for instance if range requests are not supported but the full table can be loaded. - //@ts-expect-error e.message not a property of e: unknown - console.warn(`Failed to load parquet schema bytes for ${parquetPath}: ${e.message}`); + resolvedColumns = await this.resolveParquetTableColumns( + parquetPath, + columns, + readSchema + ); + const wasmSchema = readSchema(normalizedBytes); + const arrowTableForSchema = await tableFromIPC(wasmSchema.intoIPCStream()); + const indexColumnName = tableToIndexColumnName(arrowTableForSchema); + if (indexColumnName && resolvedColumns && !resolvedColumns.includes(indexColumnName)) { + resolvedColumns = [...resolvedColumns, indexColumnName]; } } - // Load the full table bytes. - // TODO: can we avoid loading the full table bytes - // if we only need a subset of columns? - // For example, if the store supports - // getRange like above to get the schema bytes. - // See https://github.com/kylebarron/parquet-wasm/issues/758 - let parquetBytes = await this.loadParquetBytes(parquetPath); - if (!parquetBytes) { - throw new Error('Failed to load parquet data from store.'); + const wasmTable = readParquet( + normalizedBytes, + resolvedColumns?.length ? { columns: resolvedColumns } : undefined + ); + return tableFromIPC(wasmTable.intoIPCStream()); + } + + private async loadMultipartParquetTable( + parquetPath: string, + columns: string[] | undefined, + dataset: ParquetDatasetMetadata, + readParquet: ParquetModule['readParquet'], + readSchema: ParquetModule['readSchema'] + ): Promise { + const resolvedColumns = columns?.length + ? await this.resolveParquetTableColumns( + parquetPath, + columns, + readSchema, + dataset.parts[0]?.schemaBytes + ) + : undefined; + + const tables: ArrowTable[] = []; + for (const part of dataset.parts) { + const parquetBytes = await this.loadParquetFileBytesAtPath(part.path); + if (!parquetBytes) { + throw new Error(`Failed to load parquet part at ${part.path}.`); + } + const wasmTable = readParquet( + parquetBytes, + resolvedColumns?.length ? { columns: resolvedColumns } : undefined + ); + tables.push(await tableFromIPC(wasmTable.intoIPCStream())); } - if (!ArrayBuffer.isView(parquetBytes)) { - // This is required because in vitessce-python the - // experimental.invoke store wrapper can return an ArrayBuffer, - // but readParquet expects a Uint8Array. - parquetBytes = new Uint8Array(parquetBytes); + + if (tables.length === 0) { + throw new Error(`Failed to load multipart parquet data from ${parquetPath}.`); } + return tables.slice(1).reduce((merged, part) => merged.concat(part), tables[0]); + } - if (columns?.length && !indexColumnName) { - // The user requested specific columns, but we did not load the schema bytes - // to successfully get the index column name. - // Here we try again to get the index column name, but this - // time from the full table bytes (rather than only the schema-bytes). - const wasmSchema = readSchema(parquetBytes); - /** @type {import('apache-arrow').Table} */ - const arrowTableForSchema = await tableFromIPC(wasmSchema.intoIPCStream()); - indexColumnName = tableToIndexColumnName(arrowTableForSchema); + private async _loadParquetTableUncached( + parquetPath: string, + columns?: string[] + ): Promise { + const { readParquet, readSchema } = await SpatialDataTableSource.parquetModulePromise; + + const dataset = await this.loadParquetDatasetMetadata(parquetPath); + if (dataset && dataset.parts.length > 1) { + return this.loadMultipartParquetTable( + parquetPath, + columns, + dataset, + readParquet, + readSchema + ); } - if (options.columns?.length && indexColumnName) { - options.columns = [...options.columns, indexColumnName]; + const partPaths = await this.discoverMultipartPartPaths(parquetPath); + if (partPaths.length > 1) { + return this.loadMultipartParquetTableFromPartPaths( + parquetPath, + partPaths, + columns, + readParquet, + readSchema + ); } - const wasmTable = readParquet(parquetBytes, options); - /** @type {import('apache-arrow').Table} */ - const arrowTable = await tableFromIPC(wasmTable.intoIPCStream()); - return arrowTable; + let parquetBytes = await this.loadParquetBytes(parquetPath); + if (!parquetBytes) { + throw new Error('Failed to load parquet data from store.'); + } + return this.readParquetTableFromFileBytes( + parquetBytes, + columns, + readParquet, + readSchema, + parquetPath + ); } // TABLE-SPECIFIC METHODS diff --git a/packages/core/src/models/index.ts b/packages/core/src/models/index.ts index 253fef4e..aa1ba2b0 100644 --- a/packages/core/src/models/index.ts +++ b/packages/core/src/models/index.ts @@ -12,6 +12,7 @@ import { shapesAttrsSchema, tableAttrsSchema, } from '../schemas'; +import type { PointsLoadOptions } from '../pointsLoadOptions.js'; import type { ShapesRenderData } from '../shapes'; import { isSpatialData, loadFeatureRowIndexByFeatureIndex } from '../tableAssociations'; import { type BaseTransformation, Identity, parseTransforms } from '../transformations'; @@ -30,6 +31,7 @@ import { Err, Ok } from '../types'; import SpatialDataPointsSource from './VPointsSource'; import SpatialDataShapesSource from './VShapesSource'; import SpatialDataTableSource from './VTableSource'; +import type { PointsFeatureCatalog } from '../pointsTiling.js'; /** * Parameters for creating element instances. @@ -494,6 +496,10 @@ export class ShapesElement extends AbstractSpatialElement<'shapes', ShapesAttrs> }); return renderData; } + + async loadShapesInBounds(options: Parameters[1]) { + return this.vShapes.loadShapesInBounds(`shapes/${this.key}`, options); + } } // ============================================ @@ -528,11 +534,39 @@ export class PointsElement extends AbstractSpatialElement<'points', PointsAttrs> return this.attrs.coordinateTransformations; } - async loadPoints() { - //Error: Unexpected response status 500 INTERNAL SERVER ERROR - //IsADirectoryError: [Errno 21] Is a directory: '/MySpatialData.zarr/points/key/points.parquet' - //we have points.parquet/part.0.parquet etc. - return this.vPoints.loadPoints(`points/${this.key}`); + async loadPoints(options?: PointsLoadOptions) { + return this.vPoints.loadPoints(`points/${this.key}`, options); + } + + async loadRowFeatureCodes(options?: { + memoryCap?: number; + featureCatalog?: PointsFeatureCatalog | null; + }) { + return this.vPoints.loadPointsRowFeatureCodes(`points/${this.key}`, options); + } + + async loadFeatureCounts() { + return this.vPoints.loadFeatureCounts(`points/${this.key}`); + } + + async listFeaturesWithCounts() { + return this.vPoints.listPointsFeaturesWithCounts(`points/${this.key}`); + } + + async getPointsTilingMetadata() { + return this.vPoints.getPointsTilingMetadata(`points/${this.key}`); + } + + async loadPointsInBounds(options: Parameters[1]) { + return this.vPoints.loadPointsInBounds(`points/${this.key}`, options); + } + + async listFeatures() { + return this.vPoints.listPointsFeatures(`points/${this.key}`); + } + + async getParquetRowCount() { + return this.vPoints.getPointsParquetRowCount(`points/${this.key}`); } } diff --git a/packages/core/src/parquetWasmLoader.ts b/packages/core/src/parquetWasmLoader.ts new file mode 100644 index 00000000..9252738c --- /dev/null +++ b/packages/core/src/parquetWasmLoader.ts @@ -0,0 +1,121 @@ +export interface ParquetWasmTableLike { + intoIPCStream(): Uint8Array; +} + +export interface ParquetWasmFileMetadata { + numRows(): number; +} + +export interface ParquetWasmRowGroupMetadata { + numRows(): number; + fileOffset(): number | bigint; + compressedSize(): number | bigint; +} + +export interface ParquetWasmMetadata { + fileMetadata(): ParquetWasmFileMetadata; + numRowGroups(): number; + rowGroup(index: number): ParquetWasmRowGroupMetadata; +} + +export interface ParquetRowGroupReadOptions { + columns?: string[]; + limit?: number; + offset?: number; +} + +export interface ParquetModule { + readParquet: (bytes: Uint8Array, options?: ParquetRowGroupReadOptions) => ParquetWasmTableLike; + readSchema: (bytes: Uint8Array) => ParquetWasmTableLike; + readMetadata?: (bytes: Uint8Array) => ParquetWasmMetadata; + readParquetRowGroup?: ( + schemaBytes: Uint8Array, + rowGroupBytes: Uint8Array, + rowGroupIndex: number, + options?: ParquetRowGroupReadOptions + ) => ParquetWasmTableLike; +} + +function normalizeParquetModule(module: unknown): ParquetModule { + if (typeof module !== 'object' || module === null) { + throw new Error('parquet-wasm module did not load as an object'); + } + // External WASM builds have drifted API surfaces and incomplete declarations; + // keep the boundary narrow and capability-check every optional method. + const candidate = module as Record; + const { readParquet, readSchema, readMetadata, readParquetRowGroup } = candidate; + if (typeof readParquet !== 'function' || typeof readSchema !== 'function') { + throw new Error('parquet-wasm module is missing required readParquet/readSchema APIs'); + } + return { + readParquet: readParquet as ParquetModule['readParquet'], + readSchema: readSchema as ParquetModule['readSchema'], + readMetadata: + typeof readMetadata === 'function' + ? (readMetadata as ParquetModule['readMetadata']) + : undefined, + readParquetRowGroup: + typeof readParquetRowGroup === 'function' + ? (readParquetRowGroup as ParquetModule['readParquetRowGroup']) + : undefined, + }; +} + +async function initializeParquetModule(module: unknown) { + if (typeof module !== 'object' || module === null) { + return; + } + const record = module as Record; + const initSync = record.initSync; + const defaultInit = record.default; + + // Vitest/Node load the vendored browser ESM glue; initialize WASM from disk + // because undici cannot fetch file:// URLs. + if (import.meta.url.startsWith('file:') && typeof initSync === 'function') { + const [{ readFileSync }, { fileURLToPath }, { dirname, join }] = await Promise.all([ + import('node:fs'), + import('node:url'), + import('node:path'), + ]); + const wasmPath = join( + dirname(fileURLToPath(import.meta.url)), + '../vendor/parquet-wasm/parquet_wasm_bg.wasm' + ); + initSync({ module: readFileSync(wasmPath) }); + return; + } + + if (typeof defaultInit === 'function') { + await defaultInit(); + } +} + +function parquetModuleSupportsRowGroupReads(module: ParquetModule): boolean { + return ( + typeof module.readMetadata === 'function' && typeof module.readParquetRowGroup === 'function' + ); +} + +async function loadVendoredParquetModule(): Promise { + const module: unknown = await import( + /* @vite-ignore */ + '../vendor/parquet-wasm/parquet_wasm.js' + ); + await initializeParquetModule(module); + const normalized = normalizeParquetModule(module); + if (!parquetModuleSupportsRowGroupReads(normalized)) { + throw new Error( + 'Vendored parquet-wasm is missing required row-group APIs (readMetadata, readParquetRowGroup)' + ); + } + return normalized; +} + +let parquetModulePromise: Promise | undefined; + +export function getParquetModule(): Promise { + if (!parquetModulePromise) { + parquetModulePromise = loadVendoredParquetModule(); + } + return parquetModulePromise; +} diff --git a/packages/core/src/pointsFeatures.ts b/packages/core/src/pointsFeatures.ts new file mode 100644 index 00000000..4ffcd304 --- /dev/null +++ b/packages/core/src/pointsFeatures.ts @@ -0,0 +1,310 @@ +import { Type } from 'apache-arrow'; +import type { Table, Vector } from 'apache-arrow'; +import { + isMortonSentinelValue, + MORTON_CODE_2D_COLUMN, + type PointsFeatureCatalog, + type PointsFeatureEntry, +} from './pointsTiling.js'; + +function dictionaryStrings(column: Vector): string[] | null { + if (column.type.typeId !== Type.Dictionary) { + return null; + } + for (const chunk of column.data) { + const dictionary = chunk.dictionary; + if (dictionary && dictionary.length > 0) { + return dictionary.toArray().map((value: unknown) => (value == null ? '' : String(value))); + } + } + return null; +} + +function resolveFeatureName(nameValue: unknown, dictionary: string[] | null): string { + if (nameValue == null) { + return ''; + } + if (dictionary && typeof nameValue === 'number' && Number.isFinite(nameValue)) { + return dictionary[nameValue] ?? ''; + } + return String(nameValue); +} + +export function buildFeatureCatalogFromColumns( + featureKey: string, + nameColumn: Vector, + codeColumn: Vector | null, + mortonColumn: Vector | null, + numRows: number +): PointsFeatureCatalog { + const codeToName = new Map(); + const nameToCode = new Map(); + accumulateFeatureCatalogFromVectors( + codeToName, + nameToCode, + nameColumn, + codeColumn, + mortonColumn, + numRows + ); + return featureCatalogFromCodeMap(featureKey, codeToName); +} + +export function accumulateFeatureCatalogFromTable( + codeToName: Map, + nameToCode: Map, + table: Table, + featureKey: string, + featureCodeColumnName: string | undefined, + options: { skipMortonSentinels?: boolean } = {} +): void { + const nameColumn = table.getChild(featureKey); + if (!nameColumn) { + return; + } + const codeColumn = featureCodeColumnName ? table.getChild(featureCodeColumnName) : null; + const mortonColumn = + options.skipMortonSentinels === true ? table.getChild(MORTON_CODE_2D_COLUMN) : null; + accumulateFeatureCatalogFromVectors( + codeToName, + nameToCode, + nameColumn, + codeColumn, + mortonColumn, + table.numRows + ); +} + +export function featureCatalogFromCodeMap( + featureKey: string, + codeToName: Map +): PointsFeatureCatalog { + const entries: PointsFeatureEntry[] = [...codeToName.entries()] + .sort((left, right) => left[0] - right[0]) + .map(([code, name]) => ({ code, name })); + return { featureKey, entries }; +} + +/** Row-group reads can yield empty names for dictionary-only columns; prefer readParquet. */ +export function featureCatalogNeedsParquetFallback(codeToName: Map): boolean { + if (codeToName.size === 0) { + return true; + } + return [...codeToName.values()].every((name) => name.length === 0); +} + +export function featureCodeMapFromCatalog( + catalog: PointsFeatureCatalog | null | undefined +): Map | undefined { + if (!catalog) { + return undefined; + } + return new Map(catalog.entries.map((entry) => [entry.name, entry.code])); +} + +function accumulateFeatureCatalogFromVectors( + codeToName: Map, + nameToCode: Map, + nameColumn: Vector, + codeColumn: Vector | null, + mortonColumn: Vector | null, + numRows: number +): void { + const dictionary = dictionaryStrings(nameColumn); + + for (let rowIndex = 0; rowIndex < numRows; rowIndex += 1) { + if (mortonColumn && rowIndex < 4 && isMortonSentinelValue(mortonColumn.get(rowIndex))) { + continue; + } + const name = resolveFeatureName(nameColumn.get(rowIndex), dictionary); + if (codeColumn) { + const codeValue = codeColumn.get(rowIndex); + const code = typeof codeValue === 'number' ? codeValue : Number(codeValue); + if (!Number.isFinite(code)) { + continue; + } + if (!codeToName.has(code)) { + codeToName.set(code, name); + } + continue; + } + if (!nameToCode.has(name)) { + nameToCode.set(name, nameToCode.size); + } + const code = nameToCode.get(name); + if (code !== undefined && !codeToName.has(code)) { + codeToName.set(code, name); + } + } +} + +export function mergeDictionaryFeatureCatalogEntries( + codeToName: Map, + nameColumn: Vector +): boolean { + const dictionary = dictionaryStrings(nameColumn); + if (dictionary && dictionary.length > 0) { + for (let code = 0; code < dictionary.length; code += 1) { + if (!codeToName.has(code)) { + codeToName.set(code, dictionary[code] ?? ''); + } + } + return true; + } + + const numRows = nameColumn.length; + if (numRows <= 0) { + return false; + } + let added = false; + for (let row = 0; row < numRows; row += 1) { + const code = getDictionaryIndexAt(nameColumn, row); + if (code === null || !Number.isFinite(code) || codeToName.has(code)) { + continue; + } + const decoded = nameColumn.get(row); + const name = decoded == null ? '' : String(decoded); + if (name.length > 0) { + codeToName.set(code, name); + added = true; + } + } + return added; +} + +export function buildFeatureCatalogFromDictionaryOnly( + featureKey: string, + nameColumn: Vector, + _codeColumn: Vector | null +): PointsFeatureCatalog | null { + const codeToName = new Map(); + if (!mergeDictionaryFeatureCatalogEntries(codeToName, nameColumn)) { + return null; + } + if (codeToName.size === 0) { + return null; + } + return featureCatalogFromCodeMap(featureKey, codeToName); +} + +export function isDictionaryFeatureColumn(column: Vector): boolean { + return column.type.typeId === Type.Dictionary; +} + +function getDictionaryIndexAt(column: Vector, row: number): number | null { + if (!isDictionaryFeatureColumn(column) || row < 0 || row >= column.length) { + return null; + } + let currentRow = 0; + for (const chunk of column.data) { + const values = chunk.values; + if (!values) { + continue; + } + const chunkLength = chunk.length ?? values.length; + if (row < currentRow + chunkLength) { + const valueIndex = (chunk.offset ?? 0) + (row - currentRow); + if (valueIndex < 0 || valueIndex >= values.length) { + return null; + } + const index = values[valueIndex]; + if (typeof index === 'number' && Number.isFinite(index)) { + return index; + } + if (typeof index === 'bigint') { + return Number(index); + } + const asNumber = Number(index); + return Number.isFinite(asNumber) ? asNumber : null; + } + currentRow += chunkLength; + } + return null; +} + +function dictionaryIndexArray(column: Vector, numRows: number): Int32Array | null { + if (!isDictionaryFeatureColumn(column)) { + return null; + } + const rowCount = Math.min(numRows, column.length); + if (rowCount <= 0) { + return null; + } + const out = new Int32Array(rowCount); + for (let row = 0; row < rowCount; row += 1) { + const index = getDictionaryIndexAt(column, row); + out[row] = index !== null && Number.isFinite(index) ? index : 0; + } + return out; +} + +/** Per-row integer codes for feature filtering (explicit codes column or dictionary indices). */ +export function resolveRowFeatureCodesFromTable( + table: Table, + featureKey: string, + featureCodeColumnName: string | undefined, + featureCodeByName?: ReadonlyMap +): ArrayLike | undefined { + const nameColumn = table.getChild(featureKey); + if (featureCodeColumnName) { + return table.getChild(featureCodeColumnName)?.toArray(); + } + if (!nameColumn) { + return undefined; + } + const dictionary = dictionaryStrings(nameColumn); + + if (!featureCodeByName) { + return undefined; + } + + const out = new Int32Array(table.numRows); + for (let rowIndex = 0; rowIndex < table.numRows; rowIndex += 1) { + const name = resolveFeatureName(nameColumn.get(rowIndex), dictionary); + out[rowIndex] = featureCodeByName.get(name) ?? -1; + } + return out; +} + +export function featureFilterNeedsRowCodes( + featureCodes: readonly number[] | undefined, + featureCodeColumnName: string | undefined, + featureKey: string, + fields: string[] +): boolean { + if (featureCodes === undefined) { + return false; + } + if (featureCodeColumnName) { + return true; + } + return fields.includes(featureKey); +} + +/** Histogram of integer feature codes (single pass, worker-safe). */ +export function countFeatureCodesHistogram( + sourceFeatureCodes: ArrayLike +): Map { + const counts = new Map(); + for (let index = 0; index < sourceFeatureCodes.length; index += 1) { + const code = sourceFeatureCodes[index]; + if (typeof code !== 'number' || !Number.isFinite(code)) { + continue; + } + counts.set(code, (counts.get(code) ?? 0) + 1); + } + return counts; +} + +export function mergeFeatureCountsIntoCatalog( + catalog: PointsFeatureCatalog, + counts: ReadonlyMap +): PointsFeatureCatalog { + return { + ...catalog, + entries: catalog.entries.map((entry) => ({ + ...entry, + count: counts.get(entry.code) ?? entry.count, + })), + }; +} diff --git a/packages/core/src/pointsLimits.ts b/packages/core/src/pointsLimits.ts new file mode 100644 index 00000000..b9198f1d --- /dev/null +++ b/packages/core/src/pointsLimits.ts @@ -0,0 +1,109 @@ +/** Maximum rows allowed for full-table points preload (canonical scatter path). */ +export const POINTS_PRELOAD_MAX_ROWS = 4_000_000; + +/** Default in-memory row cap for preloaded scatter (layer override via props panel). */ +export const DEFAULT_POINTS_MEMORY_CAP = POINTS_PRELOAD_MAX_ROWS; + +/** Default render row cap — points kept in memory may exceed this. */ +export const DEFAULT_POINTS_RENDER_CAP = POINTS_PRELOAD_MAX_ROWS; + +export interface PointsColumnarLike { + shape: number[]; + data: ArrayLike[]; + pointCount?: number; +} + +export function resolvePointsMemoryCap(configured?: number): number { + if (configured !== undefined && Number.isFinite(configured) && configured > 0) { + return Math.floor(configured); + } + return DEFAULT_POINTS_MEMORY_CAP; +} + +export function resolvePointsRenderCap(configured?: number): number | undefined { + if (configured === undefined) { + return DEFAULT_POINTS_RENDER_CAP; + } + if (!Number.isFinite(configured) || configured <= 0) { + return undefined; + } + return Math.floor(configured); +} + +export function columnarPointCount(shape: number[], data: ArrayLike[]): number { + if (shape.length >= 2 && Number.isFinite(shape[1])) { + return shape[1]; + } + return data[0]?.length ?? shape[0] ?? 0; +} + +export function applyRenderCapToColumnar( + batch: T, + renderCap: number | undefined +): T { + if (renderCap === undefined) { + return batch; + } + const pointCount = batch.pointCount ?? columnarPointCount(batch.shape, batch.data); + if (pointCount <= renderCap) { + return batch; + } + const axisCount = batch.shape[0] ?? batch.data.length; + const nextData = batch.data.map((column) => { + if (column instanceof Float32Array) { + return column.subarray(0, renderCap); + } + return Float32Array.from(column as ArrayLike).subarray(0, renderCap); + }); + return { + ...batch, + data: nextData, + shape: [axisCount, renderCap], + pointCount: renderCap, + }; +} + +export class PointsPreloadTooLargeError extends Error { + readonly rowCount: number; + readonly maxRows: number; + + constructor(rowCount: number, maxRows: number = POINTS_PRELOAD_MAX_ROWS) { + super( + `${rowCount.toLocaleString()} points exceeds the ${maxRows.toLocaleString()} preload limit — use a Morton-sorted element or tiled path` + ); + this.name = 'PointsPreloadTooLargeError'; + this.rowCount = rowCount; + this.maxRows = maxRows; + } +} + +export function preloadedColumnarPointCount(shape: number[], data: ArrayLike[]): number { + if (shape.length >= 2 && Number.isFinite(shape[1])) { + return shape[1]; + } + const fromData = data[0]?.length; + if (typeof fromData === 'number') { + return fromData; + } + return shape[0] ?? 0; +} + +export function exceedsPointsPreloadLimit(rowCount: number): boolean { + return rowCount > POINTS_PRELOAD_MAX_ROWS; +} + +export function pointsPreloadTruncatedMessage(loadedCount: number, totalCount: number): string { + return `Showing ${loadedCount.toLocaleString()} of ${totalCount.toLocaleString()} points (preload limit ${POINTS_PRELOAD_MAX_ROWS.toLocaleString()})`; +} + +export function pointsFilteredMemoryCapMessage( + loadedCount: number, + memoryCap: number, + scannedRows?: number +): string { + const scanned = + scannedRows !== undefined + ? ` after scanning ${scannedRows.toLocaleString()} rows` + : ''; + return `Showing ${loadedCount.toLocaleString()} matching points (memory cap ${memoryCap.toLocaleString()}${scanned})`; +} diff --git a/packages/core/src/pointsLoadOptions.ts b/packages/core/src/pointsLoadOptions.ts new file mode 100644 index 00000000..0602cdb9 --- /dev/null +++ b/packages/core/src/pointsLoadOptions.ts @@ -0,0 +1,31 @@ +export interface PointsLoadProgress { + scannedRows: number; + matchedRows: number; + partIndex: number; + partCount: number; +} + +export interface PointsLoadOptions { + /** Max rows to retain in memory (unfiltered cap or filtered match cap). */ + memoryCap?: number; + /** When set, scan the dataset for matching features instead of capping raw rows first. */ + featureCodes?: readonly number[]; + /** Progress callback for filtered scans (main thread). */ + onProgress?: (progress: PointsLoadProgress) => void; + /** + * When true with {@link featureCodes}, scan the full dataset for matches (slow). + * Default UI uses in-memory runtime filtering instead. + */ + fullDatasetFeatureScan?: boolean; +} + +export interface PointsLoadResult { + shape: number[]; + data: ArrayLike[]; + featureCodes?: ArrayLike; + totalRowCount?: number; + preloadTruncated?: boolean; + /** Rows scanned when loading with an active feature filter. */ + scannedRowCount?: number; + filterActive?: boolean; +} diff --git a/packages/core/src/pointsLoader.ts b/packages/core/src/pointsLoader.ts new file mode 100644 index 00000000..67e6fc0f --- /dev/null +++ b/packages/core/src/pointsLoader.ts @@ -0,0 +1,191 @@ +import type { PointsElement } from './models/index.js'; +import type { PointsLoadMode } from './types.js'; +import type { + PointsInBoundsResponse, + PointsTilingMetadata, + SpatialBounds, +} from './pointsTiling.js'; + +export type PointsEncodingKind = + | 'preloaded-columnar' + | 'morton-tiled' + | 'geoarrow-binary' + | 'geoarrow-tiled'; + +export type PointsBatchFormat = 'columnar-ndarray' | 'arrow-record-batch'; + +export interface PointsLoaderCapabilities { + kind: PointsEncodingKind; + batchFormat: PointsBatchFormat; + bounds?: SpatialBounds; + supportsViewportTiles: boolean; + supportsFeatureCodes?: boolean; +} + +export interface ColumnarNdarrayPointsBatch { + format: 'columnar-ndarray'; + data: ArrayLike[]; + shape: number[]; + bounds?: SpatialBounds; + loadMode?: PointsLoadMode; + pointCount?: number; +} + +export type PointsBatch = ColumnarNdarrayPointsBatch; + +export interface PointsLoadInBoundsOptions { + bounds: SpatialBounds; + featureCodes?: readonly number[]; + signal?: AbortSignal; +} + +export interface CorePointsLoader { + readonly capabilities: PointsLoaderCapabilities; + loadInBounds(options: PointsLoadInBoundsOptions): Promise; + loadAll?(options?: { signal?: AbortSignal }): Promise; +} + +export interface PreloadedColumnarInput { + shape: number[]; + data: ArrayLike[]; +} + +export function resolvePointsEncoding( + preloaded: PreloadedColumnarInput | null | undefined, + metadata: PointsTilingMetadata | null | undefined, + wantsOptimized: boolean +): PointsEncodingKind { + if (preloaded) { + return 'preloaded-columnar'; + } + if (wantsOptimized && metadata?.supportsRowGroupRangeReads && metadata.bounds) { + return 'morton-tiled'; + } + return 'preloaded-columnar'; +} + +function columnarPointCount(shape: number[], data: ArrayLike[]): number { + if (shape.length >= 2 && Number.isFinite(shape[1])) { + return shape[1]; + } + const fromData = data[0]?.length; + if (typeof fromData === 'number') { + return fromData; + } + return shape[0] ?? 0; +} + +function toColumnarBatch( + result: PointsInBoundsResponse | PreloadedColumnarInput, + overrides?: Partial +): ColumnarNdarrayPointsBatch { + const shape = result.shape ?? []; + const data = result.data; + const pointCount = columnarPointCount(shape, data); + return { + format: 'columnar-ndarray', + data, + shape, + bounds: 'bounds' in result ? result.bounds : overrides?.bounds, + loadMode: 'loadMode' in result ? result.loadMode : overrides?.loadMode, + pointCount, + ...overrides, + }; +} + +export function createMortonTiledPointsLoader( + element: PointsElement, + metadata: PointsTilingMetadata +): CorePointsLoader { + const capabilities: PointsLoaderCapabilities = { + kind: 'morton-tiled', + batchFormat: 'columnar-ndarray', + bounds: metadata.bounds, + supportsViewportTiles: true, + supportsFeatureCodes: Boolean(metadata.featureKey), + }; + + return { + capabilities, + async loadInBounds(options: PointsLoadInBoundsOptions): Promise { + const result = await element.loadPointsInBounds(options); + return toColumnarBatch(result); + }, + }; +} + +export function createPreloadedColumnarPointsLoader( + element: PointsElement, + preloaded: PreloadedColumnarInput +): CorePointsLoader { + const batch = toColumnarBatch(preloaded, { loadMode: 'full-filter' }); + const capabilities: PointsLoaderCapabilities = { + kind: 'preloaded-columnar', + batchFormat: 'columnar-ndarray', + bounds: inferBoundsFromColumnar(preloaded), + supportsViewportTiles: false, + supportsFeatureCodes: true, + }; + + return { + capabilities, + async loadInBounds(options: PointsLoadInBoundsOptions): Promise { + void element; + void options; + return batch; + }, + async loadAll() { + return batch; + }, + }; +} + +function inferBoundsFromColumnar(preloaded: PreloadedColumnarInput) { + const xs = preloaded.data[0]; + const ys = preloaded.data[1]; + if (!xs || !ys || preloaded.shape[0] === 0) { + return undefined; + } + let minX = Number.POSITIVE_INFINITY; + let maxX = Number.NEGATIVE_INFINITY; + let minY = Number.POSITIVE_INFINITY; + let maxY = Number.NEGATIVE_INFINITY; + const count = preloaded.shape[0]; + for (let index = 0; index < count; index += 1) { + const x = xs[index]; + const y = ys[index]; + if (x < minX) minX = x; + if (x > maxX) maxX = x; + if (y < minY) minY = y; + if (y > maxY) maxY = y; + } + if (!Number.isFinite(minX) || !Number.isFinite(minY)) { + return undefined; + } + return { minX, minY, maxX, maxY }; +} + +export function createPointsLoaderForElement( + element: PointsElement, + options: { + preloaded?: PreloadedColumnarInput | null; + tilingMetadata?: PointsTilingMetadata | null; + wantsOptimized: boolean; + } +): CorePointsLoader | null { + const encoding = resolvePointsEncoding( + options.preloaded, + options.tilingMetadata, + options.wantsOptimized + ); + + if (encoding === 'morton-tiled' && options.tilingMetadata?.bounds) { + return createMortonTiledPointsLoader(element, options.tilingMetadata); + } + + if (options.preloaded) { + return createPreloadedColumnarPointsLoader(element, options.preloaded); + } + + return null; +} diff --git a/packages/core/src/pointsTiling.ts b/packages/core/src/pointsTiling.ts new file mode 100644 index 00000000..5d3a2cd5 --- /dev/null +++ b/packages/core/src/pointsTiling.ts @@ -0,0 +1,373 @@ +import type { Table as ArrowTable } from 'apache-arrow'; +import type { AxisAlignedBounds, PointsColumnarData } from './spatialViewFit.js'; + +export const MORTON_CODE_2D_COLUMN = 'morton_code_2d'; +export const MORTON_CODE_EXTREME_VALUE_INDICATOR = 0; +export const MORTON_CODE_BITS_PER_AXIS = 16; +export const MORTON_CODE_VALUE_MAX = 2 ** MORTON_CODE_BITS_PER_AXIS - 1; + +export type SpatialBounds = AxisAlignedBounds; + +export interface PointsFeatureEntry { + code: number; + name: string; + /** Row count in the dataset or loaded sample, when known. */ + count?: number; +} + +export interface PointsFeatureCatalog { + featureKey: string; + entries: PointsFeatureEntry[]; +} + +export interface PointsInBoundsOptions { + bounds: SpatialBounds; + /** Integer codes matching `{feature_key}_codes` in the Morton Parquet artifact. */ + featureCodes?: readonly number[]; + zoom?: number; + signal?: AbortSignal; + columns?: string[]; +} + +export interface PointsTilingMetadata { + kind: 'morton-points'; + parquetPath: string; + axisNames: string[]; + featureKey?: string; + featureCodeColumnName: string; + mortonCodeColumnName: typeof MORTON_CODE_2D_COLUMN; + totalRows: number; + totalRowGroups: number; + maxRowsPerGroup: number; + rowGroupRowCounts?: number[]; + supportsRowGroupRangeReads: boolean; + bounds?: SpatialBounds; +} + +export type PointsInBoundsResponse = PointsColumnarData & { + bounds: SpatialBounds; + loadMode: 'row-groups' | 'full-filter'; + tiling?: PointsTilingMetadata; + featureIndices?: ArrayLike; +}; + +export function origCoordToNormCoord(x: number, y: number, bbox: SpatialBounds): [number, number] { + const xRange = bbox.maxX - bbox.minX; + const yRange = bbox.maxY - bbox.minY; + if (xRange <= 0 || yRange <= 0) { + return [0, 0]; + } + return [ + Math.max( + 0, + Math.min( + MORTON_CODE_VALUE_MAX, + Math.floor(((x - bbox.minX) / xRange) * MORTON_CODE_VALUE_MAX) + ) + ), + Math.max( + 0, + Math.min( + MORTON_CODE_VALUE_MAX, + Math.floor(((y - bbox.minY) / yRange) * MORTON_CODE_VALUE_MAX) + ) + ), + ]; +} + +function intersects( + ax0: number, + ay0: number, + ax1: number, + ay1: number, + bx0: number, + by0: number, + bx1: number, + by1: number +) { + return !(ax1 < bx0 || bx1 < ax0 || ay1 < by0 || by1 < ay0); +} + +function contained( + ix0: number, + iy0: number, + ix1: number, + iy1: number, + ox0: number, + oy0: number, + ox1: number, + oy1: number +) { + return ox0 <= ix0 && ix0 <= ix1 && ix1 <= ox1 && oy0 <= iy0 && iy0 <= iy1 && iy1 <= oy1; +} + +function cellRange(prefix: number, level: number, bits: number): [number, number] { + const shift = 2 * (bits - level); + const power = 2 ** shift; + return [prefix * power, (prefix + 1) * power - 1]; +} + +export function mergeAdjacentIntervals( + intervals: Array<[number, number]> +): Array<[number, number]> { + if (intervals.length === 0) { + return []; + } + const sorted = [...intervals].sort((a, b) => a[0] - b[0]); + const merged: Array<[number, number]> = [sorted[0]]; + for (const [lo, hi] of sorted.slice(1)) { + const last = merged[merged.length - 1]; + if (lo <= last[1] + 1) { + last[1] = Math.max(last[1], hi); + } else { + merged.push([lo, hi]); + } + } + return merged; +} + +export function zcoverRectangle( + rx0: number, + ry0: number, + rx1: number, + ry1: number, + bits = MORTON_CODE_BITS_PER_AXIS +): Array<[number, number]> { + const maxCoord = 2 ** bits - 1; + const x0 = Math.max(0, Math.min(maxCoord, Math.min(rx0, rx1))); + const x1 = Math.max(0, Math.min(maxCoord, Math.max(rx0, rx1))); + const y0 = Math.max(0, Math.min(maxCoord, Math.min(ry0, ry1))); + const y1 = Math.max(0, Math.min(maxCoord, Math.max(ry0, ry1))); + + const intervals: Array<[number, number]> = []; + const stack: Array<[number, number, number, number, number, number]> = [ + [0, 0, 0, 0, maxCoord, maxCoord], + ]; + + while (stack.length > 0) { + const current = stack.pop(); + if (!current) { + continue; + } + const [prefix, level, xmin, ymin, xmax, ymax] = current; + if (!intersects(xmin, ymin, xmax, ymax, x0, y0, x1, y1)) { + continue; + } + if (contained(xmin, ymin, xmax, ymax, x0, y0, x1, y1) || level === bits) { + intervals.push(cellRange(prefix, level, bits)); + continue; + } + + const midx = Math.floor((xmin + xmax) / 2); + const midy = Math.floor((ymin + ymax) / 2); + const nextPrefix = prefix * 4; + stack.push([nextPrefix + 0, level + 1, xmin, ymin, midx, midy]); + stack.push([nextPrefix + 1, level + 1, midx + 1, ymin, xmax, midy]); + stack.push([nextPrefix + 2, level + 1, xmin, midy + 1, midx, ymax]); + stack.push([nextPrefix + 3, level + 1, midx + 1, midy + 1, xmax, ymax]); + } + + return mergeAdjacentIntervals(intervals); +} + +export function mortonIntervalsForBounds( + allPointsBounds: SpatialBounds, + queryBounds: SpatialBounds +): Array<[number, number]> { + const [x0, y0] = origCoordToNormCoord(queryBounds.minX, queryBounds.minY, allPointsBounds); + const [x1, y1] = origCoordToNormCoord(queryBounds.maxX, queryBounds.maxY, allPointsBounds); + return zcoverRectangle(x0, y0, x1, y1); +} + +function getNumericValue(value: unknown): number | null { + if (typeof value === 'number' && Number.isFinite(value)) { + return value; + } + if (typeof value === 'bigint') { + return Number(value); + } + return null; +} + +export function isMortonSentinelValue(value: unknown): boolean { + return getNumericValue(value) === MORTON_CODE_EXTREME_VALUE_INDICATOR; +} + +export function extractSentinelBoundingBox( + table: ArrowTable, + xColumnName = 'x', + yColumnName = 'y', + mortonColumnName = MORTON_CODE_2D_COLUMN +): SpatialBounds | null { + const xColumn = table.getChild(xColumnName); + const yColumn = table.getChild(yColumnName); + const mortonColumn = table.getChild(mortonColumnName); + if (!xColumn || !yColumn || !mortonColumn) { + return null; + } + + const maxRows = Math.min(4, table.numRows); + const xs: number[] = []; + const ys: number[] = []; + for (let i = 0; i < maxRows; i++) { + if (!isMortonSentinelValue(mortonColumn.get(i))) { + break; + } + const x = getNumericValue(xColumn.get(i)); + const y = getNumericValue(yColumn.get(i)); + if (x === null || y === null) { + continue; + } + xs.push(x); + ys.push(y); + } + if (xs.length < 2 || ys.length < 2) { + return null; + } + return { + minX: Math.min(...xs), + minY: Math.min(...ys), + maxX: Math.max(...xs), + maxY: Math.max(...ys), + }; +} + +export function featureCodeAllowSet( + featureCodes: readonly number[] | undefined +): Set | null { + if (featureCodes === undefined) { + return null; + } + return new Set(featureCodes); +} + +export function rowMatchesFeatureCode(code: unknown, allowed: Set | null): boolean { + if (!allowed) { + return true; + } + return typeof code === 'number' && Number.isFinite(code) && allowed.has(code); +} + +/** + * Future investigation: scan+compact loops below are hot paths for large + * preloaded datasets. Candidates include WASM SIMD and WebGPU compute (e.g. + * typegpu) for parallel index selection and column compaction. Worker offload + * is the near-term fix; GPU/WASM is a follow-up benchmark task. + * + * FBO-based render caching for viewport-stable layers should plug into the + * broader Render Stack compositing story (Group Entry, Viv/deck stacking) via + * shared cache utilities — not a points-only optimization. + */ +export function filterColumnarByFeatureCodes( + data: PointsColumnarData, + featureCodes: readonly number[] | undefined, + sourceFeatureCodes?: ArrayLike +): PointsColumnarData { + const allowedFeatureCodes = featureCodeAllowSet(featureCodes); + if (allowedFeatureCodes === null || !sourceFeatureCodes) { + return data; + } + if (allowedFeatureCodes.size === 0) { + const axisCount = data.shape?.[0] ?? data.data.length; + const empty = new Float32Array(0); + const emptyData = axisCount >= 3 && data.data[2] ? [empty, empty, empty] : [empty, empty]; + return { shape: [axisCount, 0], data: emptyData }; + } + + const xs = data.data[0]; + const ys = data.data[1]; + const zs = data.data[2]; + const keep: number[] = []; + const n = Math.min(xs?.length ?? 0, ys?.length ?? 0); + for (let index = 0; index < n; index += 1) { + if (!rowMatchesFeatureCode(sourceFeatureCodes[index], allowedFeatureCodes)) { + continue; + } + keep.push(index); + } + + if (keep.length === n) { + return data; + } + + const outX = new Float32Array(keep.length); + const outY = new Float32Array(keep.length); + const outZ = zs ? new Float32Array(keep.length) : undefined; + for (let index = 0; index < keep.length; index += 1) { + const sourceIndex = keep[index]; + outX[index] = xs[sourceIndex]; + outY[index] = ys[sourceIndex]; + if (outZ) { + outZ[index] = zs[sourceIndex] ?? 0; + } + } + + return { + shape: [outZ ? 3 : 2, keep.length], + data: outZ ? [outX, outY, outZ] : [outX, outY], + }; +} + +export function filterPointsToBounds( + data: PointsColumnarData, + bounds: SpatialBounds, + featureIndices?: ArrayLike, + featureCodes?: readonly number[], + sourceFeatureCodes?: ArrayLike +): PointsInBoundsResponse { + const allowedFeatureCodes = featureCodeAllowSet(featureCodes); + const xs = data.data[0]; + const ys = data.data[1]; + const zs = data.data[2]; + const keep: number[] = []; + const n = Math.min(xs?.length ?? 0, ys?.length ?? 0); + for (let i = 0; i < n; i++) { + const x = xs[i]; + const y = ys[i]; + if ( + !Number.isFinite(x) || + !Number.isFinite(y) || + x < bounds.minX || + x > bounds.maxX || + y < bounds.minY || + y > bounds.maxY + ) { + continue; + } + if ( + allowedFeatureCodes && + !rowMatchesFeatureCode(sourceFeatureCodes?.[i], allowedFeatureCodes) + ) { + continue; + } + keep.push(i); + } + + const outX = new Float32Array(keep.length); + const outY = new Float32Array(keep.length); + const outZ = zs ? new Float32Array(keep.length) : undefined; + const outFeatureIndices = featureIndices ? new Uint32Array(keep.length) : undefined; + for (let i = 0; i < keep.length; i++) { + const sourceIndex = keep[i]; + outX[i] = xs[sourceIndex]; + outY[i] = ys[sourceIndex]; + if (outZ) { + outZ[i] = zs?.[sourceIndex] ?? 0; + } + if (outFeatureIndices) { + outFeatureIndices[i] = featureIndices?.[sourceIndex] ?? 0; + } + } + + return { + data: outZ ? [outX, outY, outZ] : [outX, outY], + shape: [outZ ? 3 : 2, keep.length], + bounds, + loadMode: 'full-filter', + featureIndices: outFeatureIndices, + }; +} + +export function boundsFromStoredPointsBounds(bounds: SpatialBounds): SpatialBounds { + return bounds; +} diff --git a/packages/core/src/spatialViewFit.ts b/packages/core/src/spatialViewFit.ts index 3850713b..b1ebc827 100644 --- a/packages/core/src/spatialViewFit.ts +++ b/packages/core/src/spatialViewFit.ts @@ -22,7 +22,7 @@ export type OrthographicViewState2D = { /** Ndarray-style columnar points: data[0]=x, data[1]=y, optional data[2]=z. */ export type PointsColumnarData = { - data: number[][]; + data: ArrayLike[]; shape?: number[]; }; diff --git a/packages/core/src/types.ts b/packages/core/src/types.ts index f2cfe45b..a515fad5 100644 --- a/packages/core/src/types.ts +++ b/packages/core/src/types.ts @@ -94,3 +94,4 @@ export type SDataProps = { selection?: ElementName[]; rootStore: ConsolidatedStore; }; +export type PointsLoadMode = 'row-groups' | 'full-filter' | 'clipped'; diff --git a/packages/core/src/workers/index.ts b/packages/core/src/workers/index.ts new file mode 100644 index 00000000..796660f5 --- /dev/null +++ b/packages/core/src/workers/index.ts @@ -0,0 +1,26 @@ +export { + buildFeatureCatalogInWorker, + countFeatureCodesInWorker, + decodeParquetGeometryCappedInWorker, + decodeParquetPartsInWorker, + decodeParquetRowFeatureCodesInWorker, + disablePointsWorker, + enablePointsWorker, + ensurePointsWorker, + filterColumnarByFeatureCodesInWorker, + isPointsWorkerEnabled, + scanMortonRowGroupsInBoundsInWorker, + scanParquetByFeatureCodesInWorker, + scanParquetFeatureCatalogInWorker, + scanParquetFeatureCountsInWorker, + setPointsWorkerDefaultEnabled, + transferablesForParquetPayload, +} from './pointsWorkerClient.js'; + +export type { + PointsWorkerMessage, + PointsWorkerRequest, + PointsWorkerResponse, + ParquetRowGroupBytesChunk, + ParquetWorkerPayload, +} from './pointsWorkerProtocol.js'; diff --git a/packages/core/src/workers/points-worker.ts b/packages/core/src/workers/points-worker.ts new file mode 100644 index 00000000..cf701a6a --- /dev/null +++ b/packages/core/src/workers/points-worker.ts @@ -0,0 +1,472 @@ +import { tableFromIPC, tableToIPC } from 'apache-arrow'; +import { + buildFeatureCatalogFromColumns, +} from '../pointsFeatures.js'; +import { + filterColumnarByFeatureCodes, +} from '../pointsTiling.js'; +import { getParquetModule, type ParquetModule } from '../parquetWasmLoader.js'; +import type { PointsWorkerMessage, PointsWorkerRequest, PointsWorkerResponse } from './pointsWorkerProtocol.js'; +import { + countFeatureCodesFromArray, + decodeParquetPartsToTable, + decodeParquetPayloadToTable, + extractGeometryColumnar, + extractRowFeatureCodesFromTable, + histogramToSortedArrays, + scanFeatureCatalogFromPayload, + scanMortonTableInBounds, + scanTableByFeatureCodes, + scanTableFeatureCounts, +} from './pointsWorkerScan.js'; + +function toFloat32Array(values: ArrayLike): Float32Array { + if (values instanceof Float32Array) { + return values; + } + return Float32Array.from(values); +} + +function handleFilterColumnar(request: Extract) { + const filtered = filterColumnarByFeatureCodes( + { + shape: request.zs ? [3, request.xs.length] : [2, request.xs.length], + data: request.zs ? [request.xs, request.ys, request.zs] : [request.xs, request.ys], + }, + request.featureCodes, + request.sourceFeatureCodes + ); + const xs = toFloat32Array(filtered.data[0]); + const ys = toFloat32Array(filtered.data[1]); + const zs = filtered.data[2] ? toFloat32Array(filtered.data[2]) : undefined; + const shape: number[] = + filtered.shape && filtered.shape.length > 0 + ? filtered.shape + : zs + ? [3, xs.length] + : [2, xs.length]; + return { + ok: true as const, + result: { + kind: 'columnar' as const, + shape, + xs, + ys, + ...(zs ? { zs } : {}), + }, + }; +} + +async function handleDecodeParquet( + request: Extract +): Promise { + const { readParquet } = await getParquetModule(); + const merged = await decodeParquetPartsToTable( + readParquet, + request.parts, + request.columns, + request.maxRows + ); + return { + ok: true, + result: { + kind: 'parquetTable', + tableIpc: tableToIPC(merged), + }, + }; +} + +async function handleDecodeParquetRowFeatureCodes( + request: Extract +): Promise { + const parquetModule = await getParquetModule(); + const table = await decodeParquetPayloadToTable( + parquetModule.readParquet, + parquetModule.readParquetRowGroup, + request, + request.columns, + request.maxRows + ); + const featureCodeByName = request.featureCodeEntries + ? new Map(request.featureCodeEntries.map((entry) => [entry.name, entry.code])) + : undefined; + const codes = extractRowFeatureCodesFromTable( + table, + request.featureKey, + request.featureCodeColumnName, + featureCodeByName + ); + return { + ok: true, + result: { + kind: 'rowFeatureCodes', + codes, + numRows: table.numRows, + }, + }; +} + +async function handleScanParquetFeatureCatalog( + request: Extract +): Promise { + const parquetModule = await getParquetModule(); + const catalog = await scanFeatureCatalogFromPayload( + parquetModule.readParquet, + parquetModule.readParquetRowGroup, + request + ); + if (!catalog) { + return { ok: false, error: 'No features found in parquet catalog scan' }; + } + return { ok: true, result: { kind: 'catalog', catalog } }; +} + +async function handleDecodeParquetGeometryCapped( + request: Extract +): Promise { + const parquetModule = await getParquetModule(); + const table = await decodeParquetPayloadToTable( + parquetModule.readParquet, + parquetModule.readParquetRowGroup, + request, + request.columns, + request.maxRows + ); + const geometry = extractGeometryColumnar(table, request.axisNames); + const featureCodeByName = request.featureCodeEntries + ? new Map(request.featureCodeEntries.map((entry) => [entry.name, entry.code])) + : undefined; + const featureCodes = + request.featureKey !== undefined + ? extractRowFeatureCodesFromTable( + table, + request.featureKey, + request.featureCodeColumnName, + featureCodeByName + ) + : undefined; + return { + ok: true, + result: { + kind: 'columnar', + ...geometry, + ...(featureCodes ? { featureCodes } : {}), + }, + }; +} + +function handleCountFeatureCodes( + request: Extract +): PointsWorkerResponse { + const { codes, countValues } = countFeatureCodesFromArray(request.sourceFeatureCodes); + return { + ok: true, + result: { + kind: 'featureCounts', + codes, + counts: countValues, + }, + }; +} + +async function scanTablesForFeatureCounts( + parquetModule: ParquetModule, + request: Extract +): Promise> { + const columns = [ + request.featureKey, + ...(request.featureCodeColumnName ? [request.featureCodeColumnName] : []), + ]; + const counts = new Map(); + + if (request.rowGroups?.length && parquetModule.readParquetRowGroup) { + for (const chunk of request.rowGroups) { + const table = tableFromIPC( + parquetModule.readParquetRowGroup( + chunk.schemaBytes, + chunk.rowGroupBytes, + chunk.rowGroupIndex, + { columns } + ).intoIPCStream() + ); + scanTableFeatureCounts(table, request.featureKey, request.featureCodeColumnName, counts); + } + return counts; + } + + for (const part of request.parts ?? []) { + const table = tableFromIPC(parquetModule.readParquet(part, { columns }).intoIPCStream()); + scanTableFeatureCounts(table, request.featureKey, request.featureCodeColumnName, counts); + } + return counts; +} + +async function handleScanParquetFeatureCounts( + request: Extract +): Promise { + const parquetModule = await getParquetModule(); + const counts = await scanTablesForFeatureCounts(parquetModule, request); + const { codes, countValues } = histogramToSortedArrays(counts); + return { + ok: true, + result: { + kind: 'featureCounts', + codes, + counts: countValues, + }, + }; +} + +async function scanPayloadByFeatureCodes( + parquetModule: ParquetModule, + request: Extract, + input: { + matchedRows: number; + xs: number[]; + ys: number[]; + zs: number[]; + scannedRows: number; + } +): Promise<{ matchedRows: number; scannedRows: number }> { + const hasZ = request.axisNames.includes('z'); + const columns = [ + ...request.axisNames, + request.featureKey, + ...(request.featureCodeColumnName ? [request.featureCodeColumnName] : []), + ]; + + if (request.rowGroups?.length && parquetModule.readParquetRowGroup) { + for (const chunk of request.rowGroups) { + if (input.matchedRows >= request.memoryCap) { + break; + } + const table = tableFromIPC( + parquetModule.readParquetRowGroup( + chunk.schemaBytes, + chunk.rowGroupBytes, + chunk.rowGroupIndex, + { columns } + ).intoIPCStream() + ); + input.scannedRows += table.numRows; + input.matchedRows = scanTableByFeatureCodes({ + table, + axisNames: request.axisNames, + featureKey: request.featureKey, + featureCodeColumnName: request.featureCodeColumnName, + featureCodes: request.featureCodes, + memoryCap: request.memoryCap, + matchedRows: input.matchedRows, + xs: input.xs, + ys: input.ys, + zs: input.zs, + }); + } + return { matchedRows: input.matchedRows, scannedRows: input.scannedRows }; + } + + for (const part of request.parts ?? []) { + if (input.matchedRows >= request.memoryCap) { + break; + } + const table = tableFromIPC(parquetModule.readParquet(part, { columns }).intoIPCStream()); + input.scannedRows += table.numRows; + input.matchedRows = scanTableByFeatureCodes({ + table, + axisNames: request.axisNames, + featureKey: request.featureKey, + featureCodeColumnName: request.featureCodeColumnName, + featureCodes: request.featureCodes, + memoryCap: request.memoryCap, + matchedRows: input.matchedRows, + xs: input.xs, + ys: input.ys, + zs: input.zs, + }); + } + return { matchedRows: input.matchedRows, scannedRows: input.scannedRows }; +} + +async function handleScanParquetByFeatureCodes( + request: Extract +): Promise { + const parquetModule = await getParquetModule(); + const hasZ = request.axisNames.includes('z'); + const xs: number[] = []; + const ys: number[] = []; + const zs: number[] = []; + const { matchedRows, scannedRows } = await scanPayloadByFeatureCodes(parquetModule, request, { + matchedRows: 0, + xs, + ys, + zs, + scannedRows: 0, + }); + const outX = Float32Array.from(xs); + const outY = Float32Array.from(ys); + const outZ = hasZ ? Float32Array.from(zs) : undefined; + const shape = outZ ? [3, outX.length] : [2, outX.length]; + return { + ok: true, + result: { + kind: 'columnarScan', + shape, + xs: outX, + ys: outY, + ...(outZ ? { zs: outZ } : {}), + matchedRows, + scannedRows, + }, + }; +} + +async function handleScanMortonRowGroupsInBounds( + request: Extract +): Promise { + const parquetModule = await getParquetModule(); + if (!parquetModule.readParquetRowGroup) { + return { ok: false, error: 'parquet-wasm readParquetRowGroup is unavailable in points worker' }; + } + const hasZ = request.axisNames.includes('z'); + const columns = [ + 'x', + 'y', + ...(hasZ ? ['z'] : []), + request.mortonCodeColumnName, + ...(request.featureCodeColumnName ? [request.featureCodeColumnName] : []), + ]; + const xs: number[] = []; + const ys: number[] = []; + const zs: number[] = []; + for (const chunk of request.rowGroups) { + const table = tableFromIPC( + parquetModule.readParquetRowGroup( + chunk.schemaBytes, + chunk.rowGroupBytes, + chunk.rowGroupIndex, + { columns } + ).intoIPCStream() + ); + scanMortonTableInBounds({ + table, + rowGroupIndex: chunk.globalRowGroupIndex ?? chunk.rowGroupIndex, + bounds: request.bounds, + axisNames: request.axisNames, + mortonCodeColumnName: request.mortonCodeColumnName, + featureCodeColumnName: request.featureCodeColumnName, + featureCodes: request.featureCodes, + xs, + ys, + zs, + }); + } + const outX = Float32Array.from(xs); + const outY = Float32Array.from(ys); + const outZ = hasZ ? Float32Array.from(zs) : undefined; + const shape = outZ ? [3, outX.length] : [2, outX.length]; + return { + ok: true, + result: { + kind: 'columnar', + shape, + xs: outX, + ys: outY, + ...(outZ ? { zs: outZ } : {}), + }, + }; +} + +function handleBuildFeatureCatalog( + request: Extract +): PointsWorkerResponse { + const table = tableFromIPC(request.tableIpc); + const nameColumn = table.getChild(request.featureKey); + if (!nameColumn) { + return { ok: false, error: `Feature column "${request.featureKey}" not found` }; + } + const codeColumnName = table.schema.fields + .map((field) => field.name) + .find((name): name is string => typeof name === 'string' && name.endsWith('_codes')); + const codeColumn = codeColumnName ? table.getChild(codeColumnName) : null; + const mortonColumn = table.getChild('morton_code_2d'); + const catalog = buildFeatureCatalogFromColumns( + request.featureKey, + nameColumn, + codeColumn, + mortonColumn, + table.numRows + ); + return { ok: true, result: { kind: 'catalog', catalog } }; +} + +async function handleRequest(request: PointsWorkerRequest): Promise { + switch (request.type) { + case 'filterColumnarByFeatureCodes': + return handleFilterColumnar(request); + case 'decodeParquetParts': + return handleDecodeParquet(request); + case 'buildFeatureCatalog': + return handleBuildFeatureCatalog(request); + case 'decodeParquetRowFeatureCodes': + return handleDecodeParquetRowFeatureCodes(request); + case 'scanParquetFeatureCatalog': + return handleScanParquetFeatureCatalog(request); + case 'decodeParquetGeometryCapped': + return handleDecodeParquetGeometryCapped(request); + case 'countFeatureCodes': + return handleCountFeatureCodes(request); + case 'scanParquetFeatureCounts': + return handleScanParquetFeatureCounts(request); + case 'scanParquetByFeatureCodes': + return handleScanParquetByFeatureCodes(request); + case 'scanMortonRowGroupsInBounds': + return handleScanMortonRowGroupsInBounds(request); + default: { + const _exhaustive: never = request; + return { ok: false, error: `Unknown request type: ${String(_exhaustive)}` }; + } + } +} + +self.onmessage = (event: MessageEvent) => { + const message = event.data; + if (message.direction !== 'request') { + return; + } + void handleRequest(message.request) + .then((response) => { + const reply: PointsWorkerMessage = { id: message.id, direction: 'response', response }; + const transferables: Transferable[] = []; + if (response.ok) { + if (response.result.kind === 'columnar' || response.result.kind === 'columnarScan') { + transferables.push(response.result.xs.buffer, response.result.ys.buffer); + if (response.result.zs) { + transferables.push(response.result.zs.buffer); + } + if (response.result.kind === 'columnar' && response.result.featureCodes) { + transferables.push(response.result.featureCodes.buffer); + } + } else if (response.result.kind === 'parquetTable') { + transferables.push(response.result.tableIpc.buffer); + } else if (response.result.kind === 'rowFeatureCodes') { + transferables.push(response.result.codes.buffer); + } else if (response.result.kind === 'featureCounts') { + transferables.push(response.result.codes.buffer, response.result.counts.buffer); + } + } + self.postMessage(reply, transferables); + }) + .catch((error: unknown) => { + const reply: PointsWorkerMessage = { + id: message.id, + direction: 'response', + response: { + ok: false, + error: error instanceof Error ? error.message : String(error), + }, + }; + self.postMessage(reply); + }); +}; + +export {}; diff --git a/packages/core/src/workers/pointsWorkerClient.ts b/packages/core/src/workers/pointsWorkerClient.ts new file mode 100644 index 00000000..9a14c05e --- /dev/null +++ b/packages/core/src/workers/pointsWorkerClient.ts @@ -0,0 +1,465 @@ +import { tableFromIPC } from 'apache-arrow'; +import type { PointsColumnarData } from '../spatialViewFit.js'; +import type { PointsFeatureCatalog } from '../pointsTiling.js'; +import { + columnarDataFromWorkerResult, + type ParquetRowGroupBytesChunk, + type ParquetWorkerPayload, + type PointsBounds, + type PointsWorkerMessage, + type PointsWorkerRequest, + type PointsWorkerResponse, +} from './pointsWorkerProtocol.js'; + +let worker: Worker | undefined; +let nextRequestId = 0; +const pending = new Map< + number, + { resolve: (value: unknown) => void; reject: (error: Error) => void } +>(); + +let enabled = false; +let defaultEnabled = typeof window !== 'undefined'; + +function ensureWorkerListener() { + if (!worker) { + return; + } + worker.onmessage = (event: MessageEvent) => { + const message = event.data; + if (message.direction !== 'response') { + return; + } + const entry = pending.get(message.id); + if (!entry) { + return; + } + pending.delete(message.id); + if (message.response.ok) { + entry.resolve(message.response.result); + } else { + entry.reject(new Error(message.response.error)); + } + }; + worker.onerror = (event) => { + for (const [, entry] of pending) { + entry.reject(new Error(event.message || 'Points worker error')); + } + pending.clear(); + }; +} + +function postRequest(request: PointsWorkerRequest, transferables: Transferable[] = []): Promise { + const activeWorker = worker; + if (!activeWorker) { + return Promise.reject(new Error('Points worker is not enabled')); + } + const id = ++nextRequestId; + return new Promise((resolve, reject) => { + pending.set(id, { resolve: resolve as (value: unknown) => void, reject }); + const message: PointsWorkerMessage = { id, direction: 'request', request }; + if (transferables.length > 0) { + activeWorker.postMessage(message, transferables); + } else { + activeWorker.postMessage(message); + } + }); +} + +export function transferablesForParquetPayload( + parts?: Uint8Array[], + rowGroups?: ParquetRowGroupBytesChunk[] +): Transferable[] { + const transferables: Transferable[] = []; + if (parts) { + for (const part of parts) { + transferables.push(part.buffer); + } + } + if (rowGroups) { + for (const chunk of rowGroups) { + transferables.push(chunk.schemaBytes.buffer, chunk.rowGroupBytes.buffer); + } + } + return transferables; +} + +function transferablesForRequest(request: PointsWorkerRequest): Transferable[] { + switch (request.type) { + case 'decodeParquetRowFeatureCodes': + case 'scanParquetFeatureCounts': + case 'decodeParquetGeometryCapped': + case 'scanParquetByFeatureCodes': + case 'scanParquetFeatureCatalog': + return transferablesForParquetPayload(request.parts, request.rowGroups); + case 'scanMortonRowGroupsInBounds': + return transferablesForParquetPayload(undefined, request.rowGroups); + } + return []; +} + +export function isPointsWorkerEnabled(): boolean { + return enabled && worker !== undefined; +} + +export function enablePointsWorker(options: { workerUrl?: string | URL } = {}) { + if (typeof Worker === 'undefined') { + return; + } + if (worker) { + disablePointsWorker(); + } + if (options.workerUrl) { + worker = new Worker(options.workerUrl, { type: 'module' }); + } else { + // Inline URL so Vite dev apps can bundle the worker; @vite-ignore keeps lib build + // emitting a runtime relative URL to dist/points-worker.js (not /assets/...). + worker = new Worker( + new URL(/* @vite-ignore */ './points-worker.js', import.meta.url), + { type: 'module' } + ); + } + ensureWorkerListener(); + enabled = true; +} + +export function disablePointsWorker() { + enabled = false; + if (worker) { + worker.terminate(); + worker = undefined; + } + for (const [, entry] of pending) { + entry.reject(new Error('Points worker disabled')); + } + pending.clear(); +} + +export function setPointsWorkerDefaultEnabled(value: boolean) { + defaultEnabled = value; +} + +export function ensurePointsWorker(options: { workerUrl?: string | URL } = {}) { + if (!enabled && defaultEnabled) { + enablePointsWorker(options); + } +} + +export async function filterColumnarByFeatureCodesInWorker( + data: PointsColumnarData, + featureCodes: readonly number[] | undefined, + sourceFeatureCodes: ArrayLike +): Promise { + ensurePointsWorker(); + if (!isPointsWorkerEnabled()) { + const { filterColumnarByFeatureCodes } = await import('../pointsTiling.js'); + return filterColumnarByFeatureCodes(data, featureCodes, sourceFeatureCodes); + } + + const xs = data.data[0] instanceof Float32Array + ? data.data[0] + : Float32Array.from(data.data[0] as ArrayLike); + const ys = data.data[1] instanceof Float32Array + ? data.data[1] + : Float32Array.from(data.data[1] as ArrayLike); + const zs = data.data[2] + ? data.data[2] instanceof Float32Array + ? data.data[2] + : Float32Array.from(data.data[2] as ArrayLike) + : undefined; + + const result = await postRequest['result']>({ + type: 'filterColumnarByFeatureCodes', + xs, + ys, + zs, + featureCodes, + sourceFeatureCodes, + }); + + if (result.kind !== 'columnar') { + throw new Error('Unexpected points worker response for filterColumnarByFeatureCodes'); + } + return columnarDataFromWorkerResult(result); +} + +export type DecodeParquetRowFeatureCodesInput = { + parts?: Uint8Array[]; + rowGroups?: ParquetRowGroupBytesChunk[]; + columns: string[]; + maxRows?: number; + featureKey: string; + featureCodeColumnName?: string; + featureCodeEntries?: ReadonlyArray<{ name: string; code: number }>; +}; + +export async function decodeParquetRowFeatureCodesInWorker( + input: DecodeParquetRowFeatureCodesInput +): Promise { + ensurePointsWorker(); + if (!isPointsWorkerEnabled()) { + return null; + } + if (!input.parts?.length && !input.rowGroups?.length) { + return null; + } + if (input.parts?.length && input.rowGroups?.length) { + throw new Error('decodeParquetRowFeatureCodesInWorker requires parts or rowGroups, not both'); + } + const request: Extract = { + type: 'decodeParquetRowFeatureCodes', + ...input, + }; + const result = await postRequest['result']>( + request, + transferablesForRequest(request) + ); + if (result.kind !== 'rowFeatureCodes') { + throw new Error('Unexpected points worker response for decodeParquetRowFeatureCodes'); + } + return result.codes; +} + +export type ScanParquetFeatureCatalogInput = { + rowGroups?: ParquetRowGroupBytesChunk[]; + parts: Uint8Array[]; + columns: string[]; + featureKey: string; + featureCodeColumnName?: string; + skipMortonSentinels?: boolean; +}; + +export async function scanParquetFeatureCatalogInWorker( + input: ScanParquetFeatureCatalogInput +): Promise { + ensurePointsWorker(); + if (!isPointsWorkerEnabled() || input.parts.length === 0) { + return null; + } + const request: Extract = { + type: 'scanParquetFeatureCatalog', + ...input, + }; + const result = await postRequest['result']>( + request, + transferablesForRequest(request) + ); + if (result.kind !== 'catalog') { + throw new Error('Unexpected points worker response for scanParquetFeatureCatalog'); + } + return result.catalog; +} + +export type DecodeParquetGeometryCappedInput = ParquetWorkerPayload & { + axisNames: string[]; + columns: string[]; + maxRows: number; + featureKey?: string; + featureCodeColumnName?: string; + featureCodeEntries?: ReadonlyArray<{ name: string; code: number }>; +}; + +export async function decodeParquetGeometryCappedInWorker( + input: DecodeParquetGeometryCappedInput +): Promise<{ + shape: number[]; + data: ArrayLike[]; + featureCodes?: Int32Array; +} | null> { + ensurePointsWorker(); + if (!isPointsWorkerEnabled()) { + return null; + } + if (!input.parts?.length && !input.rowGroups?.length) { + return null; + } + if (input.parts?.length && input.rowGroups?.length) { + throw new Error('decodeParquetGeometryCappedInWorker requires parts or rowGroups, not both'); + } + const request: Extract = { + type: 'decodeParquetGeometryCapped', + ...input, + }; + const result = await postRequest['result']>( + request, + transferablesForRequest(request) + ); + if (result.kind !== 'columnar') { + throw new Error('Unexpected points worker response for decodeParquetGeometryCapped'); + } + const data = result.zs ? [result.xs, result.ys, result.zs] : [result.xs, result.ys]; + return { + shape: result.shape, + data, + featureCodes: result.featureCodes, + }; +} + +export async function countFeatureCodesInWorker( + sourceFeatureCodes: ArrayLike +): Promise> { + ensurePointsWorker(); + if (!isPointsWorkerEnabled()) { + const { countFeatureCodesHistogram } = await import('../pointsFeatures.js'); + return countFeatureCodesHistogram(sourceFeatureCodes); + } + const codesArray = + sourceFeatureCodes instanceof Int32Array + ? sourceFeatureCodes + : Int32Array.from(sourceFeatureCodes); + const result = await postRequest['result']>({ + type: 'countFeatureCodes', + sourceFeatureCodes: codesArray, + }); + if (result.kind !== 'featureCounts') { + throw new Error('Unexpected points worker response for countFeatureCodes'); + } + const counts = new Map(); + for (let index = 0; index < result.codes.length; index += 1) { + counts.set(result.codes[index], result.counts[index]); + } + return counts; +} + +export type ScanParquetFeatureCountsInput = ParquetWorkerPayload & { + featureKey: string; + featureCodeColumnName?: string; +}; + +export async function scanParquetFeatureCountsInWorker( + input: ScanParquetFeatureCountsInput +): Promise | null> { + ensurePointsWorker(); + if (!isPointsWorkerEnabled()) { + return null; + } + if (!input.parts?.length && !input.rowGroups?.length) { + return null; + } + const request: Extract = { + type: 'scanParquetFeatureCounts', + ...input, + }; + const result = await postRequest['result']>( + request, + transferablesForRequest(request) + ); + if (result.kind !== 'featureCounts') { + throw new Error('Unexpected points worker response for scanParquetFeatureCounts'); + } + const counts = new Map(); + for (let index = 0; index < result.codes.length; index += 1) { + counts.set(result.codes[index], result.counts[index]); + } + return counts; +} + +export type ScanParquetByFeatureCodesInput = ParquetWorkerPayload & { + axisNames: string[]; + featureKey: string; + featureCodeColumnName?: string; + featureCodes: readonly number[]; + memoryCap: number; +}; + +export async function scanParquetByFeatureCodesInWorker( + input: ScanParquetByFeatureCodesInput +): Promise<{ + data: PointsColumnarData; + matchedRows: number; + scannedRows: number; +} | null> { + ensurePointsWorker(); + if (!isPointsWorkerEnabled()) { + return null; + } + if (!input.parts?.length && !input.rowGroups?.length) { + return null; + } + const request: Extract = { + type: 'scanParquetByFeatureCodes', + ...input, + }; + const result = await postRequest['result']>( + request, + transferablesForRequest(request) + ); + if (result.kind !== 'columnarScan') { + throw new Error('Unexpected points worker response for scanParquetByFeatureCodes'); + } + return { + data: columnarDataFromWorkerResult(result), + matchedRows: result.matchedRows, + scannedRows: result.scannedRows, + }; +} + +export type ScanMortonRowGroupsInBoundsInput = { + rowGroups: ParquetRowGroupBytesChunk[]; + bounds: PointsBounds; + axisNames: string[]; + mortonCodeColumnName: string; + featureCodeColumnName?: string; + featureCodes?: readonly number[]; +}; + +export async function scanMortonRowGroupsInBoundsInWorker( + input: ScanMortonRowGroupsInBoundsInput +): Promise { + ensurePointsWorker(); + if (!isPointsWorkerEnabled() || input.rowGroups.length === 0) { + return null; + } + const request: Extract = { + type: 'scanMortonRowGroupsInBounds', + ...input, + }; + const result = await postRequest['result']>( + request, + transferablesForRequest(request) + ); + if (result.kind !== 'columnar') { + throw new Error('Unexpected points worker response for scanMortonRowGroupsInBounds'); + } + return columnarDataFromWorkerResult(result); +} + +export async function decodeParquetPartsInWorker( + parts: Uint8Array[], + columns?: string[], + maxRows?: number +): Promise> { + ensurePointsWorker(); + if (!isPointsWorkerEnabled()) { + throw new Error('Points worker is required for decodeParquetPartsInWorker'); + } + const result = await postRequest['result']>({ + type: 'decodeParquetParts', + parts, + columns, + maxRows, + }); + if (result.kind !== 'parquetTable') { + throw new Error('Unexpected points worker response for decodeParquetParts'); + } + return tableFromIPC(result.tableIpc); +} + +export async function buildFeatureCatalogInWorker( + featureKey: string, + tableIpc: Uint8Array +): Promise { + ensurePointsWorker(); + if (!isPointsWorkerEnabled()) { + throw new Error('Points worker is required for buildFeatureCatalogInWorker'); + } + const result = await postRequest['result']>({ + type: 'buildFeatureCatalog', + featureKey, + tableIpc, + }); + if (result.kind !== 'catalog') { + throw new Error('Unexpected points worker response for buildFeatureCatalog'); + } + return result.catalog; +} diff --git a/packages/core/src/workers/pointsWorkerProtocol.ts b/packages/core/src/workers/pointsWorkerProtocol.ts new file mode 100644 index 00000000..91959e71 --- /dev/null +++ b/packages/core/src/workers/pointsWorkerProtocol.ts @@ -0,0 +1,148 @@ +import type { PointsColumnarData } from '../spatialViewFit.js'; +import type { PointsFeatureCatalog } from '../pointsTiling.js'; + +export type ParquetRowGroupBytesChunk = { + schemaBytes: Uint8Array; + rowGroupBytes: Uint8Array; + rowGroupIndex: number; + /** Dataset-wide row group index (for morton sentinel handling). */ + globalRowGroupIndex?: number; +}; + +export type ParquetWorkerPayload = { + parts?: Uint8Array[]; + rowGroups?: ParquetRowGroupBytesChunk[]; +}; + +export type PointsBounds = { + minX: number; + maxX: number; + minY: number; + maxY: number; +}; + +export type PointsWorkerRequest = + | { + type: 'filterColumnarByFeatureCodes'; + xs: Float32Array; + ys: Float32Array; + zs?: Float32Array; + /** Omitted = all features; empty = none. */ + featureCodes?: readonly number[]; + sourceFeatureCodes: ArrayLike; + } + | { + type: 'decodeParquetParts'; + parts: Uint8Array[]; + columns?: string[]; + /** When set, decode stops after this many rows (across parts). */ + maxRows?: number; + } + | { + type: 'buildFeatureCatalog'; + featureKey: string; + tableIpc: Uint8Array; + } + | { + type: 'decodeParquetRowFeatureCodes'; + parts?: Uint8Array[]; + rowGroups?: ParquetRowGroupBytesChunk[]; + columns: string[]; + maxRows?: number; + featureKey: string; + featureCodeColumnName?: string; + /** Serialized catalog for dict-only elements (no *_codes column). */ + featureCodeEntries?: ReadonlyArray<{ name: string; code: number }>; + } + | { + type: 'countFeatureCodes'; + sourceFeatureCodes: ArrayLike; + } + | { + type: 'scanParquetFeatureCounts'; + parts?: Uint8Array[]; + rowGroups?: ParquetRowGroupBytesChunk[]; + featureKey: string; + featureCodeColumnName?: string; + } + | { + type: 'scanParquetFeatureCatalog'; + rowGroups?: ParquetRowGroupBytesChunk[]; + parts: Uint8Array[]; + columns: string[]; + featureKey: string; + featureCodeColumnName?: string; + skipMortonSentinels?: boolean; + } + | { + type: 'decodeParquetGeometryCapped'; + parts?: Uint8Array[]; + rowGroups?: ParquetRowGroupBytesChunk[]; + axisNames: string[]; + columns: string[]; + maxRows: number; + featureKey?: string; + featureCodeColumnName?: string; + featureCodeEntries?: ReadonlyArray<{ name: string; code: number }>; + } + | { + type: 'scanParquetByFeatureCodes'; + parts?: Uint8Array[]; + rowGroups?: ParquetRowGroupBytesChunk[]; + axisNames: string[]; + featureKey: string; + featureCodeColumnName?: string; + featureCodes: readonly number[]; + memoryCap: number; + } + | { + type: 'scanMortonRowGroupsInBounds'; + rowGroups: ParquetRowGroupBytesChunk[]; + bounds: PointsBounds; + axisNames: string[]; + mortonCodeColumnName: string; + featureCodeColumnName?: string; + featureCodes?: readonly number[]; + }; + +export type PointsWorkerColumnarResult = { + kind: 'columnar'; + shape: number[]; + xs: Float32Array; + ys: Float32Array; + zs?: Float32Array; + featureCodes?: Int32Array; +}; + +export type PointsWorkerScanResult = Omit & { + kind: 'columnarScan'; + matchedRows: number; + scannedRows: number; +}; + +export type PointsWorkerResponse = + | { + ok: true; + result: + | PointsWorkerColumnarResult + | PointsWorkerScanResult + | { kind: 'parquetTable'; tableIpc: Uint8Array } + | { kind: 'catalog'; catalog: PointsFeatureCatalog } + | { kind: 'rowFeatureCodes'; codes: Int32Array; numRows: number } + | { kind: 'featureCounts'; codes: Int32Array; counts: Uint32Array }; + } + | { ok: false; error: string }; + +export type PointsWorkerMessage = { + id: number; +} & ( + | { direction: 'request'; request: PointsWorkerRequest } + | { direction: 'response'; response: PointsWorkerResponse } +); + +export function columnarDataFromWorkerResult( + result: PointsWorkerColumnarResult | PointsWorkerScanResult +): PointsColumnarData { + const data = result.zs ? [result.xs, result.ys, result.zs] : [result.xs, result.ys]; + return { shape: result.shape, data }; +} diff --git a/packages/core/src/workers/pointsWorkerScan.ts b/packages/core/src/workers/pointsWorkerScan.ts new file mode 100644 index 00000000..323ab065 --- /dev/null +++ b/packages/core/src/workers/pointsWorkerScan.ts @@ -0,0 +1,378 @@ +import { tableFromIPC, type Table } from 'apache-arrow'; +import { + accumulateFeatureCatalogFromTable, + countFeatureCodesHistogram, + featureCatalogFromCodeMap, + featureCatalogNeedsParquetFallback, + resolveRowFeatureCodesFromTable, +} from '../pointsFeatures.js'; +import type { PointsFeatureCatalog } from '../pointsTiling.js'; +import { + featureCodeAllowSet, + isMortonSentinelValue, + rowMatchesFeatureCode, +} from '../pointsTiling.js'; + +type ParquetWasmTableLike = { intoIPCStream(): Uint8Array }; +type ParquetModule = { + readParquet: (bytes: Uint8Array, options?: { columns?: string[] }) => ParquetWasmTableLike; +}; + +export type ParquetRowGroupBytesChunk = { + schemaBytes: Uint8Array; + rowGroupBytes: Uint8Array; + rowGroupIndex: number; + globalRowGroupIndex?: number; +}; + +type ReadParquetRowGroup = ( + schemaBytes: Uint8Array, + rowGroupBytes: Uint8Array, + rowGroupIndex: number, + options?: { columns?: string[] } +) => ParquetWasmTableLike; + +export async function decodeParquetPartsToTable( + readParquet: ParquetModule['readParquet'], + parts: Uint8Array[], + columns: string[] | undefined, + maxRows?: number +): Promise { + const tables: Table[] = []; + let accumulated = 0; + for (const part of parts) { + const table = tableFromIPC(readParquet(part, { columns }).intoIPCStream()); + if (maxRows === undefined) { + tables.push(table); + continue; + } + const remaining = maxRows - accumulated; + if (table.numRows <= remaining) { + tables.push(table); + accumulated += table.numRows; + } else { + tables.push(table.slice(0, remaining)); + break; + } + if (accumulated >= maxRows) { + break; + } + } + if (tables.length === 0) { + throw new Error('No parquet tables to decode'); + } + return tables.slice(1).reduce((merged, part) => merged.concat(part), tables[0]); +} + +export async function decodeParquetRowGroupsToTable( + readParquetRowGroup: ReadParquetRowGroup, + chunks: ParquetRowGroupBytesChunk[], + columns: string[] | undefined, + maxRows?: number +): Promise
{ + const readOptions = columns?.length ? { columns } : undefined; + const tables: Table[] = []; + let accumulated = 0; + for (const chunk of chunks) { + const table = tableFromIPC( + readParquetRowGroup( + chunk.schemaBytes, + chunk.rowGroupBytes, + chunk.rowGroupIndex, + readOptions + ).intoIPCStream() + ); + if (maxRows === undefined) { + tables.push(table); + continue; + } + const remaining = maxRows - accumulated; + if (table.numRows <= remaining) { + tables.push(table); + accumulated += table.numRows; + } else { + tables.push(table.slice(0, remaining)); + break; + } + if (accumulated >= maxRows) { + break; + } + } + if (tables.length === 0) { + throw new Error('No parquet row groups to decode'); + } + return tables.slice(1).reduce((merged, part) => merged.concat(part), tables[0]); +} + +export type ParquetWorkerPayloadInput = { + parts?: Uint8Array[]; + rowGroups?: ParquetRowGroupBytesChunk[]; +}; + +export async function decodeParquetPayloadToTable( + readParquet: ParquetModule['readParquet'], + readParquetRowGroup: ReadParquetRowGroup | undefined, + payload: ParquetWorkerPayloadInput, + columns: string[] | undefined, + maxRows?: number +): Promise
{ + if (payload.rowGroups?.length) { + if (!readParquetRowGroup) { + throw new Error('readParquetRowGroup is unavailable'); + } + return decodeParquetRowGroupsToTable( + readParquetRowGroup, + payload.rowGroups, + columns, + maxRows + ); + } + if (payload.parts?.length) { + return decodeParquetPartsToTable(readParquet, payload.parts, columns, maxRows); + } + throw new Error('No parquet parts or row groups to decode'); +} + +export function extractGeometryColumnar( + table: Table, + axisNames: string[] +): { shape: number[]; xs: Float32Array; ys: Float32Array; zs?: Float32Array } { + const xColumn = table.getChild(axisNames[0]); + const yColumn = table.getChild(axisNames[1]); + if (!xColumn || !yColumn) { + throw new Error(`Geometry columns not found in parquet table`); + } + const xs = Float32Array.from(xColumn.toArray() as ArrayLike); + const ys = Float32Array.from(yColumn.toArray() as ArrayLike); + const hasZ = axisNames.includes('z'); + const zColumn = hasZ ? table.getChild('z') : null; + const zs = zColumn ? Float32Array.from(zColumn.toArray() as ArrayLike) : undefined; + const shape = zs ? [3, xs.length] : [2, xs.length]; + return { shape, xs, ys, ...(zs ? { zs } : {}) }; +} + +export async function scanFeatureCatalogFromPayload( + readParquet: ParquetModule['readParquet'], + readParquetRowGroup: ReadParquetRowGroup | undefined, + input: { + rowGroups?: ParquetRowGroupBytesChunk[]; + parts: Uint8Array[]; + columns: string[]; + featureKey: string; + featureCodeColumnName?: string; + skipMortonSentinels?: boolean; + } +): Promise { + const codeToName = new Map(); + const nameToCode = new Map(); + const catalogOptions = { skipMortonSentinels: input.skipMortonSentinels === true }; + + if (input.featureCodeColumnName && input.rowGroups?.length && readParquetRowGroup) { + const readOptions = { columns: input.columns }; + for (const chunk of input.rowGroups) { + const table = tableFromIPC( + readParquetRowGroup( + chunk.schemaBytes, + chunk.rowGroupBytes, + chunk.rowGroupIndex, + readOptions + ).intoIPCStream() + ); + if (table.numRows === 0) { + continue; + } + accumulateFeatureCatalogFromTable( + codeToName, + nameToCode, + table, + input.featureKey, + input.featureCodeColumnName, + catalogOptions + ); + } + } + + if (featureCatalogNeedsParquetFallback(codeToName)) { + codeToName.clear(); + nameToCode.clear(); + const table = await decodeParquetPartsToTable(readParquet, input.parts, input.columns); + accumulateFeatureCatalogFromTable( + codeToName, + nameToCode, + table, + input.featureKey, + input.featureCodeColumnName, + catalogOptions + ); + } + + if (codeToName.size === 0) { + return null; + } + return featureCatalogFromCodeMap(input.featureKey, codeToName); +} + +export function scanMortonTableInBounds(input: { + table: Table; + rowGroupIndex: number; + bounds: { minX: number; maxX: number; minY: number; maxY: number }; + axisNames: string[]; + mortonCodeColumnName: string; + featureCodeColumnName?: string; + featureCodes?: readonly number[]; + xs: number[]; + ys: number[]; + zs: number[]; +}): void { + const allowedFeatureCodes = featureCodeAllowSet(input.featureCodes); + const filterByFeature = allowedFeatureCodes !== null; + const hasZ = input.axisNames.includes('z'); + const xColumn = input.table.getChild('x'); + const yColumn = input.table.getChild('y'); + const zColumn = hasZ ? input.table.getChild('z') : null; + const mortonColumn = input.table.getChild(input.mortonCodeColumnName); + const featureCodeColumn = input.featureCodeColumnName + ? input.table.getChild(input.featureCodeColumnName) + : null; + if (!xColumn || !yColumn) { + return; + } + for (let rowIndex = 0; rowIndex < input.table.numRows; rowIndex += 1) { + if ( + input.rowGroupIndex === 0 && + rowIndex < 4 && + isMortonSentinelValue(mortonColumn?.get(rowIndex)) + ) { + continue; + } + if ( + filterByFeature && + featureCodeColumn && + !rowMatchesFeatureCode(featureCodeColumn.get(rowIndex), allowedFeatureCodes) + ) { + continue; + } + const x = xColumn.get(rowIndex); + const y = yColumn.get(rowIndex); + if (typeof x !== 'number' || typeof y !== 'number') { + continue; + } + if ( + x < input.bounds.minX || + x > input.bounds.maxX || + y < input.bounds.minY || + y > input.bounds.maxY + ) { + continue; + } + input.xs.push(x); + input.ys.push(y); + if (zColumn) { + const z = zColumn.get(rowIndex); + input.zs.push(typeof z === 'number' ? z : 0); + } + } +} + +export function extractRowFeatureCodesFromTable( + table: Table, + featureKey: string, + featureCodeColumnName?: string, + featureCodeByName?: ReadonlyMap +): Int32Array { + const resolved = resolveRowFeatureCodesFromTable( + table, + featureKey, + featureCodeColumnName, + featureCodeByName + ); + if (!resolved) { + return new Int32Array(0); + } + if (resolved instanceof Int32Array) { + return resolved; + } + return Int32Array.from(resolved); +} + +export function histogramToSortedArrays(counts: Map): { + codes: Int32Array; + countValues: Uint32Array; +} { + const sorted = [...counts.entries()].sort((left, right) => left[0] - right[0]); + return { + codes: Int32Array.from(sorted.map(([code]) => code)), + countValues: Uint32Array.from(sorted.map(([, count]) => count)), + }; +} + +export function scanTableFeatureCounts( + table: Table, + featureKey: string, + featureCodeColumnName: string | undefined, + counts: Map +): void { + const rowCodes = extractRowFeatureCodesFromTable(table, featureKey, featureCodeColumnName); + for (let index = 0; index < rowCodes.length; index += 1) { + const code = rowCodes[index]; + counts.set(code, (counts.get(code) ?? 0) + 1); + } +} + +export function scanTableByFeatureCodes(input: { + table: Table; + axisNames: string[]; + featureKey: string; + featureCodeColumnName?: string; + featureCodes: readonly number[]; + memoryCap: number; + matchedRows: number; + xs: number[]; + ys: number[]; + zs: number[]; +}): number { + const allowed = featureCodeAllowSet(input.featureCodes); + if (allowed !== null && allowed.size === 0) { + return input.matchedRows; + } + const rowCodes = extractRowFeatureCodesFromTable( + input.table, + input.featureKey, + input.featureCodeColumnName + ); + const xColumn = input.axisNames.includes('x') ? input.table.getChild('x') : null; + const yColumn = input.axisNames.includes('y') ? input.table.getChild('y') : null; + const zColumn = input.axisNames.includes('z') ? input.table.getChild('z') : null; + if (!xColumn || !yColumn) { + return input.matchedRows; + } + let matchedRows = input.matchedRows; + for (let rowIndex = 0; rowIndex < input.table.numRows; rowIndex += 1) { + if (matchedRows >= input.memoryCap) { + break; + } + if (allowed !== null && !rowMatchesFeatureCode(rowCodes[rowIndex], allowed)) { + continue; + } + const x = xColumn.get(rowIndex); + const y = yColumn.get(rowIndex); + if (typeof x !== 'number' || typeof y !== 'number') { + continue; + } + input.xs.push(x); + input.ys.push(y); + if (zColumn) { + const z = zColumn.get(rowIndex); + input.zs.push(typeof z === 'number' ? z : 0); + } + matchedRows += 1; + } + return matchedRows; +} + +export function countFeatureCodesFromArray(sourceFeatureCodes: ArrayLike): { + codes: Int32Array; + countValues: Uint32Array; +} { + return histogramToSortedArrays(countFeatureCodesHistogram(sourceFeatureCodes)); +} diff --git a/packages/core/tests/mortonPointsTiling.spec.ts b/packages/core/tests/mortonPointsTiling.spec.ts new file mode 100644 index 00000000..f85ad4e8 --- /dev/null +++ b/packages/core/tests/mortonPointsTiling.spec.ts @@ -0,0 +1,283 @@ +import { execSync } from 'node:child_process'; +import { mkdtemp, readFile, writeFile, mkdir, rm } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; +import SpatialDataPointsSource from '../src/models/VPointsSource.js'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const projectRoot = join(__dirname, '../../..'); +const writerRoot = join(projectRoot, 'python/spatialdata-experimental-writer'); + +async function writeSyntheticPointsZarr(root: string) { + const elementDir = join(root, 'points', 'transcripts'); + await mkdir(elementDir, { recursive: true }); + await writeFile( + join(root, 'zarr.json'), + JSON.stringify({ zarr_format: 3, node_type: 'group' }) + ); + await writeFile( + join(elementDir, 'zarr.json'), + JSON.stringify({ + attributes: { + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }, + zarr_format: 3, + node_type: 'group', + }) + ); + + execSync( + `uv run python - <<'PY' +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +root = Path(${JSON.stringify(elementDir)}) +rows = 500 +df = pd.DataFrame( + { + "x": [float(i % 100) for i in range(rows)], + "y": [float((i * 3) % 100) for i in range(rows)], + "feature_name": (["gene_a", "gene_b", "gene_c"] * rows)[:rows], + } +) +pq.write_table(pa.Table.from_pandas(df, preserve_index=False), root / "points.parquet") +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); + + execSync( + `uv run spatialdata-experimental-writer morton-points-from-zarr ${JSON.stringify(root)} --points-key transcripts --row-group-size 100`, + { cwd: writerRoot, stdio: 'pipe' } + ); +} + +async function writeBadSentinelMortonPointsZarr(root: string) { + const elementDir = join(root, 'points', 'transcripts'); + await mkdir(elementDir, { recursive: true }); + await writeFile( + join(root, 'zarr.json'), + JSON.stringify({ zarr_format: 3, node_type: 'group' }) + ); + await writeFile( + join(elementDir, 'zarr.json'), + JSON.stringify({ + attributes: { + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }, + zarr_format: 3, + node_type: 'group', + }) + ); + + execSync( + `uv run python - <<'PY' +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +root = Path(${JSON.stringify(elementDir)}) +df = pd.DataFrame( + { + "x": [0.0, 100.0, 0.1, 0.2, 0.3, 20.0, 40.0], + "y": [0.0, 100.0, 0.1, 0.2, 0.3, 20.0, 40.0], + "feature_name_codes": [0, 1, 0, 0, 0, 1, 1], + "morton_code_2d": [0, 0, 0, 0, 0, 100, 200], + "feature_name": ["gene_a", "gene_b", "gene_a", "gene_a", "gene_a", "gene_b", "gene_b"], + } +) +table = pa.Table.from_pandas(df, preserve_index=False) +writer = pq.ParquetWriter(root / "points.parquet", table.schema, compression="zstd") +try: + writer.write_table(table.slice(0, 5), row_group_size=5) + writer.write_table(table.slice(5), row_group_size=2) +finally: + writer.close() +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); +} + +function createStore(files: Record) { + let getRangeCalls = 0; + let getCalls = 0; + const store = { + getRangeCalls: () => getRangeCalls, + getCalls: () => getCalls, + resetCalls: () => { + getRangeCalls = 0; + getCalls = 0; + }, + store: { + async get(path: string) { + getCalls += 1; + return files[path.slice(1)] ?? null; + }, + async getRange( + path: string, + range: { offset?: number; length?: number; suffixLength?: number } + ) { + getRangeCalls += 1; + const bytes = files[path.slice(1)]; + if (!bytes) { + return null; + } + if (range.suffixLength !== undefined) { + const start = Math.max(0, bytes.length - range.suffixLength); + return bytes.slice(start); + } + const offset = range.offset ?? 0; + const length = range.length ?? bytes.length - offset; + return bytes.slice(offset, offset + length); + }, + }, + }; + return store; +} + +describe('Morton points tiling (canonical parquet)', () => { + let fixtureRoot: string; + let source: SpatialDataPointsSource; + let mockStore: ReturnType; + + beforeAll(async () => { + fixtureRoot = await mkdtemp(join(tmpdir(), 'morton-points-')); + await writeSyntheticPointsZarr(fixtureRoot); + + const parquetPath = join(fixtureRoot, 'points/transcripts/points.parquet'); + const elementJsonPath = join(fixtureRoot, 'points/transcripts/zarr.json'); + mockStore = createStore({ + 'points/transcripts/points.parquet': new Uint8Array(await readFile(parquetPath)), + 'points/transcripts/zarr.json': new Uint8Array(await readFile(elementJsonPath)), + }); + + source = new SpatialDataPointsSource({ + store: mockStore.store, + fileType: '.zarr', + }); + }, 120_000); + + afterAll(async () => { + await rm(fixtureRoot, { recursive: true, force: true }); + }); + + it('detects morton tiling metadata on canonical points.parquet', async () => { + mockStore.resetCalls(); + const metadata = await source.getPointsTilingMetadata('points/transcripts'); + expect(metadata).toMatchObject({ + kind: 'morton-points', + featureCodeColumnName: 'feature_name_codes', + }); + expect(mockStore.getRangeCalls()).toBeGreaterThan(0); + }); + + it('loads a bounded viewport without returning the full table', async () => { + const full = await source.loadPoints('points/transcripts'); + const xs = full.data[0]; + const ys = full.data[1]; + const minX = Math.min(...xs); + const maxX = Math.max(...xs); + const minY = Math.min(...ys); + const maxY = Math.max(...ys); + const bounds = { + minX: minX + 5, + maxX: minX + 15, + minY: minY + 5, + maxY: minY + 15, + }; + + mockStore.resetCalls(); + const loadTable = vi.spyOn(source, 'loadParquetTable'); + const result = await source.loadPointsInBounds('points/transcripts', { bounds }); + expect(result.shape[1]).toBeGreaterThan(0); + expect(result.shape[1]).toBeLessThan(full.shape[1]); + expect(['row-groups', 'full-filter']).toContain(result.loadMode); + if (result.loadMode === 'full-filter') { + expect(loadTable).toHaveBeenCalled(); + } + loadTable.mockRestore(); + }); + + it('filters loaded points by feature codes', async () => { + const full = await source.loadPoints('points/transcripts'); + const xs = full.data[0]; + const ys = full.data[1]; + const bounds = { + minX: Math.min(...xs), + maxX: Math.max(...xs), + minY: Math.min(...ys), + maxY: Math.max(...ys), + }; + + const unfiltered = await source.loadPointsInBounds('points/transcripts', { bounds }); + const filtered = await source.loadPointsInBounds('points/transcripts', { + bounds, + featureCodes: [0], + }); + expect(filtered.shape[1]).toBeGreaterThan(0); + expect(filtered.shape[1]).toBeLessThan(unfiltered.shape[1] ?? Number.MAX_SAFE_INTEGER); + }); + + it('uses row-group reads when parquet-wasm exposes row-group APIs', async () => { + const canRowGroups = await source.canLoadParquetRowGroups(); + if (!canRowGroups) { + return; + } + + const metadata = await source.getPointsTilingMetadata('points/transcripts'); + expect(metadata?.supportsRowGroupRangeReads).toBe(true); + expect(metadata?.bounds).toBeDefined(); + + mockStore.resetCalls(); + const bounds = { + minX: metadata!.bounds!.minX + 10, + maxX: metadata!.bounds!.minX + 30, + minY: metadata!.bounds!.minY + 10, + maxY: metadata!.bounds!.minY + 30, + }; + const result = await source.loadPointsInBounds('points/transcripts', { bounds }); + expect(result.shape?.[1]).toBeGreaterThan(0); + if (result.loadMode === 'row-groups') { + expect(mockStore.getRangeCalls()).toBeGreaterThan(0); + expect(mockStore.getCalls()).toBe(0); + } + }); + + it('does not enable morton tiling when sentinel row group is oversized', async () => { + const badFixtureRoot = await mkdtemp(join(tmpdir(), 'bad-morton-points-')); + try { + await writeBadSentinelMortonPointsZarr(badFixtureRoot); + const parquetPath = join(badFixtureRoot, 'points/transcripts/points.parquet'); + const elementJsonPath = join(badFixtureRoot, 'points/transcripts/zarr.json'); + const badStore = createStore({ + 'points/transcripts/points.parquet': new Uint8Array(await readFile(parquetPath)), + 'points/transcripts/zarr.json': new Uint8Array(await readFile(elementJsonPath)), + }); + const badSource = new SpatialDataPointsSource({ + store: badStore.store, + fileType: '.zarr', + }); + + const metadata = await badSource.getPointsTilingMetadata('points/transcripts'); + + expect(metadata?.supportsRowGroupRangeReads).toBe(false); + expect(metadata?.bounds).toBeUndefined(); + } finally { + execSync(`rm -rf ${JSON.stringify(badFixtureRoot)}`, { stdio: 'pipe' }); + } + }); +}); diff --git a/packages/core/tests/pointsFeatures.spec.ts b/packages/core/tests/pointsFeatures.spec.ts new file mode 100644 index 00000000..df6298ca --- /dev/null +++ b/packages/core/tests/pointsFeatures.spec.ts @@ -0,0 +1,431 @@ +import { execSync } from 'node:child_process'; +import { mkdtemp, readFile, rm, mkdir, stat } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest'; +import SpatialDataPointsSource from '../src/models/VPointsSource.js'; +import * as pointsWorkerClient from '../src/workers/pointsWorkerClient.js'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const writerRoot = join(__dirname, '../../../python/spatialdata-experimental-writer'); + +async function writePointsFeatureFixture(root: string) { + const elementDir = join(root, 'points', 'transcripts'); + await mkdir(elementDir, { recursive: true }); + + execSync( + `uv run python - <<'PY' +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +root = Path(${JSON.stringify(elementDir)}) +(root / "points.parquet").mkdir(parents=True, exist_ok=True) +table0 = pa.table( + { + "x": [0.0, 1.0, 2.0], + "y": [0.0, 1.0, 2.0], + "feature_name": ["gene_a", "gene_b", "gene_a"], + "feature_name_codes": pa.array([0, 1, 0], type=pa.int32()), + } +) +table1 = pa.table( + { + "x": [3.0, 4.0], + "y": [3.0, 4.0], + "feature_name": ["gene_c", "gene_b"], + "feature_name_codes": pa.array([2, 1], type=pa.int32()), + } +) +pq.write_table(table0, root / "points.parquet" / "part.0.parquet") +pq.write_table(table1, root / "points.parquet" / "part.1.parquet") +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); +} + +function createFilesystemStore(root: string) { + const readStoreBytes = async (relativePath: string): Promise => { + const fullPath = join(root, relativePath); + try { + const info = await stat(fullPath); + if (info.isDirectory()) { + return null; + } + return await readFile(fullPath); + } catch { + return null; + } + }; + + return { + async get(path: string) { + const relativePath = path.startsWith('/') ? path.slice(1) : path; + return readStoreBytes(relativePath); + }, + async getRange( + path: string, + range: { offset?: number; length?: number; suffixLength?: number } + ) { + const relativePath = path.startsWith('/') ? path.slice(1) : path; + const bytes = await readStoreBytes(relativePath); + if (!bytes) { + return null; + } + if (range.suffixLength != null) { + return bytes.subarray(bytes.length - range.suffixLength); + } + const offset = range.offset ?? 0; + const length = range.length ?? bytes.length - offset; + return bytes.subarray(offset, offset + length); + }, + }; +} + +describe('SpatialDataPointsSource feature catalog', () => { + let fixtureRoot: string; + let source: SpatialDataPointsSource; + + beforeAll(async () => { + fixtureRoot = await mkdtemp(join(tmpdir(), 'points-features-')); + await writePointsFeatureFixture(fixtureRoot); + source = new SpatialDataPointsSource({ + store: createFilesystemStore(fixtureRoot), + fileType: '.zarr', + }); + vi.spyOn(source, 'loadSpatialDataElementAttrs').mockResolvedValue({ + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }); + }, 120_000); + + afterAll(async () => { + await rm(fixtureRoot, { recursive: true, force: true }); + }); + + it('lists distinct feature names and codes across multipart parquet', async () => { + const catalog = await source.listPointsFeatures('points/transcripts'); + expect(catalog).toEqual({ + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'gene_a' }, + { code: 1, name: 'gene_b' }, + { code: 2, name: 'gene_c' }, + ], + }); + }); + + it('lists features for oversized datasets via feature-column scan', async () => { + vi.spyOn(source, 'resolveParquetRowCount' as keyof SpatialDataPointsSource).mockResolvedValue( + 5_000_000 + ); + const catalog = await source.listPointsFeatures('points/transcripts'); + expect(catalog).toEqual({ + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'gene_a' }, + { code: 1, name: 'gene_b' }, + { code: 2, name: 'gene_c' }, + ], + }); + }); + + it('lists features for oversized dictionary-encoded datasets via row-group dictionary read', async () => { + const elementDir = join(fixtureRoot, 'points', 'dict_large'); + await mkdir(elementDir, { recursive: true }); + execSync( + `uv run python - <<'PY' +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +root = Path(${JSON.stringify(elementDir)}) +(root / "points.parquet").mkdir(parents=True, exist_ok=True) +genes = pa.array(["gene_a", "gene_b", "gene_c"], type=pa.dictionary(pa.int32(), pa.string())) +names = (["gene_a", "gene_b", "gene_c"] * 34)[:100] +table = pa.table( + { + "x": [float(i) for i in range(100)], + "y": [float(i) for i in range(100)], + "feature_name": pa.array(names, type=pa.dictionary(pa.int32(), pa.string())), + } +) +pq.write_table(table, root / "points.parquet" / "part.0.parquet") +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); + + const dictSource = new SpatialDataPointsSource({ + store: createFilesystemStore(fixtureRoot), + fileType: '.zarr', + }); + vi.spyOn(dictSource, 'loadSpatialDataElementAttrs').mockResolvedValue({ + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }); + vi.spyOn( + dictSource, + 'resolveParquetRowCount' as keyof SpatialDataPointsSource + ).mockResolvedValue(5_000_000); + + const catalog = await dictSource.listPointsFeatures('points/dict_large'); + expect(catalog?.entries).toEqual([ + { code: 0, name: 'gene_a' }, + { code: 1, name: 'gene_b' }, + { code: 2, name: 'gene_c' }, + ]); + }); + + it('loads feature code column with full points preload via loadPointsRowFeatureCodes', async () => { + const points = await source.loadPoints('points/transcripts'); + expect(points.shape[1]).toBe(5); + expect(points.featureCodes).toBeUndefined(); + const featureCodes = await source.loadPointsRowFeatureCodes('points/transcripts'); + expect(featureCodes?.length).toBe(5); + }); + + it('uses explicit feature code columns instead of dictionary indices', async () => { + const elementDir = join(fixtureRoot, 'points', 'dict_with_codes'); + await mkdir(elementDir, { recursive: true }); + execSync( + `uv run python - <<'PY' +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +root = Path(${JSON.stringify(elementDir)}) +(root / "points.parquet").mkdir(parents=True, exist_ok=True) +names = pa.DictionaryArray.from_arrays( + pa.array([0, 1, 0], type=pa.int32()), + pa.array(["ABCC11", "TP53"]), +) +table = pa.table( + { + "x": [0.0, 1.0, 2.0], + "y": [0.0, 1.0, 2.0], + "feature_name": names, + "feature_name_codes": pa.array([1, 0, 1], type=pa.int32()), + } +) +pq.write_table(table, root / "points.parquet" / "part.0.parquet") +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); + + const dictSource = new SpatialDataPointsSource({ + store: createFilesystemStore(fixtureRoot), + fileType: '.zarr', + }); + vi.spyOn(dictSource, 'loadSpatialDataElementAttrs').mockResolvedValue({ + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }); + + const catalog = await dictSource.listPointsFeatures('points/dict_with_codes'); + expect(catalog?.entries).toEqual([ + { code: 0, name: 'TP53' }, + { code: 1, name: 'ABCC11' }, + ]); + + const featureCodes = await dictSource.loadPointsRowFeatureCodes('points/dict_with_codes'); + expect([...featureCodes!]).toEqual([1, 0, 1]); + }); + + it('omits counts for dictionary-only feature columns without explicit code mapping', async () => { + const elementDir = join(fixtureRoot, 'points', 'dict_counts_untrusted'); + await mkdir(elementDir, { recursive: true }); + execSync( + `uv run python - <<'PY' +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +root = Path(${JSON.stringify(elementDir)}) +(root / "points.parquet").mkdir(parents=True, exist_ok=True) +names = ["ABCC11", "TP53", "TP53", "EGFR"] +table = pa.table( + { + "x": [0.0, 1.0, 2.0, 3.0], + "y": [0.0, 1.0, 2.0, 3.0], + "feature_name": pa.array(names, type=pa.dictionary(pa.int32(), pa.string())), + } +) +pq.write_table(table, root / "points.parquet" / "part.0.parquet") +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); + + const dictSource = new SpatialDataPointsSource({ + store: createFilesystemStore(fixtureRoot), + fileType: '.zarr', + }); + vi.spyOn(dictSource, 'loadSpatialDataElementAttrs').mockResolvedValue({ + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }); + + const counts = await dictSource.loadFeatureCounts('points/dict_counts_untrusted'); + expect(counts.size).toBe(0); + + const catalog = await dictSource.listPointsFeaturesWithCounts('points/dict_counts_untrusted'); + expect(catalog?.entries).toEqual([ + { code: 0, name: 'ABCC11' }, + { code: 1, name: 'TP53' }, + { code: 2, name: 'EGFR' }, + ]); + }); + + it('derives row feature codes from dictionary-encoded feature names', async () => { + const elementDir = join(fixtureRoot, 'points', 'dict_only'); + await mkdir(elementDir, { recursive: true }); + execSync( + `uv run python - <<'PY' +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +root = Path(${JSON.stringify(elementDir)}) +(root / "points.parquet").mkdir(parents=True, exist_ok=True) +genes = pa.array(["gene_a", "gene_b", "gene_a"], type=pa.dictionary(pa.int32(), pa.string())) +table = pa.table({"x": [0.0, 1.0, 2.0], "y": [0.0, 1.0, 2.0], "feature_name": genes}) +pq.write_table(table, root / "points.parquet" / "part.0.parquet") +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); + + const dictSource = new SpatialDataPointsSource({ + store: createFilesystemStore(fixtureRoot), + fileType: '.zarr', + }); + vi.spyOn(dictSource, 'loadSpatialDataElementAttrs').mockResolvedValue({ + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }); + + const points = await dictSource.loadPoints('points/dict_only'); + expect(points.featureCodes).toBeUndefined(); + const featureCodes = await dictSource.loadPointsRowFeatureCodes('points/dict_only'); + expect(featureCodes?.length).toBe(3); + expect([...featureCodes!]).toEqual([0, 1, 0]); + }); + + it('derives dictionary-only row codes from decoded names, not local dictionary indices', async () => { + const elementDir = join(fixtureRoot, 'points', 'dict_local_indices'); + await mkdir(elementDir, { recursive: true }); + execSync( + `uv run python - <<'PY' +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +root = Path(${JSON.stringify(elementDir)}) +(root / "points.parquet").mkdir(parents=True, exist_ok=True) +part0_names = pa.DictionaryArray.from_arrays( + pa.array([0, 0, 1], type=pa.int32()), + pa.array(["ABCC11", "TP53"]), +) +part1_names = pa.DictionaryArray.from_arrays( + pa.array([0, 0, 1], type=pa.int32()), + pa.array(["TP53", "EGFR"]), +) +part0 = pa.table( + { + "x": [0.0, 1.0, 2.0], + "y": [0.0, 1.0, 2.0], + "feature_name": part0_names, + } +) +part1 = pa.table( + { + "x": [3.0, 4.0, 5.0], + "y": [3.0, 4.0, 5.0], + "feature_name": part1_names, + } +) +pq.write_table(part0, root / "points.parquet" / "part.0.parquet") +pq.write_table(part1, root / "points.parquet" / "part.1.parquet") +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); + + const dictSource = new SpatialDataPointsSource({ + store: createFilesystemStore(fixtureRoot), + fileType: '.zarr', + }); + vi.spyOn(dictSource, 'loadSpatialDataElementAttrs').mockResolvedValue({ + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }); + + const catalog = await dictSource.listPointsFeatures('points/dict_local_indices'); + expect(catalog?.entries).toEqual([ + { code: 0, name: 'ABCC11' }, + { code: 1, name: 'TP53' }, + { code: 2, name: 'EGFR' }, + ]); + + const featureCodes = await dictSource.loadPointsRowFeatureCodes('points/dict_local_indices'); + expect([...featureCodes!]).toEqual([0, 0, 1, 1, 1, 2]); + }); + + it('delegates row feature code decode to the points worker when enabled', async () => { + const workerCodes = Int32Array.from([0, 1, 0, 1, 2]); + vi.spyOn(pointsWorkerClient, 'ensurePointsWorker').mockImplementation(() => {}); + vi.spyOn(pointsWorkerClient, 'isPointsWorkerEnabled').mockReturnValue(true); + const decodeSpy = vi + .spyOn(pointsWorkerClient, 'decodeParquetRowFeatureCodesInWorker') + .mockResolvedValue(workerCodes); + vi.spyOn(source, 'canLoadParquetRowGroups').mockResolvedValue(false); + + const featureCodes = await source.loadPointsRowFeatureCodes('points/transcripts'); + expect(decodeSpy).toHaveBeenCalled(); + expect([...featureCodes!]).toEqual([...workerCodes]); + }); + + it('delegates oversized feature catalog scan to the points worker when enabled', async () => { + const workerCatalog = { + featureKey: 'feature_name', + entries: [ + { code: 0, name: 'gene_a' }, + { code: 1, name: 'gene_b' }, + ], + }; + vi.spyOn(pointsWorkerClient, 'ensurePointsWorker').mockImplementation(() => {}); + vi.spyOn(pointsWorkerClient, 'isPointsWorkerEnabled').mockReturnValue(true); + const catalogSpy = vi + .spyOn(pointsWorkerClient, 'scanParquetFeatureCatalogInWorker') + .mockResolvedValue(workerCatalog); + vi.spyOn(source, 'resolveParquetRowCount' as keyof SpatialDataPointsSource).mockResolvedValue( + 5_000_000 + ); + + const catalog = await source.listPointsFeatures('points/transcripts'); + expect(catalogSpy).toHaveBeenCalled(); + expect(catalog).toEqual(workerCatalog); + }); +}); diff --git a/packages/core/tests/pointsLoader.spec.ts b/packages/core/tests/pointsLoader.spec.ts new file mode 100644 index 00000000..7c7a04c3 --- /dev/null +++ b/packages/core/tests/pointsLoader.spec.ts @@ -0,0 +1,62 @@ +import { describe, expect, it } from 'vitest'; + +import { createMortonTiledPointsLoader, resolvePointsEncoding } from '../src/pointsLoader.js'; +import type { PointsElement } from '../src/models/index.js'; + +describe('resolvePointsEncoding', () => { + it('prefers preloaded data when present', () => { + expect( + resolvePointsEncoding({ shape: [1], data: [[0], [0]] }, null, true) + ).toBe('preloaded-columnar'); + }); + + it('selects morton tiling when metadata supports row-group reads', () => { + expect( + resolvePointsEncoding(null, { + kind: 'morton-points', + parquetPath: 'points/a/points.parquet', + axisNames: ['x', 'y'], + featureCodeColumnName: 'feature_name_codes', + mortonCodeColumnName: 'morton_code_2d', + totalRows: 10, + totalRowGroups: 1, + maxRowsPerGroup: 10, + supportsRowGroupRangeReads: true, + bounds: { minX: 0, minY: 0, maxX: 10, maxY: 10 }, + }, true) + ).toBe('morton-tiled'); + }); +}); + +describe('createMortonTiledPointsLoader', () => { + it('uses columnar shape[1] as point count, not shape[0] axis count', async () => { + const element = { + async loadPointsInBounds() { + return { + shape: [2, 1_000], + data: [new Float64Array(1_000), new Float64Array(1_000)], + bounds: { minX: 0, minY: 0, maxX: 10, maxY: 10 }, + loadMode: 'row-groups', + }; + }, + } as unknown as PointsElement; + + const loader = createMortonTiledPointsLoader(element, { + kind: 'morton-points', + parquetPath: 'points/a/points.parquet', + axisNames: ['x', 'y'], + featureCodeColumnName: 'feature_name_codes', + mortonCodeColumnName: 'morton_code_2d', + totalRows: 1_000, + totalRowGroups: 1, + maxRowsPerGroup: 1_000, + supportsRowGroupRangeReads: true, + bounds: { minX: 0, minY: 0, maxX: 10, maxY: 10 }, + }); + + const batch = await loader.loadInBounds({ + bounds: { minX: 0, minY: 0, maxX: 10, maxY: 10 }, + }); + expect(batch?.pointCount).toBe(1_000); + }); +}); diff --git a/packages/core/tests/pointsPreloadGuard.spec.ts b/packages/core/tests/pointsPreloadGuard.spec.ts new file mode 100644 index 00000000..5ef7596e --- /dev/null +++ b/packages/core/tests/pointsPreloadGuard.spec.ts @@ -0,0 +1,52 @@ +import { describe, expect, it, vi } from 'vitest'; +import { tableFromArrays } from 'apache-arrow'; +import SpatialDataPointsSource from '../src/models/VPointsSource.js'; +import { + POINTS_PRELOAD_MAX_ROWS, + preloadedColumnarPointCount, +} from '../src/pointsLimits.js'; + +describe('points preload cap', () => { + it('loads a capped subset when parquet row count exceeds the cap', async () => { + const source = new SpatialDataPointsSource({ + store: { get: async () => null }, + fileType: '.zarr', + }); + + vi.spyOn(source, 'loadSpatialDataElementAttrs').mockResolvedValue({ + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }); + vi.spyOn(source, 'loadParquetDatasetMetadata').mockResolvedValue({ + totalNumRows: POINTS_PRELOAD_MAX_ROWS + 1, + totalNumRowGroups: 1, + numRowsByPart: [POINTS_PRELOAD_MAX_ROWS + 1], + numRowGroupsByPart: [1], + numRowsPerGroupByPart: [POINTS_PRELOAD_MAX_ROWS + 1], + rowGroupRows: [POINTS_PRELOAD_MAX_ROWS + 1], + schema: null, + parts: [], + }); + vi.spyOn(source, 'resolveParquetRowCount').mockResolvedValue(POINTS_PRELOAD_MAX_ROWS + 1); + + const cappedTable = tableFromArrays({ + x: new Float32Array(POINTS_PRELOAD_MAX_ROWS), + y: new Float32Array(POINTS_PRELOAD_MAX_ROWS), + feature_name: new Array(POINTS_PRELOAD_MAX_ROWS).fill('gene'), + }); + vi.spyOn(source, 'loadParquetTableCapped').mockResolvedValue({ + table: cappedTable, + totalRows: POINTS_PRELOAD_MAX_ROWS + 1, + truncated: true, + }); + + const result = await source.loadPoints('points/transcripts'); + expect(preloadedColumnarPointCount(result.shape, result.data)).toBe(POINTS_PRELOAD_MAX_ROWS); + expect(result.totalRowCount).toBe(POINTS_PRELOAD_MAX_ROWS + 1); + expect(result.preloadTruncated).toBe(true); + }); +}); diff --git a/packages/core/tests/pointsPreloadReadStrategy.spec.ts b/packages/core/tests/pointsPreloadReadStrategy.spec.ts new file mode 100644 index 00000000..9c51c215 --- /dev/null +++ b/packages/core/tests/pointsPreloadReadStrategy.spec.ts @@ -0,0 +1,75 @@ +import { describe, expect, it, vi } from 'vitest'; +import { tableFromArrays } from 'apache-arrow'; +import SpatialDataPointsSource from '../src/models/VPointsSource.js'; +import * as pointsWorkerClient from '../src/workers/pointsWorkerClient.js'; + +describe('points preload read strategy', () => { + it('does not prefetch row-group bytes for geometry preload', async () => { + const source = new SpatialDataPointsSource({ + store: { get: async () => null }, + fileType: '.zarr', + }); + + vi.spyOn(source, 'loadSpatialDataElementAttrs').mockResolvedValue({ + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }); + vi.spyOn(source, 'resolveParquetRowCount').mockResolvedValue(100); + vi.spyOn(source, 'canLoadParquetRowGroups').mockResolvedValue(true); + + const rowGroupBytesSpy = vi.spyOn(source, 'readParquetRowGroupsBytesCapped'); + const payloadSpy = vi.spyOn(source, 'readParquetWorkerPayload'); + + vi.spyOn(pointsWorkerClient, 'isPointsWorkerEnabled').mockReturnValue(true); + vi.spyOn(pointsWorkerClient, 'decodeParquetGeometryCappedInWorker').mockResolvedValue({ + shape: [2, 100], + data: [new Float32Array(100), new Float32Array(100)], + }); + + await source.loadPoints('points/transcripts'); + + expect(rowGroupBytesSpy).not.toHaveBeenCalled(); + expect(payloadSpy).toHaveBeenCalledWith('points/transcripts/points.parquet', { maxRows: 100 }); + expect(payloadSpy.mock.calls[0]?.[1]?.includeRowGroups).not.toBe(true); + }); + + it('uses full-file capped decode for preload fallback, not row groups', async () => { + const source = new SpatialDataPointsSource({ + store: { get: async () => null }, + fileType: '.zarr', + }); + + vi.spyOn(source, 'loadSpatialDataElementAttrs').mockResolvedValue({ + 'encoding-type': 'ngff:points', + axes: ['x', 'y'], + spatialdata_attrs: { + feature_key: 'feature_name', + version: '0.2', + }, + }); + vi.spyOn(source, 'resolveParquetRowCount').mockResolvedValue(100); + vi.spyOn(source, 'canLoadParquetRowGroups').mockResolvedValue(true); + vi.spyOn(pointsWorkerClient, 'isPointsWorkerEnabled').mockReturnValue(false); + + const cappedSpy = vi.spyOn(source, 'loadParquetTableCapped').mockResolvedValue({ + table: tableFromArrays({ + x: new Float32Array(100), + y: new Float32Array(100), + }), + totalRows: 100, + truncated: false, + }); + + await source.loadPoints('points/transcripts'); + + expect(cappedSpy).toHaveBeenCalledWith( + 'points/transcripts/points.parquet', + ['x', 'y'], + 100 + ); + }); +}); diff --git a/packages/core/tests/pointsTiling.spec.ts b/packages/core/tests/pointsTiling.spec.ts new file mode 100644 index 00000000..ceeaafd6 --- /dev/null +++ b/packages/core/tests/pointsTiling.spec.ts @@ -0,0 +1,151 @@ +import type { Table as ArrowTable } from 'apache-arrow'; +import { describe, expect, it } from 'vitest'; +import { + extractSentinelBoundingBox, + filterColumnarByFeatureCodes, + filterPointsToBounds, + mergeAdjacentIntervals, + mortonIntervalsForBounds, + zcoverRectangle, +} from '../src/pointsTiling.js'; + +function vector(values: unknown[]) { + return { + length: values.length, + get: (index: number) => values[index], + }; +} + +function table(columns: Record): ArrowTable { + const first = Object.values(columns)[0] ?? []; + return { + numRows: first.length, + getChild: (name: string) => { + const values = columns[name]; + return values ? vector(values) : null; + }, + } as unknown as ArrowTable; +} + +describe('points tiling helpers', () => { + it('extracts the Vitessce sentinel bounding box from the leading rows', () => { + const arrowTable = table({ + x: [10, 20, 15, 17, 99], + y: [5, 8, 40, 12, 99], + morton_code_2d: [0, 0, 0, 0, 123], + }); + + expect(extractSentinelBoundingBox(arrowTable)).toEqual({ + minX: 10, + minY: 5, + maxX: 20, + maxY: 40, + }); + }); + + it('accepts bigint morton sentinel values', () => { + const arrowTable = table({ + x: [10, 20, 15, 17], + y: [5, 8, 40, 12], + morton_code_2d: [0n, 0n, 0n, 0n], + }); + + expect(extractSentinelBoundingBox(arrowTable)).toEqual({ + minX: 10, + minY: 5, + maxX: 20, + maxY: 40, + }); + }); + + it('rejects missing or incomplete sentinel bounds', () => { + expect( + extractSentinelBoundingBox( + table({ + x: [10, 20], + y: [5, 8], + morton_code_2d: [7, 8], + }) + ) + ).toBeNull(); + }); + + it('merges adjacent Morton intervals', () => { + expect( + mergeAdjacentIntervals([ + [10, 12], + [13, 15], + [20, 21], + ]) + ).toEqual([ + [10, 15], + [20, 21], + ]); + }); + + it('covers a full rectangle with the full Morton range', () => { + expect(zcoverRectangle(0, 0, 65535, 65535)).toEqual([[0, 4294967295]]); + }); + + it('produces intervals for a query rectangle inside a stored bbox', () => { + const intervals = mortonIntervalsForBounds( + { minX: 0, minY: 0, maxX: 100, maxY: 100 }, + { minX: 10, minY: 10, maxX: 20, maxY: 20 } + ); + expect(intervals.length).toBeGreaterThan(0); + expect(intervals.every(([lo, hi]) => lo <= hi)).toBe(true); + }); + + it('filters columnar points to bounds without changing source arrays', () => { + const xs = new Float32Array([0, 5, 10]); + const ys = new Float32Array([0, 5, 20]); + const filtered = filterPointsToBounds( + { data: [xs, ys], shape: [2, 3] }, + { minX: 1, minY: 1, maxX: 10, maxY: 10 } + ); + expect(Array.from(filtered.data[0])).toEqual([5]); + expect(Array.from(filtered.data[1])).toEqual([5]); + expect(filtered.shape).toEqual([2, 1]); + }); + + it('filters columnar points by feature codes after spatial bounds', () => { + const xs = new Float32Array([5, 5, 5]); + const ys = new Float32Array([5, 5, 5]); + const featureCodes = new Int32Array([0, 1, 2]); + const filtered = filterPointsToBounds( + { data: [xs, ys], shape: [2, 3] }, + { minX: 0, minY: 0, maxX: 10, maxY: 10 }, + undefined, + [1], + featureCodes + ); + expect(Array.from(filtered.data[0])).toEqual([5]); + expect(filtered.shape).toEqual([2, 1]); + }); + + it('filters columnar points by feature codes without bounds', () => { + const xs = new Float32Array([0, 1, 2]); + const ys = new Float32Array([0, 1, 2]); + const sourceFeatureCodes = new Int32Array([0, 1, 0]); + const filtered = filterColumnarByFeatureCodes( + { data: [xs, ys], shape: [2, 3] }, + [0], + sourceFeatureCodes + ); + expect(Array.from(filtered.data[0])).toEqual([0, 2]); + expect(filtered.shape).toEqual([2, 2]); + }); + + it('returns no rows when feature filter is an empty selection', () => { + const xs = new Float32Array([0, 1, 2]); + const ys = new Float32Array([0, 1, 2]); + const sourceFeatureCodes = new Int32Array([0, 1, 0]); + const filtered = filterColumnarByFeatureCodes( + { data: [xs, ys], shape: [2, 3] }, + [], + sourceFeatureCodes + ); + expect(filtered.data[0].length).toBe(0); + expect(filtered.shape).toEqual([2, 0]); + }); +}); diff --git a/packages/core/tests/pointsWorker.spec.ts b/packages/core/tests/pointsWorker.spec.ts new file mode 100644 index 00000000..cea441d2 --- /dev/null +++ b/packages/core/tests/pointsWorker.spec.ts @@ -0,0 +1,50 @@ +import { describe, expect, it, vi } from 'vitest'; +import { + decodeParquetRowFeatureCodesInWorker, + disablePointsWorker, + filterColumnarByFeatureCodesInWorker, + scanParquetFeatureCatalogInWorker, + setPointsWorkerDefaultEnabled, +} from '../src/workers/pointsWorkerClient.js'; +import { filterColumnarByFeatureCodes as filterSync } from '../src/pointsTiling.js'; + +describe('points worker client', () => { + it('falls back to main-thread filtering when the worker is disabled', async () => { + setPointsWorkerDefaultEnabled(false); + const data = { + shape: [2, 4] as [number, number], + data: [ + Float32Array.from([0, 1, 2, 3]), + Float32Array.from([0, 1, 2, 3]), + ], + }; + const sourceFeatureCodes = Int32Array.from([0, 1, 0, 2]); + const filtered = await filterColumnarByFeatureCodesInWorker(data, [1], sourceFeatureCodes); + const expected = filterSync(data, [1], sourceFeatureCodes); + expect(filtered.shape).toEqual(expected.shape); + expect(Array.from(filtered.data[0])).toEqual(Array.from(expected.data[0])); + expect(Array.from(filtered.data[1])).toEqual(Array.from(expected.data[1])); + }); + + it('returns null for row feature code decode when the worker is disabled', async () => { + disablePointsWorker(); + setPointsWorkerDefaultEnabled(false); + const result = await decodeParquetRowFeatureCodesInWorker({ + parts: [new Uint8Array([1, 2, 3])], + columns: ['feature_name'], + featureKey: 'feature_name', + }); + expect(result).toBeNull(); + }); + + it('returns null for feature catalog scan when the worker is disabled', async () => { + disablePointsWorker(); + setPointsWorkerDefaultEnabled(false); + const result = await scanParquetFeatureCatalogInWorker({ + parts: [new Uint8Array([1, 2, 3])], + columns: ['feature_name'], + featureKey: 'feature_name', + }); + expect(result).toBeNull(); + }); +}); diff --git a/packages/core/tests/pointsWorkerScan.spec.ts b/packages/core/tests/pointsWorkerScan.spec.ts new file mode 100644 index 00000000..d389553d --- /dev/null +++ b/packages/core/tests/pointsWorkerScan.spec.ts @@ -0,0 +1,122 @@ +import { tableFromArrays, tableToIPC } from 'apache-arrow'; +import { describe, expect, it } from 'vitest'; +import { + decodeParquetRowGroupsToTable, + extractGeometryColumnar, + extractRowFeatureCodesFromTable, + scanFeatureCatalogFromPayload, +} from '../src/workers/pointsWorkerScan.js'; + +function mockReadParquetRowGroup( + chunks: Array> +): ( + schemaBytes: Uint8Array, + rowGroupBytes: Uint8Array, + rowGroupIndex: number, + options?: { columns?: string[] } +) => { intoIPCStream(): Uint8Array } { + return (_schemaBytes, _rowGroupBytes, rowGroupIndex) => { + const rows = chunks[rowGroupIndex] ?? []; + const table = tableFromArrays({ + feature_name: rows.map((row) => row.name), + feature_name_codes: Int32Array.from(rows.map((row) => row.code)), + }); + return { intoIPCStream: () => tableToIPC(table) }; + }; +} + +describe('decodeParquetRowGroupsToTable', () => { + it('merges row groups and respects maxRows', async () => { + const table = await decodeParquetRowGroupsToTable( + mockReadParquetRowGroup([ + [ + { name: 'a', code: 0 }, + { name: 'b', code: 1 }, + ], + [ + { name: 'c', code: 2 }, + { name: 'd', code: 3 }, + ], + ]), + [ + { schemaBytes: new Uint8Array(0), rowGroupBytes: new Uint8Array(0), rowGroupIndex: 0 }, + { schemaBytes: new Uint8Array(0), rowGroupBytes: new Uint8Array(0), rowGroupIndex: 1 }, + ], + ['feature_name', 'feature_name_codes'], + 3 + ); + expect(table.numRows).toBe(3); + }); +}); + +describe('extractRowFeatureCodesFromTable with featureCodeByName', () => { + it('maps dictionary feature names to catalog codes', () => { + const names = ['gene_a', 'gene_b', 'gene_a']; + const table = tableFromArrays({ + feature_name: names, + }); + const featureCodeByName = new Map([ + ['gene_a', 0], + ['gene_b', 1], + ]); + const codes = extractRowFeatureCodesFromTable( + table, + 'feature_name', + undefined, + featureCodeByName + ); + expect([...codes]).toEqual([0, 1, 0]); + }); +}); + +describe('extractGeometryColumnar', () => { + it('returns float32 axis columns', () => { + const table = tableFromArrays({ + x: [0, 1], + y: [2, 3], + }); + const geometry = extractGeometryColumnar(table, ['x', 'y']); + expect(geometry.shape).toEqual([2, 2]); + expect([...geometry.xs]).toEqual([0, 1]); + expect([...geometry.ys]).toEqual([2, 3]); + }); +}); + +function mockReadParquet( + rows: Array<{ name: string; code: number }> +): (bytes: Uint8Array, options?: { columns?: string[] }) => { intoIPCStream(): Uint8Array } { + return () => { + const table = tableFromArrays({ + feature_name: rows.map((row) => row.name), + feature_name_codes: Int32Array.from(rows.map((row) => row.code)), + }); + return { intoIPCStream: () => tableToIPC(table) }; + }; +} + +describe('scanFeatureCatalogFromPayload', () => { + it('accumulates catalog entries from row groups', async () => { + const catalog = await scanFeatureCatalogFromPayload( + mockReadParquet([]), + mockReadParquetRowGroup([ + [ + { name: 'gene_a', code: 0 }, + { name: 'gene_b', code: 1 }, + ], + ]), + { + rowGroups: [ + { schemaBytes: new Uint8Array(0), rowGroupBytes: new Uint8Array(0), rowGroupIndex: 0 }, + ], + parts: [new Uint8Array(0)], + columns: ['feature_name', 'feature_name_codes'], + featureKey: 'feature_name', + featureCodeColumnName: 'feature_name_codes', + } + ); + expect(catalog?.entries).toEqual([ + { code: 0, name: 'gene_a' }, + { code: 1, name: 'gene_b' }, + ]); + }); +}); diff --git a/packages/core/tests/vtableMultipart.spec.ts b/packages/core/tests/vtableMultipart.spec.ts new file mode 100644 index 00000000..36d567e2 --- /dev/null +++ b/packages/core/tests/vtableMultipart.spec.ts @@ -0,0 +1,121 @@ +import { execSync } from 'node:child_process'; +import { mkdtemp, readFile, rm, stat } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join, dirname } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import SpatialDataTableSource from '../src/models/VTableSource.js'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const writerRoot = join(__dirname, '../../../python/spatialdata-experimental-writer'); + +async function writeMultipartParquetFixture(root: string, partRows: [number, number]) { + execSync( + `uv run python - <<'PY' +import pyarrow as pa +import pyarrow.parquet as pq +from pathlib import Path + +root = Path(${JSON.stringify(root)}) +root.mkdir(parents=True, exist_ok=True) + +def write_part(path: Path, start: int, count: int) -> None: + table = pa.table( + { + "x": [float(start + i) for i in range(count)], + "y": [float(i) for i in range(count)], + "feature_name": [f"gene_{i % 3}" for i in range(count)], + "feature_name_codes": pa.array([(i % 3) for i in range(count)], type=pa.int32()), + } + ) + pq.write_table(table, path) + +write_part(root / "part.0.parquet", 0, ${partRows[0]}) +write_part(root / "part.1.parquet", ${partRows[0]}, ${partRows[1]}) +PY`, + { cwd: writerRoot, stdio: 'pipe' } + ); +} + +function createFilesystemStore(root: string) { + const readStoreBytes = async (relativePath: string): Promise => { + const fullPath = join(root, relativePath); + try { + const info = await stat(fullPath); + if (info.isDirectory()) { + return null; + } + return await readFile(fullPath); + } catch { + return null; + } + }; + + return { + async get(path: string) { + const relativePath = path.startsWith('/') ? path.slice(1) : path; + return readStoreBytes(relativePath); + }, + async getRange(path: string, range: { offset?: number; length?: number; suffixLength?: number }) { + const relativePath = path.startsWith('/') ? path.slice(1) : path; + const bytes = await readStoreBytes(relativePath); + if (!bytes) { + return null; + } + if (range.suffixLength != null) { + return bytes.subarray(bytes.length - range.suffixLength); + } + const offset = range.offset ?? 0; + const length = range.length ?? bytes.length - offset; + return bytes.subarray(offset, offset + length); + }, + }; +} + +describe('SpatialDataTableSource multipart parquet reads', () => { + let fixtureRoot: string; + let source: SpatialDataTableSource; + const parquetPath = 'points/transcripts/points.parquet'; + + beforeAll(async () => { + fixtureRoot = await mkdtemp(join(tmpdir(), 'multipart-parquet-')); + await writeMultipartParquetFixture(join(fixtureRoot, parquetPath), [100, 50]); + source = new SpatialDataTableSource({ + store: createFilesystemStore(fixtureRoot), + fileType: '.zarr', + }); + }, 120_000); + + afterAll(async () => { + await rm(fixtureRoot, { recursive: true, force: true }); + }); + + it('concatenates all multipart parquet files for full-table reads', async () => { + const table = await source.loadParquetTable(parquetPath); + expect(table.numRows).toBe(150); + }); + + it('loads feature columns across all parts for catalog-style reads', async () => { + const table = await source.loadParquetTable(parquetPath, [ + 'feature_name', + 'feature_name_codes', + ]); + expect(table.numRows).toBe(150); + const codes = table.getChild('feature_name_codes')?.toArray(); + expect(codes?.length).toBe(150); + }); + + it('loads capped column subset via row-group range reads', async () => { + const { table, truncated, totalRows } = await source.loadParquetTableCapped( + parquetPath, + ['x', 'y'], + 120, + { useRowGroupReads: true } + ); + expect(totalRows).toBe(150); + expect(truncated).toBe(true); + expect(table.numRows).toBe(120); + expect(table.getChild('x')?.length).toBe(120); + expect(table.getChild('y')?.length).toBe(120); + }); +}); diff --git a/packages/core/tsconfig.json b/packages/core/tsconfig.json index fab2f55d..48b787f7 100644 --- a/packages/core/tsconfig.json +++ b/packages/core/tsconfig.json @@ -2,7 +2,7 @@ "compilerOptions": { "target": "ES2020", "module": "ESNext", - "lib": ["ES2020"], + "lib": ["ES2020", "WebWorker", "DOM"], "moduleResolution": "bundler", "baseUrl": ".", "paths": { diff --git a/packages/core/vendor/parquet-wasm/README.md b/packages/core/vendor/parquet-wasm/README.md new file mode 100644 index 00000000..6404c852 --- /dev/null +++ b/packages/core/vendor/parquet-wasm/README.md @@ -0,0 +1,17 @@ +# Vendored parquet-wasm (browser ESM build) + + +Copied from the Vitessce CDN build: + +- `https://cdn.vitessce.io/parquet-wasm@2c23652/esm/parquet_wasm.js` +- `https://cdn.vitessce.io/parquet-wasm@2c23652/esm/parquet_wasm_bg.wasm` + +This build includes row-group APIs (`readMetadata`, `readParquetRowGroup`) that are +not present in the published `parquet-wasm@0.6.1` npm package. + +https://github.com/kylebarron/parquet-wasm/issues/804 + +Loaded by `packages/core/src/parquetWasmLoader.ts`. + +Upstream: [kylebarron/parquet-wasm](https://github.com/kylebarron/parquet-wasm) +(MIT OR Apache-2.0). diff --git a/packages/core/vendor/parquet-wasm/parquet_wasm.d.ts b/packages/core/vendor/parquet-wasm/parquet_wasm.d.ts new file mode 100644 index 00000000..900939c3 --- /dev/null +++ b/packages/core/vendor/parquet-wasm/parquet_wasm.d.ts @@ -0,0 +1,15 @@ +declare const init: (moduleOrPath?: unknown) => Promise; +export function initSync(module?: unknown): unknown; +export function readParquet( + bytes: Uint8Array, + options?: { columns?: string[]; limit?: number; offset?: number } +): { intoIPCStream(): Uint8Array }; +export function readSchema(bytes: Uint8Array): { intoIPCStream(): Uint8Array }; +export function readMetadata(bytes: Uint8Array): unknown; +export function readParquetRowGroup( + footerBytes: Uint8Array, + rowGroupBytes: Uint8Array, + rowGroupIndex: number, + options?: { columns?: string[]; limit?: number; offset?: number } +): { intoIPCStream(): Uint8Array }; +export default init; diff --git a/packages/core/vendor/parquet-wasm/parquet_wasm.js b/packages/core/vendor/parquet-wasm/parquet_wasm.js new file mode 100644 index 00000000..37a78a37 --- /dev/null +++ b/packages/core/vendor/parquet-wasm/parquet_wasm.js @@ -0,0 +1,2983 @@ +let wasm; + +let cachedUint8ArrayMemory0 = null; + +function getUint8ArrayMemory0() { + if (cachedUint8ArrayMemory0 === null || cachedUint8ArrayMemory0.byteLength === 0) { + cachedUint8ArrayMemory0 = new Uint8Array(wasm.memory.buffer); + } + return cachedUint8ArrayMemory0; +} + +let cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true }); + +cachedTextDecoder.decode(); + +const MAX_SAFARI_DECODE_BYTES = 2146435072; +let numBytesDecoded = 0; +function decodeText(ptr, len) { + numBytesDecoded += len; + if (numBytesDecoded >= MAX_SAFARI_DECODE_BYTES) { + cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true }); + cachedTextDecoder.decode(); + numBytesDecoded = len; + } + return cachedTextDecoder.decode(getUint8ArrayMemory0().subarray(ptr, ptr + len)); +} + +function getStringFromWasm0(ptr, len) { + ptr = ptr >>> 0; + return decodeText(ptr, len); +} + +let WASM_VECTOR_LEN = 0; + +const cachedTextEncoder = new TextEncoder(); + +if (!('encodeInto' in cachedTextEncoder)) { + cachedTextEncoder.encodeInto = function (arg, view) { + const buf = cachedTextEncoder.encode(arg); + view.set(buf); + return { + read: arg.length, + written: buf.length + }; + } +} + +function passStringToWasm0(arg, malloc, realloc) { + + if (realloc === undefined) { + const buf = cachedTextEncoder.encode(arg); + const ptr = malloc(buf.length, 1) >>> 0; + getUint8ArrayMemory0().subarray(ptr, ptr + buf.length).set(buf); + WASM_VECTOR_LEN = buf.length; + return ptr; + } + + let len = arg.length; + let ptr = malloc(len, 1) >>> 0; + + const mem = getUint8ArrayMemory0(); + + let offset = 0; + + for (; offset < len; offset++) { + const code = arg.charCodeAt(offset); + if (code > 0x7F) break; + mem[ptr + offset] = code; + } + + if (offset !== len) { + if (offset !== 0) { + arg = arg.slice(offset); + } + ptr = realloc(ptr, len, len = offset + arg.length * 3, 1) >>> 0; + const view = getUint8ArrayMemory0().subarray(ptr + offset, ptr + len); + const ret = cachedTextEncoder.encodeInto(arg, view); + + offset += ret.written; + ptr = realloc(ptr, len, offset, 1) >>> 0; + } + + WASM_VECTOR_LEN = offset; + return ptr; +} + +let cachedDataViewMemory0 = null; + +function getDataViewMemory0() { + if (cachedDataViewMemory0 === null || cachedDataViewMemory0.buffer.detached === true || (cachedDataViewMemory0.buffer.detached === undefined && cachedDataViewMemory0.buffer !== wasm.memory.buffer)) { + cachedDataViewMemory0 = new DataView(wasm.memory.buffer); + } + return cachedDataViewMemory0; +} + +function addToExternrefTable0(obj) { + const idx = wasm.__externref_table_alloc(); + wasm.__wbindgen_export_4.set(idx, obj); + return idx; +} + +function handleError(f, args) { + try { + return f.apply(this, args); + } catch (e) { + const idx = addToExternrefTable0(e); + wasm.__wbindgen_exn_store(idx); + } +} + +function isLikeNone(x) { + return x === undefined || x === null; +} + +function getArrayU8FromWasm0(ptr, len) { + ptr = ptr >>> 0; + return getUint8ArrayMemory0().subarray(ptr / 1, ptr / 1 + len); +} + +function debugString(val) { + // primitive types + const type = typeof val; + if (type == 'number' || type == 'boolean' || val == null) { + return `${val}`; + } + if (type == 'string') { + return `"${val}"`; + } + if (type == 'symbol') { + const description = val.description; + if (description == null) { + return 'Symbol'; + } else { + return `Symbol(${description})`; + } + } + if (type == 'function') { + const name = val.name; + if (typeof name == 'string' && name.length > 0) { + return `Function(${name})`; + } else { + return 'Function'; + } + } + // objects + if (Array.isArray(val)) { + const length = val.length; + let debug = '['; + if (length > 0) { + debug += debugString(val[0]); + } + for(let i = 1; i < length; i++) { + debug += ', ' + debugString(val[i]); + } + debug += ']'; + return debug; + } + // Test for built-in + const builtInMatches = /\[object ([^\]]+)\]/.exec(toString.call(val)); + let className; + if (builtInMatches && builtInMatches.length > 1) { + className = builtInMatches[1]; + } else { + // Failed to match the standard '[object ClassName]' + return toString.call(val); + } + if (className == 'Object') { + // we're a user defined class or Object + // JSON.stringify avoids problems with cycles, and is generally much + // easier than looping through ownProperties of `val`. + try { + return 'Object(' + JSON.stringify(val) + ')'; + } catch (_) { + return 'Object'; + } + } + // errors + if (val instanceof Error) { + return `${val.name}: ${val.message}\n${val.stack}`; + } + // TODO we could test for more things here, like `Set`s and `Map`s. + return className; +} + +const CLOSURE_DTORS = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry( +state => { + wasm.__wbindgen_export_5.get(state.dtor)(state.a, state.b); +} +); + +function makeMutClosure(arg0, arg1, dtor, f) { + const state = { a: arg0, b: arg1, cnt: 1, dtor }; + const real = (...args) => { + + // First up with a closure we increment the internal reference + // count. This ensures that the Rust closure environment won't + // be deallocated while we're invoking it. + state.cnt++; + const a = state.a; + state.a = 0; + try { + return f(a, state.b, ...args); + } finally { + if (--state.cnt === 0) { + wasm.__wbindgen_export_5.get(state.dtor)(a, state.b); + CLOSURE_DTORS.unregister(state); + } else { + state.a = a; + } + } + }; + real.original = state; + CLOSURE_DTORS.register(real, state, state); + return real; +} + +function takeFromExternrefTable0(idx) { + const value = wasm.__wbindgen_export_4.get(idx); + wasm.__externref_table_dealloc(idx); + return value; +} + +function passArray8ToWasm0(arg, malloc) { + const ptr = malloc(arg.length * 1, 1) >>> 0; + getUint8ArrayMemory0().set(arg, ptr / 1); + WASM_VECTOR_LEN = arg.length; + return ptr; +} +/** + * Read a Parquet file into Arrow data. + * + * This returns an Arrow table in WebAssembly memory. To transfer the Arrow table to JavaScript + * memory you have two options: + * + * - (Easier): Call {@linkcode Table.intoIPCStream} to construct a buffer that can be parsed with + * Arrow JS's `tableFromIPC` function. + * - (More performant but bleeding edge): Call {@linkcode Table.intoFFI} to construct a data + * representation that can be parsed zero-copy from WebAssembly with + * [arrow-js-ffi](https://github.com/kylebarron/arrow-js-ffi) using `parseTable`. + * + * Example with IPC stream: + * + * ```js + * import { tableFromIPC } from "apache-arrow"; + * import initWasm, {readParquet} from "parquet-wasm"; + * + * // Instantiate the WebAssembly context + * await initWasm(); + * + * const resp = await fetch("https://example.com/file.parquet"); + * const parquetUint8Array = new Uint8Array(await resp.arrayBuffer()); + * const arrowWasmTable = readParquet(parquetUint8Array); + * const arrowTable = tableFromIPC(arrowWasmTable.intoIPCStream()); + * ``` + * + * Example with `arrow-js-ffi`: + * + * ```js + * import { parseTable } from "arrow-js-ffi"; + * import initWasm, {readParquet, wasmMemory} from "parquet-wasm"; + * + * // Instantiate the WebAssembly context + * await initWasm(); + * const WASM_MEMORY = wasmMemory(); + * + * const resp = await fetch("https://example.com/file.parquet"); + * const parquetUint8Array = new Uint8Array(await resp.arrayBuffer()); + * const arrowWasmTable = readParquet(parquetUint8Array); + * const ffiTable = arrowWasmTable.intoFFI(); + * const arrowTable = parseTable( + * WASM_MEMORY.buffer, + * ffiTable.arrayAddrs(), + * ffiTable.schemaAddr() + * ); + * ``` + * + * @param parquet_file Uint8Array containing Parquet data + * @param options + * + * Options for reading Parquet data. Optional keys include: + * + * - `batchSize`: The number of rows in each batch. If not provided, the upstream parquet + * default is 1024. + * - `rowGroups`: Only read data from the provided row group indexes. + * - `limit`: Provide a limit to the number of rows to be read. + * - `offset`: Provide an offset to skip over the given number of rows. + * - `columns`: The column names from the file to read. + * @param {Uint8Array} parquet_file + * @param {ReaderOptions | null} [options] + * @returns {Table} + */ +export function readParquet(parquet_file, options) { + const ptr0 = passArray8ToWasm0(parquet_file, wasm.__wbindgen_malloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.readParquet(ptr0, len0, isLikeNone(options) ? 0 : addToExternrefTable0(options)); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return Table.__wrap(ret[0]); +} + +/** + * @param {Uint8Array} footer_bytes + * @param {Uint8Array} row_group_bytes + * @param {number} row_group_index + * @param {ReaderOptions | null} [options] + * @returns {Table} + */ +export function readParquetRowGroup(footer_bytes, row_group_bytes, row_group_index, options) { + const ptr0 = passArray8ToWasm0(footer_bytes, wasm.__wbindgen_malloc); + const len0 = WASM_VECTOR_LEN; + const ptr1 = passArray8ToWasm0(row_group_bytes, wasm.__wbindgen_malloc); + const len1 = WASM_VECTOR_LEN; + const ret = wasm.readParquetRowGroup(ptr0, len0, ptr1, len1, row_group_index, isLikeNone(options) ? 0 : addToExternrefTable0(options)); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return Table.__wrap(ret[0]); +} + +/** + * Read an Arrow schema from a Parquet file in memory. + * + * This returns an Arrow schema in WebAssembly memory. To transfer the Arrow schema to JavaScript + * memory you have two options: + * + * - (Easier): Call {@linkcode Schema.intoIPCStream} to construct a buffer that can be parsed with + * Arrow JS's `tableFromIPC` function. This results in an Arrow JS Table with zero rows but a + * valid schema. + * - (More performant but bleeding edge): Call {@linkcode Schema.intoFFI} to construct a data + * representation that can be parsed zero-copy from WebAssembly with + * [arrow-js-ffi](https://github.com/kylebarron/arrow-js-ffi) using `parseSchema`. + * + * Example with IPC Stream: + * + * ```js + * import { tableFromIPC } from "apache-arrow"; + * import initWasm, {readSchema} from "parquet-wasm"; + * + * // Instantiate the WebAssembly context + * await initWasm(); + * + * const resp = await fetch("https://example.com/file.parquet"); + * const parquetUint8Array = new Uint8Array(await resp.arrayBuffer()); + * const arrowWasmSchema = readSchema(parquetUint8Array); + * const arrowTable = tableFromIPC(arrowWasmSchema.intoIPCStream()); + * const arrowSchema = arrowTable.schema; + * ``` + * + * Example with `arrow-js-ffi`: + * + * ```js + * import { parseSchema } from "arrow-js-ffi"; + * import initWasm, {readSchema, wasmMemory} from "parquet-wasm"; + * + * // Instantiate the WebAssembly context + * await initWasm(); + * const WASM_MEMORY = wasmMemory(); + * + * const resp = await fetch("https://example.com/file.parquet"); + * const parquetUint8Array = new Uint8Array(await resp.arrayBuffer()); + * const arrowWasmSchema = readSchema(parquetUint8Array); + * const ffiSchema = arrowWasmSchema.intoFFI(); + * const arrowTable = parseSchema(WASM_MEMORY.buffer, ffiSchema.addr()); + * const arrowSchema = arrowTable.schema; + * ``` + * + * @param parquet_file Uint8Array containing Parquet data + * @param {Uint8Array} parquet_file + * @returns {Schema} + */ +export function readSchema(parquet_file) { + const ptr0 = passArray8ToWasm0(parquet_file, wasm.__wbindgen_malloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.readSchema(ptr0, len0); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return Schema.__wrap(ret[0]); +} + +/** + * Read Parquet metadata from a Parquet file (or footer-only) bytes in memory. + * @param {Uint8Array} parquet_file + * @returns {ParquetMetaData} + */ +export function readMetadata(parquet_file) { + const ptr0 = passArray8ToWasm0(parquet_file, wasm.__wbindgen_malloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.readMetadata(ptr0, len0); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return ParquetMetaData.__wrap(ret[0]); +} + +function _assertClass(instance, klass) { + if (!(instance instanceof klass)) { + throw new Error(`expected instance of ${klass.name}`); + } +} +/** + * Write Arrow data to a Parquet file. + * + * For example, to create a Parquet file with Snappy compression: + * + * ```js + * import { tableToIPC } from "apache-arrow"; + * // Edit the `parquet-wasm` import as necessary + * import initWasm, { + * Table, + * WriterPropertiesBuilder, + * Compression, + * writeParquet, + * } from "parquet-wasm"; + * + * // Instantiate the WebAssembly context + * await initWasm(); + * + * // Given an existing arrow JS table under `table` + * const wasmTable = Table.fromIPCStream(tableToIPC(table, "stream")); + * const writerProperties = new WriterPropertiesBuilder() + * .setCompression(Compression.SNAPPY) + * .build(); + * const parquetUint8Array = writeParquet(wasmTable, writerProperties); + * ``` + * + * If `writerProperties` is not provided or is `null`, the default writer properties will be used. + * This is equivalent to `new WriterPropertiesBuilder().build()`. + * + * @param table A {@linkcode Table} representation in WebAssembly memory. + * @param writer_properties (optional) Configuration for writing to Parquet. Use the {@linkcode + * WriterPropertiesBuilder} to build a writing configuration, then call `.build()` to create an + * immutable writer properties to pass in here. + * @returns Uint8Array containing written Parquet data. + * @param {Table} table + * @param {WriterProperties | null} [writer_properties] + * @returns {Uint8Array} + */ +export function writeParquet(table, writer_properties) { + _assertClass(table, Table); + var ptr0 = table.__destroy_into_raw(); + let ptr1 = 0; + if (!isLikeNone(writer_properties)) { + _assertClass(writer_properties, WriterProperties); + ptr1 = writer_properties.__destroy_into_raw(); + } + const ret = wasm.writeParquet(ptr0, ptr1); + if (ret[3]) { + throw takeFromExternrefTable0(ret[2]); + } + var v3 = getArrayU8FromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 1, 1); + return v3; +} + +/** + * Read a Parquet file into a stream of Arrow `RecordBatch`es. + * + * This returns a ReadableStream containing RecordBatches in WebAssembly memory. To transfer the + * Arrow table to JavaScript memory you have two options: + * + * - (Easier): Call {@linkcode RecordBatch.intoIPCStream} to construct a buffer that can be parsed + * with Arrow JS's `tableFromIPC` function. (The table will have a single internal record + * batch). + * - (More performant but bleeding edge): Call {@linkcode RecordBatch.intoFFI} to construct a data + * representation that can be parsed zero-copy from WebAssembly with + * [arrow-js-ffi](https://github.com/kylebarron/arrow-js-ffi) using `parseRecordBatch`. + * + * Example with IPC stream: + * + * ```js + * import { tableFromIPC, Table } from "apache-arrow"; + * import initWasm, {readParquetStream} from "parquet-wasm"; + * + * // Instantiate the WebAssembly context + * await initWasm(); + * + * const stream = await readParquetStream(url); + * + * const batches = []; + * for await (const wasmRecordBatch of stream) { + * const arrowTable = tableFromIPC(wasmRecordBatch.intoIPCStream()); + * batches.push(...arrowTable.batches); + * } + * const table = new Table(batches); + * ``` + * + * Example with `arrow-js-ffi`: + * + * ```js + * import { Table } from "apache-arrow"; + * import { parseRecordBatch } from "arrow-js-ffi"; + * import initWasm, {readParquetStream, wasmMemory} from "parquet-wasm"; + * + * // Instantiate the WebAssembly context + * await initWasm(); + * const WASM_MEMORY = wasmMemory(); + * + * const stream = await readParquetStream(url); + * + * const batches = []; + * for await (const wasmRecordBatch of stream) { + * const ffiRecordBatch = wasmRecordBatch.intoFFI(); + * const recordBatch = parseRecordBatch( + * WASM_MEMORY.buffer, + * ffiRecordBatch.arrayAddr(), + * ffiRecordBatch.schemaAddr(), + * true + * ); + * batches.push(recordBatch); + * } + * const table = new Table(batches); + * ``` + * + * @param url URL to Parquet file + * @param {string} url + * @param {number | null} [content_length] + * @returns {Promise} + */ +export function readParquetStream(url, content_length) { + const ptr0 = passStringToWasm0(url, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.readParquetStream(ptr0, len0, isLikeNone(content_length) ? 0x100000001 : (content_length) >>> 0); + return ret; +} + +/** + * Transform a ReadableStream of RecordBatches to a ReadableStream of bytes + * + * Browser example with piping to a file via the File System API: + * + * ```js + * import initWasm, {ParquetFile, transformParquetStream} from "parquet-wasm"; + * + * // Instantiate the WebAssembly context + * await initWasm(); + * + * const fileInstance = await ParquetFile.fromUrl("https://example.com/file.parquet"); + * const recordBatchStream = await fileInstance.stream(); + * const serializedParquetStream = await transformParquetStream(recordBatchStream); + * // NB: requires transient user activation - you would typically do this before ☝️ + * const handle = await window.showSaveFilePicker(); + * const writable = await handle.createWritable(); + * await serializedParquetStream.pipeTo(writable); + * ``` + * + * NodeJS (ESM) example with piping to a file: + * ```js + * import { open } from "node:fs/promises"; + * import { Writable } from "node:stream"; + * import initWasm, {ParquetFile, transformParquetStream} from "parquet-wasm"; + * + * // Instantiate the WebAssembly context + * await initWasm(); + * + * const fileInstance = await ParquetFile.fromUrl("https://example.com/file.parquet"); + * const recordBatchStream = await fileInstance.stream(); + * const serializedParquetStream = await transformParquetStream(recordBatchStream); + * + * // grab a file handle via fsPromises + * const handle = await open("file.parquet"); + * const destinationStream = Writable.toWeb(handle.createWriteStream()); + * await serializedParquetStream.pipeTo(destinationStream); + * + * ``` + * NB: the above is a little contrived - `await writeFile("file.parquet", serializedParquetStream)` + * is enough for most use cases. + * + * Browser kitchen sink example - teeing to the Cache API, using as a streaming post body, transferring + * to a Web Worker: + * ```js + * // prelude elided - see above + * const serializedParquetStream = await transformParquetStream(recordBatchStream); + * const [cacheStream, bodyStream] = serializedParquetStream.tee(); + * const postProm = fetch(targetUrl, { + * method: "POST", + * duplex: "half", + * body: bodyStream + * }); + * const targetCache = await caches.open("foobar"); + * await targetCache.put("https://example.com/file.parquet", new Response(cacheStream)); + * // this could have been done with another tee, but beware of buffering + * const workerStream = await targetCache.get("https://example.com/file.parquet").body; + * const worker = new Worker("worker.js"); + * worker.postMessage(workerStream, [workerStream]); + * await postProm; + * ``` + * + * @param stream A {@linkcode ReadableStream} of {@linkcode RecordBatch} instances + * @param writer_properties (optional) Configuration for writing to Parquet. Use the {@linkcode + * WriterPropertiesBuilder} to build a writing configuration, then call `.build()` to create an + * immutable writer properties to pass in here. + * @returns ReadableStream containing serialized Parquet data. + * @param {ReadableStream} stream + * @param {WriterProperties | null} [writer_properties] + * @returns {Promise} + */ +export function transformParquetStream(stream, writer_properties) { + let ptr0 = 0; + if (!isLikeNone(writer_properties)) { + _assertClass(writer_properties, WriterProperties); + ptr0 = writer_properties.__destroy_into_raw(); + } + const ret = wasm.transformParquetStream(stream, ptr0); + return ret; +} + +function getArrayJsValueFromWasm0(ptr, len) { + ptr = ptr >>> 0; + const mem = getDataViewMemory0(); + const result = []; + for (let i = ptr; i < ptr + 4 * len; i += 4) { + result.push(wasm.__wbindgen_export_4.get(mem.getUint32(i, true))); + } + wasm.__externref_drop_slice(ptr, len); + return result; +} + +let cachedUint32ArrayMemory0 = null; + +function getUint32ArrayMemory0() { + if (cachedUint32ArrayMemory0 === null || cachedUint32ArrayMemory0.byteLength === 0) { + cachedUint32ArrayMemory0 = new Uint32Array(wasm.memory.buffer); + } + return cachedUint32ArrayMemory0; +} + +function getArrayU32FromWasm0(ptr, len) { + ptr = ptr >>> 0; + return getUint32ArrayMemory0().subarray(ptr / 4, ptr / 4 + len); +} +/** + * Returns a handle to this wasm instance's `WebAssembly.Memory` + * @returns {Memory} + */ +export function wasmMemory() { + const ret = wasm.wasmMemory(); + return ret; +} + +/** + * Returns a handle to this wasm instance's `WebAssembly.Table` which is the indirect function + * table used by Rust + * @returns {FunctionTable} + */ +export function _functionTable() { + const ret = wasm._functionTable(); + return ret; +} + +function __wbg_adapter_6(arg0, arg1) { + wasm.wasm_bindgen__convert__closures_____invoke__ha714fc933ee2ad72(arg0, arg1); +} + +function __wbg_adapter_11(arg0, arg1, arg2) { + wasm.closure3814_externref_shim(arg0, arg1, arg2); +} + +function __wbg_adapter_271(arg0, arg1, arg2, arg3) { + wasm.closure3828_externref_shim(arg0, arg1, arg2, arg3); +} + +/** + * Supported compression algorithms. + * + * Codecs added in format version X.Y can be read by readers based on X.Y and later. + * Codec support may vary between readers based on the format version and + * libraries available at runtime. + * @enum {0 | 1 | 2 | 3 | 4 | 5 | 6 | 7} + */ +export const Compression = Object.freeze({ + UNCOMPRESSED: 0, "0": "UNCOMPRESSED", + SNAPPY: 1, "1": "SNAPPY", + GZIP: 2, "2": "GZIP", + BROTLI: 3, "3": "BROTLI", + /** + * @deprecated as of Parquet 2.9.0. + * Switch to LZ4_RAW + */ + LZ4: 4, "4": "LZ4", + ZSTD: 5, "5": "ZSTD", + LZ4_RAW: 6, "6": "LZ4_RAW", + LZO: 7, "7": "LZO", +}); +/** + * Controls the level of statistics to be computed by the writer + * @enum {0 | 1 | 2} + */ +export const EnabledStatistics = Object.freeze({ + /** + * Compute no statistics + */ + None: 0, "0": "None", + /** + * Compute chunk-level statistics but not page-level + */ + Chunk: 1, "1": "Chunk", + /** + * Compute page-level and chunk-level statistics + */ + Page: 2, "2": "Page", +}); +/** + * Encodings supported by Parquet. + * Not all encodings are valid for all types. These enums are also used to specify the + * encoding of definition and repetition levels. + * @enum {0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8} + */ +export const Encoding = Object.freeze({ + /** + * Default byte encoding. + * - BOOLEAN - 1 bit per value, 0 is false; 1 is true. + * - INT32 - 4 bytes per value, stored as little-endian. + * - INT64 - 8 bytes per value, stored as little-endian. + * - FLOAT - 4 bytes per value, stored as little-endian. + * - DOUBLE - 8 bytes per value, stored as little-endian. + * - BYTE_ARRAY - 4 byte length stored as little endian, followed by bytes. + * - FIXED_LEN_BYTE_ARRAY - just the bytes are stored. + */ + PLAIN: 0, "0": "PLAIN", + /** + * **Deprecated** dictionary encoding. + * + * The values in the dictionary are encoded using PLAIN encoding. + * Since it is deprecated, RLE_DICTIONARY encoding is used for a data page, and + * PLAIN encoding is used for dictionary page. + */ + PLAIN_DICTIONARY: 1, "1": "PLAIN_DICTIONARY", + /** + * Group packed run length encoding. + * + * Usable for definition/repetition levels encoding and boolean values. + */ + RLE: 2, "2": "RLE", + /** + * Bit packed encoding. + * + * This can only be used if the data has a known max width. + * Usable for definition/repetition levels encoding. + */ + BIT_PACKED: 3, "3": "BIT_PACKED", + /** + * Delta encoding for integers, either INT32 or INT64. + * + * Works best on sorted data. + */ + DELTA_BINARY_PACKED: 4, "4": "DELTA_BINARY_PACKED", + /** + * Encoding for byte arrays to separate the length values and the data. + * + * The lengths are encoded using DELTA_BINARY_PACKED encoding. + */ + DELTA_LENGTH_BYTE_ARRAY: 5, "5": "DELTA_LENGTH_BYTE_ARRAY", + /** + * Incremental encoding for byte arrays. + * + * Prefix lengths are encoded using DELTA_BINARY_PACKED encoding. + * Suffixes are stored using DELTA_LENGTH_BYTE_ARRAY encoding. + */ + DELTA_BYTE_ARRAY: 6, "6": "DELTA_BYTE_ARRAY", + /** + * Dictionary encoding. + * + * The ids are encoded using the RLE encoding. + */ + RLE_DICTIONARY: 7, "7": "RLE_DICTIONARY", + /** + * Encoding for floating-point data. + * + * K byte-streams are created where K is the size in bytes of the data type. + * The individual bytes of an FP value are scattered to the corresponding stream and + * the streams are concatenated. + * This itself does not reduce the size of the data but can lead to better compression + * afterwards. + */ + BYTE_STREAM_SPLIT: 8, "8": "BYTE_STREAM_SPLIT", +}); +/** + * The Parquet version to use when writing + * @enum {0 | 1} + */ +export const WriterVersion = Object.freeze({ + V1: 0, "0": "V1", + V2: 1, "1": "V2", +}); + +const __wbindgen_enum_ReadableStreamType = ["bytes"]; + +const __wbindgen_enum_RequestCache = ["default", "no-store", "reload", "no-cache", "force-cache", "only-if-cached"]; + +const __wbindgen_enum_RequestCredentials = ["omit", "same-origin", "include"]; + +const __wbindgen_enum_RequestMode = ["same-origin", "no-cors", "cors", "navigate"]; + +const ColumnChunkMetaDataFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_columnchunkmetadata_free(ptr >>> 0, 1)); +/** + * Metadata for a Parquet column chunk. + */ +export class ColumnChunkMetaData { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(ColumnChunkMetaData.prototype); + obj.__wbg_ptr = ptr; + ColumnChunkMetaDataFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + ColumnChunkMetaDataFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_columnchunkmetadata_free(ptr, 0); + } + /** + * File where the column chunk is stored. + * + * If not set, assumed to belong to the same file as the metadata. + * This path is relative to the current file. + * @returns {string | undefined} + */ + filePath() { + const ret = wasm.columnchunkmetadata_filePath(this.__wbg_ptr); + let v1; + if (ret[0] !== 0) { + v1 = getStringFromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 1, 1); + } + return v1; + } + /** + * Byte offset in `file_path()`. + * @returns {bigint} + */ + fileOffset() { + const ret = wasm.columnchunkmetadata_fileOffset(this.__wbg_ptr); + return ret; + } + /** + * Path (or identifier) of this column. + * @returns {string[]} + */ + columnPath() { + const ret = wasm.columnchunkmetadata_columnPath(this.__wbg_ptr); + var v1 = getArrayJsValueFromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 4, 4); + return v1; + } + /** + * All encodings used for this column. + * @returns {any[]} + */ + encodings() { + const ret = wasm.columnchunkmetadata_encodings(this.__wbg_ptr); + var v1 = getArrayJsValueFromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 4, 4); + return v1; + } + /** + * Total number of values in this column chunk. + * @returns {number} + */ + numValues() { + const ret = wasm.columnchunkmetadata_numValues(this.__wbg_ptr); + return ret; + } + /** + * Compression for this column. + * @returns {Compression} + */ + compression() { + const ret = wasm.columnchunkmetadata_compression(this.__wbg_ptr); + return ret; + } + /** + * Returns the total compressed data size of this column chunk. + * @returns {number} + */ + compressedSize() { + const ret = wasm.columnchunkmetadata_compressedSize(this.__wbg_ptr); + return ret; + } + /** + * Returns the total uncompressed data size of this column chunk. + * @returns {number} + */ + uncompressedSize() { + const ret = wasm.columnchunkmetadata_uncompressedSize(this.__wbg_ptr); + return ret; + } +} +if (Symbol.dispose) ColumnChunkMetaData.prototype[Symbol.dispose] = ColumnChunkMetaData.prototype.free; + +const FFIDataFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_ffidata_free(ptr >>> 0, 1)); +/** + * An Arrow array exported to FFI. + * + * Using [`arrow-js-ffi`](https://github.com/kylebarron/arrow-js-ffi), you can view or copy Arrow + * these objects to JavaScript. + * + * Note that this also includes an ArrowSchema C struct as well, so that extension type + * information can be maintained. + * ## Memory management + * + * Note that this array will not be released automatically. You need to manually call `.free()` to + * release memory. + */ +export class FFIData { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(FFIData.prototype); + obj.__wbg_ptr = ptr; + FFIDataFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + FFIDataFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_ffidata_free(ptr, 0); + } + /** + * Access the pointer to the + * [`ArrowArray`](https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions) + * struct. This can be viewed or copied (without serialization) to an Arrow JS `RecordBatch` by + * using [`arrow-js-ffi`](https://github.com/kylebarron/arrow-js-ffi). You can access the + * [`WebAssembly.Memory`](https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/Memory) + * instance by using {@linkcode wasmMemory}. + * + * **Example**: + * + * ```ts + * import { parseRecordBatch } from "arrow-js-ffi"; + * + * const wasmRecordBatch: FFIRecordBatch = ... + * const wasmMemory: WebAssembly.Memory = wasmMemory(); + * + * // Pass `true` to copy arrays across the boundary instead of creating views. + * const jsRecordBatch = parseRecordBatch( + * wasmMemory.buffer, + * wasmRecordBatch.arrayAddr(), + * wasmRecordBatch.schemaAddr(), + * true + * ); + * ``` + * @returns {number} + */ + arrayAddr() { + const ret = wasm.ffidata_arrayAddr(this.__wbg_ptr); + return ret >>> 0; + } + /** + * Access the pointer to the + * [`ArrowSchema`](https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions) + * struct. This can be viewed or copied (without serialization) to an Arrow JS `Field` by + * using [`arrow-js-ffi`](https://github.com/kylebarron/arrow-js-ffi). You can access the + * [`WebAssembly.Memory`](https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/Memory) + * instance by using {@linkcode wasmMemory}. + * + * **Example**: + * + * ```ts + * import { parseRecordBatch } from "arrow-js-ffi"; + * + * const wasmRecordBatch: FFIRecordBatch = ... + * const wasmMemory: WebAssembly.Memory = wasmMemory(); + * + * // Pass `true` to copy arrays across the boundary instead of creating views. + * const jsRecordBatch = parseRecordBatch( + * wasmMemory.buffer, + * wasmRecordBatch.arrayAddr(), + * wasmRecordBatch.schemaAddr(), + * true + * ); + * ``` + * @returns {number} + */ + schemaAddr() { + const ret = wasm.ffidata_schemaAddr(this.__wbg_ptr); + return ret >>> 0; + } +} +if (Symbol.dispose) FFIData.prototype[Symbol.dispose] = FFIData.prototype.free; + +const FFISchemaFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_ffischema_free(ptr >>> 0, 1)); + +export class FFISchema { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(FFISchema.prototype); + obj.__wbg_ptr = ptr; + FFISchemaFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + FFISchemaFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_ffischema_free(ptr, 0); + } + /** + * Access the pointer to the + * [`ArrowSchema`](https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions) + * struct. This can be viewed or copied (without serialization) to an Arrow JS `Field` by + * using [`arrow-js-ffi`](https://github.com/kylebarron/arrow-js-ffi). You can access the + * [`WebAssembly.Memory`](https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/Memory) + * instance by using {@linkcode wasmMemory}. + * + * **Example**: + * + * ```ts + * import { parseRecordBatch } from "arrow-js-ffi"; + * + * const wasmRecordBatch: FFIRecordBatch = ... + * const wasmMemory: WebAssembly.Memory = wasmMemory(); + * + * // Pass `true` to copy arrays across the boundary instead of creating views. + * const jsRecordBatch = parseRecordBatch( + * wasmMemory.buffer, + * wasmRecordBatch.arrayAddr(), + * wasmRecordBatch.schemaAddr(), + * true + * ); + * ``` + * @returns {number} + */ + addr() { + const ret = wasm.ffischema_addr(this.__wbg_ptr); + return ret >>> 0; + } +} +if (Symbol.dispose) FFISchema.prototype[Symbol.dispose] = FFISchema.prototype.free; + +const FFIStreamFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_ffistream_free(ptr >>> 0, 1)); +/** + * A representation of an Arrow C Stream in WebAssembly memory exposed as FFI-compatible + * structs through the Arrow C Data Interface. + * + * Unlike other Arrow implementations outside of JS, this always stores the "stream" fully + * materialized as a sequence of Arrow chunks. + */ +export class FFIStream { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(FFIStream.prototype); + obj.__wbg_ptr = ptr; + FFIStreamFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + FFIStreamFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_ffistream_free(ptr, 0); + } + /** + * Get the total number of elements in this stream + * @returns {number} + */ + numArrays() { + const ret = wasm.ffistream_numArrays(this.__wbg_ptr); + return ret >>> 0; + } + /** + * Get the pointer to the ArrowSchema FFI struct + * @returns {number} + */ + schemaAddr() { + const ret = wasm.ffistream_schemaAddr(this.__wbg_ptr); + return ret >>> 0; + } + /** + * Get the pointer to one ArrowArray FFI struct for a given chunk index and column index + * + * Access the pointer to one + * [`ArrowArray`](https://arrow.apache.org/docs/format/CDataInterface.html#structure-definitions) + * struct representing one of the internal `RecordBatch`es. This can be viewed or copied (without serialization) to an Arrow JS `RecordBatch` by + * using [`arrow-js-ffi`](https://github.com/kylebarron/arrow-js-ffi). You can access the + * [`WebAssembly.Memory`](https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/Memory) + * instance by using {@linkcode wasmMemory}. + * + * **Example**: + * + * ```ts + * import * as arrow from "apache-arrow"; + * import { parseRecordBatch } from "arrow-js-ffi"; + * + * const wasmTable: FFITable = ... + * const wasmMemory: WebAssembly.Memory = wasmMemory(); + * + * const jsBatches: arrow.RecordBatch[] = [] + * for (let i = 0; i < wasmTable.numBatches(); i++) { + * // Pass `true` to copy arrays across the boundary instead of creating views. + * const jsRecordBatch = parseRecordBatch( + * wasmMemory.buffer, + * wasmTable.arrayAddr(i), + * wasmTable.schemaAddr(), + * true + * ); + * jsBatches.push(jsRecordBatch); + * } + * const jsTable = new arrow.Table(jsBatches); + * ``` + * + * @param chunk number The chunk index to use + * @returns number pointer to an ArrowArray FFI struct in Wasm memory + * @param {number} chunk + * @returns {number} + */ + arrayAddr(chunk) { + const ret = wasm.ffistream_arrayAddr(this.__wbg_ptr, chunk); + return ret >>> 0; + } + /** + * @returns {Uint32Array} + */ + arrayAddrs() { + const ret = wasm.ffistream_arrayAddrs(this.__wbg_ptr); + var v1 = getArrayU32FromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 4, 4); + return v1; + } + drop() { + const ptr = this.__destroy_into_raw(); + wasm.ffistream_drop(ptr); + } +} +if (Symbol.dispose) FFIStream.prototype[Symbol.dispose] = FFIStream.prototype.free; + +const FileMetaDataFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_filemetadata_free(ptr >>> 0, 1)); +/** + * Metadata for a Parquet file. + */ +export class FileMetaData { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(FileMetaData.prototype); + obj.__wbg_ptr = ptr; + FileMetaDataFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + FileMetaDataFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_filemetadata_free(ptr, 0); + } + /** + * Returns version of this file. + * @returns {number} + */ + version() { + const ret = wasm.filemetadata_version(this.__wbg_ptr); + return ret; + } + /** + * Returns number of rows in the file. + * @returns {number} + */ + numRows() { + const ret = wasm.filemetadata_numRows(this.__wbg_ptr); + return ret; + } + /** + * String message for application that wrote this file. + * + * This should have the following format: + * ` version (build )`. + * + * ```shell + * parquet-mr version 1.8.0 (build 0fda28af84b9746396014ad6a415b90592a98b3b) + * ``` + * @returns {string | undefined} + */ + createdBy() { + const ret = wasm.filemetadata_createdBy(this.__wbg_ptr); + let v1; + if (ret[0] !== 0) { + v1 = getStringFromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 1, 1); + } + return v1; + } + /** + * Returns key_value_metadata of this file. + * @returns {Map} + */ + keyValueMetadata() { + const ret = wasm.filemetadata_keyValueMetadata(this.__wbg_ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return takeFromExternrefTable0(ret[0]); + } +} +if (Symbol.dispose) FileMetaData.prototype[Symbol.dispose] = FileMetaData.prototype.free; + +const IntoUnderlyingByteSourceFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_intounderlyingbytesource_free(ptr >>> 0, 1)); + +export class IntoUnderlyingByteSource { + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + IntoUnderlyingByteSourceFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_intounderlyingbytesource_free(ptr, 0); + } + /** + * @returns {ReadableStreamType} + */ + get type() { + const ret = wasm.intounderlyingbytesource_type(this.__wbg_ptr); + return __wbindgen_enum_ReadableStreamType[ret]; + } + /** + * @returns {number} + */ + get autoAllocateChunkSize() { + const ret = wasm.intounderlyingbytesource_autoAllocateChunkSize(this.__wbg_ptr); + return ret >>> 0; + } + /** + * @param {ReadableByteStreamController} controller + */ + start(controller) { + wasm.intounderlyingbytesource_start(this.__wbg_ptr, controller); + } + /** + * @param {ReadableByteStreamController} controller + * @returns {Promise} + */ + pull(controller) { + const ret = wasm.intounderlyingbytesource_pull(this.__wbg_ptr, controller); + return ret; + } + cancel() { + const ptr = this.__destroy_into_raw(); + wasm.intounderlyingbytesource_cancel(ptr); + } +} +if (Symbol.dispose) IntoUnderlyingByteSource.prototype[Symbol.dispose] = IntoUnderlyingByteSource.prototype.free; + +const IntoUnderlyingSinkFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_intounderlyingsink_free(ptr >>> 0, 1)); + +export class IntoUnderlyingSink { + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + IntoUnderlyingSinkFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_intounderlyingsink_free(ptr, 0); + } + /** + * @param {any} chunk + * @returns {Promise} + */ + write(chunk) { + const ret = wasm.intounderlyingsink_write(this.__wbg_ptr, chunk); + return ret; + } + /** + * @returns {Promise} + */ + close() { + const ptr = this.__destroy_into_raw(); + const ret = wasm.intounderlyingsink_close(ptr); + return ret; + } + /** + * @param {any} reason + * @returns {Promise} + */ + abort(reason) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.intounderlyingsink_abort(ptr, reason); + return ret; + } +} +if (Symbol.dispose) IntoUnderlyingSink.prototype[Symbol.dispose] = IntoUnderlyingSink.prototype.free; + +const IntoUnderlyingSourceFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_intounderlyingsource_free(ptr >>> 0, 1)); + +export class IntoUnderlyingSource { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(IntoUnderlyingSource.prototype); + obj.__wbg_ptr = ptr; + IntoUnderlyingSourceFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + IntoUnderlyingSourceFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_intounderlyingsource_free(ptr, 0); + } + /** + * @param {ReadableStreamDefaultController} controller + * @returns {Promise} + */ + pull(controller) { + const ret = wasm.intounderlyingsource_pull(this.__wbg_ptr, controller); + return ret; + } + cancel() { + const ptr = this.__destroy_into_raw(); + wasm.intounderlyingsource_cancel(ptr); + } +} +if (Symbol.dispose) IntoUnderlyingSource.prototype[Symbol.dispose] = IntoUnderlyingSource.prototype.free; + +const ParquetFileFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_parquetfile_free(ptr >>> 0, 1)); + +export class ParquetFile { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(ParquetFile.prototype); + obj.__wbg_ptr = ptr; + ParquetFileFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + ParquetFileFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_parquetfile_free(ptr, 0); + } + /** + * Construct a ParquetFile from a new URL. + * @param {string} url + * @returns {Promise} + */ + static fromUrl(url) { + const ptr0 = passStringToWasm0(url, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.parquetfile_fromUrl(ptr0, len0); + return ret; + } + /** + * Construct a ParquetFile from a new [Blob] or [File] handle. + * + * [Blob]: https://developer.mozilla.org/en-US/docs/Web/API/Blob + * [File]: https://developer.mozilla.org/en-US/docs/Web/API/File + * + * Safety: Do not use this in a multi-threaded environment, + * (transitively depends on `!Send` `web_sys::Blob`) + * @param {Blob} handle + * @returns {Promise} + */ + static fromFile(handle) { + const ret = wasm.parquetfile_fromFile(handle); + return ret; + } + /** + * @returns {ParquetMetaData} + */ + metadata() { + const ret = wasm.parquetfile_metadata(this.__wbg_ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return ParquetMetaData.__wrap(ret[0]); + } + /** + * @returns {Schema} + */ + schema() { + const ret = wasm.parquetfile_schema(this.__wbg_ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return Schema.__wrap(ret[0]); + } + /** + * Read from the Parquet file in an async fashion. + * + * @param options + * + * Options for reading Parquet data. Optional keys include: + * + * - `batchSize`: The number of rows in each batch. If not provided, the upstream parquet + * default is 1024. + * - `rowGroups`: Only read data from the provided row group indexes. + * - `limit`: Provide a limit to the number of rows to be read. + * - `offset`: Provide an offset to skip over the given number of rows. + * - `columns`: The column names from the file to read. + * @param {ReaderOptions | null} [options] + * @returns {Promise
} + */ + read(options) { + const ret = wasm.parquetfile_read(this.__wbg_ptr, isLikeNone(options) ? 0 : addToExternrefTable0(options)); + return ret; + } + /** + * Create a readable stream of record batches. + * + * Each item in the stream will be a {@linkcode RecordBatch}. + * + * @param options + * + * Options for reading Parquet data. Optional keys include: + * + * - `batchSize`: The number of rows in each batch. If not provided, the upstream parquet + * default is 1024. + * - `rowGroups`: Only read data from the provided row group indexes. + * - `limit`: Provide a limit to the number of rows to be read. + * - `offset`: Provide an offset to skip over the given number of rows. + * - `columns`: The column names from the file to read. + * - `concurrency`: The number of concurrent requests to make + * @param {ReaderOptions | null} [options] + * @returns {Promise} + */ + stream(options) { + const ret = wasm.parquetfile_stream(this.__wbg_ptr, isLikeNone(options) ? 0 : addToExternrefTable0(options)); + return ret; + } +} +if (Symbol.dispose) ParquetFile.prototype[Symbol.dispose] = ParquetFile.prototype.free; + +const ParquetMetaDataFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_parquetmetadata_free(ptr >>> 0, 1)); +/** + * Global Parquet metadata. + */ +export class ParquetMetaData { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(ParquetMetaData.prototype); + obj.__wbg_ptr = ptr; + ParquetMetaDataFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + ParquetMetaDataFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_parquetmetadata_free(ptr, 0); + } + /** + * Returns file metadata as reference. + * @returns {FileMetaData} + */ + fileMetadata() { + const ret = wasm.parquetmetadata_fileMetadata(this.__wbg_ptr); + return FileMetaData.__wrap(ret); + } + /** + * Returns number of row groups in this file. + * @returns {number} + */ + numRowGroups() { + const ret = wasm.parquetmetadata_numRowGroups(this.__wbg_ptr); + return ret >>> 0; + } + /** + * Returns row group metadata for `i`th position. + * Position should be less than number of row groups `num_row_groups`. + * @param {number} i + * @returns {RowGroupMetaData} + */ + rowGroup(i) { + const ret = wasm.parquetmetadata_rowGroup(this.__wbg_ptr, i); + return RowGroupMetaData.__wrap(ret); + } + /** + * Returns row group metadata for all row groups + * @returns {RowGroupMetaData[]} + */ + rowGroups() { + const ret = wasm.parquetmetadata_rowGroups(this.__wbg_ptr); + var v1 = getArrayJsValueFromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 4, 4); + return v1; + } +} +if (Symbol.dispose) ParquetMetaData.prototype[Symbol.dispose] = ParquetMetaData.prototype.free; + +const RecordBatchFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_recordbatch_free(ptr >>> 0, 1)); +/** + * A group of columns of equal length in WebAssembly memory with an associated {@linkcode Schema}. + */ +export class RecordBatch { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(RecordBatch.prototype); + obj.__wbg_ptr = ptr; + RecordBatchFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + static __unwrap(jsValue) { + if (!(jsValue instanceof RecordBatch)) { + return 0; + } + return jsValue.__destroy_into_raw(); + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + RecordBatchFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_recordbatch_free(ptr, 0); + } + /** + * The number of rows in this RecordBatch. + * @returns {number} + */ + get numRows() { + const ret = wasm.recordbatch_numRows(this.__wbg_ptr); + return ret >>> 0; + } + /** + * The number of columns in this RecordBatch. + * @returns {number} + */ + get numColumns() { + const ret = wasm.recordbatch_numColumns(this.__wbg_ptr); + return ret >>> 0; + } + /** + * The {@linkcode Schema} of this RecordBatch. + * @returns {Schema} + */ + get schema() { + const ret = wasm.recordbatch_schema(this.__wbg_ptr); + return Schema.__wrap(ret); + } + /** + * Export this RecordBatch to FFI structs according to the Arrow C Data Interface. + * + * This method **does not consume** the RecordBatch, so you must remember to call {@linkcode + * RecordBatch.free} to release the resources. The underlying arrays are reference counted, so + * this method does not copy data, it only prevents the data from being released. + * @returns {FFIData} + */ + toFFI() { + const ret = wasm.recordbatch_toFFI(this.__wbg_ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return FFIData.__wrap(ret[0]); + } + /** + * Export this RecordBatch to FFI structs according to the Arrow C Data Interface. + * + * This method **does consume** the RecordBatch, so the original RecordBatch will be + * inaccessible after this call. You must still call {@linkcode FFIRecordBatch.free} after + * you've finished using the FFIRecordBatch. + * @returns {FFIData} + */ + intoFFI() { + const ptr = this.__destroy_into_raw(); + const ret = wasm.recordbatch_intoFFI(ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return FFIData.__wrap(ret[0]); + } + /** + * Consume this RecordBatch and convert to an Arrow IPC Stream buffer + * @returns {Uint8Array} + */ + intoIPCStream() { + const ptr = this.__destroy_into_raw(); + const ret = wasm.recordbatch_intoIPCStream(ptr); + if (ret[3]) { + throw takeFromExternrefTable0(ret[2]); + } + var v1 = getArrayU8FromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 1, 1); + return v1; + } + /** + * Override the schema of this [`RecordBatch`] + * + * Returns an error if `schema` is not a superset of the current schema + * as determined by [`Schema::contains`] + * @param {Schema} schema + * @returns {RecordBatch} + */ + withSchema(schema) { + _assertClass(schema, Schema); + var ptr0 = schema.__destroy_into_raw(); + const ret = wasm.recordbatch_withSchema(this.__wbg_ptr, ptr0); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return RecordBatch.__wrap(ret[0]); + } + /** + * Return a new RecordBatch where each column is sliced + * according to `offset` and `length` + * @param {number} offset + * @param {number} length + * @returns {RecordBatch} + */ + slice(offset, length) { + const ret = wasm.recordbatch_slice(this.__wbg_ptr, offset, length); + return RecordBatch.__wrap(ret); + } + /** + * Returns the total number of bytes of memory occupied physically by this batch. + * @returns {number} + */ + getArrayMemorySize() { + const ret = wasm.recordbatch_getArrayMemorySize(this.__wbg_ptr); + return ret >>> 0; + } +} +if (Symbol.dispose) RecordBatch.prototype[Symbol.dispose] = RecordBatch.prototype.free; + +const RowGroupMetaDataFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_rowgroupmetadata_free(ptr >>> 0, 1)); +/** + * Metadata for a Parquet row group. + */ +export class RowGroupMetaData { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(RowGroupMetaData.prototype); + obj.__wbg_ptr = ptr; + RowGroupMetaDataFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + RowGroupMetaDataFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_rowgroupmetadata_free(ptr, 0); + } + /** + * Number of columns in this row group. + * @returns {number} + */ + numColumns() { + const ret = wasm.rowgroupmetadata_numColumns(this.__wbg_ptr); + return ret >>> 0; + } + /** + * Returns column chunk metadata for `i`th column. + * @param {number} i + * @returns {ColumnChunkMetaData} + */ + column(i) { + const ret = wasm.rowgroupmetadata_column(this.__wbg_ptr, i); + return ColumnChunkMetaData.__wrap(ret); + } + /** + * Returns column chunk metadata for all columns + * @returns {ColumnChunkMetaData[]} + */ + columns() { + const ret = wasm.rowgroupmetadata_columns(this.__wbg_ptr); + var v1 = getArrayJsValueFromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 4, 4); + return v1; + } + /** + * Number of rows in this row group. + * @returns {number} + */ + numRows() { + const ret = wasm.rowgroupmetadata_numRows(this.__wbg_ptr); + return ret; + } + /** + * Total byte size of all uncompressed column data in this row group. + * @returns {number} + */ + totalByteSize() { + const ret = wasm.rowgroupmetadata_totalByteSize(this.__wbg_ptr); + return ret; + } + /** + * Total size of all compressed column data in this row group. + * @returns {number} + */ + compressedSize() { + const ret = wasm.rowgroupmetadata_compressedSize(this.__wbg_ptr); + return ret; + } + /** + * File offset of this row group in file. + * @returns {number | undefined} + */ + fileOffset() { + const ret = wasm.rowgroupmetadata_fileOffset(this.__wbg_ptr); + return ret[0] === 0 ? undefined : ret[1]; + } +} +if (Symbol.dispose) RowGroupMetaData.prototype[Symbol.dispose] = RowGroupMetaData.prototype.free; + +const SchemaFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_schema_free(ptr >>> 0, 1)); +/** + * A named collection of types that defines the column names and types in a RecordBatch or Table + * data structure. + * + * A Schema can also contain extra user-defined metadata either at the Table or Column level. + * Column-level metadata is often used to define [extension + * types](https://arrow.apache.org/docs/format/Columnar.html#extension-types). + */ +export class Schema { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(Schema.prototype); + obj.__wbg_ptr = ptr; + SchemaFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + SchemaFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_schema_free(ptr, 0); + } + /** + * Export this schema to an FFISchema object, which can be read with arrow-js-ffi. + * + * This method **does not consume** the Schema, so you must remember to call {@linkcode + * Schema.free} to release the resources. The underlying arrays are reference counted, so + * this method does not copy data, it only prevents the data from being released. + * @returns {FFISchema} + */ + toFFI() { + const ret = wasm.schema_toFFI(this.__wbg_ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return FFISchema.__wrap(ret[0]); + } + /** + * Export this Table to FFI structs according to the Arrow C Data Interface. + * + * This method **does consume** the Table, so the original Table will be + * inaccessible after this call. You must still call {@linkcode FFITable.free} after + * you've finished using the FFITable. + * @returns {FFISchema} + */ + intoFFI() { + const ptr = this.__destroy_into_raw(); + const ret = wasm.schema_intoFFI(ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return FFISchema.__wrap(ret[0]); + } + /** + * Consume this schema and convert to an Arrow IPC Stream buffer + * @returns {Uint8Array} + */ + intoIPCStream() { + const ptr = this.__destroy_into_raw(); + const ret = wasm.schema_intoIPCStream(ptr); + if (ret[3]) { + throw takeFromExternrefTable0(ret[2]); + } + var v1 = getArrayU8FromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 1, 1); + return v1; + } + /** + * Sets the metadata of this `Schema` to be `metadata` and returns a new object + * @param {SchemaMetadata} metadata + * @returns {Schema} + */ + withMetadata(metadata) { + const ret = wasm.schema_withMetadata(this.__wbg_ptr, metadata); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return Schema.__wrap(ret[0]); + } + /** + * Find the index of the column with the given name. + * @param {string} name + * @returns {number} + */ + indexOf(name) { + const ptr0 = passStringToWasm0(name, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.schema_indexOf(this.__wbg_ptr, ptr0, len0); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return ret[0] >>> 0; + } + /** + * Returns an immutable reference to the Map of custom metadata key-value pairs. + * @returns {SchemaMetadata} + */ + metadata() { + const ret = wasm.schema_metadata(this.__wbg_ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return takeFromExternrefTable0(ret[0]); + } +} +if (Symbol.dispose) Schema.prototype[Symbol.dispose] = Schema.prototype.free; + +const TableFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_table_free(ptr >>> 0, 1)); +/** + * A Table in WebAssembly memory conforming to the Apache Arrow spec. + * + * A Table consists of one or more {@linkcode RecordBatch} objects plus a {@linkcode Schema} that + * each RecordBatch conforms to. + */ +export class Table { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(Table.prototype); + obj.__wbg_ptr = ptr; + TableFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + TableFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_table_free(ptr, 0); + } + /** + * Access the Table's {@linkcode Schema}. + * @returns {Schema} + */ + get schema() { + const ret = wasm.table_schema(this.__wbg_ptr); + return Schema.__wrap(ret); + } + /** + * Access a RecordBatch from the Table by index. + * + * @param index The positional index of the RecordBatch to retrieve. + * @returns a RecordBatch or `null` if out of range. + * @param {number} index + * @returns {RecordBatch | undefined} + */ + recordBatch(index) { + const ret = wasm.table_recordBatch(this.__wbg_ptr, index); + return ret === 0 ? undefined : RecordBatch.__wrap(ret); + } + /** + * @returns {RecordBatch[]} + */ + recordBatches() { + const ret = wasm.table_recordBatches(this.__wbg_ptr); + var v1 = getArrayJsValueFromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 4, 4); + return v1; + } + /** + * The number of batches in the Table + * @returns {number} + */ + get numBatches() { + const ret = wasm.table_numBatches(this.__wbg_ptr); + return ret >>> 0; + } + /** + * Export this Table to FFI structs according to the Arrow C Data Interface. + * + * This method **does not consume** the Table, so you must remember to call {@linkcode + * Table.free} to release the resources. The underlying arrays are reference counted, so + * this method does not copy data, it only prevents the data from being released. + * @returns {FFIStream} + */ + toFFI() { + const ret = wasm.table_toFFI(this.__wbg_ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return FFIStream.__wrap(ret[0]); + } + /** + * Export this Table to FFI structs according to the Arrow C Data Interface. + * + * This method **does consume** the Table, so the original Table will be + * inaccessible after this call. You must still call {@linkcode FFITable.free} after + * you've finished using the FFITable. + * @returns {FFIStream} + */ + intoFFI() { + const ptr = this.__destroy_into_raw(); + const ret = wasm.table_intoFFI(ptr); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return FFIStream.__wrap(ret[0]); + } + /** + * Consume this table and convert to an Arrow IPC Stream buffer + * @returns {Uint8Array} + */ + intoIPCStream() { + const ptr = this.__destroy_into_raw(); + const ret = wasm.table_intoIPCStream(ptr); + if (ret[3]) { + throw takeFromExternrefTable0(ret[2]); + } + var v1 = getArrayU8FromWasm0(ret[0], ret[1]).slice(); + wasm.__wbindgen_free(ret[0], ret[1] * 1, 1); + return v1; + } + /** + * Create a table from an Arrow IPC Stream buffer + * @param {Uint8Array} buf + * @returns {Table} + */ + static fromIPCStream(buf) { + const ptr0 = passArray8ToWasm0(buf, wasm.__wbindgen_malloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.table_fromIPCStream(ptr0, len0); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return Table.__wrap(ret[0]); + } + /** + * Returns the total number of bytes of memory occupied physically by all batches in this + * table. + * @returns {number} + */ + getArrayMemorySize() { + const ret = wasm.table_getArrayMemorySize(this.__wbg_ptr); + return ret >>> 0; + } +} +if (Symbol.dispose) Table.prototype[Symbol.dispose] = Table.prototype.free; + +const WriterPropertiesFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_writerproperties_free(ptr >>> 0, 1)); +/** + * Immutable struct to hold writing configuration for `writeParquet`. + * + * Use {@linkcode WriterPropertiesBuilder} to create a configuration, then call {@linkcode + * WriterPropertiesBuilder.build} to create an instance of `WriterProperties`. + */ +export class WriterProperties { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(WriterProperties.prototype); + obj.__wbg_ptr = ptr; + WriterPropertiesFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + WriterPropertiesFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_writerproperties_free(ptr, 0); + } +} +if (Symbol.dispose) WriterProperties.prototype[Symbol.dispose] = WriterProperties.prototype.free; + +const WriterPropertiesBuilderFinalization = (typeof FinalizationRegistry === 'undefined') + ? { register: () => {}, unregister: () => {} } + : new FinalizationRegistry(ptr => wasm.__wbg_writerpropertiesbuilder_free(ptr >>> 0, 1)); +/** + * Builder to create a writing configuration for `writeParquet` + * + * Call {@linkcode build} on the finished builder to create an immputable {@linkcode WriterProperties} to pass to `writeParquet` + */ +export class WriterPropertiesBuilder { + + static __wrap(ptr) { + ptr = ptr >>> 0; + const obj = Object.create(WriterPropertiesBuilder.prototype); + obj.__wbg_ptr = ptr; + WriterPropertiesBuilderFinalization.register(obj, obj.__wbg_ptr, obj); + return obj; + } + + __destroy_into_raw() { + const ptr = this.__wbg_ptr; + this.__wbg_ptr = 0; + WriterPropertiesBuilderFinalization.unregister(this); + return ptr; + } + + free() { + const ptr = this.__destroy_into_raw(); + wasm.__wbg_writerpropertiesbuilder_free(ptr, 0); + } + /** + * Returns default state of the builder. + */ + constructor() { + const ret = wasm.writerpropertiesbuilder_new(); + this.__wbg_ptr = ret >>> 0; + WriterPropertiesBuilderFinalization.register(this, this.__wbg_ptr, this); + return this; + } + /** + * Finalizes the configuration and returns immutable writer properties struct. + * @returns {WriterProperties} + */ + build() { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_build(ptr); + return WriterProperties.__wrap(ret); + } + /** + * Sets writer version. + * @param {WriterVersion} value + * @returns {WriterPropertiesBuilder} + */ + setWriterVersion(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setWriterVersion(ptr, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets data page size limit. + * @param {number} value + * @returns {WriterPropertiesBuilder} + */ + setDataPageSizeLimit(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setDataPageSizeLimit(ptr, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets dictionary page size limit. + * @param {number} value + * @returns {WriterPropertiesBuilder} + */ + setDictionaryPageSizeLimit(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setDictionaryPageSizeLimit(ptr, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets write batch size. + * @param {number} value + * @returns {WriterPropertiesBuilder} + */ + setWriteBatchSize(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setWriteBatchSize(ptr, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets maximum number of rows in a row group. + * @param {number} value + * @returns {WriterPropertiesBuilder} + */ + setMaxRowGroupSize(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setMaxRowGroupSize(ptr, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets "created by" property. + * @param {string} value + * @returns {WriterPropertiesBuilder} + */ + setCreatedBy(value) { + const ptr = this.__destroy_into_raw(); + const ptr0 = passStringToWasm0(value, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.writerpropertiesbuilder_setCreatedBy(ptr, ptr0, len0); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets "key_value_metadata" property. + * @param {KeyValueMetadata} value + * @returns {WriterPropertiesBuilder} + */ + setKeyValueMetadata(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setKeyValueMetadata(ptr, value); + if (ret[2]) { + throw takeFromExternrefTable0(ret[1]); + } + return WriterPropertiesBuilder.__wrap(ret[0]); + } + /** + * Sets encoding for any column. + * + * If dictionary is not enabled, this is treated as a primary encoding for all + * columns. In case when dictionary is enabled for any column, this value is + * considered to be a fallback encoding for that column. + * + * Panics if user tries to set dictionary encoding here, regardless of dictionary + * encoding flag being set. + * @param {Encoding} value + * @returns {WriterPropertiesBuilder} + */ + setEncoding(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setEncoding(ptr, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets compression codec for any column. + * @param {Compression} value + * @returns {WriterPropertiesBuilder} + */ + setCompression(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setCompression(ptr, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets flag to enable/disable dictionary encoding for any column. + * + * Use this method to set dictionary encoding, instead of explicitly specifying + * encoding in `set_encoding` method. + * @param {boolean} value + * @returns {WriterPropertiesBuilder} + */ + setDictionaryEnabled(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setDictionaryEnabled(ptr, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets flag to enable/disable statistics for any column. + * @param {EnabledStatistics} value + * @returns {WriterPropertiesBuilder} + */ + setStatisticsEnabled(value) { + const ptr = this.__destroy_into_raw(); + const ret = wasm.writerpropertiesbuilder_setStatisticsEnabled(ptr, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets encoding for a column. + * Takes precedence over globally defined settings. + * + * If dictionary is not enabled, this is treated as a primary encoding for this + * column. In case when dictionary is enabled for this column, either through + * global defaults or explicitly, this value is considered to be a fallback + * encoding for this column. + * + * Panics if user tries to set dictionary encoding here, regardless of dictionary + * encoding flag being set. + * @param {string} col + * @param {Encoding} value + * @returns {WriterPropertiesBuilder} + */ + setColumnEncoding(col, value) { + const ptr = this.__destroy_into_raw(); + const ptr0 = passStringToWasm0(col, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.writerpropertiesbuilder_setColumnEncoding(ptr, ptr0, len0, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets compression codec for a column. + * Takes precedence over globally defined settings. + * @param {string} col + * @param {Compression} value + * @returns {WriterPropertiesBuilder} + */ + setColumnCompression(col, value) { + const ptr = this.__destroy_into_raw(); + const ptr0 = passStringToWasm0(col, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.writerpropertiesbuilder_setColumnCompression(ptr, ptr0, len0, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets flag to enable/disable dictionary encoding for a column. + * Takes precedence over globally defined settings. + * @param {string} col + * @param {boolean} value + * @returns {WriterPropertiesBuilder} + */ + setColumnDictionaryEnabled(col, value) { + const ptr = this.__destroy_into_raw(); + const ptr0 = passStringToWasm0(col, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.writerpropertiesbuilder_setColumnDictionaryEnabled(ptr, ptr0, len0, value); + return WriterPropertiesBuilder.__wrap(ret); + } + /** + * Sets flag to enable/disable statistics for a column. + * Takes precedence over globally defined settings. + * @param {string} col + * @param {EnabledStatistics} value + * @returns {WriterPropertiesBuilder} + */ + setColumnStatisticsEnabled(col, value) { + const ptr = this.__destroy_into_raw(); + const ptr0 = passStringToWasm0(col, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len0 = WASM_VECTOR_LEN; + const ret = wasm.writerpropertiesbuilder_setColumnStatisticsEnabled(ptr, ptr0, len0, value); + return WriterPropertiesBuilder.__wrap(ret); + } +} +if (Symbol.dispose) WriterPropertiesBuilder.prototype[Symbol.dispose] = WriterPropertiesBuilder.prototype.free; + +const EXPECTED_RESPONSE_TYPES = new Set(['basic', 'cors', 'default']); + +async function __wbg_load(module, imports) { + if (typeof Response === 'function' && module instanceof Response) { + if (typeof WebAssembly.instantiateStreaming === 'function') { + try { + return await WebAssembly.instantiateStreaming(module, imports); + + } catch (e) { + const validResponse = module.ok && EXPECTED_RESPONSE_TYPES.has(module.type); + + if (validResponse && module.headers.get('Content-Type') !== 'application/wasm') { + console.warn("`WebAssembly.instantiateStreaming` failed because your server does not serve Wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n", e); + + } else { + throw e; + } + } + } + + const bytes = await module.arrayBuffer(); + return await WebAssembly.instantiate(bytes, imports); + + } else { + const instance = await WebAssembly.instantiate(module, imports); + + if (instance instanceof WebAssembly.Instance) { + return { instance, module }; + + } else { + return instance; + } + } +} + +function __wbg_get_imports() { + const imports = {}; + imports.wbg = {}; + imports.wbg.__wbg_Error_e17e777aac105295 = function(arg0, arg1) { + const ret = Error(getStringFromWasm0(arg0, arg1)); + return ret; + }; + imports.wbg.__wbg_Number_998bea33bd87c3e0 = function(arg0) { + const ret = Number(arg0); + return ret; + }; + imports.wbg.__wbg_String_8f0eb39a4a4c2f66 = function(arg0, arg1) { + const ret = String(arg1); + const ptr1 = passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len1 = WASM_VECTOR_LEN; + getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true); + getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true); + }; + imports.wbg.__wbg_abort_67e1b49bf6614565 = function(arg0) { + arg0.abort(); + }; + imports.wbg.__wbg_abort_d830bf2e9aa6ec5b = function(arg0, arg1) { + arg0.abort(arg1); + }; + imports.wbg.__wbg_append_72a3c0addd2bce38 = function() { return handleError(function (arg0, arg1, arg2, arg3, arg4) { + arg0.append(getStringFromWasm0(arg1, arg2), getStringFromWasm0(arg3, arg4)); + }, arguments) }; + imports.wbg.__wbg_arrayBuffer_2c907ed8e8ef4e35 = function(arg0) { + const ret = arg0.arrayBuffer(); + return ret; + }; + imports.wbg.__wbg_arrayBuffer_9c99b8e2809e8cbb = function() { return handleError(function (arg0) { + const ret = arg0.arrayBuffer(); + return ret; + }, arguments) }; + imports.wbg.__wbg_buffer_8d40b1d762fb3c66 = function(arg0) { + const ret = arg0.buffer; + return ret; + }; + imports.wbg.__wbg_byobRequest_2c036bceca1e6037 = function(arg0) { + const ret = arg0.byobRequest; + return isLikeNone(ret) ? 0 : addToExternrefTable0(ret); + }; + imports.wbg.__wbg_byteLength_331a6b5545834024 = function(arg0) { + const ret = arg0.byteLength; + return ret; + }; + imports.wbg.__wbg_byteOffset_49a5b5608000358b = function(arg0) { + const ret = arg0.byteOffset; + return ret; + }; + imports.wbg.__wbg_call_13410aac570ffff7 = function() { return handleError(function (arg0, arg1) { + const ret = arg0.call(arg1); + return ret; + }, arguments) }; + imports.wbg.__wbg_call_a5400b25a865cfd8 = function() { return handleError(function (arg0, arg1, arg2) { + const ret = arg0.call(arg1, arg2); + return ret; + }, arguments) }; + imports.wbg.__wbg_cancel_8bb5b8f4906b658a = function(arg0) { + const ret = arg0.cancel(); + return ret; + }; + imports.wbg.__wbg_catch_c80ecae90cb8ed4e = function(arg0, arg1) { + const ret = arg0.catch(arg1); + return ret; + }; + imports.wbg.__wbg_clearTimeout_6222fede17abcb1a = function(arg0) { + const ret = clearTimeout(arg0); + return ret; + }; + imports.wbg.__wbg_close_a1918cff3cac355b = function(arg0) { + const ret = arg0.close(); + return ret; + }; + imports.wbg.__wbg_close_cccada6053ee3a65 = function() { return handleError(function (arg0) { + arg0.close(); + }, arguments) }; + imports.wbg.__wbg_close_d71a78219dc23e91 = function() { return handleError(function (arg0) { + arg0.close(); + }, arguments) }; + imports.wbg.__wbg_columnchunkmetadata_new = function(arg0) { + const ret = ColumnChunkMetaData.__wrap(arg0); + return ret; + }; + imports.wbg.__wbg_done_75ed0ee6dd243d9d = function(arg0) { + const ret = arg0.done; + return ret; + }; + imports.wbg.__wbg_enqueue_452bc2343d1c2ff9 = function() { return handleError(function (arg0, arg1) { + arg0.enqueue(arg1); + }, arguments) }; + imports.wbg.__wbg_entries_2be2f15bd5554996 = function(arg0) { + const ret = Object.entries(arg0); + return ret; + }; + imports.wbg.__wbg_fetch_87aed7f306ec6d63 = function(arg0, arg1) { + const ret = arg0.fetch(arg1); + return ret; + }; + imports.wbg.__wbg_fetch_f156d10be9a5c88a = function(arg0) { + const ret = fetch(arg0); + return ret; + }; + imports.wbg.__wbg_getReader_48e00749fe3f6089 = function() { return handleError(function (arg0) { + const ret = arg0.getReader(); + return ret; + }, arguments) }; + imports.wbg.__wbg_getWriter_03d7689e275ac6a4 = function() { return handleError(function (arg0) { + const ret = arg0.getWriter(); + return ret; + }, arguments) }; + imports.wbg.__wbg_get_0da715ceaecea5c8 = function(arg0, arg1) { + const ret = arg0[arg1 >>> 0]; + return ret; + }; + imports.wbg.__wbg_get_458e874b43b18b25 = function() { return handleError(function (arg0, arg1) { + const ret = Reflect.get(arg0, arg1); + return ret; + }, arguments) }; + imports.wbg.__wbg_getdone_f026246f6bbe58d3 = function(arg0) { + const ret = arg0.done; + return isLikeNone(ret) ? 0xFFFFFF : ret ? 1 : 0; + }; + imports.wbg.__wbg_getvalue_31e5a08f61e5aa42 = function(arg0) { + const ret = arg0.value; + return ret; + }; + imports.wbg.__wbg_getwithrefkey_1dc361bd10053bfe = function(arg0, arg1) { + const ret = arg0[arg1]; + return ret; + }; + imports.wbg.__wbg_has_b89e451f638123e3 = function() { return handleError(function (arg0, arg1) { + const ret = Reflect.has(arg0, arg1); + return ret; + }, arguments) }; + imports.wbg.__wbg_headers_29fec3c72865cd75 = function(arg0) { + const ret = arg0.headers; + return ret; + }; + imports.wbg.__wbg_instanceof_ArrayBuffer_67f3012529f6a2dd = function(arg0) { + let result; + try { + result = arg0 instanceof ArrayBuffer; + } catch (_) { + result = false; + } + const ret = result; + return ret; + }; + imports.wbg.__wbg_instanceof_Response_50fde2cd696850bf = function(arg0) { + let result; + try { + result = arg0 instanceof Response; + } catch (_) { + result = false; + } + const ret = result; + return ret; + }; + imports.wbg.__wbg_instanceof_Uint8Array_9a8378d955933db7 = function(arg0) { + let result; + try { + result = arg0 instanceof Uint8Array; + } catch (_) { + result = false; + } + const ret = result; + return ret; + }; + imports.wbg.__wbg_isArray_030cce220591fb41 = function(arg0) { + const ret = Array.isArray(arg0); + return ret; + }; + imports.wbg.__wbg_isSafeInteger_1c0d1af5542e102a = function(arg0) { + const ret = Number.isSafeInteger(arg0); + return ret; + }; + imports.wbg.__wbg_iterator_f370b34483c71a1c = function() { + const ret = Symbol.iterator; + return ret; + }; + imports.wbg.__wbg_length_186546c51cd61acd = function(arg0) { + const ret = arg0.length; + return ret; + }; + imports.wbg.__wbg_length_6bb7e81f9d7713e4 = function(arg0) { + const ret = arg0.length; + return ret; + }; + imports.wbg.__wbg_new_19c25a3f2fa63a02 = function() { + const ret = new Object(); + return ret; + }; + imports.wbg.__wbg_new_2e3c58a15f39f5f9 = function(arg0, arg1) { + try { + var state0 = {a: arg0, b: arg1}; + var cb0 = (arg0, arg1) => { + const a = state0.a; + state0.a = 0; + try { + return __wbg_adapter_271(a, state0.b, arg0, arg1); + } finally { + state0.a = a; + } + }; + const ret = new Promise(cb0); + return ret; + } finally { + state0.a = state0.b = 0; + } + }; + imports.wbg.__wbg_new_2ff1f68f3676ea53 = function() { + const ret = new Map(); + return ret; + }; + imports.wbg.__wbg_new_638ebfaedbf32a5e = function(arg0) { + const ret = new Uint8Array(arg0); + return ret; + }; + imports.wbg.__wbg_new_66b9434b4e59b63e = function() { return handleError(function () { + const ret = new AbortController(); + return ret; + }, arguments) }; + imports.wbg.__wbg_new_99a6a948d5b3f607 = function() { return handleError(function () { + const ret = new TransformStream(); + return ret; + }, arguments) }; + imports.wbg.__wbg_new_da9dc54c5db29dfa = function(arg0, arg1) { + const ret = new Error(getStringFromWasm0(arg0, arg1)); + return ret; + }; + imports.wbg.__wbg_new_f6e53210afea8e45 = function() { return handleError(function () { + const ret = new Headers(); + return ret; + }, arguments) }; + imports.wbg.__wbg_newfromslice_074c56947bd43469 = function(arg0, arg1) { + const ret = new Uint8Array(getArrayU8FromWasm0(arg0, arg1)); + return ret; + }; + imports.wbg.__wbg_newnoargs_254190557c45b4ec = function(arg0, arg1) { + const ret = new Function(getStringFromWasm0(arg0, arg1)); + return ret; + }; + imports.wbg.__wbg_newwithbyteoffset_6bd4b2a4ca518883 = function(arg0, arg1) { + const ret = new Uint8Array(arg0, arg1 >>> 0); + return ret; + }; + imports.wbg.__wbg_newwithbyteoffsetandlength_e8f53910b4d42b45 = function(arg0, arg1, arg2) { + const ret = new Uint8Array(arg0, arg1 >>> 0, arg2 >>> 0); + return ret; + }; + imports.wbg.__wbg_newwithintounderlyingsource_b47f6a6a596a7f24 = function(arg0, arg1) { + const ret = new ReadableStream(IntoUnderlyingSource.__wrap(arg0), arg1); + return ret; + }; + imports.wbg.__wbg_newwithstrandinit_b5d168a29a3fd85f = function() { return handleError(function (arg0, arg1, arg2) { + const ret = new Request(getStringFromWasm0(arg0, arg1), arg2); + return ret; + }, arguments) }; + imports.wbg.__wbg_next_5b3530e612fde77d = function(arg0) { + const ret = arg0.next; + return ret; + }; + imports.wbg.__wbg_next_692e82279131b03c = function() { return handleError(function (arg0) { + const ret = arg0.next(); + return ret; + }, arguments) }; + imports.wbg.__wbg_parquetfile_new = function(arg0) { + const ret = ParquetFile.__wrap(arg0); + return ret; + }; + imports.wbg.__wbg_prototypesetcall_3d4a26c1ed734349 = function(arg0, arg1, arg2) { + Uint8Array.prototype.set.call(getArrayU8FromWasm0(arg0, arg1), arg2); + }; + imports.wbg.__wbg_queueMicrotask_25d0739ac89e8c88 = function(arg0) { + queueMicrotask(arg0); + }; + imports.wbg.__wbg_queueMicrotask_4488407636f5bf24 = function(arg0) { + const ret = arg0.queueMicrotask; + return ret; + }; + imports.wbg.__wbg_read_bc925c758aa4d897 = function(arg0) { + const ret = arg0.read(); + return ret; + }; + imports.wbg.__wbg_readable_e82cff27b968ed1c = function(arg0) { + const ret = arg0.readable; + return ret; + }; + imports.wbg.__wbg_ready_4186da3cb500ae7d = function(arg0) { + const ret = arg0.ready; + return ret; + }; + imports.wbg.__wbg_recordbatch_new = function(arg0) { + const ret = RecordBatch.__wrap(arg0); + return ret; + }; + imports.wbg.__wbg_recordbatch_unwrap = function(arg0) { + const ret = RecordBatch.__unwrap(arg0); + return ret; + }; + imports.wbg.__wbg_releaseLock_62151472ae632176 = function(arg0) { + arg0.releaseLock(); + }; + imports.wbg.__wbg_releaseLock_ff29b586502a8221 = function(arg0) { + arg0.releaseLock(); + }; + imports.wbg.__wbg_resolve_4055c623acdd6a1b = function(arg0) { + const ret = Promise.resolve(arg0); + return ret; + }; + imports.wbg.__wbg_respond_6c2c4e20ef85138e = function() { return handleError(function (arg0, arg1) { + arg0.respond(arg1 >>> 0); + }, arguments) }; + imports.wbg.__wbg_rowgroupmetadata_new = function(arg0) { + const ret = RowGroupMetaData.__wrap(arg0); + return ret; + }; + imports.wbg.__wbg_setTimeout_2b339866a2aa3789 = function(arg0, arg1) { + const ret = setTimeout(arg0, arg1); + return ret; + }; + imports.wbg.__wbg_set_1353b2a5e96bc48c = function(arg0, arg1, arg2) { + arg0.set(getArrayU8FromWasm0(arg1, arg2)); + }; + imports.wbg.__wbg_set_3f1d0b984ed272ed = function(arg0, arg1, arg2) { + arg0[arg1] = arg2; + }; + imports.wbg.__wbg_set_b7f1cf4fae26fe2a = function(arg0, arg1, arg2) { + const ret = arg0.set(arg1, arg2); + return ret; + }; + imports.wbg.__wbg_setbody_c8460bdf44147df8 = function(arg0, arg1) { + arg0.body = arg1; + }; + imports.wbg.__wbg_setcache_90ca4ad8a8ad40d3 = function(arg0, arg1) { + arg0.cache = __wbindgen_enum_RequestCache[arg1]; + }; + imports.wbg.__wbg_setcredentials_9cd60d632c9d5dfc = function(arg0, arg1) { + arg0.credentials = __wbindgen_enum_RequestCredentials[arg1]; + }; + imports.wbg.__wbg_setheaders_0052283e2f3503d1 = function(arg0, arg1) { + arg0.headers = arg1; + }; + imports.wbg.__wbg_sethighwatermark_3d5961f834647d41 = function(arg0, arg1) { + arg0.highWaterMark = arg1; + }; + imports.wbg.__wbg_setmethod_9b504d5b855b329c = function(arg0, arg1, arg2) { + arg0.method = getStringFromWasm0(arg1, arg2); + }; + imports.wbg.__wbg_setmode_a23e1a2ad8b512f8 = function(arg0, arg1) { + arg0.mode = __wbindgen_enum_RequestMode[arg1]; + }; + imports.wbg.__wbg_setsignal_8c45ad1247a74809 = function(arg0, arg1) { + arg0.signal = arg1; + }; + imports.wbg.__wbg_signal_da4d466ce86118b5 = function(arg0) { + const ret = arg0.signal; + return ret; + }; + imports.wbg.__wbg_size_8f84e7768fba0589 = function(arg0) { + const ret = arg0.size; + return ret; + }; + imports.wbg.__wbg_slice_224856d46230c13c = function() { return handleError(function (arg0, arg1, arg2) { + const ret = arg0.slice(arg1, arg2); + return ret; + }, arguments) }; + imports.wbg.__wbg_static_accessor_GLOBAL_8921f820c2ce3f12 = function() { + const ret = typeof global === 'undefined' ? null : global; + return isLikeNone(ret) ? 0 : addToExternrefTable0(ret); + }; + imports.wbg.__wbg_static_accessor_GLOBAL_THIS_f0a4409105898184 = function() { + const ret = typeof globalThis === 'undefined' ? null : globalThis; + return isLikeNone(ret) ? 0 : addToExternrefTable0(ret); + }; + imports.wbg.__wbg_static_accessor_SELF_995b214ae681ff99 = function() { + const ret = typeof self === 'undefined' ? null : self; + return isLikeNone(ret) ? 0 : addToExternrefTable0(ret); + }; + imports.wbg.__wbg_static_accessor_WINDOW_cde3890479c675ea = function() { + const ret = typeof window === 'undefined' ? null : window; + return isLikeNone(ret) ? 0 : addToExternrefTable0(ret); + }; + imports.wbg.__wbg_status_3fea3036088621d6 = function(arg0) { + const ret = arg0.status; + return ret; + }; + imports.wbg.__wbg_stringify_b98c93d0a190446a = function() { return handleError(function (arg0) { + const ret = JSON.stringify(arg0); + return ret; + }, arguments) }; + imports.wbg.__wbg_table_new = function(arg0) { + const ret = Table.__wrap(arg0); + return ret; + }; + imports.wbg.__wbg_then_b33a773d723afa3e = function(arg0, arg1, arg2) { + const ret = arg0.then(arg1, arg2); + return ret; + }; + imports.wbg.__wbg_then_e22500defe16819f = function(arg0, arg1) { + const ret = arg0.then(arg1); + return ret; + }; + imports.wbg.__wbg_toString_78df35411a4fd40c = function(arg0) { + const ret = arg0.toString(); + return ret; + }; + imports.wbg.__wbg_url_e5720dfacf77b05e = function(arg0, arg1) { + const ret = arg1.url; + const ptr1 = passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len1 = WASM_VECTOR_LEN; + getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true); + getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true); + }; + imports.wbg.__wbg_value_dd9372230531eade = function(arg0) { + const ret = arg0.value; + return ret; + }; + imports.wbg.__wbg_view_91cc97d57ab30530 = function(arg0) { + const ret = arg0.view; + return isLikeNone(ret) ? 0 : addToExternrefTable0(ret); + }; + imports.wbg.__wbg_wbindgenbigintgetasi64_ac743ece6ab9bba1 = function(arg0, arg1) { + const v = arg1; + const ret = typeof(v) === 'bigint' ? v : undefined; + getDataViewMemory0().setBigInt64(arg0 + 8 * 1, isLikeNone(ret) ? BigInt(0) : ret, true); + getDataViewMemory0().setInt32(arg0 + 4 * 0, !isLikeNone(ret), true); + }; + imports.wbg.__wbg_wbindgenbooleanget_3fe6f642c7d97746 = function(arg0) { + const v = arg0; + const ret = typeof(v) === 'boolean' ? v : undefined; + return isLikeNone(ret) ? 0xFFFFFF : ret ? 1 : 0; + }; + imports.wbg.__wbg_wbindgencbdrop_eb10308566512b88 = function(arg0) { + const obj = arg0.original; + if (obj.cnt-- == 1) { + obj.a = 0; + return true; + } + const ret = false; + return ret; + }; + imports.wbg.__wbg_wbindgendebugstring_99ef257a3ddda34d = function(arg0, arg1) { + const ret = debugString(arg1); + const ptr1 = passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + const len1 = WASM_VECTOR_LEN; + getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true); + getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true); + }; + imports.wbg.__wbg_wbindgenfunctiontable_aa1084b2969a9cbe = function() { + const ret = wasm.__wbindgen_export_5; + return ret; + }; + imports.wbg.__wbg_wbindgenin_d7a1ee10933d2d55 = function(arg0, arg1) { + const ret = arg0 in arg1; + return ret; + }; + imports.wbg.__wbg_wbindgenisbigint_ecb90cc08a5a9154 = function(arg0) { + const ret = typeof(arg0) === 'bigint'; + return ret; + }; + imports.wbg.__wbg_wbindgenisfunction_8cee7dce3725ae74 = function(arg0) { + const ret = typeof(arg0) === 'function'; + return ret; + }; + imports.wbg.__wbg_wbindgenisobject_307a53c6bd97fbf8 = function(arg0) { + const val = arg0; + const ret = typeof(val) === 'object' && val !== null; + return ret; + }; + imports.wbg.__wbg_wbindgenisstring_d4fa939789f003b0 = function(arg0) { + const ret = typeof(arg0) === 'string'; + return ret; + }; + imports.wbg.__wbg_wbindgenisundefined_c4b71d073b92f3c5 = function(arg0) { + const ret = arg0 === undefined; + return ret; + }; + imports.wbg.__wbg_wbindgenjsvaleq_e6f2ad59ccae1b58 = function(arg0, arg1) { + const ret = arg0 === arg1; + return ret; + }; + imports.wbg.__wbg_wbindgenjsvallooseeq_9bec8c9be826bed1 = function(arg0, arg1) { + const ret = arg0 == arg1; + return ret; + }; + imports.wbg.__wbg_wbindgenmemory_d84da70f7c42d172 = function() { + const ret = wasm.memory; + return ret; + }; + imports.wbg.__wbg_wbindgennumberget_f74b4c7525ac05cb = function(arg0, arg1) { + const obj = arg1; + const ret = typeof(obj) === 'number' ? obj : undefined; + getDataViewMemory0().setFloat64(arg0 + 8 * 1, isLikeNone(ret) ? 0 : ret, true); + getDataViewMemory0().setInt32(arg0 + 4 * 0, !isLikeNone(ret), true); + }; + imports.wbg.__wbg_wbindgenstringget_0f16a6ddddef376f = function(arg0, arg1) { + const obj = arg1; + const ret = typeof(obj) === 'string' ? obj : undefined; + var ptr1 = isLikeNone(ret) ? 0 : passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc); + var len1 = WASM_VECTOR_LEN; + getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true); + getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true); + }; + imports.wbg.__wbg_wbindgenthrow_451ec1a8469d7eb6 = function(arg0, arg1) { + throw new Error(getStringFromWasm0(arg0, arg1)); + }; + imports.wbg.__wbg_writable_e5202c9fd57615db = function(arg0) { + const ret = arg0.writable; + return ret; + }; + imports.wbg.__wbg_write_2e39e04a4c8c9e9d = function(arg0, arg1) { + const ret = arg0.write(arg1); + return ret; + }; + imports.wbg.__wbindgen_cast_2241b6af4c4b2941 = function(arg0, arg1) { + // Cast intrinsic for `Ref(String) -> Externref`. + const ret = getStringFromWasm0(arg0, arg1); + return ret; + }; + imports.wbg.__wbindgen_cast_4625c577ab2ec9ee = function(arg0) { + // Cast intrinsic for `U64 -> Externref`. + const ret = BigInt.asUintN(64, arg0); + return ret; + }; + imports.wbg.__wbindgen_cast_7cc3531aa3e3e0fe = function(arg0, arg1) { + // Cast intrinsic for `Closure(Closure { dtor_idx: 3803, function: Function { arguments: [Externref], shim_idx: 3814, ret: Unit, inner_ret: Some(Unit) }, mutable: true }) -> Externref`. + const ret = makeMutClosure(arg0, arg1, 3803, __wbg_adapter_11); + return ret; + }; + imports.wbg.__wbindgen_cast_d6cd19b81560fd6e = function(arg0) { + // Cast intrinsic for `F64 -> Externref`. + const ret = arg0; + return ret; + }; + imports.wbg.__wbindgen_cast_fc73a81ca98ca911 = function(arg0, arg1) { + // Cast intrinsic for `Closure(Closure { dtor_idx: 335, function: Function { arguments: [], shim_idx: 336, ret: Unit, inner_ret: Some(Unit) }, mutable: true }) -> Externref`. + const ret = makeMutClosure(arg0, arg1, 335, __wbg_adapter_6); + return ret; + }; + imports.wbg.__wbindgen_init_externref_table = function() { + const table = wasm.__wbindgen_export_4; + const offset = table.grow(4); + table.set(0, undefined); + table.set(offset + 0, undefined); + table.set(offset + 1, null); + table.set(offset + 2, true); + table.set(offset + 3, false); + ; + }; + + return imports; +} + +function __wbg_init_memory(imports, memory) { + +} + +function __wbg_finalize_init(instance, module) { + wasm = instance.exports; + __wbg_init.__wbindgen_wasm_module = module; + cachedDataViewMemory0 = null; + cachedUint32ArrayMemory0 = null; + cachedUint8ArrayMemory0 = null; + + + wasm.__wbindgen_start(); + return wasm; +} + +function initSync(module) { + if (wasm !== undefined) return wasm; + + + if (typeof module !== 'undefined') { + if (Object.getPrototypeOf(module) === Object.prototype) { + ({module} = module) + } else { + console.warn('using deprecated parameters for `initSync()`; pass a single object instead') + } + } + + const imports = __wbg_get_imports(); + + __wbg_init_memory(imports); + + if (!(module instanceof WebAssembly.Module)) { + module = new WebAssembly.Module(module); + } + + const instance = new WebAssembly.Instance(module, imports); + + return __wbg_finalize_init(instance, module); +} + +async function __wbg_init(module_or_path) { + if (wasm !== undefined) return wasm; + + + if (typeof module_or_path !== 'undefined') { + if (Object.getPrototypeOf(module_or_path) === Object.prototype) { + ({module_or_path} = module_or_path) + } else { + console.warn('using deprecated parameters for the initialization function; pass a single object instead') + } + } + + if (typeof module_or_path === 'undefined') { + module_or_path = new URL('parquet_wasm_bg.wasm', import.meta.url); + } + const imports = __wbg_get_imports(); + + if (typeof module_or_path === 'string' || (typeof Request === 'function' && module_or_path instanceof Request) || (typeof URL === 'function' && module_or_path instanceof URL)) { + module_or_path = fetch(module_or_path); + } + + __wbg_init_memory(imports); + + const { instance, module } = await __wbg_load(await module_or_path, imports); + + return __wbg_finalize_init(instance, module); +} + +export { initSync }; +export default __wbg_init; diff --git a/packages/core/vendor/parquet-wasm/parquet_wasm_bg.wasm b/packages/core/vendor/parquet-wasm/parquet_wasm_bg.wasm new file mode 100644 index 00000000..438b94b0 Binary files /dev/null and b/packages/core/vendor/parquet-wasm/parquet_wasm_bg.wasm differ diff --git a/packages/core/vite.config.ts b/packages/core/vite.config.ts index a661a4d7..ce221f35 100644 --- a/packages/core/vite.config.ts +++ b/packages/core/vite.config.ts @@ -1,20 +1,60 @@ +import { cpSync } from 'node:fs'; import { resolve } from 'node:path'; import { defineConfig } from 'vitest/config'; +const rollupExternals = new Set(['zarrita', 'zod', 'anndata.js', 'zarrextra', 'apache-arrow']); + export default defineConfig({ build: { lib: { - entry: resolve(__dirname, 'src/index.ts'), + entry: { + index: resolve(__dirname, 'src/index.ts'), + workers: resolve(__dirname, 'src/workers/index.ts'), + 'points-worker': resolve(__dirname, 'src/workers/points-worker.ts'), + }, name: 'SpatialDataCore', formats: ['es', 'cjs'], - fileName: (format) => `index.${format === 'es' ? 'js' : 'cjs'}`, + fileName: (format, entryName) => { + if (entryName === 'index') { + return `index.${format === 'es' ? 'js' : 'cjs'}`; + } + return `${entryName}.js`; + }, }, rollupOptions: { - external: ['zarrita', 'zod', 'anndata.js', 'parquet-wasm', 'zarrextra'], + external: (id) => { + const normalizedId = id.replace(/\\/g, '/'); + if (normalizedId.includes('vendor/parquet-wasm/parquet_wasm.js')) { + return true; + } + return rollupExternals.has(id); + }, }, sourcemap: true, target: 'es2020', }, + plugins: [ + { + name: 'externalize-vendored-parquet-wasm', + resolveId(source) { + const normalizedSource = source.replace(/\\/g, '/'); + if (normalizedSource.includes('vendor/parquet-wasm/parquet_wasm.js')) { + return { id: source, external: true }; + } + return null; + }, + }, + { + name: 'copy-vendored-parquet-wasm', + closeBundle() { + cpSync( + resolve(__dirname, 'vendor/parquet-wasm'), + resolve(__dirname, 'dist/vendor/parquet-wasm'), + { recursive: true } + ); + }, + }, + ], test: { globals: true, environment: 'node', diff --git a/packages/layers/package.json b/packages/layers/package.json index de0f939f..505835ac 100644 --- a/packages/layers/package.json +++ b/packages/layers/package.json @@ -29,6 +29,7 @@ "@deck.gl/core": "catalog:", "@hms-dbmi/viv": "catalog:", "@math.gl/core": "catalog:", + "@spatialdata/core": "workspace:*", "zod": "catalog:" }, "devDependencies": { diff --git a/packages/layers/src/PointsLayer.ts b/packages/layers/src/PointsLayer.ts new file mode 100644 index 00000000..fbd5d5d6 --- /dev/null +++ b/packages/layers/src/PointsLayer.ts @@ -0,0 +1,213 @@ +import type { Matrix4 } from '@math.gl/core'; +import type { UpdateParameters } from '@deck.gl/core'; +import { filterColumnarByFeatureCodesInWorker } from '@spatialdata/core'; +import { CompositeLayer } from 'deck.gl'; +import type { Layer, LayersList } from 'deck.gl'; +import type { PointsRenderResource } from './pointsLoader.js'; +import type { TileDebugStore } from './pointsTiledDebugHooks.js'; +import type { ColumnarNdarrayPointsBatch } from './pointsLoader.js'; +import { filterBatchSignature, featureFilterAwaitingRowCodes, hasPreloadedRowFeatureCodes } from './pointsFeatureCodes.js'; +import { resolvePointsRenderStrategy } from './pointsRenderStrategies.js'; +import { applyRenderCapToColumnar } from '@spatialdata/core'; +import { + DEFAULT_POINT_RADIUS_MAX_PIXELS, + DEFAULT_POINT_RADIUS_MIN_PIXELS, + DEFAULT_POINT_SIZE, +} from './pointsScatterLayer.js'; + +export interface PointsLayerProps { + id: string; + resource: PointsRenderResource; + visible?: boolean; + opacity?: number; + modelMatrix: Matrix4; + pointSize?: number; + pointRadiusMinPixels?: number; + pointRadiusMaxPixels?: number; + pointMinSizeScale?: number; + viewZoom?: number | null; + color?: [number, number, number, number]; + featureCodes?: readonly number[]; + /** Source-side integer codes aligned with the preloaded table rows. */ + preloadedFeatureCodes?: ArrayLike; + /** Max rows to draw after feature filtering. */ + renderCap?: number; + showTileDebugOverlay?: boolean; + tileDebugStore?: TileDebugStore; + /** Bumps when {@link tileDebugStore} contents change; forces debug overlay refresh. */ + tileDebugSignature?: string; + use3d?: boolean; +} + +interface PointsLayerState { + preloadedBatch?: ColumnarNdarrayPointsBatch; + filteredBatch?: ColumnarNdarrayPointsBatch; + filteredBatchSignature?: string; + filterGeneration?: number; +} + +function emptyFilteredBatch(batch: ColumnarNdarrayPointsBatch): ColumnarNdarrayPointsBatch { + const axisCount = batch.shape[0] ?? batch.data.length; + const empty = new Float32Array(0); + const emptyData = axisCount >= 3 && batch.data[2] ? [empty, empty, empty] : [empty, empty]; + return { + ...batch, + data: emptyData, + shape: [axisCount, 0], + pointCount: 0, + }; +} + +async function filterPreloadedBatch( + batch: ColumnarNdarrayPointsBatch, + featureCodes: readonly number[] | undefined, + preloadedFeatureCodes: ArrayLike | undefined +): Promise { + if (featureCodes === undefined) { + return batch; + } + if (featureCodes.length === 0) { + return emptyFilteredBatch(batch); + } + if (!hasPreloadedRowFeatureCodes(preloadedFeatureCodes)) { + return batch; + } + const filtered = await filterColumnarByFeatureCodesInWorker( + { shape: batch.shape, data: batch.data }, + featureCodes, + preloadedFeatureCodes ?? [] + ); + const filteredShape = filtered.shape ?? [filtered.data.length, filtered.data[0]?.length ?? 0]; + const pointCount = filteredShape[1] ?? filtered.data[0]?.length ?? 0; + return { + ...batch, + data: filtered.data, + shape: filteredShape, + pointCount, + }; +} + +export class PointsLayer extends CompositeLayer { + static layerName = 'PointsLayer'; + + static defaultProps = { + visible: true, + opacity: 1, + pointSize: DEFAULT_POINT_SIZE, + pointRadiusMinPixels: DEFAULT_POINT_RADIUS_MIN_PIXELS, + pointRadiusMaxPixels: DEFAULT_POINT_RADIUS_MAX_PIXELS, + showTileDebugOverlay: true, + } satisfies Partial; + + initializeState(): void { + this.state = { filterGeneration: 0 }; + void this.ensurePreloadedBatch(); + } + + updateState(params: UpdateParameters): void { + const { props, oldProps } = params; + if ( + props.resource.loader !== oldProps.resource?.loader || + props.resource.element !== oldProps.resource?.element + ) { + this.setState({ + preloadedBatch: undefined, + filteredBatch: undefined, + filteredBatchSignature: undefined, + filterGeneration: 0, + }); + void this.ensurePreloadedBatch(); + return; + } + + const signature = filterBatchSignature( + props.featureCodes, + props.preloadedFeatureCodes, + props.renderCap + ); + const state = this.state as PointsLayerState; + const preloadedBatch = state.preloadedBatch; + const awaitingRowCodes = featureFilterAwaitingRowCodes( + props.featureCodes, + props.preloadedFeatureCodes + ); + const canFilter = !awaitingRowCodes; + const rowCodesBecameReady = + hasPreloadedRowFeatureCodes(props.preloadedFeatureCodes) && + !hasPreloadedRowFeatureCodes(oldProps.preloadedFeatureCodes); + if ( + preloadedBatch && + canFilter && + (rowCodesBecameReady || + signature !== state.filteredBatchSignature || + !state.filteredBatch) + ) { + void this.ensureFilteredBatch(preloadedBatch, signature); + } + } + + private async ensurePreloadedBatch(): Promise { + const { resource } = this.props; + if (resource.loader.capabilities.kind !== 'preloaded-columnar') { + return; + } + const existing = (this.state as PointsLayerState).preloadedBatch; + if (existing) { + return; + } + const batch = await resource.loader.loadAll?.(); + if (batch?.format === 'columnar-ndarray') { + this.setState({ preloadedBatch: batch }); + const awaitingRowCodes = featureFilterAwaitingRowCodes( + this.props.featureCodes, + this.props.preloadedFeatureCodes + ); + if (!awaitingRowCodes) { + void this.ensureFilteredBatch( + batch, + filterBatchSignature( + this.props.featureCodes, + this.props.preloadedFeatureCodes, + this.props.renderCap + ) + ); + } + } + } + + private async ensureFilteredBatch( + batch: ColumnarNdarrayPointsBatch, + signature: string + ): Promise { + const generation = ((this.state as PointsLayerState).filterGeneration ?? 0) + 1; + this.setState({ filterGeneration: generation }); + const { featureCodes, preloadedFeatureCodes, renderCap } = this.props; + let filtered = await filterPreloadedBatch(batch, featureCodes, preloadedFeatureCodes); + filtered = applyRenderCapToColumnar(filtered, renderCap); + const state = this.state as PointsLayerState; + if (state.filterGeneration !== generation) { + return; + } + this.setState({ + filteredBatch: filtered, + filteredBatchSignature: signature, + }); + this.setNeedsUpdate(); + } + + /** Public wrapper for strategy modules outside this class. */ + subLayerProps

>(props: P & { id: string }): P { + return this.getSubLayerProps(props); + } + + renderLayers(): Layer | null | LayersList { + const { visible = true, resource } = this.props; + if (!visible || !resource?.loader) { + return null; + } + return resolvePointsRenderStrategy(resource.loader).renderLayers(this); + } +} + +export { filterPreloadedBatch }; +export { featureCodesSignature, filterBatchSignature } from './pointsFeatureCodes.js'; diff --git a/packages/layers/src/geoArrowStrategies.ts b/packages/layers/src/geoArrowStrategies.ts new file mode 100644 index 00000000..a2b68f97 --- /dev/null +++ b/packages/layers/src/geoArrowStrategies.ts @@ -0,0 +1,24 @@ +import type { Layer, LayersList } from 'deck.gl'; +import type { PointsLayer } from './PointsLayer.js'; +import type { PointsRenderStrategy } from './pointsRenderStrategies.js'; + +export const geoArrowBinaryStrategy: PointsRenderStrategy = { + renderLayers(): Layer | null | LayersList { + return null; + }, +}; + +export const geoArrowTiledStrategy: PointsRenderStrategy = { + renderLayers(): Layer | null | LayersList { + return null; + }, +}; + +export const unsupportedPointsStrategy: PointsRenderStrategy = { + renderLayers(layer): Layer | null | LayersList { + console.debug( + `[PointsLayer] Unsupported points encoding for element "${layer.props.resource.element.key}"` + ); + return null; + }, +}; diff --git a/packages/layers/src/index.ts b/packages/layers/src/index.ts index e7ed66ff..735e1804 100644 --- a/packages/layers/src/index.ts +++ b/packages/layers/src/index.ts @@ -68,3 +68,51 @@ export type { RenderStackSpatialElementType, RenderStackSpatialEntry, } from './renderStack'; +export { PointsLayer } from './PointsLayer'; +export type { PointsLayerProps } from './PointsLayer'; +export { + columnarBatchFromPointData, + pointDataFromColumnarBatch, + type ArrowRecordBatchPointsBatch, + type ColumnarNdarrayPointsBatch, + type PointData, + type PointsBatch, + type PointsBatchFormat, + type PointsEncodingKind, + type PointsLoadInBoundsOptions, + type PointsLoader, + type PointsLoaderCapabilities, + type PointsRenderResource, +} from './pointsLoader.js'; +export { + createPointsRenderResource, + coreLoaderToPointsLoader, +} from './pointsLoaderAdapter.js'; +export { + DEFAULT_POINT_RADIUS_MAX_PIXELS, + DEFAULT_POINT_RADIUS_MIN_PIXELS, + DEFAULT_POINT_SIZE, + MIN_POINT_SIZE_SCALE, + POINT_SIZE_ZOOM_REFERENCE, + zoomScaledPointSize, +} from './pointsScatterLayer.js'; +export type { PointsTileHandle, PointsTileLoadResult } from './pointsTileLoadCallbacks.js'; +export { + createTileDebugStore, + createTiledPointsDebugHooks, + type TileDebugStore, + type TiledPointsDebugState, +} from './pointsTiledDebugHooks.js'; +export { + POINTS_TILE_DEBUG_PICK_KIND, + formatPointsTileDebugTooltip, + isPointsTileDebugPickObject, + reduceTileDebugEntries, + tileDebugEntriesSignature, + tileDebugStatusFillColor, + tileDebugStatusLineColor, + type PointsTileDebugEntry, + type PointsTileDebugPickObject, + type PointsTileLoadProgress, + type PointsTileStatus, +} from './pointsTileDebug.js'; diff --git a/packages/layers/src/mortonTiledStrategy.ts b/packages/layers/src/mortonTiledStrategy.ts new file mode 100644 index 00000000..1c7dc687 --- /dev/null +++ b/packages/layers/src/mortonTiledStrategy.ts @@ -0,0 +1,255 @@ +import { COORDINATE_SYSTEM } from '@deck.gl/core'; +import { PolygonLayer, TileLayer } from 'deck.gl'; +import type { Layer, LayersList } from 'deck.gl'; +import type { PointsLayer } from './PointsLayer.js'; +import { + boundsFromTileBbox, + intersectBounds, + isPointTileBbox, + scatterBoundsFromTileBbox, + tileHandleFromDeckTile, +} from './pointsBbox.js'; +import type { ColumnarNdarrayPointsBatch } from './pointsLoader.js'; +import { + DEFAULT_POINT_RADIUS_MAX_PIXELS, + DEFAULT_POINT_RADIUS_MIN_PIXELS, + DEFAULT_POINT_SIZE, + renderColumnarScatterLayer, +} from './pointsScatterLayer.js'; +import type { PointsRenderStrategy } from './pointsRenderStrategies.js'; +import { featureCodesSignature } from './pointsFeatureCodes.js'; +import { createTiledPointsDebugHooks } from './pointsTiledDebugHooks.js'; +import { + POINTS_TILE_DEBUG_PICK_KIND, + pointsTileDebugPolygonData, + tileDebugStatusFillColor, + tileDebugStatusLineColor, +} from './pointsTileDebug.js'; + +function isAbortError(error: unknown) { + return error instanceof DOMException && error.name === 'AbortError'; +} + +function isColumnarBatch(value: unknown): value is ColumnarNdarrayPointsBatch { + return ( + !!value && + typeof value === 'object' && + (value as ColumnarNdarrayPointsBatch).format === 'columnar-ndarray' + ); +} + +function renderedPointCount(batch: ColumnarNdarrayPointsBatch): number { + if (batch.pointCount !== undefined) { + return batch.pointCount; + } + if (batch.shape.length >= 2 && Number.isFinite(batch.shape[1])) { + return batch.shape[1]; + } + return batch.data[0]?.length ?? 0; +} + +export const mortonTiledStrategy: PointsRenderStrategy = { + renderLayers(layer: PointsLayer): Layer | null | LayersList { + const { + resource, + featureCodes, + showTileDebugOverlay, + opacity = 1, + visible = true, + pointSize = DEFAULT_POINT_SIZE, + pointRadiusMinPixels, + pointRadiusMaxPixels, + color = [255, 100, 100, 200], + use3d, + } = layer.props; + + const localBounds = resource.loader.capabilities.bounds; + if (!localBounds) { + return null; + } + + const debugHooks = createTiledPointsDebugHooks(layer.props.tileDebugStore); + const scatterStyleProps = { + color, + pointSize, + pointRadiusMinPixels, + pointRadiusMaxPixels, + opacity, + modelMatrix: layer.props.modelMatrix, + use3d, + }; + + const layers: LayersList = [ + new TileLayer( + layer.subLayerProps({ + id: 'tiles', + coordinateSystem: COORDINATE_SYSTEM.CARTESIAN, + modelMatrix: layer.props.modelMatrix, + extent: [localBounds.minX, localBounds.minY, localBounds.maxX, localBounds.maxY], + opacity, + visible, + tileSize: 512, + minZoom: -1, + maxZoom: -1, + refinementStrategy: 'best-available', + updateTriggers: { + getTileData: [resource.element.key, featureCodesSignature(featureCodes)], + renderSubLayers: [ + pointSize, + pointRadiusMinPixels, + pointRadiusMaxPixels, + color, + opacity, + layer.props.modelMatrix, + use3d, + ], + }, + onViewportLoad( + tiles: Array<{ + index?: { x: number; y: number; z: number }; + id?: string; + bbox?: unknown; + }> | null + ) { + const handles = (tiles ?? []) + .map( + (tile: { + index?: { x: number; y: number; z: number }; + id?: string; + bbox?: unknown; + }) => tileHandleFromDeckTile(tile) + ) + .filter( + (handle): handle is NonNullable> => + handle != null + ); + debugHooks.onViewportTilesRequested(handles); + }, + async getTileData(tileProps: { + index?: { x: number; y: number; z: number }; + id?: string; + bbox?: unknown; + signal?: AbortSignal; + }) { + const tile = tileHandleFromDeckTile(tileProps); + if (!tile || !isPointTileBbox(tileProps.bbox)) { + return null; + } + debugHooks.onTileLoadStart(tile); + const rawBounds = boundsFromTileBbox(tile.bbox); + const bounds = intersectBounds(rawBounds, localBounds); + if (!bounds) { + debugHooks.onTileLoadEnd( + tile, + { success: true, clippedBounds: null, pointCount: 0, loadMode: 'clipped' }, + rawBounds + ); + return null; + } + try { + const batch = await resource.loader.loadInBounds({ + bounds, + featureCodes, + signal: tileProps.signal, + }); + if (!batch || !isColumnarBatch(batch)) { + debugHooks.onTileLoadEnd( + tile, + { success: true, clippedBounds: bounds, pointCount: 0 }, + rawBounds + ); + return null; + } + debugHooks.onTileLoadEnd( + tile, + { + success: true, + clippedBounds: bounds, + pointCount: renderedPointCount(batch), + loadMode: batch.loadMode, + }, + rawBounds + ); + return batch; + } catch (error) { + const aborted = Boolean(tileProps.signal?.aborted) || isAbortError(error); + debugHooks.onTileLoadEnd( + tile, + { + success: false, + aborted, + clippedBounds: bounds, + errorMessage: aborted ? 'aborted' : String(error), + }, + rawBounds + ); + if (aborted) { + return null; + } + throw error; + } + }, + renderSubLayers: (props: { + id: string; + data?: ColumnarNdarrayPointsBatch | null; + tile?: { bbox?: unknown }; + }) => { + if (!props.data || !isColumnarBatch(props.data)) { + return null; + } + const tileBbox = isPointTileBbox(props.tile?.bbox) ? props.tile.bbox : null; + return renderColumnarScatterLayer(`${props.id}-scatter`, props.data, { + ...scatterStyleProps, + tileBounds: tileBbox ? scatterBoundsFromTileBbox(tileBbox) : undefined, + tileSubLayer: true, + }); + }, + }) + ), + ]; + + if (showTileDebugOverlay) { + const entries = debugHooks.getTileDebugEntries(); + const debugSignature = layer.props.tileDebugSignature ?? debugHooks.getTileDebugSignature(); + const polygonData = pointsTileDebugPolygonData(entries).map(({ polygon, entry }) => ({ + polygon, + entry, + kind: POINTS_TILE_DEBUG_PICK_KIND as typeof POINTS_TILE_DEBUG_PICK_KIND, + })); + layers.push( + new PolygonLayer( + layer.subLayerProps({ + id: 'tile-debug', + coordinateSystem: COORDINATE_SYSTEM.CARTESIAN, + modelMatrix: layer.props.modelMatrix, + data: polygonData, + pickable: true, + autoHighlight: true, + highlightColor: [255, 255, 255, 120], + getPolygon: (d: { polygon: [number, number][] }) => d.polygon, + getFillColor: (d: { + entry: { status: import('./pointsTileDebug.js').PointsTileStatus }; + }) => tileDebugStatusFillColor(d.entry.status), + getLineColor: (d: { + entry: { status: import('./pointsTileDebug.js').PointsTileStatus }; + }) => tileDebugStatusLineColor(d.entry.status), + getLineWidth: 2, + lineWidthUnits: 'pixels', + filled: true, + stroked: true, + opacity: Math.min(1, opacity + 0.15), + visible, + updateTriggers: { + data: [debugSignature], + getFillColor: [debugSignature], + getLineColor: [debugSignature], + getPolygon: [debugSignature], + }, + }) + ) + ); + } + + return layers; + }, +}; diff --git a/packages/layers/src/pointsBbox.ts b/packages/layers/src/pointsBbox.ts new file mode 100644 index 00000000..3cad54ff --- /dev/null +++ b/packages/layers/src/pointsBbox.ts @@ -0,0 +1,67 @@ +import type { SpatialBounds } from '@spatialdata/core'; +import type { PointsTileHandle } from './pointsTileLoadCallbacks.js'; + +export type PointTileBbox = { + left: number; + right: number; + top: number; + bottom: number; +}; + +export function isPointTileBbox(value: unknown): value is PointTileBbox { + if (!value || typeof value !== 'object') { + return false; + } + const candidate = value as Record; + return ( + typeof candidate.left === 'number' && + typeof candidate.right === 'number' && + typeof candidate.top === 'number' && + typeof candidate.bottom === 'number' + ); +} + +export function intersectBounds( + query: SpatialBounds, + clip: SpatialBounds +): SpatialBounds | null { + const minX = Math.max(query.minX, clip.minX); + const maxX = Math.min(query.maxX, clip.maxX); + const minY = Math.max(query.minY, clip.minY); + const maxY = Math.min(query.maxY, clip.maxY); + if (minX > maxX || minY > maxY) { + return null; + } + return { minX, minY, maxX, maxY }; +} + +export function boundsFromTileBbox(bbox: PointTileBbox): SpatialBounds { + return { + minX: Math.min(bbox.left, bbox.right), + maxX: Math.max(bbox.left, bbox.right), + minY: Math.min(bbox.top, bbox.bottom), + maxY: Math.max(bbox.top, bbox.bottom), + }; +} + +export function scatterBoundsFromTileBbox( + bbox: PointTileBbox +): [number, number, number, number] { + return [bbox.left, bbox.top, bbox.right, bbox.bottom]; +} + +export function tileHandleFromDeckTile(tile: { + index?: { x: number; y: number; z: number }; + id?: string; + bbox?: unknown; +}): PointsTileHandle | null { + if (!tile.index || !isPointTileBbox(tile.bbox)) { + return null; + } + const { x, y, z } = tile.index; + return { + tileId: tile.id ?? `${x}-${y}-${z}`, + index: { x, y, z }, + bbox: tile.bbox, + }; +} diff --git a/packages/layers/src/pointsFeatureCodes.ts b/packages/layers/src/pointsFeatureCodes.ts new file mode 100644 index 00000000..32acefb5 --- /dev/null +++ b/packages/layers/src/pointsFeatureCodes.ts @@ -0,0 +1,49 @@ +export function hasPreloadedRowFeatureCodes( + preloadedFeatureCodes: ArrayLike | undefined +): boolean { + return preloadedFeatureCodes !== undefined && preloadedFeatureCodes.length > 0; +} + +export function featureFilterAwaitingRowCodes( + featureCodes: readonly number[] | undefined, + preloadedFeatureCodes: ArrayLike | undefined +): boolean { + return ( + featureCodes !== undefined && + featureCodes.length > 0 && + !hasPreloadedRowFeatureCodes(preloadedFeatureCodes) + ); +} + +export function featureCodesSignature(featureCodes: readonly number[] | undefined): string { + if (featureCodes === undefined) { + return 'all'; + } + if (featureCodes.length === 0) { + return 'none'; + } + return featureCodes.slice().sort((left, right) => left - right).join(','); +} + +export function preloadedFeatureCodesSignature( + featureCodes: ArrayLike | undefined +): string { + if (!featureCodes) { + return 'nocodes'; + } + const length = featureCodes.length; + if (length === 0) { + return 'len:0'; + } + return `len:${length}:${featureCodes[0]}:${featureCodes[length - 1]}`; +} + +export function filterBatchSignature( + featureCodes: readonly number[] | undefined, + preloadedFeatureCodes: ArrayLike | undefined, + renderCap?: number +): string { + const renderPart = + renderCap === undefined ? 'default' : renderCap <= 0 ? 'none' : String(renderCap); + return `${featureCodesSignature(featureCodes)}|${preloadedFeatureCodesSignature(preloadedFeatureCodes)}|r:${renderPart}`; +} diff --git a/packages/layers/src/pointsLoader.ts b/packages/layers/src/pointsLoader.ts new file mode 100644 index 00000000..16b0edb9 --- /dev/null +++ b/packages/layers/src/pointsLoader.ts @@ -0,0 +1,92 @@ +import type { SpatialBounds, PointsElement, PointsLoadMode } from '@spatialdata/core'; + +export type PointsEncodingKind = + | 'preloaded-columnar' + | 'morton-tiled' + | 'geoarrow-binary' + | 'geoarrow-tiled'; + +export type PointsBatchFormat = 'columnar-ndarray' | 'arrow-record-batch'; + +export interface PointsLoaderCapabilities { + kind: PointsEncodingKind; + batchFormat: PointsBatchFormat; + bounds?: SpatialBounds; + supportsViewportTiles: boolean; + supportsFeatureCodes?: boolean; +} + +export interface ColumnarNdarrayPointsBatch { + format: 'columnar-ndarray'; + data: ArrayLike[]; + shape: number[]; + bounds?: SpatialBounds; + loadMode?: PointsLoadMode; + pointCount?: number; +} + +/** Placeholder for future GeoArrow strategies. */ +export interface ArrowRecordBatchPointsBatch { + format: 'arrow-record-batch'; + batch: unknown; + bounds?: SpatialBounds; + loadMode?: string; + pointCount?: number; +} + +export type PointsBatch = ColumnarNdarrayPointsBatch | ArrowRecordBatchPointsBatch; + +export interface PointsLoadInBoundsOptions { + bounds: SpatialBounds; + featureCodes?: readonly number[]; + signal?: AbortSignal; +} + +export interface PointsLoader { + readonly capabilities: PointsLoaderCapabilities; + loadInBounds(options: PointsLoadInBoundsOptions): Promise; + loadAll?(options?: { signal?: AbortSignal }): Promise; +} + +export interface PointsRenderResource { + element: PointsElement; + loader: PointsLoader; +} + +export interface PointData { + shape: number[]; + data: ArrayLike[]; + featureCodes?: ArrayLike; + /** Full dataset row count when preload was truncated. */ + totalRowCount?: number; + preloadTruncated?: boolean; + /** Rows scanned when loading with an active feature filter. */ + scannedRowCount?: number; + /** Data was loaded with a source-side feature filter. */ + filterActive?: boolean; +} + +export function columnarBatchFromPointData( + data: PointData, + options?: { loadMode?: PointsLoadMode; bounds?: SpatialBounds } +): ColumnarNdarrayPointsBatch { + const pointCount = + data.shape.length >= 2 && Number.isFinite(data.shape[1]) + ? data.shape[1] + : (data.data[0]?.length ?? data.shape[0] ?? 0); + return { + format: 'columnar-ndarray', + data: data.data, + shape: data.shape, + bounds: options?.bounds, + loadMode: options?.loadMode, + pointCount, + }; +} + +export function pointDataFromColumnarBatch(batch: ColumnarNdarrayPointsBatch): PointData { + return { + data: batch.data, + shape: batch.shape, + }; +} diff --git a/packages/layers/src/pointsLoaderAdapter.ts b/packages/layers/src/pointsLoaderAdapter.ts new file mode 100644 index 00000000..d4b9a238 --- /dev/null +++ b/packages/layers/src/pointsLoaderAdapter.ts @@ -0,0 +1,46 @@ +import type { PointsElement } from '@spatialdata/core'; +import type { + PointsBatch, + PointsLoader, + PointsLoaderCapabilities, + PointsLoadInBoundsOptions, + PointsRenderResource, +} from './pointsLoader.js'; + +type CorePointsLoader = { + readonly capabilities: PointsLoaderCapabilities; + loadInBounds(options: PointsLoadInBoundsOptions): Promise; + loadAll?(options?: { signal?: AbortSignal }): Promise; +}; + +export type { + ArrowRecordBatchPointsBatch, + ColumnarNdarrayPointsBatch, + PointData, + PointsBatch, + PointsBatchFormat, + PointsEncodingKind, + PointsLoadInBoundsOptions, + PointsLoader, + PointsLoaderCapabilities, + PointsRenderResource, +} from './pointsLoader.js'; + +export { + columnarBatchFromPointData, + pointDataFromColumnarBatch, +} from './pointsLoader.js'; + +export function coreLoaderToPointsLoader(loader: CorePointsLoader): PointsLoader { + return loader; +} + +export function createPointsRenderResource( + element: PointsElement, + loader: CorePointsLoader +): PointsRenderResource { + return { + element, + loader: coreLoaderToPointsLoader(loader), + }; +} diff --git a/packages/layers/src/pointsRenderStrategies.ts b/packages/layers/src/pointsRenderStrategies.ts new file mode 100644 index 00000000..0dba7a61 --- /dev/null +++ b/packages/layers/src/pointsRenderStrategies.ts @@ -0,0 +1,21 @@ +import type { Layer, LayersList } from 'deck.gl'; +import type { PointsEncodingKind, PointsLoader } from './pointsLoader.js'; +import type { PointsLayer } from './PointsLayer.js'; +import { geoArrowBinaryStrategy, geoArrowTiledStrategy, unsupportedPointsStrategy } from './geoArrowStrategies.js'; +import { mortonTiledStrategy } from './mortonTiledStrategy.js'; +import { preloadedScatterStrategy } from './preloadedScatterStrategy.js'; + +export interface PointsRenderStrategy { + renderLayers(layer: PointsLayer): Layer | null | LayersList; +} + +const STRATEGIES: Record = { + 'preloaded-columnar': preloadedScatterStrategy, + 'morton-tiled': mortonTiledStrategy, + 'geoarrow-binary': geoArrowBinaryStrategy, + 'geoarrow-tiled': geoArrowTiledStrategy, +}; + +export function resolvePointsRenderStrategy(loader: PointsLoader): PointsRenderStrategy { + return STRATEGIES[loader.capabilities.kind] ?? unsupportedPointsStrategy; +} diff --git a/packages/layers/src/pointsScatterLayer.ts b/packages/layers/src/pointsScatterLayer.ts new file mode 100644 index 00000000..3c85e35c --- /dev/null +++ b/packages/layers/src/pointsScatterLayer.ts @@ -0,0 +1,95 @@ +import type { Matrix4 } from '@math.gl/core'; +import { COORDINATE_SYSTEM } from '@deck.gl/core'; +import { ScatterplotLayer } from 'deck.gl'; +import type { ColumnarNdarrayPointsBatch } from './pointsLoader.js'; +import { pointDataFromColumnarBatch } from './pointsLoader.js'; + +/** Orthographic zoom at which configured pointSize applies at full scale. */ +export const POINT_SIZE_ZOOM_REFERENCE = 0; +/** Minimum radius multiplier when zoomed out (reduces fragment overdraw). */ +export const MIN_POINT_SIZE_SCALE = 0.15; +export const DEFAULT_POINT_SIZE = 0.1; +export const DEFAULT_POINT_RADIUS_MIN_PIXELS = 0.1; +export const DEFAULT_POINT_RADIUS_MAX_PIXELS = 3; + +export function zoomScaledPointSize( + pointSize: number, + zoom: number | null | undefined, + zoomReference = POINT_SIZE_ZOOM_REFERENCE, + minScale = MIN_POINT_SIZE_SCALE +): number { + if (zoom === null || zoom === undefined || !Number.isFinite(zoom)) { + return pointSize; + } + const scale = 2 ** (zoom - zoomReference); + return pointSize * Math.min(1, Math.max(minScale, scale)); +} + +export interface PointsScatterStyleProps { + color: [number, number, number, number]; + pointSize: number; + pointRadiusMinPixels?: number; + pointRadiusMaxPixels?: number; + pointMinSizeScale?: number; + viewZoom?: number | null; + opacity: number; + modelMatrix: Matrix4; + use3d?: boolean; + tileBounds?: [number, number, number, number]; + tileSubLayer?: boolean; +} + +export function renderColumnarScatterLayer( + id: string, + batch: ColumnarNdarrayPointsBatch, + props: PointsScatterStyleProps +) { + const pointData = pointDataFromColumnarBatch(batch); + const d = pointData.data; + const effectivePointSize = props.tileSubLayer + ? props.pointSize + : zoomScaledPointSize( + props.pointSize, + props.viewZoom, + POINT_SIZE_ZOOM_REFERENCE, + props.pointMinSizeScale ?? MIN_POINT_SIZE_SCALE + ); + + const pointCount = batch.pointCount ?? batch.shape[1] ?? d[0]?.length ?? 0; + + return new ScatterplotLayer({ + id, + coordinateSystem: COORDINATE_SYSTEM.CARTESIAN, + data: d[0], + ...(props.tileBounds ? { bounds: props.tileBounds } : {}), + getPosition: (_d, { index, target }) => [ + d[0][index], + d[1][index], + props.use3d ? d[2]?.[index] || 0 : 0, + ], + getRadius: effectivePointSize, + ...(props.tileSubLayer + ? { + radiusMinPixels: props.pointRadiusMinPixels ?? DEFAULT_POINT_RADIUS_MIN_PIXELS, + radiusMaxPixels: props.pointRadiusMaxPixels ?? DEFAULT_POINT_RADIUS_MAX_PIXELS, + } + : {}), + radiusUnits: 'pixels', + getFillColor: props.color, + opacity: props.opacity, + modelMatrix: props.modelMatrix, + pickable: true, + autoHighlight: true, + highlightColor: [255, 255, 0, 200], + updateTriggers: { + getPosition: [pointCount, d[0], d[1], d[2]], + getRadius: [ + props.pointSize, + props.viewZoom, + props.pointRadiusMinPixels, + props.pointRadiusMaxPixels, + props.pointMinSizeScale, + ], + }, + }); +} diff --git a/packages/layers/src/pointsTileDebug.ts b/packages/layers/src/pointsTileDebug.ts new file mode 100644 index 00000000..62a45bb5 --- /dev/null +++ b/packages/layers/src/pointsTileDebug.ts @@ -0,0 +1,368 @@ +import type { SpatialBounds } from '@spatialdata/core'; +import type { PointsTileHandle, PointsTileLoadResult } from './pointsTileLoadCallbacks.js'; + +export type PointsTileStatus = + | 'pending' + | 'loading' + | 'loaded' + | 'empty' + | 'error' + | 'aborted'; + +export interface PointsTileLoadProgress { + inFlight: number; + loaded: number; + loadedPoints: number; + viewportTotal: number; +} + +export interface PointsTileDebugEntry { + tileId: string; + index: { x: number; y: number; z: number }; + bbox: SpatialBounds; + clippedBounds: SpatialBounds | null; + status: PointsTileStatus; + requestedAt: number; + startedAt?: number; + completedAt?: number; + pointCount?: number; + loadMode?: string; + errorMessage?: string; +} + +export const POINTS_TILE_DEBUG_PICK_KIND = 'spatialdata-points-tile-debug' as const; + +export interface PointsTileDebugPickObject { + kind: typeof POINTS_TILE_DEBUG_PICK_KIND; + entry: PointsTileDebugEntry; +} + +export function isPointsTileDebugPickObject( + value: unknown +): value is PointsTileDebugPickObject { + if (!value || typeof value !== 'object') { + return false; + } + const candidate = value as Partial; + return candidate.kind === POINTS_TILE_DEBUG_PICK_KIND && candidate.entry != null; +} + +export interface PointsTileCompletedSnapshot { + status: PointsTileStatus; + pointCount?: number; + loadMode?: string; + clippedBounds: SpatialBounds | null; + errorMessage?: string; + startedAt?: number; + completedAt: number; +} + +export interface PointsTileDebugViewportContext { + loadingTileIds: ReadonlySet; + completedTilesById: ReadonlyMap; + tileHandlesById: ReadonlyMap; +} + +export type PointsTileDebugEvent = + | { + type: 'viewport'; + tiles: readonly PointsTileHandle[]; + at: number; + context: PointsTileDebugViewportContext; + } + | { type: 'start'; tile: PointsTileHandle; at: number } + | { + type: 'end'; + tile: PointsTileHandle; + result: PointsTileLoadResult; + at: number; + clipBounds: SpatialBounds; + }; + +export function completedSnapshotFromLoadResult( + result: PointsTileLoadResult, + clipBounds: SpatialBounds, + completedAt: number, + startedAt?: number +): PointsTileCompletedSnapshot { + let status: PointsTileStatus = 'error'; + if (result.aborted) { + status = 'aborted'; + } else if (result.success) { + status = (result.pointCount ?? 0) > 0 ? 'loaded' : 'empty'; + } + + return { + status, + pointCount: result.pointCount, + loadMode: result.loadMode, + clippedBounds: result.clippedBounds ?? clipBounds, + errorMessage: result.errorMessage, + startedAt, + completedAt, + }; +} + +function resolveViewportTileStatus( + tileId: string, + existing: PointsTileDebugEntry | undefined, + context: PointsTileDebugViewportContext +): PointsTileStatus { + if (context.loadingTileIds.has(tileId)) { + return 'loading'; + } + const completed = context.completedTilesById.get(tileId); + if (completed) { + return completed.status; + } + if (existing?.status === 'loaded' || existing?.status === 'empty') { + return existing.status; + } + return 'pending'; +} + +function applyCompletedSnapshot( + entry: PointsTileDebugEntry, + completed: PointsTileCompletedSnapshot +): PointsTileDebugEntry { + return { + ...entry, + status: completed.status, + clippedBounds: completed.clippedBounds, + pointCount: completed.pointCount, + loadMode: completed.loadMode, + errorMessage: completed.errorMessage, + startedAt: completed.startedAt ?? entry.startedAt, + completedAt: completed.completedAt, + }; +} + +export function reduceTileDebugEntries( + previous: readonly PointsTileDebugEntry[], + event: PointsTileDebugEvent +): PointsTileDebugEntry[] { + const byId = new Map(previous.map((entry) => [entry.tileId, entry])); + + if (event.type === 'viewport') { + const tileHandlesById = new Map(event.context.tileHandlesById); + for (const tile of event.tiles) { + tileHandlesById.set(tile.tileId, tile); + } + const activeTileIds = new Set([ + ...event.tiles.map((tile) => tile.tileId), + ...event.context.loadingTileIds, + ...event.context.completedTilesById.keys(), + ]); + const next = new Map(); + for (const tileId of activeTileIds) { + const tile = tileHandlesById.get(tileId); + if (!tile) { + continue; + } + const rawBounds = boundsFromHandle(tile); + const existing = byId.get(tile.tileId); + const completed = event.context.completedTilesById.get(tile.tileId); + const status = resolveViewportTileStatus(tile.tileId, existing, event.context); + let entry: PointsTileDebugEntry = { + tileId: tile.tileId, + index: tile.index, + bbox: rawBounds, + clippedBounds: existing?.clippedBounds ?? completed?.clippedBounds ?? null, + status, + requestedAt: existing?.requestedAt ?? event.at, + startedAt: existing?.startedAt ?? completed?.startedAt, + completedAt: existing?.completedAt ?? completed?.completedAt, + pointCount: existing?.pointCount ?? completed?.pointCount, + loadMode: existing?.loadMode ?? completed?.loadMode, + errorMessage: existing?.errorMessage ?? completed?.errorMessage, + }; + if (completed && (status === 'loaded' || status === 'empty' || status === 'error' || status === 'aborted')) { + entry = applyCompletedSnapshot(entry, completed); + } + next.set(tile.tileId, entry); + } + return [...next.values()]; + } + + if (event.type === 'start') { + const rawBounds = boundsFromHandle(event.tile); + const existing = byId.get(event.tile.tileId); + byId.set(event.tile.tileId, { + tileId: event.tile.tileId, + index: event.tile.index, + bbox: rawBounds, + clippedBounds: existing?.clippedBounds ?? null, + status: 'loading', + requestedAt: existing?.requestedAt ?? event.at, + startedAt: event.at, + completedAt: undefined, + pointCount: undefined, + loadMode: undefined, + errorMessage: undefined, + }); + return [...byId.values()]; + } + + const rawBounds = boundsFromHandle(event.tile); + const { result } = event; + const snapshot = completedSnapshotFromLoadResult( + result, + event.clipBounds, + event.at, + byId.get(event.tile.tileId)?.startedAt ?? event.at + ); + + byId.set(event.tile.tileId, { + tileId: event.tile.tileId, + index: event.tile.index, + bbox: rawBounds, + clippedBounds: snapshot.clippedBounds, + status: snapshot.status, + requestedAt: byId.get(event.tile.tileId)?.requestedAt ?? event.at, + startedAt: snapshot.startedAt, + completedAt: snapshot.completedAt, + pointCount: snapshot.pointCount, + loadMode: snapshot.loadMode, + errorMessage: snapshot.errorMessage, + }); + return [...byId.values()]; +} + +function boundsFromHandle(tile: PointsTileHandle): SpatialBounds { + const { bbox } = tile; + return { + minX: Math.min(bbox.left, bbox.right), + maxX: Math.max(bbox.left, bbox.right), + minY: Math.min(bbox.top, bbox.bottom), + maxY: Math.max(bbox.top, bbox.bottom), + }; +} + +export interface PointsTileDebugPolygonDatum { + polygon: [number, number][]; + entry: PointsTileDebugEntry; +} + +export function pointsTileDebugPolygonData( + entries: readonly PointsTileDebugEntry[] +): PointsTileDebugPolygonDatum[] { + return entries.map((entry) => { + const bounds = entry.clippedBounds ?? entry.bbox; + const { minX, minY, maxX, maxY } = bounds; + return { + entry, + polygon: [ + [minX, minY], + [maxX, minY], + [maxX, maxY], + [minX, maxY], + ], + }; + }); +} + +function formatBounds(bounds: SpatialBounds): string { + return `[${bounds.minX.toFixed(1)}, ${bounds.minY.toFixed(1)}]–[${bounds.maxX.toFixed(1)}, ${bounds.maxY.toFixed(1)}]`; +} + +function formatDuration(ms: number): string { + if (ms < 1000) { + return `${Math.round(ms)}ms`; + } + return `${(ms / 1000).toFixed(2)}s`; +} + +export function formatPointsTileDebugTooltip( + entry: PointsTileDebugEntry, + batchProgress: PointsTileLoadProgress, + now = Date.now() +): { title: string; items: Array<{ label: string; value: string }> } { + const items: Array<{ label: string; value: string }> = [ + { label: 'tile', value: entry.tileId }, + { label: 'status', value: entry.status }, + { + label: 'batch', + value: + batchProgress.loadedPoints > 0 + ? `${batchProgress.loaded}/${batchProgress.viewportTotal} (${batchProgress.inFlight} in flight, ${batchProgress.loadedPoints.toLocaleString()} points)` + : `${batchProgress.loaded}/${batchProgress.viewportTotal} (${batchProgress.inFlight} in flight)`, + }, + { label: 'index', value: `x=${entry.index.x} y=${entry.index.y} z=${entry.index.z}` }, + { label: 'bbox', value: formatBounds(entry.bbox) }, + ]; + + if (entry.clippedBounds) { + items.push({ label: 'clipped', value: formatBounds(entry.clippedBounds) }); + } + if (entry.startedAt !== undefined) { + if (entry.completedAt !== undefined) { + items.push({ + label: 'duration', + value: formatDuration(entry.completedAt - entry.startedAt), + }); + } else { + items.push({ + label: 'elapsed', + value: formatDuration(now - entry.startedAt), + }); + } + } + if (entry.pointCount !== undefined) { + items.push({ label: 'points', value: String(entry.pointCount) }); + } + if (entry.loadMode) { + items.push({ label: 'load mode', value: entry.loadMode }); + } + if (entry.errorMessage) { + items.push({ label: 'error', value: entry.errorMessage }); + } + + return { + title: `Tile ${entry.tileId}`, + items, + }; +} + +export function tileDebugStatusFillColor( + status: PointsTileStatus +): [number, number, number, number] { + switch (status) { + case 'pending': + return [120, 120, 120, 30]; + case 'loading': + return [255, 180, 0, 80]; + case 'loaded': + return [80, 200, 80, 25]; + case 'empty': + return [120, 160, 200, 35]; + case 'error': + return [220, 60, 60, 70]; + case 'aborted': + return [180, 80, 80, 45]; + } +} + +export function tileDebugStatusLineColor( + status: PointsTileStatus +): [number, number, number, number] { + switch (status) { + case 'pending': + return [180, 180, 180, 180]; + case 'loading': + return [255, 200, 0, 255]; + case 'loaded': + return [80, 220, 80, 220]; + case 'empty': + return [140, 180, 220, 220]; + case 'error': + return [255, 80, 80, 255]; + case 'aborted': + return [220, 120, 120, 220]; + } +} + +export function tileDebugEntriesSignature(entries: readonly PointsTileDebugEntry[]): string { + return entries + .map((entry) => `${entry.tileId}:${entry.status}:${entry.pointCount ?? ''}`) + .join('|'); +} diff --git a/packages/layers/src/pointsTileLoadCallbacks.ts b/packages/layers/src/pointsTileLoadCallbacks.ts new file mode 100644 index 00000000..43b93fd0 --- /dev/null +++ b/packages/layers/src/pointsTileLoadCallbacks.ts @@ -0,0 +1,17 @@ +import type { SpatialBounds, PointsLoadMode } from '@spatialdata/core'; +import type { PointTileBbox } from './pointsBbox.js'; + +export interface PointsTileHandle { + tileId: string; + index: { x: number; y: number; z: number }; + bbox: PointTileBbox; +} + +export interface PointsTileLoadResult { + success: boolean; + aborted?: boolean; + clippedBounds?: SpatialBounds | null; + pointCount?: number; + loadMode?: PointsLoadMode; + errorMessage?: string; +} diff --git a/packages/layers/src/pointsTiledDebugHooks.ts b/packages/layers/src/pointsTiledDebugHooks.ts new file mode 100644 index 00000000..8daf978f --- /dev/null +++ b/packages/layers/src/pointsTiledDebugHooks.ts @@ -0,0 +1,179 @@ +import type { PointsTileHandle, PointsTileLoadResult } from './pointsTileLoadCallbacks.js'; +import { + completedSnapshotFromLoadResult, + reduceTileDebugEntries, + tileDebugEntriesSignature, + type PointsTileCompletedSnapshot, + type PointsTileDebugEntry, +} from './pointsTileDebug.js'; + +export interface TiledPointsDebugState { + tileDebugEntries: PointsTileDebugEntry[]; + completedTilesById?: Record; + loadingTileIds?: string[]; + lastViewportTiles?: readonly PointsTileHandle[]; + tileHandlesById?: Record; +} + +function rememberTileHandle( + state: TiledPointsDebugState, + tile: PointsTileHandle +): Record { + return { ...(state.tileHandlesById ?? {}), [tile.tileId]: tile }; +} + +function rebuildActiveDebugEntries( + entries: readonly PointsTileDebugEntry[], + state: TiledPointsDebugState, + at: number +): PointsTileDebugEntry[] { + return reduceTileDebugEntries(entries, { + type: 'viewport', + tiles: state.lastViewportTiles ?? [], + at, + context: { + loadingTileIds: new Set(state.loadingTileIds ?? []), + completedTilesById: new Map(Object.entries(state.completedTilesById ?? {})), + tileHandlesById: new Map(Object.entries(state.tileHandlesById ?? {})), + }, + }); +} + +export interface TileDebugStore { + getState(): TiledPointsDebugState; + update(updater: (state: TiledPointsDebugState) => TiledPointsDebugState): void; +} + +function emptyDebugState(): TiledPointsDebugState { + return { tileDebugEntries: [], completedTilesById: {}, loadingTileIds: [], tileHandlesById: {} }; +} + +function debugStateSignature(state: TiledPointsDebugState): string { + const completedKeys = Object.keys(state.completedTilesById ?? {}) + .sort() + .join(','); + const loadingKeys = [...(state.loadingTileIds ?? [])].sort().join(','); + const handleKeys = Object.keys(state.tileHandlesById ?? {}) + .sort() + .join(','); + return `${tileDebugEntriesSignature(state.tileDebugEntries)}|${loadingKeys}|${completedKeys}|${handleKeys}`; +} + +export function createTileDebugStore(onChange?: () => void): TileDebugStore { + let state = emptyDebugState(); + return { + getState() { + return state; + }, + update(updater) { + const next = updater(state); + if (debugStateSignature(state) === debugStateSignature(next)) { + return; + } + state = next; + onChange?.(); + }, + }; +} + +export function createTiledPointsDebugHooks(store: TileDebugStore | undefined) { + if (!store) { + return { + onViewportTilesRequested(_tiles: readonly PointsTileHandle[]) {}, + onTileLoadStart(_tile: PointsTileHandle) {}, + onTileLoadEnd( + _tile: PointsTileHandle, + _result: PointsTileLoadResult, + _clipBounds: { minX: number; minY: number; maxX: number; maxY: number } + ) {}, + getTileDebugEntries(): PointsTileDebugEntry[] { + return []; + }, + getTileDebugSignature(): string { + return ''; + }, + }; + } + + return { + onViewportTilesRequested(tiles: readonly PointsTileHandle[]) { + store.update((state) => { + const at = Date.now(); + const tileHandlesById = { ...(state.tileHandlesById ?? {}) }; + for (const tile of tiles) { + tileHandlesById[tile.tileId] = tile; + } + const nextState: TiledPointsDebugState = { + ...state, + lastViewportTiles: tiles, + tileHandlesById, + }; + return { + ...nextState, + tileDebugEntries: rebuildActiveDebugEntries(state.tileDebugEntries, nextState, at), + }; + }); + }, + onTileLoadStart(tile: PointsTileHandle) { + store.update((state) => { + const at = Date.now(); + const nextState: TiledPointsDebugState = { + ...state, + tileHandlesById: rememberTileHandle(state, tile), + loadingTileIds: [...new Set([...(state.loadingTileIds ?? []), tile.tileId])], + completedTilesById: Object.fromEntries( + Object.entries(state.completedTilesById ?? {}).filter( + ([tileId]) => tileId !== tile.tileId + ) + ), + }; + const afterStart = reduceTileDebugEntries(state.tileDebugEntries, { + type: 'start', + tile, + at, + }); + return { + ...nextState, + tileDebugEntries: rebuildActiveDebugEntries(afterStart, nextState, at), + }; + }); + }, + onTileLoadEnd( + tile: PointsTileHandle, + result: PointsTileLoadResult, + clipBounds: { minX: number; minY: number; maxX: number; maxY: number } + ) { + const at = Date.now(); + store.update((state) => { + const loadingTileIds = (state.loadingTileIds ?? []).filter( + (tileId) => tileId !== tile.tileId + ); + const completedTilesById = { ...(state.completedTilesById ?? {}) }; + const startedAt = + state.tileDebugEntries.find((entry) => entry.tileId === tile.tileId)?.startedAt ?? at; + completedTilesById[tile.tileId] = completedSnapshotFromLoadResult( + result, + clipBounds, + at, + startedAt + ); + const nextState: TiledPointsDebugState = { + ...state, + tileHandlesById: rememberTileHandle(state, tile), + loadingTileIds, + completedTilesById, + }; + return { + ...nextState, + tileDebugEntries: rebuildActiveDebugEntries(state.tileDebugEntries, nextState, at), + }; + }); + }, + getTileDebugEntries(): PointsTileDebugEntry[] { + return store.getState().tileDebugEntries; + }, + getTileDebugSignature(): string { + return tileDebugEntriesSignature(store.getState().tileDebugEntries); + }, + }; +} diff --git a/packages/layers/src/preloadedScatterStrategy.ts b/packages/layers/src/preloadedScatterStrategy.ts new file mode 100644 index 00000000..8fa746bb --- /dev/null +++ b/packages/layers/src/preloadedScatterStrategy.ts @@ -0,0 +1,72 @@ +import { applyRenderCapToColumnar } from '@spatialdata/core'; +import type { Layer, LayersList } from 'deck.gl'; +import type { PointsLayer } from './PointsLayer.js'; +import type { PointsRenderStrategy } from './pointsRenderStrategies.js'; +import { featureFilterAwaitingRowCodes, filterBatchSignature } from './pointsFeatureCodes.js'; +import { + DEFAULT_POINT_SIZE, + renderColumnarScatterLayer, +} from './pointsScatterLayer.js'; +import type { ColumnarNdarrayPointsBatch } from './pointsLoader.js'; + +function resolveScatterBatch(layer: PointsLayer): ColumnarNdarrayPointsBatch | undefined { + const { featureCodes, preloadedFeatureCodes, renderCap } = layer.props; + const state = layer.state as { + preloadedBatch?: ColumnarNdarrayPointsBatch; + filteredBatch?: ColumnarNdarrayPointsBatch; + filteredBatchSignature?: string; + }; + const signature = filterBatchSignature(featureCodes, preloadedFeatureCodes, renderCap); + const awaitingRowCodes = featureFilterAwaitingRowCodes(featureCodes, preloadedFeatureCodes); + if (awaitingRowCodes) { + if (!state.preloadedBatch) { + return undefined; + } + return applyRenderCapToColumnar(state.preloadedBatch, renderCap); + } + if (state.filteredBatch && state.filteredBatchSignature === signature) { + return state.filteredBatch; + } + if (!state.preloadedBatch) { + return undefined; + } + return applyRenderCapToColumnar(state.preloadedBatch, renderCap); +} + +export const preloadedScatterStrategy: PointsRenderStrategy = { + renderLayers(layer): Layer | null | LayersList { + const { + resource, + opacity = 1, + visible = true, + pointSize = DEFAULT_POINT_SIZE, + pointRadiusMinPixels, + pointRadiusMaxPixels, + pointMinSizeScale, + viewZoom, + color = [255, 100, 100, 200], + use3d, + } = layer.props; + + if (!visible) { + return null; + } + + const batch = resolveScatterBatch(layer); + if (!batch) { + return null; + } + + return renderColumnarScatterLayer(layer.props.id, batch, { + color, + pointSize, + pointRadiusMinPixels, + pointRadiusMaxPixels, + pointMinSizeScale, + viewZoom, + opacity, + modelMatrix: layer.props.modelMatrix, + use3d, + }); + }, +}; diff --git a/packages/layers/tests/pointsLayerFilter.spec.ts b/packages/layers/tests/pointsLayerFilter.spec.ts new file mode 100644 index 00000000..12fbd074 --- /dev/null +++ b/packages/layers/tests/pointsLayerFilter.spec.ts @@ -0,0 +1,60 @@ +import { describe, expect, it } from 'vitest'; +import { + featureCodesSignature, + featureFilterAwaitingRowCodes, + hasPreloadedRowFeatureCodes, +} from '../src/pointsFeatureCodes.js'; +import { filterPreloadedBatch } from '../src/PointsLayer.js'; +import type { ColumnarNdarrayPointsBatch } from '../src/pointsLoader.js'; + +describe('PointsLayer preloaded filtering', () => { + const batch: ColumnarNdarrayPointsBatch = { + format: 'columnar-ndarray', + shape: [2, 4], + data: [Float32Array.from([0, 1, 2, 3]), Float32Array.from([10, 11, 12, 13])], + pointCount: 4, + }; + + it('builds stable feature code signatures', () => { + expect(featureCodesSignature(undefined)).toBe('all'); + expect(featureCodesSignature([])).toBe('none'); + expect(featureCodesSignature([2, 0, 1])).toBe('0,1,2'); + }); + + it('keeps the preloaded batch visible while row feature codes are still loading', async () => { + const filtered = await filterPreloadedBatch(batch, [1], undefined); + expect(filtered.pointCount).toBe(4); + expect(filtered.data[0].length).toBe(4); + }); + + it('treats empty row feature code arrays as still loading', async () => { + expect(hasPreloadedRowFeatureCodes(undefined)).toBe(false); + expect(hasPreloadedRowFeatureCodes(new Int32Array(0))).toBe(false); + expect(featureFilterAwaitingRowCodes([1], new Int32Array(0))).toBe(true); + + const filtered = await filterPreloadedBatch(batch, [1], new Int32Array(0)); + expect(filtered.pointCount).toBe(4); + expect(filtered.data[0].length).toBe(4); + }); + + it('returns an empty batch when all features are deselected without row codes', async () => { + const filtered = await filterPreloadedBatch(batch, [], undefined); + expect(filtered.pointCount).toBe(0); + expect(filtered.data[0].length).toBe(0); + }); + + it('returns an empty batch when all features are deselected', async () => { + const sourceFeatureCodes = Int32Array.from([0, 1, 0, 2]); + const filtered = await filterPreloadedBatch(batch, [], sourceFeatureCodes); + expect(filtered.pointCount).toBe(0); + expect(filtered.data[0].length).toBe(0); + }); + + it('filters preloaded batches by feature codes', async () => { + const sourceFeatureCodes = Int32Array.from([0, 1, 0, 2]); + const filtered = await filterPreloadedBatch(batch, [1], sourceFeatureCodes); + expect(filtered.pointCount).toBe(1); + expect(filtered.data[0][0]).toBe(1); + expect(filtered.data[1][0]).toBe(11); + }); +}); diff --git a/packages/layers/tests/pointsRenderStrategies.spec.ts b/packages/layers/tests/pointsRenderStrategies.spec.ts new file mode 100644 index 00000000..16a7b1bb --- /dev/null +++ b/packages/layers/tests/pointsRenderStrategies.spec.ts @@ -0,0 +1,35 @@ +import { describe, expect, it } from 'vitest'; + +import { resolvePointsRenderStrategy } from '../src/pointsRenderStrategies.js'; + +describe('resolvePointsRenderStrategy', () => { + it('selects morton and preloaded strategies by encoding kind', () => { + expect( + resolvePointsRenderStrategy({ + capabilities: { + kind: 'morton-tiled', + batchFormat: 'columnar-ndarray', + supportsViewportTiles: true, + }, + loadInBounds: async () => null, + }).renderLayers + ).toBeTypeOf('function'); + + expect( + resolvePointsRenderStrategy({ + capabilities: { + kind: 'preloaded-columnar', + batchFormat: 'columnar-ndarray', + supportsViewportTiles: false, + }, + loadAll: async () => ({ + format: 'columnar-ndarray', + data: [[0], [0]], + shape: [1], + pointCount: 1, + }), + loadInBounds: async () => null, + }).renderLayers + ).toBeTypeOf('function'); + }); +}); diff --git a/packages/layers/tests/pointsTileDebug.spec.ts b/packages/layers/tests/pointsTileDebug.spec.ts new file mode 100644 index 00000000..874cd90d --- /dev/null +++ b/packages/layers/tests/pointsTileDebug.spec.ts @@ -0,0 +1,139 @@ +import { describe, expect, it } from 'vitest'; + +import { + completedSnapshotFromLoadResult, + formatPointsTileDebugTooltip, + reduceTileDebugEntries, +} from '../src/pointsTileDebug.js'; + +const sampleTile = { + tileId: '1-2--1', + index: { x: 1, y: 2, z: -1 }, + bbox: { left: 512, top: 1024, right: 1024, bottom: 512 }, +}; + +const emptyViewportContext = { + loadingTileIds: new Set(), + completedTilesById: new Map(), + tileHandlesById: new Map(), +}; + +const sampleTile2 = { + tileId: '3-4--1', + index: { x: 3, y: 4, z: -1 }, + bbox: { left: 1536, top: 2048, right: 2048, bottom: 1536 }, +}; + +describe('pointsTileDebug', () => { + it('transitions tile status through viewport, start, and end events', () => { + const at = 1_000; + let entries = reduceTileDebugEntries([], { + type: 'viewport', + tiles: [sampleTile], + at, + context: { + ...emptyViewportContext, + tileHandlesById: new Map([[sampleTile.tileId, sampleTile]]), + }, + }); + expect(entries[0]?.status).toBe('pending'); + + entries = reduceTileDebugEntries(entries, { type: 'start', tile: sampleTile, at: at + 10 }); + expect(entries[0]?.status).toBe('loading'); + expect(entries[0]?.startedAt).toBe(at + 10); + + entries = reduceTileDebugEntries(entries, { + type: 'end', + tile: sampleTile, + at: at + 100, + clipBounds: { minX: 512, minY: 512, maxX: 1024, maxY: 1024 }, + result: { success: true, pointCount: 42, loadMode: 'row-groups' }, + }); + expect(entries[0]?.status).toBe('loaded'); + expect(entries[0]?.pointCount).toBe(42); + expect(entries[0]?.completedAt).toBe(at + 100); + }); + + it('restores completed tiles after they re-enter the viewport', () => { + const at = 1_000; + const completedTilesById = new Map([ + [ + sampleTile.tileId, + completedSnapshotFromLoadResult( + { success: true, pointCount: 42, loadMode: 'row-groups' }, + { minX: 512, minY: 512, maxX: 1024, maxY: 1024 }, + at + 100, + at + 10 + ), + ], + ]); + + const entries = reduceTileDebugEntries([], { + type: 'viewport', + tiles: [sampleTile], + at: at + 200, + context: { + loadingTileIds: new Set(), + completedTilesById, + tileHandlesById: new Map([[sampleTile.tileId, sampleTile]]), + }, + }); + + expect(entries[0]?.status).toBe('loaded'); + expect(entries[0]?.pointCount).toBe(42); + }); + + it('includes loading and completed tiles not reported in the latest viewport event', () => { + const at = 1_000; + const completedTilesById = new Map([ + [ + sampleTile2.tileId, + completedSnapshotFromLoadResult( + { success: true, pointCount: 99, loadMode: 'row-groups' }, + { minX: 1536, minY: 1536, maxX: 2048, maxY: 2048 }, + at + 50, + at + 10 + ), + ], + ]); + + const entries = reduceTileDebugEntries([], { + type: 'viewport', + tiles: [sampleTile], + at: at + 100, + context: { + loadingTileIds: new Set([sampleTile.tileId]), + completedTilesById, + tileHandlesById: new Map([ + [sampleTile.tileId, sampleTile], + [sampleTile2.tileId, sampleTile2], + ]), + }, + }); + + expect(entries.map((entry) => entry.tileId).sort()).toEqual( + [sampleTile.tileId, sampleTile2.tileId].sort() + ); + expect(entries.find((entry) => entry.tileId === sampleTile.tileId)?.status).toBe('loading'); + expect(entries.find((entry) => entry.tileId === sampleTile2.tileId)?.pointCount).toBe(99); + }); + + it('formats tooltip with elapsed time for in-flight tiles', () => { + const tooltip = formatPointsTileDebugTooltip( + { + tileId: sampleTile.tileId, + index: sampleTile.index, + bbox: { minX: 512, minY: 512, maxX: 1024, maxY: 1024 }, + clippedBounds: null, + status: 'loading', + requestedAt: 1_000, + startedAt: 1_500, + }, + { inFlight: 1, loaded: 0, loadedPoints: 0, viewportTotal: 3 }, + 2_000 + ); + expect(tooltip.items.some((item) => item.label === 'elapsed' && item.value === '500ms')).toBe( + true + ); + }); +}); diff --git a/packages/layers/vite.config.ts b/packages/layers/vite.config.ts index 2cdfa994..5ebfce03 100644 --- a/packages/layers/vite.config.ts +++ b/packages/layers/vite.config.ts @@ -26,7 +26,14 @@ export default defineConfig({ formats: ['es'], }, rollupOptions: { - external: ['@deck.gl/core', '@hms-dbmi/viv', '@math.gl/core', 'deck.gl', 'zod'], + external: [ + '@deck.gl/core', + '@hms-dbmi/viv', + '@math.gl/core', + '@spatialdata/core', + 'deck.gl', + 'zod', + ], }, }, test: { diff --git a/packages/vis/demo/src/enableDemoPointsWorker.ts b/packages/vis/demo/src/enableDemoPointsWorker.ts new file mode 100644 index 00000000..72c5f495 --- /dev/null +++ b/packages/vis/demo/src/enableDemoPointsWorker.ts @@ -0,0 +1,13 @@ +import { enablePointsWorker } from '@spatialdata/core'; + +let enabled = false; + +/** Enable points worker decode/filter once for all vis demo routes. */ +export function ensureDemoPointsWorker() { + if (enabled || typeof Worker === 'undefined') { + return; + } + + enablePointsWorker(); + enabled = true; +} diff --git a/packages/vis/demo/src/main.tsx b/packages/vis/demo/src/main.tsx index 56fd1f67..4ca939ba 100644 --- a/packages/vis/demo/src/main.tsx +++ b/packages/vis/demo/src/main.tsx @@ -2,9 +2,11 @@ import React from 'react'; import ReactDOM from 'react-dom/client'; import App from './App'; import { ensureDemoWorkerChunkDecode } from './enableDemoWorkerChunkDecode'; +import { ensureDemoPointsWorker } from './enableDemoPointsWorker'; import './index.css'; ensureDemoWorkerChunkDecode(); +ensureDemoPointsWorker(); const root = document.getElementById('root'); if (!root) { diff --git a/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx b/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx new file mode 100644 index 00000000..2ab6e3fa --- /dev/null +++ b/packages/vis/src/SpatialCanvas/PointsFeatureFilterPanel.tsx @@ -0,0 +1,246 @@ +import type { CSSProperties } from 'react'; +import { useMemo, useState } from 'react'; +import type { PointsFeatureCatalog } from '@spatialdata/core'; +import type { PointsLayerConfig } from './types'; + +const panelStyle: CSSProperties = { + display: 'flex', + flexDirection: 'column', + gap: 6, + color: '#ccc', + fontSize: '12px', +}; + +const listStyle: CSSProperties = { + display: 'flex', + flexDirection: 'column', + gap: 4, + maxHeight: 180, + overflowY: 'auto', + padding: '4px 0', +}; + +const checkboxLabelStyle: CSSProperties = { + display: 'flex', + alignItems: 'center', + gap: 6, +}; + +const helperStyle: CSSProperties = { + color: '#888', + fontSize: '11px', +}; + +const countStyle: CSSProperties = { + color: '#888', + fontSize: '11px', + marginLeft: 'auto', + flexShrink: 0, +}; + +const searchStyle: CSSProperties = { + color: '#ccc', + fontSize: '12px', + padding: '4px 6px', + borderRadius: 4, + border: '1px solid #444', + background: '#1a1a1a', +}; + +const buttonStyle: CSSProperties = { + alignSelf: 'flex-start', + color: '#ddd', + fontSize: '12px', + padding: '4px 8px', + borderRadius: 4, + border: '1px solid #555', + background: '#2a2a2a', + cursor: 'pointer', +}; + +const FEATURE_LIST_SEARCH_THRESHOLD = 100; + +function formatFeatureCount(count: number | undefined): string { + if (count === undefined) { + return '—'; + } + return count.toLocaleString(); +} + +export interface PointsFeatureFilterPanelProps { + layerId: string; + config: PointsLayerConfig; + catalog?: PointsFeatureCatalog | null; + catalogLoading?: boolean; + onRequestCatalog: (layerId: string) => void; + updateLayer: (id: string, updates: Partial) => void; +} + +export function PointsFeatureFilterPanel({ + layerId, + config, + catalog, + catalogLoading = false, + onRequestCatalog, + updateLayer, +}: PointsFeatureFilterPanelProps) { + const [searchQuery, setSearchQuery] = useState(''); + const entries = catalog?.entries ?? []; + const hasCounts = entries.some((entry) => entry.count !== undefined); + const allSelected = config.featureCodes === undefined; + const noneSelected = config.featureCodes !== undefined && config.featureCodes.length === 0; + const selectedCodes = allSelected + ? new Set(entries.map((entry) => entry.code)) + : new Set(config.featureCodes ?? []); + + const sortedEntries = useMemo(() => { + const list = [...entries]; + if (hasCounts) { + list.sort((left, right) => { + const countDiff = (right.count ?? -1) - (left.count ?? -1); + if (countDiff !== 0) { + return countDiff; + } + return left.name.localeCompare(right.name); + }); + } else { + list.sort((left, right) => left.name.localeCompare(right.name)); + } + return list; + }, [entries, hasCounts]); + + const visibleEntries = useMemo(() => { + const query = searchQuery.trim().toLowerCase(); + if (!query) { + return sortedEntries; + } + return sortedEntries.filter((entry) => entry.name.toLowerCase().includes(query)); + }, [sortedEntries, searchQuery]); + + const setFeatureCodes = (nextCodes: number[] | undefined) => { + updateLayer(layerId, { featureCodes: nextCodes }); + }; + + const toggleFeature = (code: number, checked: boolean) => { + const current = new Set( + allSelected ? entries.map((entry) => entry.code) : (config.featureCodes ?? []) + ); + if (checked) { + current.add(code); + } else { + current.delete(code); + } + if (current.size === 0) { + setFeatureCodes([]); + return; + } + if (current.size === entries.length) { + setFeatureCodes(undefined); + return; + } + setFeatureCodes([...current].sort((left, right) => left - right)); + }; + + if (catalogLoading) { + return ( +

+
Loading features…
+
+ ); + } + + if (catalog === undefined) { + return ( +
+
Feature list not loaded.
+ +
+ ); + } + + if (!catalog || entries.length === 0) { + return ( +
+
+ {catalog === null + ? 'No feature catalog available for this points layer (missing feature_key or unsupported encoding for this dataset size).' + : 'No features found in the feature catalog.'} +
+
+ ); + } + + const selectedCount = noneSelected ? 0 : allSelected ? entries.length : selectedCodes.size; + const showSearch = entries.length > FEATURE_LIST_SEARCH_THRESHOLD; + + return ( +
+
+ Features ({catalog.featureKey}) + + {' '} + · {selectedCount}/{entries.length} selected + {hasCounts ? ' · sorted by count' : ''} + +
+ + + {showSearch ? ( + setSearchQuery(event.target.value)} + style={searchStyle} + /> + ) : null} +
+ {visibleEntries.map((entry) => { + const checked = !noneSelected && (allSelected || selectedCodes.has(entry.code)); + return ( + + ); + })} + {showSearch && visibleEntries.length === 0 ? ( +
No features match your search.
+ ) : null} +
+
+ ); +} diff --git a/packages/vis/src/SpatialCanvas/PointsStylePanel.tsx b/packages/vis/src/SpatialCanvas/PointsStylePanel.tsx new file mode 100644 index 00000000..2b76f577 --- /dev/null +++ b/packages/vis/src/SpatialCanvas/PointsStylePanel.tsx @@ -0,0 +1,213 @@ +import type { CSSProperties } from 'react'; +import { + DEFAULT_POINTS_MEMORY_CAP, + DEFAULT_POINTS_RENDER_CAP, + POINTS_PRELOAD_MAX_ROWS, +} from '@spatialdata/core'; +import { + DEFAULT_POINT_RADIUS_MAX_PIXELS, + DEFAULT_POINT_RADIUS_MIN_PIXELS, + DEFAULT_POINT_SIZE, +} from './renderers/pointsRenderer'; +import type { PointsLayerConfig } from './types'; +import { GeometryLoadStats } from './geometryLoadStats'; +import type { LayerLoadState } from './useLayerData'; + +const rangeLabelStyle: CSSProperties = { + color: '#ccc', + fontSize: '12px', + display: 'flex', + flexDirection: 'column', + gap: 4, +}; + +const numberInputStyle: CSSProperties = { + color: '#ccc', + fontSize: '12px', + padding: '4px 6px', + borderRadius: 4, + border: '1px solid #444', + background: '#1a1a1a', + width: '100%', +}; + +const helperStyle: CSSProperties = { + color: '#888', + fontSize: '11px', +}; + +const tileProgressStyle: CSSProperties = { + color: '#aaa', + fontSize: '11px', +}; + +export function preloadedPointCount(data: { shape: number[]; data: ArrayLike[] }): number { + if (data.shape.length >= 2 && Number.isFinite(data.shape[1])) { + return data.shape[1]; + } + return data.data[0]?.length ?? data.shape[0] ?? 0; +} + +export function preloadedPointCountSuffix( + data: { + shape: number[]; + data: ArrayLike[]; + totalRowCount?: number; + preloadTruncated?: boolean; + filterActive?: boolean; + scannedRowCount?: number; + } +): string | undefined { + const loaded = preloadedPointCount(data); + if (data.filterActive && data.scannedRowCount !== undefined) { + return ` · ${loaded.toLocaleString()} matching (scanned ${data.scannedRowCount.toLocaleString()} rows)`; + } + if (data.preloadTruncated && data.totalRowCount !== undefined) { + return ` · ${loaded.toLocaleString()} of ${data.totalRowCount.toLocaleString()} points loaded`; + } + return ` · ${loaded.toLocaleString()} points loaded`; +} + +const checkboxLabelStyle: CSSProperties = { + color: '#ccc', + fontSize: '12px', + display: 'flex', + alignItems: 'center', + gap: 6, +}; + +export interface PointsStylePanelProps { + layerId: string; + config: PointsLayerConfig; + loadState?: LayerLoadState; + pointCountSuffix?: string; + tileLoadingMessage?: string | null; + supportsTileDebugOverlay?: boolean; + updateLayer: (id: string, updates: Partial) => void; +} + +export function PointsStylePanel({ + layerId, + config, + loadState, + pointCountSuffix, + tileLoadingMessage, + supportsTileDebugOverlay = false, + updateLayer, +}: PointsStylePanelProps) { + const memoryCap = config.pointsMemoryCap ?? DEFAULT_POINTS_MEMORY_CAP; + const renderCap = config.pointsRenderCap ?? DEFAULT_POINTS_RENDER_CAP; + + return ( + <> + + + + + + + {supportsTileDebugOverlay ? ( + + ) : null} + {tileLoadingMessage ? ( +
{tileLoadingMessage}
+ ) : null} + + ); +} diff --git a/packages/vis/src/SpatialCanvas/ShapesStylePanel.tsx b/packages/vis/src/SpatialCanvas/ShapesStylePanel.tsx new file mode 100644 index 00000000..26a5d1b1 --- /dev/null +++ b/packages/vis/src/SpatialCanvas/ShapesStylePanel.tsx @@ -0,0 +1,28 @@ +import type { ShapesGeometryKind } from '@spatialdata/core'; +import { GeometryLoadStats } from './geometryLoadStats'; +import type { LayerLoadState, ShapesLayerLoadedSummary } from './useLayerData'; + +export function formatShapesGeometryKindLabel(kind: ShapesGeometryKind): string { + switch (kind) { + case 'polygon': + return 'polygons'; + case 'circle': + return 'circles'; + case 'point': + return 'points'; + } +} + +export interface ShapesStylePanelProps { + loadState?: LayerLoadState; + loadedSummary?: ShapesLayerLoadedSummary; +} + +export function ShapesStylePanel({ loadState, loadedSummary }: ShapesStylePanelProps) { + const geometryDetails = + loadedSummary !== undefined + ? ` · ${loadedSummary.featureCount.toLocaleString()} ${formatShapesGeometryKindLabel(loadedSummary.geometryKind)}` + : undefined; + + return ; +} diff --git a/packages/vis/src/SpatialCanvas/SpatialCanvasViewer.tsx b/packages/vis/src/SpatialCanvas/SpatialCanvasViewer.tsx index 8d46cce1..5313aefc 100644 --- a/packages/vis/src/SpatialCanvas/SpatialCanvasViewer.tsx +++ b/packages/vis/src/SpatialCanvas/SpatialCanvasViewer.tsx @@ -21,13 +21,13 @@ import { SpatialViewer } from './SpatialViewer'; import { VivLoaderRegistryProvider } from './VivLoaderRegistry'; import { getDeckFromDeckGlRef, resolveHoverFeatureTooltip } from './featureTooltipHover'; import { + type RenderStackHostLayerResolver, + type RenderStackLayerInputs, + type UnknownRenderStackHostLayerHandler, renderStackOrder, renderStackToLayerInputs, resolveRenderStackHostLayers, sortLayersByRenderStackOrder, - type RenderStackHostLayerResolver, - type RenderStackLayerInputs, - type UnknownRenderStackHostLayerHandler, } from './renderStackAdapters'; import type { ElementsByType, LayerConfig, ShapesLayerPickEvent, ViewState } from './types'; import { @@ -85,6 +85,7 @@ export interface SpatialCanvasViewerProps { * When true (default), hover tooltips aggregate picks from all layers under the cursor. */ aggregateHoverTooltips?: boolean; + experimentalOptimizations?: 'auto' | 'off'; } interface AutoFitInput { @@ -146,6 +147,7 @@ export interface UseSpatialCanvasRendererOptions { hostLayerResolver?: RenderStackHostLayerResolver; onUnknownHostLayer?: UnknownRenderStackHostLayerHandler; autoFit?: boolean; + experimentalOptimizations?: 'auto' | 'off'; } interface UseSpatialCanvasRendererFromLayerInputsOptions { @@ -154,6 +156,8 @@ interface UseSpatialCanvasRendererFromLayerInputsOptions { layerInputs: RenderStackLayerInputs; renderOrder?: string[]; viewState?: ViewState | null; + /** Orthographic zoom for points layer radius scaling (without subscribing to pan target). */ + viewZoom?: number | null; onViewStateChange?: (viewState: ViewState) => void; width: number; height: number; @@ -161,6 +165,7 @@ interface UseSpatialCanvasRendererFromLayerInputsOptions { externalDeckLayers?: Layer[]; sortDeckLayers?: boolean; autoFit?: boolean; + experimentalOptimizations?: 'auto' | 'off'; } export function useSpatialCanvasRendererFromLayerInputs({ @@ -169,6 +174,7 @@ export function useSpatialCanvasRendererFromLayerInputs({ layerInputs, renderOrder, viewState, + viewZoom: viewZoomProp, onViewStateChange, width, height, @@ -176,6 +182,7 @@ export function useSpatialCanvasRendererFromLayerInputs({ externalDeckLayers, sortDeckLayers, autoFit = true, + experimentalOptimizations = 'auto', }: UseSpatialCanvasRendererFromLayerInputsOptions) { const availableElements = useMemo(() => { if (!spatialData || !coordinateSystem) { @@ -191,7 +198,9 @@ export function useSpatialCanvasRendererFromLayerInputs({ layerInputs.layerOrder, availableElements, coordinateSystem, - spatialData ?? undefined + spatialData ?? undefined, + experimentalOptimizations, + viewZoomProp ?? viewState?.zoom ?? null ); const generatedDeckLayers = layerData.getLayers(); @@ -269,6 +278,7 @@ export function useSpatialCanvasRenderer({ hostLayerResolver, onUnknownHostLayer, autoFit = true, + experimentalOptimizations = 'auto', }: UseSpatialCanvasRendererOptions) { const layerInputs = useMemo(() => renderStackToLayerInputs(renderStack), [renderStack]); const hostDeckLayers = useMemo( @@ -292,6 +302,7 @@ export function useSpatialCanvasRenderer({ hostDeckLayers, sortDeckLayers: true, autoFit, + experimentalOptimizations, }); } @@ -348,6 +359,7 @@ function SpatialCanvasViewerInner({ autoFit = true, style, aggregateHoverTooltips = true, + experimentalOptimizations = 'auto', }: SpatialCanvasViewerProps) { const [measureRef, { width, height }] = useMeasure(); const viewerContainerRef = useRef(null); @@ -385,7 +397,9 @@ function SpatialCanvasViewerInner({ externalDeckLayers, sortDeckLayers: Boolean(renderStack), autoFit, + experimentalOptimizations, }); + const overlayStatusMessage = renderer.getOverlayStatusMessage(); const hoverPickLayerIds = useMemo( () => Array.from(renderer.enabledLayerIds), [renderer.enabledLayerIds] @@ -556,9 +570,9 @@ function SpatialCanvasViewerInner({ {showLoadingOverlay && renderer.isBlocking && (
Loading layer data...
)} - {showLoadingOverlay && renderer.isLoading && !renderer.isBlocking && ( + {showLoadingOverlay && !renderer.isBlocking && overlayStatusMessage && (
- Refreshing layer metadata... + {overlayStatusMessage}
)} {!renderer.hasLayersDrawn && !renderer.isBlocking && ( diff --git a/packages/vis/src/SpatialCanvas/geometryLoadStats.tsx b/packages/vis/src/SpatialCanvas/geometryLoadStats.tsx new file mode 100644 index 00000000..1fa611ed --- /dev/null +++ b/packages/vis/src/SpatialCanvas/geometryLoadStats.tsx @@ -0,0 +1,50 @@ +import type { CSSProperties } from 'react'; +import { formatLoadDurationMs, type LayerLoadState } from './useLayerData'; + +const loadStatsStyle: CSSProperties = { + color: '#888', + fontSize: '11px', +}; + +const errorStyle: CSSProperties = { + color: '#c96', + fontSize: '11px', +}; + +const noticeStyle: CSSProperties = { + color: '#ca8', + fontSize: '11px', +}; + +export interface GeometryLoadStatsProps { + loadState?: LayerLoadState; + detailsSuffix?: string; +} + +export function GeometryLoadStats({ loadState, detailsSuffix }: GeometryLoadStatsProps) { + if (!loadState?.geometry) { + return null; + } + + const geometryDuration = + loadState.geometryLoadDurationMs !== undefined + ? formatLoadDurationMs(loadState.geometryLoadDurationMs) + : null; + + return ( +
+ Geometry: {loadState.geometry} + {geometryDuration ? ` (${geometryDuration})` : ''} + {detailsSuffix ?? ''} + {loadState.geometry === 'ready' && loadState.geometryNotice ? ( +
{loadState.geometryNotice}
+ ) : null} + {loadState.geometry === 'loading' && loadState.geometryNotice ? ( +
{loadState.geometryNotice}
+ ) : null} + {loadState.geometry === 'error' && loadState.geometryError ? ( +
{loadState.geometryError}
+ ) : null} +
+ ); +} diff --git a/packages/vis/src/SpatialCanvas/index.tsx b/packages/vis/src/SpatialCanvas/index.tsx index e15650e8..54d3de65 100644 --- a/packages/vis/src/SpatialCanvas/index.tsx +++ b/packages/vis/src/SpatialCanvas/index.tsx @@ -24,7 +24,10 @@ import { createPortal } from 'react-dom'; import { ImageChannelPanel } from './ImageChannelPanel'; import { LabelsChannelPanel } from './LabelsChannelPanel'; import { LayerOrderList } from './LayerOrderList'; +import { PointsFeatureFilterPanel } from './PointsFeatureFilterPanel'; +import { PointsStylePanel, preloadedPointCountSuffix } from './PointsStylePanel'; import { ShapeFillColorPanel } from './ShapeFillColorPanel'; +import { ShapesStylePanel } from './ShapesStylePanel'; import { shouldAutoFitSpatialView, useSpatialCanvasRendererFromLayerInputs, @@ -39,10 +42,11 @@ import { TooltipFieldsPanel } from './TooltipFieldsPanel'; import { VivLoaderRegistryProvider } from './VivLoaderRegistry'; import { SpatialCanvasProvider, useSpatialCanvasActions, useSpatialCanvasStore } from './context'; import { getDeckFromDeckGlRef, resolveHoverFeatureTooltip } from './featureTooltipHover'; -import type { SpatialCanvasStoreApi } from './stores'; import { layerConfig } from './layerConfig'; +import { pointsTileLoadingMessage as formatPointsTileLoadingMessage } from './pointsTileProgress'; +import type { SpatialCanvasStoreApi } from './stores'; import type { AvailableElement, ElementsByType, ViewState } from './types'; -import type { ImageLayerConfig } from './useLayerData'; +import { formatLoadDurationMs, type ImageLayerConfig } from './useLayerData'; import { generateLayerId, getAllCoordinateSystems } from './utils'; // ============================================ @@ -212,7 +216,7 @@ interface ViewerSectionProps { vivLayerProps: ImageLayerConfig[]; hasEnabledLayers: boolean; isBlocking: boolean; - isLoading: boolean; + overlayStatusMessage: string | null; hasLayersDrawn: boolean; getWorldBoundsForVisibleLayers: () => import('@spatialdata/core').AxisAlignedBounds | null; vw: number; @@ -228,7 +232,7 @@ function ViewerSection({ vivLayerProps, hasEnabledLayers, isBlocking, - isLoading, + overlayStatusMessage, hasLayersDrawn, getWorldBoundsForVisibleLayers, vw, @@ -318,7 +322,7 @@ function ViewerSection({ Loading layer data... )} - {isLoading && !isBlocking && ( + {!isBlocking && overlayStatusMessage && (
- Refreshing layer metadata... + {overlayStatusMessage}
)} {!hasLayersDrawn && !isBlocking && ( @@ -365,12 +369,14 @@ interface SpatialCanvasInnerProps { * When true (default), hover tooltips include picks from all layers under the cursor. */ aggregateHoverTooltips?: boolean; + experimentalOptimizations?: 'auto' | 'off'; } function SpatialCanvasInner({ tooltipContainer, renderTooltip, aggregateHoverTooltips = true, + experimentalOptimizations = 'auto', }: SpatialCanvasInnerProps) { const { spatialData, loading: sdLoading } = useSpatialData(); const [measureRef, { width, height }] = useMeasure(); @@ -392,6 +398,7 @@ function SpatialCanvasInner({ // viewState is intentionally NOT subscribed here. It is consumed only by // ViewerSection, which is the sole component that re-renders on every pan. const selectedLayerId = useSpatialCanvasStore((s) => s.selectedLayerId); + const viewZoom = useSpatialCanvasStore((s) => s.viewState?.zoom ?? null); const actions = useSpatialCanvasActions(); @@ -409,24 +416,35 @@ function SpatialCanvasInner({ getFeatureTooltip, getImageLayerLoadedData, getLabelsLayerLoadedData, + getPointsLayerLoadedData, + getPointsFeatureCatalog, + getShapesLayerLoadedData, getLayerLoadState, + getPointsTileLoadProgress, + getPointsTileLoadingMessage, + getOverlayStatusMessage, + getPointsLayerSupportsTileDebug, + isPointsFeatureCatalogLoading, + requestPointsFeatureCatalog, getWorldBoundsForLayer, getWorldBoundsForVisibleLayers, hasEnabledLayers, hasLayersDrawn, hasRenderableLayerData, isBlocking, - isLoading, vivLayerProps, } = useSpatialCanvasRendererFromLayerInputs({ spatialData, coordinateSystem, layerInputs: { layers, layerOrder }, - // viewState and onViewStateChange are omitted: auto-fit and pan handling - // are managed entirely by ViewerSection so this hook never re-runs on pan. + // viewState target is not subscribed here; zoom alone drives point-size scaling. + viewZoom, width: vw, height: vh, + experimentalOptimizations, }); + const overlayStatusMessage = getOverlayStatusMessage(); + const hoverPickLayerIds = useMemo(() => Array.from(enabledLayerIds), [enabledLayerIds]); useEffect(() => { @@ -457,16 +475,16 @@ function SpatialCanvasInner({ vw, ]); - useEffect(() => { - if (coordinateSystems.length > 0 && !coordinateSystem) { - actions.setCoordinateSystem(coordinateSystems[0]); - } - }, [coordinateSystems, coordinateSystem, actions]); - useEffect(() => { actions.reset(); - if (coordinateSystem && coordinateSystems.includes(coordinateSystem)) { - actions.setCoordinateSystem(coordinateSystem); + const nextCoordinateSystem = + coordinateSystem && coordinateSystems.includes(coordinateSystem) + ? coordinateSystem + : coordinateSystems.length === 1 + ? coordinateSystems[0] + : null; + if (nextCoordinateSystem) { + actions.setCoordinateSystem(nextCoordinateSystem); } }, [coordinateSystem, coordinateSystems, actions]); @@ -496,6 +514,7 @@ function SpatialCanvasInner({ elementKey: element.key, visible: true, opacity: 1, + ...(element.type === 'points' ? { showTileDebugOverlay: true } : {}), }); actions.addLayer(config); } @@ -511,6 +530,14 @@ function SpatialCanvasInner({ ? spatialData?.getAssociatedTable('labels', selectedConfig.elementKey)?.[1] : undefined; const selectedLayerLoadState = getLayerLoadState(selectedConfig?.id); + const selectedPointsLoadedData = + selectedConfig?.type === 'points' ? getPointsLayerLoadedData(selectedConfig.id) : undefined; + const selectedPointsPointCountSuffix = + selectedPointsLoadedData === undefined + ? undefined + : preloadedPointCountSuffix(selectedPointsLoadedData); + const selectedShapesLoadedSummary = + selectedConfig?.type === 'shapes' ? getShapesLayerLoadedData(selectedConfig.id) : undefined; const selectedLayerCanCenter = !!selectedConfig?.id && @@ -677,7 +704,7 @@ function SpatialCanvasInner({ vivLayerProps={vivLayerProps} hasEnabledLayers={hasEnabledLayers} isBlocking={isBlocking} - isLoading={isLoading} + overlayStatusMessage={overlayStatusMessage} hasLayersDrawn={hasLayersDrawn} getWorldBoundsForVisibleLayers={getWorldBoundsForVisibleLayers} vw={vw} @@ -761,15 +788,23 @@ function SpatialCanvasInner({ fontSize: '11px', }} > - {selectedConfig.type !== 'image' && selectedLayerLoadState.geometry && ( -
- Geometry: {selectedLayerLoadState.geometry} - {!hasRenderableLayerData(selectedConfig.id) && - selectedLayerLoadState.geometry === 'loading' - ? ' (blocking)' - : ''} -
- )} + {selectedConfig.type !== 'image' && + selectedConfig.type !== 'points' && + selectedConfig.type !== 'shapes' && + selectedLayerLoadState.geometry && ( +
+ Geometry: {selectedLayerLoadState.geometry} + {selectedLayerLoadState.geometryLoadDurationMs !== undefined && + (selectedLayerLoadState.geometry === 'ready' || + selectedLayerLoadState.geometry === 'error') + ? ` (${formatLoadDurationMs(selectedLayerLoadState.geometryLoadDurationMs)})` + : ''} + {!hasRenderableLayerData(selectedConfig.id) && + selectedLayerLoadState.geometry === 'loading' + ? ' (blocking)' + : ''} +
+ )} {(selectedConfig.type === 'image' || selectedConfig.type === 'labels') && selectedLayerLoadState.image && (
@@ -803,16 +838,45 @@ function SpatialCanvasInner({ updateLayer={actions.updateLayer} /> )} + {selectedConfig.type === 'points' && ( + <> + + + + )} {selectedConfig.type === 'shapes' && ( - { - actions.updateLayer(selectedConfig.id, { fillColorByColumn }); - }} - noAssociatedTableMessage="No associated table found for this shapes layer" - /> + <> + + { + actions.updateLayer(selectedConfig.id, { fillColorByColumn }); + }} + noAssociatedTableMessage="No associated table found for this shapes layer" + /> + )} {(selectedConfig.type === 'shapes' || selectedConfig.type === 'labels') && ( @@ -932,6 +998,7 @@ export default function SpatialCanvas({ tooltipContainer={tooltipContainer} renderTooltip={renderTooltip} aggregateHoverTooltips={aggregateHoverTooltips} + experimentalOptimizations={experimentalOptimizations} /> diff --git a/packages/vis/src/SpatialCanvas/pointsLoadPlan.ts b/packages/vis/src/SpatialCanvas/pointsLoadPlan.ts new file mode 100644 index 00000000..b20e8a57 --- /dev/null +++ b/packages/vis/src/SpatialCanvas/pointsLoadPlan.ts @@ -0,0 +1,122 @@ +import { resolvePointsMemoryCap, type PointsTilingMetadata } from '@spatialdata/core'; + +export interface PointsPreloadCacheKeyInput { + pointsMemoryCap?: number; +} + +/** Cache key for preloaded scatter data (per element + memory cap). */ +export function pointsPreloadCacheKey( + elementKey: string, + config: PointsPreloadCacheKeyInput +): string { + const memoryCap = resolvePointsMemoryCap(config.pointsMemoryCap); + return `${elementKey}|m${memoryCap}`; +} + +export function deletePointsPreloadCacheForElement( + cache: Map, + elementKey: string +): void { + for (const key of [...cache.keys()]) { + if (key === elementKey || key.startsWith(`${elementKey}|`)) { + cache.delete(key); + } + } +} + +export function hasPointsPreloadForElement( + cache: Map, + elementKey: string +): boolean { + for (const key of cache.keys()) { + if (key === elementKey || key.startsWith(`${elementKey}|`)) { + return true; + } + } + return false; +} + +export function resolvePointsPreloadData( + cache: Map, + elementKey: string, + preloadCacheKey: string +): T | undefined { + return cache.get(preloadCacheKey) ?? cache.get(elementKey); +} + +export interface PointsLoadPlanInput { + wantsOptimized: boolean; + metadataKnown: boolean; + tiledMetadata: PointsTilingMetadata | null | undefined; + hasPreloaded: boolean; + /** Known row count from parquet metadata, when available. */ + totalRows?: number; +} + +export interface PointsLoadPlan { + probeMetadata: boolean; + preloadFullTable: boolean; +} + +/** Decide which points loads to schedule at the start of a load pass. */ +export function planPointsLoads(input: PointsLoadPlanInput): PointsLoadPlan { + const { wantsOptimized, metadataKnown, tiledMetadata, hasPreloaded } = input; + const probeMetadata = wantsOptimized && !metadataKnown; + const preloadFullTable = + !hasPreloaded && (!wantsOptimized || (metadataKnown && tiledMetadata === null)); + return { probeMetadata, preloadFullTable }; +} + +export interface ShouldPreloadAfterMetadataProbeInput { + probeRan: boolean; + renderableMetadata: boolean; + hasPreloaded: boolean; + totalRows?: number; +} + +/** + * After a metadata probe completes, preload may still be required even when + * `planPointsLoads` did not schedule it (metadata was unknown at plan time). + */ +export function shouldPreloadAfterMetadataProbe( + input: ShouldPreloadAfterMetadataProbeInput | boolean, + renderableMetadata?: boolean, + hasPreloaded?: boolean, + totalRows?: number +): boolean { + const normalized: ShouldPreloadAfterMetadataProbeInput = + typeof input === 'boolean' + ? { + probeRan: input, + renderableMetadata: renderableMetadata ?? false, + hasPreloaded: hasPreloaded ?? false, + totalRows, + } + : input; + + if (!normalized.probeRan || normalized.renderableMetadata || normalized.hasPreloaded) { + return false; + } + return true; +} + +export interface ShouldLoadPointsRowFeatureCodesInput { + hasPreloaded: boolean; + hasCached: boolean; + inFlight: boolean; + featureCodes?: readonly number[]; +} + +export function shouldLoadPointsRowFeatureCodes( + input: ShouldLoadPointsRowFeatureCodesInput +): boolean { + return input.hasPreloaded && !input.hasCached && !input.inFlight; +} + +export function pointsPreloadBlockedMessage(totalRows: number): string { + return `${totalRows.toLocaleString()} points exceeds the preload limit — use a Morton-sorted element or tiled path`; +} + +export function pointsTilingUnavailableMessage(totalRows: number): string { + return `${totalRows.toLocaleString()} points cannot be tiled with this store (range reads unavailable) and exceeds the preload limit`; +} diff --git a/packages/vis/src/SpatialCanvas/pointsTileProgress.ts b/packages/vis/src/SpatialCanvas/pointsTileProgress.ts new file mode 100644 index 00000000..3df55078 --- /dev/null +++ b/packages/vis/src/SpatialCanvas/pointsTileProgress.ts @@ -0,0 +1,89 @@ +import type { + PointsTileLoadProgress, + TileDebugStore, + TiledPointsDebugState, +} from '@spatialdata/layers'; + +export type { PointsTileLoadProgress }; + +export function emptyPointsTileLoadProgress(): PointsTileLoadProgress { + return { inFlight: 0, loaded: 0, loadedPoints: 0, viewportTotal: 0 }; +} + +export function pointsTileLoadProgressFromDebugState( + state: TiledPointsDebugState | undefined +): PointsTileLoadProgress { + if (!state) { + return emptyPointsTileLoadProgress(); + } + + const viewportTileIds = new Set((state.lastViewportTiles ?? []).map((tile) => tile.tileId)); + const loadingTileIds = new Set(state.loadingTileIds ?? []); + const completedTilesById = state.completedTilesById ?? {}; + + let inFlight = 0; + let loaded = 0; + let loadedPoints = 0; + for (const tileId of viewportTileIds) { + if (loadingTileIds.has(tileId)) { + inFlight += 1; + } + const completed = completedTilesById[tileId]; + if (completed?.status === 'loaded' || completed?.status === 'empty') { + loaded += 1; + loadedPoints += completed.pointCount ?? 0; + } + } + + return { + inFlight, + loaded, + loadedPoints, + viewportTotal: viewportTileIds.size, + }; +} + +export function pointsTileLoadProgressFromStore( + store: TileDebugStore | undefined +): PointsTileLoadProgress { + return pointsTileLoadProgressFromDebugState(store?.getState()); +} + +export function aggregatePointsTileLoadProgress( + progressByLayer: ReadonlyMap +): PointsTileLoadProgress { + let inFlight = 0; + let loaded = 0; + let loadedPoints = 0; + let viewportTotal = 0; + for (const progress of progressByLayer.values()) { + inFlight += progress.inFlight; + loaded += progress.loaded; + loadedPoints += progress.loadedPoints; + viewportTotal += progress.viewportTotal; + } + return { inFlight, loaded, loadedPoints, viewportTotal }; +} + +function formatLoadedPointCount(pointCount: number): string { + return pointCount.toLocaleString(); +} + +export function pointsTileLoadingMessage(progress: PointsTileLoadProgress): string | null { + const { inFlight, loaded, loadedPoints, viewportTotal } = progress; + if (inFlight <= 0) { + return null; + } + const pointsSuffix = loaded > 0 ? `, ${formatLoadedPointCount(loadedPoints)} points` : ''; + const message = + viewportTotal > 0 + ? `Loading points… (${loaded}/${viewportTotal} tiles${pointsSuffix})` + : inFlight > 0 + ? 'Loading points…' + : null; + return message; +} + +export function isPointsTileLoading(progress: PointsTileLoadProgress): boolean { + return pointsTileLoadingMessage(progress) !== null; +} diff --git a/packages/vis/src/SpatialCanvas/renderers/pointsRenderer.ts b/packages/vis/src/SpatialCanvas/renderers/pointsRenderer.ts index e8bda519..cb6070a3 100644 --- a/packages/vis/src/SpatialCanvas/renderers/pointsRenderer.ts +++ b/packages/vis/src/SpatialCanvas/renderers/pointsRenderer.ts @@ -1,31 +1,25 @@ /** - * Points layer renderer using deck.gl ScatterplotLayer - * - * Renders point cloud data from SpatialData points elements. + * Points layer renderer adapter for SpatialCanvas. */ -import { ScatterplotLayer } from 'deck.gl'; import type { Matrix4 } from '@math.gl/core'; -import type { PointsElement } from '@spatialdata/core'; +import { PointsLayer, type PointsRenderResource, type TileDebugStore } from '@spatialdata/layers'; import type { Layer } from 'deck.gl'; -export interface PointDataX { - position: [number, number] | [number, number, number]; - // Additional properties can be added for coloring, sizing, etc. - [key: string]: unknown; -} +export { + DEFAULT_POINT_RADIUS_MAX_PIXELS, + DEFAULT_POINT_RADIUS_MIN_PIXELS, + DEFAULT_POINT_SIZE, + MIN_POINT_SIZE_SCALE, + POINT_SIZE_ZOOM_REFERENCE, + zoomScaledPointSize, +} from '@spatialdata/layers'; -// this is ndarray and should be defined elsewhere -// not that we wouldn't also want to be able to have other data & accessors -export interface PointData { - shape: number[]; - // this should most definitely be TypedArray... - data: number[][]; -} +export type { PointData } from '@spatialdata/layers'; export interface PointsLayerRenderConfig { - /** The points element to render */ - element: PointsElement; + /** Resolved points render resource from the Resource Resolver. */ + resource: PointsRenderResource; /** Unique layer ID */ id: string; /** Transformation matrix to target coordinate system */ @@ -36,64 +30,77 @@ export interface PointsLayerRenderConfig { visible: boolean; /** Point radius in pixels */ pointSize?: number; + pointRadiusMinPixels?: number; + pointRadiusMaxPixels?: number; + pointMinSizeScale?: number; + /** Orthographic view zoom used to scale pointSize when zoomed out */ + viewZoom?: number | null; /** Point color [r, g, b, a] (0-255) */ color?: [number, number, number, number]; - /** ndarray - if we want other data for properties like color/radius etc they will be handled differently */ - pointData?: PointData; + /** Integer codes matching `{feature_key}_codes` in the Morton Parquet artifact. */ + featureCodes?: readonly number[]; + preloadedFeatureCodes?: ArrayLike; + renderCap?: number; + showTileDebugOverlay?: boolean; + tileDebugStore?: TileDebugStore; + tileDebugSignature?: string; use3d?: boolean; } -/** - * Create a deck.gl ScatterplotLayer for points data. - * - * Note: This requires the point data to be pre-loaded since deck.gl layers - * are synchronous. The data loading should happen at a higher level. - */ export function renderPointsLayer(config: PointsLayerRenderConfig): Layer | null { const { - element, + resource, id, modelMatrix, opacity, visible, - pointSize = 1, - color = [255, 100, 100, 200], - pointData, + pointSize, + pointRadiusMinPixels, + pointRadiusMaxPixels, + pointMinSizeScale, + viewZoom, + color, + featureCodes, + preloadedFeatureCodes, + renderCap, + showTileDebugOverlay, + tileDebugStore, + tileDebugSignature, use3d, } = config; - if (!visible) return null; + if (!visible) { + return null; + } - if (!pointData) { - // Data not loaded yet + if ( + !resource.loader.capabilities.bounds && + resource.loader.capabilities.kind === 'morton-tiled' + ) { console.debug( - `[PointsRenderer] No point data for layer "${id}" from ${element.url ?? element.path}` + `[PointsRenderer] No tiling bounds for layer "${id}" from ${resource.element.path}` ); return null; } - const d = pointData.data; - return new ScatterplotLayer({ + + return new PointsLayer({ id, - data: d[0], //just for index really - // todo: more robust ndarray handling, be more efficient with target - // see https://deck.gl/docs/developer-guide/performance#supply-attributes-directly - // spatial data-structure (quad/oct-tree) vs pushing raw attributes. - // with ways of querying within view. - // also allow accessors for other props - getPosition: (_d, { index, target }) => [ - d[0][index], - d[1][index], - use3d ? d[2]?.[index] || 0 : 0, - ], - getRadius: pointSize, - radiusUnits: 'pixels', - getFillColor: color, - opacity, - // Apply coordinate transformation + resource, modelMatrix, - // Picking - pickable: true, - autoHighlight: true, - highlightColor: [255, 255, 0, 200], + opacity, + visible, + pointSize, + pointRadiusMinPixels, + pointRadiusMaxPixels, + pointMinSizeScale, + viewZoom, + color, + featureCodes, + preloadedFeatureCodes, + renderCap, + showTileDebugOverlay: showTileDebugOverlay ?? true, + tileDebugStore, + tileDebugSignature, + use3d, }); } diff --git a/packages/vis/src/SpatialCanvas/resolvePointsRenderResource.ts b/packages/vis/src/SpatialCanvas/resolvePointsRenderResource.ts new file mode 100644 index 00000000..051fa609 --- /dev/null +++ b/packages/vis/src/SpatialCanvas/resolvePointsRenderResource.ts @@ -0,0 +1,64 @@ +import { + createPointsLoaderForElement, + type PointsElement, + type PointsTilingMetadata, +} from '@spatialdata/core'; +import { + createPointsRenderResource, + type PointsRenderResource, +} from '@spatialdata/layers'; + +export interface ResolvePointsRenderResourceCache { + preloaded?: { shape: number[]; data: ArrayLike[] } | null; + tilingMetadata?: PointsTilingMetadata | null; + metadataKnown?: boolean; +} + +export interface ResolvePointsRenderResourceOptions { + experimentalOptimizations: 'auto' | 'off'; +} + +export function resolvePointsRenderResource( + element: PointsElement, + cache: ResolvePointsRenderResourceCache, + options: ResolvePointsRenderResourceOptions +): PointsRenderResource | null { + const wantsOptimized = options.experimentalOptimizations !== 'off'; + const canTile = + wantsOptimized && + cache.metadataKnown && + cache.tilingMetadata?.supportsRowGroupRangeReads && + cache.tilingMetadata.bounds; + + const loader = createPointsLoaderForElement(element, { + preloaded: cache.preloaded ?? null, + tilingMetadata: canTile ? cache.tilingMetadata : null, + wantsOptimized, + }); + + if (!loader) { + return null; + } + + return createPointsRenderResource(element, loader); +} + +export function pointsRenderResourceSignature( + element: PointsElement, + cache: ResolvePointsRenderResourceCache, + options: ResolvePointsRenderResourceOptions & { preloadCacheKey?: string } +): string { + const rowCount = + cache.preloaded && cache.preloaded.shape.length >= 2 + ? cache.preloaded.shape[1] + : cache.preloaded?.data[0]?.length ?? 0; + return [ + element.key, + options.preloadCacheKey ?? '', + options.experimentalOptimizations, + cache.metadataKnown ? 'meta' : 'nometa', + cache.tilingMetadata?.parquetPath ?? '', + cache.tilingMetadata?.supportsRowGroupRangeReads ? 'rg' : '', + cache.preloaded ? `pre:${rowCount}` : 'nopre', + ].join('|'); +} diff --git a/packages/vis/src/SpatialCanvas/types.ts b/packages/vis/src/SpatialCanvas/types.ts index 78fae21e..7f2c0537 100644 --- a/packages/vis/src/SpatialCanvas/types.ts +++ b/packages/vis/src/SpatialCanvas/types.ts @@ -93,8 +93,24 @@ export interface PointsLayerConfig extends BaseLayerConfig { // Points-specific settings // TODO: these should be accessors for getColor etc based on e.g. transcript type // should be able to filter etc. Some kind of LOD... + /** Base point radius in pixels (scatter path; tile sublayer getRadius). */ pointSize?: number; + /** Minimum radius in pixels for tiled points (deck.gl radiusMinPixels). */ + pointRadiusMinPixels?: number; + /** Maximum radius in pixels for tiled points (deck.gl radiusMaxPixels). */ + pointRadiusMaxPixels?: number; + /** Minimum pointSize multiplier when zoomed out on the non-tiled scatter path. */ + pointMinSizeScale?: number; color?: [number, number, number, number]; + /** Filter to these feature code(s). Future: string[] resolved via codebook. */ + featureCodes?: number[]; + /** Max rows to retain in memory for preloaded scatter (default 4M). */ + pointsMemoryCap?: number; + /** Max rows to render after filtering (default 4M; set 0 to disable). */ + pointsRenderCap?: number; + experimentalOptimizations?: 'auto' | 'off'; + /** Show viewport tile polygons and loading stats for tiled points layers. */ + showTileDebugOverlay?: boolean; } export interface LabelsLayerConfig extends BaseLayerConfig { diff --git a/packages/vis/src/SpatialCanvas/useLayerData.ts b/packages/vis/src/SpatialCanvas/useLayerData.ts index 63d2cbf2..5d5a1322 100644 --- a/packages/vis/src/SpatialCanvas/useLayerData.ts +++ b/packages/vis/src/SpatialCanvas/useLayerData.ts @@ -23,11 +23,19 @@ import { type LabelsElement, type LabelsTooltipMetadata, type PointsElement, + type PointsFeatureCatalog, + type PointsTilingMetadata, type ShapesElement, + type ShapesGeometryKind, type ShapesRenderData, type ShapesTooltipMetadata, type SpatialData, type SpatialFeatureTooltipData, + pointsPreloadTruncatedMessage, + preloadedColumnarPointCount, + mergeFeatureCountsIntoCatalog, + resolvePointsMemoryCap, + resolvePointsRenderCap, attachTooltipElementContext, boundsFromCircles, boundsFromImagePixelExtents, @@ -50,9 +58,14 @@ import { buildShapeFeatureStateRuntime, buildShapeFillColorByFeatureId, buildShapesPrebuiltData, + formatPointsTileDebugTooltip, + isPointsTileDebugPickObject, + tileDebugEntriesSignature, resolveShapeFeatureFromPick, resolveShapeTooltipFromPickInfo, resolveShapeTooltipRowIndex, + createTileDebugStore, + type TileDebugStore, } from '@spatialdata/layers'; import type { Layer } from 'deck.gl'; import { useCallback, useEffect, useMemo, useRef, useState } from 'react'; @@ -64,6 +77,25 @@ import { import { createImageLoader } from './renderers/imageRenderer'; import { renderLabelsLayer } from './renderers/labelsRenderer'; import { type PointData, renderPointsLayer } from './renderers/pointsRenderer'; +import { + aggregatePointsTileLoadProgress, + isPointsTileLoading, + pointsTileLoadProgressFromStore, + pointsTileLoadingMessage, + type PointsTileLoadProgress, +} from './pointsTileProgress'; +import { + deletePointsPreloadCacheForElement, + hasPointsPreloadForElement, + planPointsLoads, + pointsPreloadCacheKey, + resolvePointsPreloadData, + shouldPreloadAfterMetadataProbe, +} from './pointsLoadPlan'; +import { + pointsRenderResourceSignature, + resolvePointsRenderResource, +} from './resolvePointsRenderResource'; import { loadShapesData, renderShapesLayer } from './renderers/shapesRenderer'; import type { AvailableElement, ElementsByType, LayerConfig, ShapesLayerConfig } from './types'; @@ -101,6 +133,7 @@ export interface WorldBoundsCacheEntry { interface LoadedData { shapes: Map; points: Map; + pointTilingMetadata: Map; images: Map; // Viv loaders with computed channel data labels: Map; /** @@ -128,10 +161,29 @@ type RasterSelection = Partial<{ z: number; c: number; t: number }>; export interface LayerLoadState { geometry?: ResourceLoadStatus; + /** User-facing geometry load error, when geometry status is `error`. */ + geometryError?: string; + /** Non-fatal geometry notice shown while geometry is ready (e.g. truncated preload). */ + geometryNotice?: string; + /** Wall-clock ms from first geometry load start to ready/error. */ + geometryLoadDurationMs?: number; image?: ResourceLoadStatus; tooltip?: ResourceLoadStatus; } +export function formatLoadDurationMs(ms: number): string { + if (ms < 1000) { + return `${ms} ms`; + } + const seconds = ms / 1000; + return seconds >= 10 ? `${Math.round(seconds)} s` : `${seconds.toFixed(1)} s`; +} + +export interface ShapesLayerLoadedSummary { + geometryKind: ShapesGeometryKind; + featureCount: number; +} + export interface ImageLayerConfig { id: string; loader: unknown; // Viv PixelSource @@ -190,8 +242,18 @@ interface UseLayerDataResult { getImageLayerLoadedData: (layerId: string) => ImageLoaderData | undefined; /** Raw loaded labels pipeline data (defaults) for the properties UI */ getLabelsLayerLoadedData: (layerId: string) => LabelsLoaderData | undefined; + /** Raw loaded preloaded points data for the properties UI */ + getPointsLayerLoadedData: (layerId: string) => PointData | undefined; + /** Loaded shapes geometry summary for the properties UI */ + getShapesLayerLoadedData: (layerId: string) => ShapesLayerLoadedSummary | undefined; /** Current load state for a given layer. */ getLayerLoadState: (layerId?: string) => LayerLoadState | undefined; + /** Feature catalog for a visible points layer, when loaded. */ + getPointsFeatureCatalog: (layerId: string) => PointsFeatureCatalog | null | undefined; + /** Whether the feature catalog is still loading for a points layer. */ + isPointsFeatureCatalogLoading: (layerId: string) => boolean; + /** Request the feature catalog for a points layer. */ + requestPointsFeatureCatalog: (layerId: string) => void; /** Whether a layer already has enough data to render. */ hasRenderableLayerData: (layerId: string) => boolean; /** Resolve a feature tooltip lazily from the picked row index. */ @@ -222,6 +284,14 @@ interface UseLayerDataResult { isLoading: boolean; /** Whether any visible layer is still waiting on its first renderable resource. */ isBlocking: boolean; + /** Tile fetch progress for Morton-tiled points layers. */ + getPointsTileLoadProgress: (layerId?: string) => PointsTileLoadProgress; + /** User-facing message while tiled points are loading, if any. */ + getPointsTileLoadingMessage: () => string | null; + /** Combined non-blocking overlay message (tiles, filter reload, other refresh). */ + getOverlayStatusMessage: () => string | null; + /** Whether a points layer uses viewport tile loading (tile debug overlay eligible). */ + getPointsLayerSupportsTileDebug: (layerId: string) => boolean; /** Trigger a reload of data for a specific element */ reloadElement: (type: string, key: string) => void; /** World-space axis-aligned bounds for one visible layer with loaded data, or null. */ @@ -315,6 +385,33 @@ function getWorldBoundsCacheKey(elem: AvailableElement): string { return `${elem.type}:${elem.key}`; } +function transformAxisAlignedBounds( + bounds: AxisAlignedBounds, + modelMatrix: Matrix4 +): AxisAlignedBounds | null { + const corners: [number, number, number][] = [ + [bounds.minX, bounds.minY, 0], + [bounds.maxX, bounds.minY, 0], + [bounds.maxX, bounds.maxY, 0], + [bounds.minX, bounds.maxY, 0], + ]; + let minX = Number.POSITIVE_INFINITY; + let minY = Number.POSITIVE_INFINITY; + let maxX = Number.NEGATIVE_INFINITY; + let maxY = Number.NEGATIVE_INFINITY; + for (const corner of corners) { + const transformed = modelMatrix.transformAsPoint(corner); + if (!Number.isFinite(transformed[0]) || !Number.isFinite(transformed[1])) { + return null; + } + minX = Math.min(minX, transformed[0]); + minY = Math.min(minY, transformed[1]); + maxX = Math.max(maxX, transformed[0]); + maxY = Math.max(maxY, transformed[1]); + } + return { minX, minY, maxX, maxY }; +} + export function resolveLayerElement( layerId: string, config: LayerConfig | undefined, @@ -448,7 +545,9 @@ export function useLayerData( layerOrder: string[], availableElements: ElementsByType, coordinateSystem: string | null, - spatialData?: SpatialData + spatialData?: SpatialData, + experimentalOptimizations: 'auto' | 'off' = 'auto', + viewZoom: number | null = null ): UseLayerDataResult { const { getOmeZarrMultiscalesData } = useVivLoaderRegistry(); @@ -456,6 +555,7 @@ export function useLayerData( const loadedDataRef = useRef({ shapes: new Map(), points: new Map(), + pointTilingMetadata: new Map(), images: new Map(), labels: new Map(), shapePrebuiltData: new Map(), @@ -473,12 +573,97 @@ export function useLayerData( layersRef.current = layers; const [layerLoadStates, setLayerLoadStates] = useState>({}); - const [, setLoadedDataRevision] = useState(0); + const geometryLoadStartRef = useRef>(new Map()); + const geometryLoadDurationRef = useRef>(new Map()); + const [loadedDataRevision, setLoadedDataRevision] = useState(0); + const pointsTileDebugStoreRef = useRef(new Map()); + const pointsRenderResourceCacheRef = useRef( + new Map< + string, + { signature: string; resource: ReturnType } + >() + ); + const [pointsTileLayersRevision, setPointsTileLayersRevision] = useState(0); + const pointsTileLayersFrameRef = useRef(null); + const pointsFeatureCatalogRef = useRef(new Map()); + const pointsFeatureCatalogInFlightRef = useRef(new Set()); + const [pointsFeatureCatalogRevision, setPointsFeatureCatalogRevision] = useState(0); + const pointsRowFeatureCodesRef = useRef(new Map>()); + const pointsRowFeatureCodesInFlightRef = useRef(new Set()); + const [pointsRowFeatureCodesRevision, setPointsRowFeatureCodesRevision] = useState(0); + + const notifyPointsTileLayersChanged = useCallback(() => { + if (pointsTileLayersFrameRef.current != null) { + return; + } + const schedule = + typeof requestAnimationFrame === 'function' + ? requestAnimationFrame + : (callback: FrameRequestCallback) => setTimeout(callback, 0); + pointsTileLayersFrameRef.current = schedule(() => { + pointsTileLayersFrameRef.current = null; + setPointsTileLayersRevision((revision) => revision + 1); + }); + }, []); const notifyLoadedDataChanged = useCallback(() => { setLoadedDataRevision((revision) => revision + 1); }, []); + const getTileDebugStore = useCallback( + (layerId: string): TileDebugStore => { + let store = pointsTileDebugStoreRef.current.get(layerId); + if (!store) { + store = createTileDebugStore(notifyPointsTileLayersChanged); + pointsTileDebugStoreRef.current.set(layerId, store); + } + return store; + }, + [notifyPointsTileLayersChanged] + ); + + const getPointsTileLoadProgress = useCallback( + (layerId?: string): PointsTileLoadProgress => { + void pointsTileLayersRevision; + if (layerId) { + return pointsTileLoadProgressFromStore(pointsTileDebugStoreRef.current.get(layerId)); + } + const visibleProgress = new Map(); + for (const id of layerOrder) { + const config = layers[id]; + if (!config?.visible || config.type !== 'points') continue; + const store = pointsTileDebugStoreRef.current.get(id); + if (store) { + visibleProgress.set(id, pointsTileLoadProgressFromStore(store)); + } + } + return aggregatePointsTileLoadProgress(visibleProgress); + }, + [layerOrder, layers, pointsTileLayersRevision] + ); + + const getPointsTileLoadingMessage = useCallback((): string | null => { + return pointsTileLoadingMessage(getPointsTileLoadProgress()); + }, [getPointsTileLoadProgress]); + + const prevExperimentalOptimizationsRef = useRef(experimentalOptimizations); + useEffect(() => { + const prev = prevExperimentalOptimizationsRef.current; + prevExperimentalOptimizationsRef.current = experimentalOptimizations; + if (prev === 'off' && experimentalOptimizations !== 'off') { + const loaded = loadedDataRef.current; + for (const layerId of layerOrder) { + const config = layers[layerId]; + if (config?.type !== 'points' || !config.visible) continue; + const elem = resolveLayerElement(layerId, config, elementMap.current); + if (elem) { + deletePointsPreloadCacheForElement(loaded.points, elem.key); + } + } + notifyLoadedDataChanged(); + } + }, [experimentalOptimizations, layerOrder, layers, notifyLoadedDataChanged]); + // Build a map of element key -> AvailableElement for quick lookup const elementMap = useRef>(new Map()); @@ -492,25 +677,195 @@ export function useLayerData( elementMap.current = map; }, [availableElements]); + const requestPointsFeatureCatalog = useCallback((layerId: string) => { + const elem = resolveLayerElement(layerId, layersRef.current[layerId], elementMap.current); + if (!elem || elem.type !== 'points') { + return; + } + if ( + (pointsFeatureCatalogRef.current.has(elem.key) && + pointsFeatureCatalogRef.current.get(elem.key) !== null) || + pointsFeatureCatalogInFlightRef.current.has(elem.key) + ) { + return; + } + + const elementKey = elem.key; + const element = elem.element as PointsElement; + pointsFeatureCatalogInFlightRef.current.add(elementKey); + setPointsFeatureCatalogRevision((revision) => revision + 1); + + void (async () => { + try { + const catalog = await element.listFeatures(); + pointsFeatureCatalogRef.current.set(elementKey, catalog); + setPointsFeatureCatalogRevision((revision) => revision + 1); + + if (catalog) { + void (async () => { + try { + const counts = await element.loadFeatureCounts(); + pointsFeatureCatalogRef.current.set( + elementKey, + mergeFeatureCountsIntoCatalog(catalog, counts) + ); + setPointsFeatureCatalogRevision((revision) => revision + 1); + } catch (error) { + console.warn(`Failed to load feature counts for ${elementKey}:`, error); + } + })(); + } + } catch (error) { + pointsFeatureCatalogRef.current.set(elementKey, null); + console.error(`Failed to load points feature catalog for ${elementKey}:`, error); + setPointsFeatureCatalogRevision((revision) => revision + 1); + } finally { + pointsFeatureCatalogInFlightRef.current.delete(elementKey); + setPointsFeatureCatalogRevision((revision) => revision + 1); + } + })(); + }, []); + + const ensurePointsRowFeatureCodes = useCallback( + (preloadCacheKey: string, elementKey: string, element: PointsElement, memoryCap?: number) => { + if (!loadedDataRef.current.points.has(preloadCacheKey)) { + return; + } + if (pointsRowFeatureCodesRef.current.has(preloadCacheKey)) { + return; + } + if (pointsRowFeatureCodesInFlightRef.current.has(preloadCacheKey)) { + return; + } + + pointsRowFeatureCodesInFlightRef.current.add(preloadCacheKey); + setPointsRowFeatureCodesRevision((revision) => revision + 1); + + const featureCatalog = pointsFeatureCatalogRef.current.has(elementKey) + ? pointsFeatureCatalogRef.current.get(elementKey) + : undefined; + + void (async () => { + try { + //pjt subject to review + const rowCodes = await element.loadRowFeatureCodes({ + memoryCap, + featureCatalog, + }); + if (rowCodes && rowCodes.length > 0) { + pointsRowFeatureCodesRef.current.set(preloadCacheKey, rowCodes); + } + setPointsRowFeatureCodesRevision((revision) => revision + 1); + } catch (error) { + console.error(`Failed to load row feature codes for ${preloadCacheKey}:`, error); + } finally { + pointsRowFeatureCodesInFlightRef.current.delete(preloadCacheKey); + setPointsRowFeatureCodesRevision((revision) => revision + 1); + } + })(); + }, + [] + ); + + useEffect(() => { + for (const layerId of layerOrder) { + const config = layersRef.current[layerId]; + if (!config?.visible || config.type !== 'points') continue; + const elem = resolveLayerElement(layerId, config, elementMap.current); + if (!elem || elem.type !== 'points' || elem.element.kind !== 'points') continue; + const preloadCacheKey = pointsPreloadCacheKey(elem.key, config); + ensurePointsRowFeatureCodes( + preloadCacheKey, + elem.key, + elem.element, + config.pointsMemoryCap + ); + } + }, [layerOrder, loadedDataRevision, ensurePointsRowFeatureCodes]); + const setLayerResourceStatus = useCallback( (layerId: string, resource: keyof LayerLoadState, status: ResourceLoadStatus) => { setLayerLoadStates((prev) => { const existing = prev[layerId] ?? {}; - if (existing[resource] === status) { + const statusUnchanged = existing[resource] === status; + const canPatchGeometryDuration = + resource === 'geometry' && + (status === 'ready' || status === 'error') && + existing.geometryLoadDurationMs === undefined && + geometryLoadStartRef.current.has(layerId); + + if (statusUnchanged && !canPatchGeometryDuration) { return prev; } + const next: LayerLoadState = { ...existing, [resource]: status }; + if (resource === 'geometry') { + if (status === 'loading' || status === 'ready') { + delete next.geometryError; + } + if (status === 'loading') { + delete next.geometryNotice; + if (!geometryLoadStartRef.current.has(layerId)) { + geometryLoadStartRef.current.set(layerId, performance.now()); + geometryLoadDurationRef.current.delete(layerId); + } + delete next.geometryLoadDurationMs; + } else if (status === 'ready' || status === 'error') { + const start = geometryLoadStartRef.current.get(layerId); + if (start !== undefined) { + const duration = Math.round(performance.now() - start); + next.geometryLoadDurationMs = duration; + geometryLoadDurationRef.current.set(layerId, duration); + geometryLoadStartRef.current.delete(layerId); + } else { + const cached = + existing.geometryLoadDurationMs ?? geometryLoadDurationRef.current.get(layerId); + if (cached !== undefined) { + next.geometryLoadDurationMs = cached; + } + } + } + } return { ...prev, - [layerId]: { - ...existing, - [resource]: status, - }, + [layerId]: next, }; }); }, [] ); + const setLayerGeometryNotice = useCallback((layerId: string, message: string | undefined) => { + setLayerLoadStates((prev) => { + const existing = prev[layerId] ?? {}; + if (existing.geometryNotice === message) { + return prev; + } + const next: LayerLoadState = { ...existing }; + if (message) { + next.geometryNotice = message; + } else { + delete next.geometryNotice; + } + return { ...prev, [layerId]: next }; + }); + }, []); + + const setLayerGeometryError = useCallback((layerId: string, message: string | undefined) => { + setLayerLoadStates((prev) => { + const existing = prev[layerId] ?? {}; + if (existing.geometryError === message) { + return prev; + } + const next: LayerLoadState = { ...existing }; + if (message) { + next.geometryError = message; + } else { + delete next.geometryError; + } + return { ...prev, [layerId]: next }; + }); + }, []); + // Load data for enabled layers that don't have data yet useEffect(() => { const loadData = async () => { @@ -546,6 +901,7 @@ export function useLayerData( loadFillColor: boolean; loadImage: boolean; loadPoints: boolean; + loadPointTilingMetadata: boolean; loadLabels: boolean; }> = []; @@ -575,6 +931,7 @@ export function useLayerData( loadFillColor, loadImage: false, loadPoints: false, + loadPointTilingMetadata: false, loadLabels: false, }); } @@ -592,20 +949,36 @@ export function useLayerData( loadFillColor: false, loadImage: false, loadPoints: false, + loadPointTilingMetadata: false, loadLabels, }); } - } else if (config.type === 'points' && !loaded.points.has(elem.key)) { - toLoad.push({ - layerId, - element: elem, - loadGeometry: false, - loadTooltip: false, - loadFillColor: false, - loadImage: false, - loadPoints: true, - loadLabels: false, - }); + } else if (config.type === 'points') { + const wantsOptimized = + experimentalOptimizations !== 'off' && config.experimentalOptimizations !== 'off'; + const metadataKnown = loaded.pointTilingMetadata.has(elem.key); + const tiledMetadata = loaded.pointTilingMetadata.get(elem.key); + const preloadCacheKey = pointsPreloadCacheKey(elem.key, config); + const { probeMetadata: loadPointTilingMetadata, preloadFullTable: loadPoints } = + planPointsLoads({ + wantsOptimized, + metadataKnown, + tiledMetadata, + hasPreloaded: loaded.points.has(preloadCacheKey), + }); + if (loadPointTilingMetadata || loadPoints) { + toLoad.push({ + layerId, + element: elem, + loadGeometry: false, + loadTooltip: false, + loadFillColor: false, + loadImage: false, + loadPoints, + loadPointTilingMetadata, + loadLabels: false, + }); + } } else if (config.type === 'image' && !loaded.images.has(elem.key)) { toLoad.push({ layerId, @@ -615,6 +988,7 @@ export function useLayerData( loadFillColor: false, loadImage: true, loadPoints: false, + loadPointTilingMetadata: false, loadLabels: false, }); } @@ -633,6 +1007,7 @@ export function useLayerData( loadFillColor, loadImage, loadPoints, + loadPointTilingMetadata, loadLabels, }) => { if (element.type === 'shapes') { @@ -769,17 +1144,98 @@ export function useLayerData( } } } - } else if (element.type === 'points' && loadPoints) { - try { - setLayerResourceStatus(layerId, 'geometry', 'loading'); - // todo better type-guards etc here. - const e = element.element as PointsElement; - const data = await e.loadPoints(); - loadedDataRef.current.points.set(element.key, data); - setLayerResourceStatus(layerId, 'geometry', 'ready'); - } catch (error) { - setLayerResourceStatus(layerId, 'geometry', 'error'); - console.error(`Failed to load points for ${layerId}:`, error); + } else if (element.type === 'points') { + const e = element.element as PointsElement; + const pointsConfig = + layersRef.current[layerId]?.type === 'points' + ? layersRef.current[layerId] + : undefined; + const preloadCacheKey = pointsPreloadCacheKey(element.key, pointsConfig ?? {}); + const memoryCap = resolvePointsMemoryCap(pointsConfig?.pointsMemoryCap); + const loadPreloadedPoints = async (options?: { continueLoading?: boolean }) => { + const nonBlocking = + options?.continueLoading ?? + hasPointsPreloadForElement(loadedDataRef.current.points, element.key); + try { + if (!nonBlocking) { + setLayerResourceStatus(layerId, 'geometry', 'loading'); + setLayerGeometryNotice(layerId, undefined); + } + setLayerGeometryError(layerId, undefined); + const data = await e.loadPoints({ memoryCap }); + loadedDataRef.current.points.set(preloadCacheKey, data); + setLayerResourceStatus(layerId, 'geometry', 'ready'); + const loadedCount = preloadedColumnarPointCount(data.shape, data.data); + if (data.preloadTruncated && data.totalRowCount !== undefined) { + setLayerGeometryNotice( + layerId, + pointsPreloadTruncatedMessage(loadedCount, data.totalRowCount) + ); + } + notifyLoadedDataChanged(); + requestPointsFeatureCatalog(layerId); + ensurePointsRowFeatureCodes( + preloadCacheKey, + element.key, + e, + pointsConfig?.pointsMemoryCap + ); + } catch (error) { + setLayerResourceStatus(layerId, 'geometry', 'error'); + setLayerGeometryNotice(layerId, undefined); + setLayerGeometryError(layerId, undefined); + console.error(`Failed to load points for ${layerId}:`, error); + notifyLoadedDataChanged(); + } + }; + if (loadPointTilingMetadata) { + let renderableMetadata: PointsTilingMetadata | null = null; + let probedTotalRows = 0; + try { + setLayerResourceStatus(layerId, 'geometry', 'loading'); + setLayerGeometryError(layerId, undefined); + const metadata = await e.getPointsTilingMetadata(); + probedTotalRows = metadata?.totalRows ?? (await e.getParquetRowCount()); + renderableMetadata = + metadata?.supportsRowGroupRangeReads && metadata.bounds ? metadata : null; + loadedDataRef.current.pointTilingMetadata.set(element.key, renderableMetadata); + if (renderableMetadata) { + deletePointsPreloadCacheForElement(loadedDataRef.current.points, element.key); + setLayerResourceStatus(layerId, 'geometry', 'ready'); + notifyLoadedDataChanged(); + } else if ( + shouldPreloadAfterMetadataProbe({ + probeRan: true, + renderableMetadata: false, + hasPreloaded: loadedDataRef.current.points.has(preloadCacheKey), + totalRows: probedTotalRows, + }) || + probedTotalRows > 0 + ) { + await loadPreloadedPoints({ continueLoading: true }); + } else { + setLayerResourceStatus(layerId, 'geometry', 'idle'); + notifyLoadedDataChanged(); + } + } catch (error) { + loadedDataRef.current.pointTilingMetadata.set(element.key, null); + setLayerResourceStatus(layerId, 'geometry', 'error'); + setLayerGeometryError(layerId, undefined); + console.error(`Failed to inspect point tiling metadata for ${layerId}:`, error); + notifyLoadedDataChanged(); + if ( + shouldPreloadAfterMetadataProbe({ + probeRan: true, + renderableMetadata: false, + hasPreloaded: loadedDataRef.current.points.has(preloadCacheKey), + totalRows: probedTotalRows, + }) + ) { + await loadPreloadedPoints({ continueLoading: true }); + } + } + } else if (loadPoints) { + await loadPreloadedPoints(); } } else if (element.type === 'image' && loadImage) { try { @@ -1048,6 +1504,9 @@ export function useLayerData( spatialData, setLayerResourceStatus, notifyLoadedDataChanged, + experimentalOptimizations, + requestPointsFeatureCatalog, + ensurePointsRowFeatureCodes, ]); const reloadElement = useCallback((type: string, key: string) => { @@ -1063,8 +1522,23 @@ export function useLayerData( } } } else if (type === 'points') { - loaded.points.delete(key); + deletePointsPreloadCacheForElement(loaded.points, key); + loaded.pointTilingMetadata.delete(key); loaded.worldBounds.delete(`points:${key}`); + pointsRenderResourceCacheRef.current.delete(key); + pointsFeatureCatalogRef.current.delete(key); + pointsFeatureCatalogInFlightRef.current.delete(key); + pointsRowFeatureCodesRef.current.delete(key); + for (const cacheKey of [...pointsRowFeatureCodesRef.current.keys()]) { + if (cacheKey.startsWith(`${key}|`)) { + pointsRowFeatureCodesRef.current.delete(cacheKey); + } + } + for (const cacheKey of [...pointsRowFeatureCodesInFlightRef.current]) { + if (cacheKey === key || cacheKey.startsWith(`${key}|`)) { + pointsRowFeatureCodesInFlightRef.current.delete(cacheKey); + } + } } else if (type === 'image') { loaded.images.delete(key); loaded.worldBounds.delete(`image:${key}`); @@ -1093,7 +1567,14 @@ export function useLayerData( return loadedDataRef.current.shapes.has(elem.key); } if (elem.type === 'points') { - return loadedDataRef.current.points.has(elem.key); + const config = layersRef.current[layerId]; + const preloadCacheKey = + config?.type === 'points' ? pointsPreloadCacheKey(elem.key, config) : elem.key; + return ( + loadedDataRef.current.points.has(preloadCacheKey) || + hasPointsPreloadForElement(loadedDataRef.current.points, elem.key) || + Boolean(loadedDataRef.current.pointTilingMetadata.get(elem.key)?.bounds) + ); } if (elem.type === 'image') { return loadedDataRef.current.images.has(elem.key); @@ -1133,14 +1614,23 @@ export function useLayerData( ); } if (elem.type === 'points') { - const pointData = loaded.points.get(elem.key); - if (!pointData) return null; + const config = layers[layerId]; + const preloadCacheKey = + config?.type === 'points' ? pointsPreloadCacheKey(elem.key, config) : elem.key; + const pointData = resolvePointsPreloadData(loaded.points, elem.key, preloadCacheKey); + const tilingMetadata = loaded.pointTilingMetadata.get(elem.key); + if (!pointData && !tilingMetadata?.bounds) return null; return getCachedWorldBounds( loaded.worldBounds, getWorldBoundsCacheKey(elem), - pointData, + pointData ?? tilingMetadata, elem.transform, - () => boundsFromPoints(pointData, elem.transform, false) + () => + pointData + ? boundsFromPoints(pointData, elem.transform, false) + : tilingMetadata?.bounds + ? transformAxisAlignedBounds(tilingMetadata.bounds, elem.transform) + : null ); } if (elem.type === 'image') { @@ -1200,6 +1690,7 @@ export function useLayerData( }, [layerOrder, layers, getWorldBoundsForLayer]); const getLayers = useCallback((): Layer[] => { + void loadedDataRevision; const deckLayers: Layer[] = []; const loaded = loadedDataRef.current; @@ -1237,17 +1728,78 @@ export function useLayerData( if (layer) deckLayers.push(layer); } } else if (config.type === 'points') { - const pointData = loaded.points.get(elem.key); - if (pointData) { + const preloadCacheKey = pointsPreloadCacheKey(elem.key, config); + const pointData = resolvePointsPreloadData(loaded.points, elem.key, preloadCacheKey); + const pointTilingMetadata = loaded.pointTilingMetadata.get(elem.key); + const metadataKnown = loaded.pointTilingMetadata.has(elem.key); + const wantsOptimized = + experimentalOptimizations !== 'off' && config.experimentalOptimizations !== 'off'; + const signature = pointsRenderResourceSignature( + elem.element as PointsElement, + { + preloaded: pointData ?? null, + tilingMetadata: pointTilingMetadata, + metadataKnown, + }, + { + experimentalOptimizations: wantsOptimized ? 'auto' : 'off', + preloadCacheKey, + } + ); + let cachedResource = pointsRenderResourceCacheRef.current.get(elem.key); + if (!cachedResource || cachedResource.signature !== signature) { + pointsTileDebugStoreRef.current.delete(layerId); + const resource = resolvePointsRenderResource( + elem.element as PointsElement, + { + preloaded: pointData ?? null, + tilingMetadata: pointTilingMetadata, + metadataKnown, + }, + { experimentalOptimizations: wantsOptimized ? 'auto' : 'off' } + ); + if (resource) { + cachedResource = { signature, resource }; + pointsRenderResourceCacheRef.current.set(elem.key, cachedResource); + } else { + pointsRenderResourceCacheRef.current.delete(elem.key); + } + } + if (cachedResource?.resource) { + if ( + config.featureCodes !== undefined && + !pointsRowFeatureCodesRef.current.has(preloadCacheKey) + ) { + ensurePointsRowFeatureCodes( + preloadCacheKey, + elem.key, + elem.element as PointsElement, + config.pointsMemoryCap + ); + } + const supportsViewportTiles = + cachedResource.resource.loader.capabilities.supportsViewportTiles; + const tileDebugStore = supportsViewportTiles ? getTileDebugStore(layerId) : undefined; const layer = renderPointsLayer({ - element: elem.element as PointsElement, + resource: cachedResource.resource, id: layerId, modelMatrix: elem.transform, opacity: config.opacity, visible: config.visible, pointSize: config.pointSize, + pointRadiusMinPixels: config.pointRadiusMinPixels, + pointRadiusMaxPixels: config.pointRadiusMaxPixels, + pointMinSizeScale: config.pointMinSizeScale, + viewZoom, color: config.color, - pointData, + featureCodes: config.featureCodes, + preloadedFeatureCodes: pointsRowFeatureCodesRef.current.get(preloadCacheKey), + renderCap: resolvePointsRenderCap(config.pointsRenderCap), + showTileDebugOverlay: config.showTileDebugOverlay ?? true, + tileDebugStore, + tileDebugSignature: tileDebugStore + ? tileDebugEntriesSignature(tileDebugStore.getState().tileDebugEntries) + : undefined, }); if (layer) deckLayers.push(layer); } @@ -1299,7 +1851,18 @@ export function useLayerData( } return deckLayers; - }, [layers, layerOrder, getStableSelections]); + }, [ + layers, + layerOrder, + getStableSelections, + viewZoom, + getTileDebugStore, + experimentalOptimizations, + loadedDataRevision, + pointsTileLayersRevision, + pointsRowFeatureCodesRevision, + ensurePointsRowFeatureCodes, + ]); const getImageLayerLoadedData = useCallback((layerId: string): ImageLoaderData | undefined => { const elem = resolveLayerElement(layerId, layersRef.current[layerId], elementMap.current); @@ -1313,10 +1876,39 @@ export function useLayerData( return loadedDataRef.current.labels.get(elem.key); }, []); + const getPointsLayerLoadedData = useCallback((layerId: string): PointData | undefined => { + const config = layersRef.current[layerId]; + const elem = resolveLayerElement(layerId, config, elementMap.current); + if (!elem || elem.type !== 'points') return undefined; + const preloadCacheKey = + config?.type === 'points' ? pointsPreloadCacheKey(elem.key, config) : elem.key; + return resolvePointsPreloadData(loadedDataRef.current.points, elem.key, preloadCacheKey); + }, []); + + const getShapesLayerLoadedData = useCallback( + (layerId: string): ShapesLayerLoadedSummary | undefined => { + const elem = resolveLayerElement(layerId, layersRef.current[layerId], elementMap.current); + if (!elem || elem.type !== 'shapes') return undefined; + const loaded = loadedDataRef.current.shapes.get(elem.key); + if (!loaded?.renderData) return undefined; + return { + geometryKind: loaded.renderData.geometryKind, + featureCount: loaded.renderData.featureIds.length, + }; + }, + [] + ); + const getLayerLoadState = useCallback( (layerId?: string): LayerLoadState | undefined => { if (layerId === undefined) return undefined; - return layerLoadStates[layerId]; + const state = layerLoadStates[layerId]; + if (!state) return undefined; + const cachedDuration = geometryLoadDurationRef.current.get(layerId); + if (cachedDuration !== undefined && state.geometryLoadDurationMs === undefined) { + return { ...state, geometryLoadDurationMs: cachedDuration }; + } + return state; }, [layerLoadStates] ); @@ -1376,6 +1968,17 @@ export function useLayerData( ); } + if (elem.type === 'points') { + if (isPointsTileDebugPickObject(pickInfo.object)) { + const progress = pointsTileLoadProgressFromStore( + pointsTileDebugStoreRef.current.get(layerId) + ); + const tooltip = formatPointsTileDebugTooltip(pickInfo.object.entry, progress); + return attachTooltipElementContext(tooltip, elementContext); + } + return undefined; + } + if (!isShapesAvailableElement(elem)) { return undefined; } @@ -1558,13 +2161,26 @@ export function useLayerData( return vivProps; }, [layers, layerOrder, getStableSelections]); - const isLoading = useMemo( - () => - Object.values(layerLoadStates).some((state) => - Object.values(state).some((status) => status === 'loading') - ), - [layerLoadStates] - ); + const isLoading = useMemo(() => { + const resourceLoading = Object.values(layerLoadStates).some((state) => + Object.values(state).some((status) => status === 'loading') + ); + if (resourceLoading) { + return true; + } + void pointsTileLayersRevision; + for (const layerId of layerOrder) { + const config = layers[layerId]; + if (!config?.visible || config.type !== 'points') continue; + const progress = pointsTileLoadProgressFromStore( + pointsTileDebugStoreRef.current.get(layerId) + ); + if (isPointsTileLoading(progress)) { + return true; + } + } + return false; + }, [layerLoadStates, layerOrder, layers, pointsTileLayersRevision]); const isBlocking = useMemo( () => @@ -1587,11 +2203,63 @@ export function useLayerData( [layerLoadStates, layerOrder, layers, hasRenderableLayerData] ); + const getPointsLayerSupportsTileDebug = useCallback((layerId: string): boolean => { + const elem = resolveLayerElement(layerId, layersRef.current[layerId], elementMap.current); + if (!elem || elem.type !== 'points') { + return false; + } + const cached = pointsRenderResourceCacheRef.current.get(elem.key); + return cached?.resource?.loader.capabilities.supportsViewportTiles ?? false; + }, []); + + const getPointsFeatureCatalog = useCallback( + (layerId: string): PointsFeatureCatalog | null | undefined => { + void pointsFeatureCatalogRevision; + const elem = resolveLayerElement(layerId, layersRef.current[layerId], elementMap.current); + if (!elem || elem.type !== 'points') { + return undefined; + } + if (!pointsFeatureCatalogRef.current.has(elem.key)) { + return undefined; + } + return pointsFeatureCatalogRef.current.get(elem.key) ?? null; + }, + [pointsFeatureCatalogRevision] + ); + + const isPointsFeatureCatalogLoading = useCallback( + (layerId: string): boolean => { + void pointsFeatureCatalogRevision; + const elem = resolveLayerElement(layerId, layersRef.current[layerId], elementMap.current); + if (!elem || elem.type !== 'points') { + return false; + } + return pointsFeatureCatalogInFlightRef.current.has(elem.key); + }, + [pointsFeatureCatalogRevision] + ); + + const getOverlayStatusMessage = useCallback((): string | null => { + const tileMessage = getPointsTileLoadingMessage(); + if (tileMessage) { + return tileMessage; + } + const resourceLoading = Object.values(layerLoadStates).some((state) => + Object.values(state).some((status) => status === 'loading') + ); + if (resourceLoading) { + return 'Refreshing layer metadata…'; + } + return null; + }, [getPointsTileLoadingMessage, layerLoadStates]); + return { getLayers, getVivLayerProps, getImageLayerLoadedData, getLabelsLayerLoadedData, + getPointsLayerLoadedData, + getShapesLayerLoadedData, getLayerLoadState, hasRenderableLayerData, getFeatureTooltip, @@ -1599,6 +2267,13 @@ export function useLayerData( getShapePickEvent, isLoading, isBlocking, + getPointsTileLoadProgress, + getPointsTileLoadingMessage, + getOverlayStatusMessage, + getPointsLayerSupportsTileDebug, + getPointsFeatureCatalog, + isPointsFeatureCatalogLoading, + requestPointsFeatureCatalog, reloadElement, getWorldBoundsForLayer, getWorldBoundsForVisibleLayers, diff --git a/packages/vis/tests/formatLoadDuration.spec.ts b/packages/vis/tests/formatLoadDuration.spec.ts new file mode 100644 index 00000000..2f2ff997 --- /dev/null +++ b/packages/vis/tests/formatLoadDuration.spec.ts @@ -0,0 +1,17 @@ +import { describe, expect, it } from 'vitest'; + +import { formatLoadDurationMs } from '../src/SpatialCanvas/useLayerData.js'; + +describe('formatLoadDurationMs', () => { + it('formats sub-second durations in milliseconds', () => { + expect(formatLoadDurationMs(850)).toBe('850 ms'); + }); + + it('formats seconds with one decimal place', () => { + expect(formatLoadDurationMs(1234)).toBe('1.2 s'); + }); + + it('formats long durations as whole seconds', () => { + expect(formatLoadDurationMs(12_500)).toBe('13 s'); + }); +}); diff --git a/packages/vis/tests/pointsLoadPlan.spec.ts b/packages/vis/tests/pointsLoadPlan.spec.ts new file mode 100644 index 00000000..6a43f835 --- /dev/null +++ b/packages/vis/tests/pointsLoadPlan.spec.ts @@ -0,0 +1,184 @@ +import { describe, expect, it } from 'vitest'; + +import { + planPointsLoads, + pointsPreloadCacheKey, + shouldLoadPointsRowFeatureCodes, + shouldPreloadAfterMetadataProbe, +} from '../src/SpatialCanvas/pointsLoadPlan.js'; + +describe('planPointsLoads', () => { + it('schedules metadata probe only when optimized and metadata unknown', () => { + expect( + planPointsLoads({ + wantsOptimized: true, + metadataKnown: false, + tiledMetadata: undefined, + hasPreloaded: false, + }) + ).toEqual({ probeMetadata: true, preloadFullTable: false }); + }); + + it('schedules preload when metadata known and non-tileable', () => { + expect( + planPointsLoads({ + wantsOptimized: true, + metadataKnown: true, + tiledMetadata: null, + hasPreloaded: false, + }) + ).toEqual({ probeMetadata: false, preloadFullTable: true }); + }); + + it('schedules preload immediately when optimizations off', () => { + expect( + planPointsLoads({ + wantsOptimized: false, + metadataKnown: false, + tiledMetadata: undefined, + hasPreloaded: false, + }) + ).toEqual({ probeMetadata: false, preloadFullTable: true }); + }); + + it('schedules probe only when Morton metadata is present', () => { + const tiledMetadata = { + supportsRowGroupRangeReads: true, + bounds: { minX: 0, minY: 0, maxX: 1, maxY: 1 }, + }; + expect( + planPointsLoads({ + wantsOptimized: true, + metadataKnown: true, + tiledMetadata, + hasPreloaded: false, + }) + ).toEqual({ probeMetadata: false, preloadFullTable: false }); + }); +}); + +describe('pointsPreloadCacheKey', () => { + it('includes memory cap only (feature filter is runtime)', () => { + expect( + pointsPreloadCacheKey('points:transcripts', { + pointsMemoryCap: 1_000_000, + }) + ).toBe('points:transcripts|m1000000'); + }); + + it('uses default memory cap when unset', () => { + expect(pointsPreloadCacheKey('points:transcripts', {})).toMatch(/m\d+$/); + }); +}); + +describe('shouldPreloadAfterMetadataProbe', () => { + it('requires preload after non-tileable probe result', () => { + expect( + shouldPreloadAfterMetadataProbe({ + probeRan: true, + renderableMetadata: false, + hasPreloaded: false, + }) + ).toBe(true); + }); + + it('skips preload when probe found renderable Morton metadata', () => { + expect( + shouldPreloadAfterMetadataProbe({ + probeRan: true, + renderableMetadata: true, + hasPreloaded: false, + }) + ).toBe(false); + }); + + it('skips preload when data already cached', () => { + expect( + shouldPreloadAfterMetadataProbe({ + probeRan: true, + renderableMetadata: false, + hasPreloaded: true, + }) + ).toBe(false); + }); + + it('skips preload when no probe ran', () => { + expect( + shouldPreloadAfterMetadataProbe({ + probeRan: false, + renderableMetadata: false, + hasPreloaded: false, + }) + ).toBe(false); + }); + + it('still requires preload after probe when row count exceeds the cap', () => { + expect( + shouldPreloadAfterMetadataProbe({ + probeRan: true, + renderableMetadata: false, + hasPreloaded: false, + totalRows: 4_000_001, + }) + ).toBe(true); + }); +}); + +describe('shouldLoadPointsRowFeatureCodes', () => { + it('does not load row codes before preloaded points exist', () => { + expect( + shouldLoadPointsRowFeatureCodes({ + hasPreloaded: false, + hasCached: false, + inFlight: false, + featureCodes: [1], + }) + ).toBe(false); + }); + + it('loads row codes once preload is ready, regardless of filter state', () => { + expect( + shouldLoadPointsRowFeatureCodes({ + hasPreloaded: true, + hasCached: false, + inFlight: false, + featureCodes: undefined, + }) + ).toBe(true); + expect( + shouldLoadPointsRowFeatureCodes({ + hasPreloaded: true, + hasCached: false, + inFlight: false, + featureCodes: [], + }) + ).toBe(true); + expect( + shouldLoadPointsRowFeatureCodes({ + hasPreloaded: true, + hasCached: false, + inFlight: false, + featureCodes: [1, 2], + }) + ).toBe(true); + }); + + it('skips row codes when cached or already in flight', () => { + expect( + shouldLoadPointsRowFeatureCodes({ + hasPreloaded: true, + hasCached: true, + inFlight: false, + featureCodes: [1, 2], + }) + ).toBe(false); + expect( + shouldLoadPointsRowFeatureCodes({ + hasPreloaded: true, + hasCached: false, + inFlight: true, + featureCodes: [1, 2], + }) + ).toBe(false); + }); +}); diff --git a/packages/vis/tests/pointsRenderer.spec.ts b/packages/vis/tests/pointsRenderer.spec.ts new file mode 100644 index 00000000..f0eae5b5 --- /dev/null +++ b/packages/vis/tests/pointsRenderer.spec.ts @@ -0,0 +1,30 @@ +import { describe, expect, it } from 'vitest'; + +import { + MIN_POINT_SIZE_SCALE, + POINT_SIZE_ZOOM_REFERENCE, + zoomScaledPointSize, +} from '@spatialdata/layers'; + +describe('zoomScaledPointSize', () => { + it('returns base size at the reference zoom', () => { + expect(zoomScaledPointSize(4, POINT_SIZE_ZOOM_REFERENCE)).toBe(4); + }); + + it('shrinks points when zoomed out', () => { + expect(zoomScaledPointSize(4, -2)).toBe(1); + }); + + it('does not grow beyond the configured size when zoomed in', () => { + expect(zoomScaledPointSize(4, 4)).toBe(4); + }); + + it('clamps to the minimum scale when zoomed far out', () => { + expect(zoomScaledPointSize(4, -10)).toBe(4 * MIN_POINT_SIZE_SCALE); + }); + + it('returns base size when zoom is unavailable', () => { + expect(zoomScaledPointSize(3, null)).toBe(3); + expect(zoomScaledPointSize(3, undefined)).toBe(3); + }); +}); diff --git a/packages/vis/tests/pointsTileProgress.spec.ts b/packages/vis/tests/pointsTileProgress.spec.ts new file mode 100644 index 00000000..e4b5cace --- /dev/null +++ b/packages/vis/tests/pointsTileProgress.spec.ts @@ -0,0 +1,130 @@ +import type { PointsTileHandle, TiledPointsDebugState } from '@spatialdata/layers'; +import { describe, expect, it } from 'vitest'; + +import { + aggregatePointsTileLoadProgress, + isPointsTileLoading, + pointsTileLoadProgressFromDebugState, + pointsTileLoadingMessage, +} from '../src/SpatialCanvas/pointsTileProgress.js'; + +const sampleTile: PointsTileHandle = { + tileId: '0-0--1', + index: { x: 0, y: 0, z: -1 }, + bbox: { left: 0, top: 512, right: 512, bottom: 0 }, +}; + +const otherTile: PointsTileHandle = { + tileId: '1-0--1', + index: { x: 1, y: 0, z: -1 }, + bbox: { left: 512, top: 512, right: 1024, bottom: 0 }, +}; + +function debugState(overrides: Partial): TiledPointsDebugState { + return { + tileDebugEntries: [], + completedTilesById: {}, + loadingTileIds: [], + tileHandlesById: {}, + ...overrides, + }; +} + +describe('pointsTileProgress', () => { + it('aggregates progress across layers', () => { + const aggregate = aggregatePointsTileLoadProgress( + new Map([ + ['a', { inFlight: 2, loaded: 1, loadedPoints: 100, viewportTotal: 4 }], + ['b', { inFlight: 1, loaded: 3, loadedPoints: 250, viewportTotal: 6 }], + ]) + ); + expect(aggregate).toEqual({ + inFlight: 3, + loaded: 4, + loadedPoints: 350, + viewportTotal: 10, + }); + }); + + it('reports loading while tiles are in flight', () => { + expect( + pointsTileLoadingMessage({ inFlight: 2, loaded: 1, loadedPoints: 42, viewportTotal: 6 }) + ).toBe('Loading points… (1/6 tiles, 42 points)'); + expect( + isPointsTileLoading({ inFlight: 2, loaded: 1, loadedPoints: 42, viewportTotal: 6 }) + ).toBe(true); + }); + + it('includes zero loaded points while later tiles are still loading', () => { + expect( + pointsTileLoadingMessage({ inFlight: 1, loaded: 1, loadedPoints: 0, viewportTotal: 2 }) + ).toBe('Loading points… (1/2 tiles, 0 points)'); + }); + + it('clears stale viewport messages when nothing is in flight', () => { + expect( + pointsTileLoadingMessage({ inFlight: 0, loaded: 0, loadedPoints: 0, viewportTotal: 4 }) + ).toBeNull(); + expect( + pointsTileLoadingMessage({ inFlight: 0, loaded: 4, loadedPoints: 900, viewportTotal: 4 }) + ).toBeNull(); + }); + + it('derives loaded and point totals from current viewport debug state', () => { + const progress = pointsTileLoadProgressFromDebugState( + debugState({ + lastViewportTiles: [sampleTile, otherTile], + loadingTileIds: [otherTile.tileId], + completedTilesById: { + [sampleTile.tileId]: { + status: 'loaded', + pointCount: 10, + clippedBounds: null, + completedAt: 10, + }, + }, + }) + ); + expect(progress).toEqual({ inFlight: 1, loaded: 1, loadedPoints: 10, viewportTotal: 2 }); + }); + + it('counts cached empty tiles as loaded after viewport refresh', () => { + const progress = pointsTileLoadProgressFromDebugState( + debugState({ + lastViewportTiles: [sampleTile], + completedTilesById: { + [sampleTile.tileId]: { + status: 'empty', + pointCount: 0, + clippedBounds: null, + completedAt: 10, + }, + }, + }) + ); + expect(progress).toEqual({ inFlight: 0, loaded: 1, loadedPoints: 0, viewportTotal: 1 }); + }); + + it('does not let stale completed tiles inflate the current viewport total', () => { + const progress = pointsTileLoadProgressFromDebugState( + debugState({ + lastViewportTiles: [sampleTile], + completedTilesById: { + [sampleTile.tileId]: { + status: 'loaded', + pointCount: 10, + clippedBounds: null, + completedAt: 10, + }, + [otherTile.tileId]: { + status: 'loaded', + pointCount: 20, + clippedBounds: null, + completedAt: 10, + }, + }, + }) + ); + expect(progress).toEqual({ inFlight: 0, loaded: 1, loadedPoints: 10, viewportTotal: 1 }); + }); +}); diff --git a/packages/vis/tests/resolvePointsRenderResource.spec.ts b/packages/vis/tests/resolvePointsRenderResource.spec.ts new file mode 100644 index 00000000..38203797 --- /dev/null +++ b/packages/vis/tests/resolvePointsRenderResource.spec.ts @@ -0,0 +1,34 @@ +import { describe, expect, it } from 'vitest'; + +import { pointsRenderResourceSignature } from '../src/SpatialCanvas/resolvePointsRenderResource.js'; + +describe('pointsRenderResourceSignature', () => { + it('changes when preload or metadata inputs change', () => { + const element = { key: 'transcripts' } as { key: string }; + const base = pointsRenderResourceSignature( + element as never, + { metadataKnown: true, tilingMetadata: null, preloaded: null }, + { experimentalOptimizations: 'auto' } + ); + const withPreload = pointsRenderResourceSignature( + element as never, + { + metadataKnown: true, + tilingMetadata: null, + preloaded: { shape: [2, 100], data: [new Float32Array(100), new Float32Array(100)] }, + }, + { experimentalOptimizations: 'auto', preloadCacheKey: 'points:transcripts|m4000000|fall' } + ); + const withMoreRows = pointsRenderResourceSignature( + element as never, + { + metadataKnown: true, + tilingMetadata: null, + preloaded: { shape: [2, 200], data: [new Float32Array(200), new Float32Array(200)] }, + }, + { experimentalOptimizations: 'auto', preloadCacheKey: 'points:transcripts|m4000000|fall' } + ); + expect(base).not.toEqual(withPreload); + expect(withPreload).not.toEqual(withMoreRows); + }); +}); diff --git a/packages/vis/tests/shapesStylePanel.spec.ts b/packages/vis/tests/shapesStylePanel.spec.ts new file mode 100644 index 00000000..9d989517 --- /dev/null +++ b/packages/vis/tests/shapesStylePanel.spec.ts @@ -0,0 +1,11 @@ +import { describe, expect, it } from 'vitest'; + +import { formatShapesGeometryKindLabel } from '../src/SpatialCanvas/ShapesStylePanel.js'; + +describe('formatShapesGeometryKindLabel', () => { + it('maps geometry kinds to display labels', () => { + expect(formatShapesGeometryKindLabel('polygon')).toBe('polygons'); + expect(formatShapesGeometryKindLabel('circle')).toBe('circles'); + expect(formatShapesGeometryKindLabel('point')).toBe('points'); + }); +}); diff --git a/packages/vis/vite.config.demo.ts b/packages/vis/vite.config.demo.ts index 6c5db8e9..8f5f19e0 100644 --- a/packages/vis/vite.config.demo.ts +++ b/packages/vis/vite.config.demo.ts @@ -25,6 +25,9 @@ export default defineConfig({ worker: { format: 'es', }, + optimizeDeps: { + exclude: ['zarrextra/workers'], + }, assetsInclude: ['**/*.wasm'], server: { host: '127.0.0.1', diff --git a/packages/zarrextra/README.md b/packages/zarrextra/README.md index 8317b608..072eedb5 100644 --- a/packages/zarrextra/README.md +++ b/packages/zarrextra/README.md @@ -117,6 +117,13 @@ required for that path. | Node / CI | `registerJpeg2kCodec()` / `registerExperimentalHtj2kCodec()` on the main thread | | Browser | `enableWorkerChunkDecode()` from `zarrextra/workers` before loading JP2K or HTJ2K data | +### Vite apps + +See the [Browser workers guide](https://taylor-ccb-group.github.io/SpatialData.js/docs/vis/browser-workers) in the main docs for the full integration contract (call-once semantics, default worker URLs, and bundler config). In short: + +- `optimizeDeps: { exclude: ['zarrextra/workers'] }` in `vite.config.ts` +- After upgrading `zarrextra`, delete `node_modules/.vite` + Optional dependencies: `@fideus-labs/fizarrita`, `@fideus-labs/worker-pool`, `@cornerstonejs/codec-openjpeg`, and `@cornerstonejs/codec-openjph` (bundled into the default worker script). Future worker entries may let applications opt into diff --git a/packages/zarrextra/src/result.ts b/packages/zarrextra/src/result.ts index 1b9d96b0..347c18ce 100644 --- a/packages/zarrextra/src/result.ts +++ b/packages/zarrextra/src/result.ts @@ -1,13 +1,10 @@ /** - * Result type for explicit error handling without exceptions. + * Minimal Result type for explicit error handling without exceptions. * Inspired by Rust's Result. * - * This type is useful for operations that can fail, especially in zarr operations - * where errors should be handled explicitly rather than thrown as exceptions. - * - * Note: This is a custom implementation for simplicity. We may review using - * an existing Result library (such as neverthrow) in the future, - * but for now this provides a lightweight, dependency-free solution. + * Adoption in this monorepo is intentionally narrow. If we expand Result use, + * we would likely adopt an established library (e.g. neverthrow) rather than + * grow this in-house API — treat these helpers as provisional. */ /** diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 784796f6..b55a2f77 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -48,9 +48,6 @@ catalogs: jsdom: specifier: ^27.4.0 version: 27.4.0 - parquet-wasm: - specifier: ^0.6.1 - version: 0.6.1 react: specifier: ^19.2.1 version: 19.2.1 @@ -131,9 +128,6 @@ importers: clsx: specifier: ^2.0.0 version: 2.1.1 - parquet-wasm: - specifier: 'catalog:' - version: 0.6.1 prism-react-renderer: specifier: ^2.3.0 version: 2.4.1(react@19.2.1) @@ -226,9 +220,6 @@ importers: ol: specifier: ^10.6.1 version: 10.6.1 - parquet-wasm: - specifier: 'catalog:' - version: 0.6.1 zarrextra: specifier: workspace:* version: link:../zarrextra @@ -269,6 +260,9 @@ importers: '@math.gl/core': specifier: 'catalog:' version: 4.1.0 + '@spatialdata/core': + specifier: workspace:* + version: link:../core zod: specifier: 'catalog:' version: 4.1.13 @@ -6061,9 +6055,6 @@ packages: resolution: {integrity: sha512-GQ2EWRpQV8/o+Aw8YqtfZZPfNRWZYkbidE9k5rpl/hC3vtHHBfGm2Ifi6qWV+coDGkrUKZAxE3Lot5kcsRlh+g==} engines: {node: '>=6'} - parquet-wasm@0.6.1: - resolution: {integrity: sha512-wTM/9Y4EHny8i0qgcOlL9UHsTXftowwCqDsAD8axaZbHp0Opp3ue8oxexbzTVNhqBjFhyhLiU3MT0rnEYnYU0Q==} - parse-entities@4.0.2: resolution: {integrity: sha512-GG2AQYWoLgL877gQIKeRPGO1xF9+eG1ujIb5soS5gPvLQ1y2o8FL90w2QWNdf9I361Mpp7726c+lj3U0qK1uGw==} @@ -15191,8 +15182,6 @@ snapshots: dependencies: callsites: 3.1.0 - parquet-wasm@0.6.1: {} - parse-entities@4.0.2: dependencies: '@types/unist': 2.0.11 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 909ec003..1d46dab9 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -16,7 +16,6 @@ catalog: anndata.js: ^0.0.2 apache-arrow: ^17.0.0 deck.gl: ~9.2.9 - parquet-wasm: ^0.6.1 react: ^19.2.1 react-dom: ^19.2.1 typescript: ^5.7.3 diff --git a/python/spatialdata-experimental-writer/README.md b/python/spatialdata-experimental-writer/README.md new file mode 100644 index 00000000..f7aef368 --- /dev/null +++ b/python/spatialdata-experimental-writer/README.md @@ -0,0 +1,132 @@ +# spatialdata-experimental-writer + +Experimental vector optimization writers for browser-oriented SpatialData +rendering. + +The initial writer targets Vitessce-compatible Morton-sorted Points Parquet: + +- `x`, `y`, optional `z` coordinates are preserved. +- `morton_code_2d` is added using 16 bits per axis. +- the first 2–4 rows are sentinel/extreme rows with `morton_code_2d == 0`; + readers can infer the full point bounding box from these rows. +- `{feature_key}_codes` (for example `feature_name_codes`) are added when + `feature_key` is set in element attrs. +- string/categorical columns are placed at the right side of the table. +- row-group size is controlled when writing Parquet. +- intermediate Morton uint columns are not persisted in the output Parquet. + +Morton v1 belongs on the **canonical** element path +`points//points.parquet`. Use `--experimental` only for layouts that +standard readers cannot consume (see +[ADR 0002](../../docs/adr/0002-spatially-aware-vector-loading.md)). + +Feature / gene filtering in the browser is documented in ADR 0002; pass integer +`featureCodes` through `@spatialdata/core` `loadPointsInBounds()` and +`PointsLayerConfig.featureCodes` in `@spatialdata/vis`. + +## Install + +```bash +cd python/spatialdata-experimental-writer +uv sync +``` + +For the interactive TUI: + +```bash +uv sync --group tui +``` + +## Interactive TUI + +```bash +uv run spatialdata-experimental-writer tui +uv run spatialdata-experimental-writer tui ~/data/xenium_rep1_io.zarr +``` + +The TUI wraps all writer commands: + +1. Pick a command from the home menu. +2. Enter paths and options on guided forms (Zarr store path is pre-filled when + passed on the command line). +3. Confirm before any in-place overwrite of canonical `points//points.parquet`. +4. Watch run output, then review post-write verification checks. + +Morton verification checks after Morton writes: + +| Check | Meaning | +|-------|---------| +| `column_present` | `morton_code_2d` column exists | +| `sentinel_prefix` | First 2–4 rows have `morton_code_2d == 0` | +| `sentinel_bbox` | Sentinel rows encode full dataset x/y bounds | +| `morton_monotonic` | Morton codes non-decreasing after sentinels | +| `row_group_sentinels` | Row group 0 contains only sentinel rows | +| `no_uint_intermediates` | No persisted `*_uint` staging columns | + +Multiscale and index-permutation runs show schema/manifest checks instead. + +## Commands + +```bash +# List Points elements in a store +uv run spatialdata-experimental-writer list-points ~/data/xenium.zarr + +# Morton-sort transcripts in-place on canonical points//points.parquet +uv run spatialdata-experimental-writer morton-points-from-zarr \ + ~/data/xenium.zarr --points-key transcripts + +# Optional: write to points.experimental/ instead of canonical path +uv run spatialdata-experimental-writer morton-points-from-zarr \ + ~/data/xenium.zarr --points-key transcripts --experimental + +# Build a derivative store with transcript index sort permutations +uv run spatialdata-experimental-writer write-index-permutations \ + ~/data/xenium_rep1_io.zarr \ + ~/data/xenium_rep1_index-permutations.zarr + +# Morton-sort a CSV or Parquet file +uv run spatialdata-experimental-writer morton-points input.csv output.parquet \ + --feature-key feature_name +``` + +## Xenium workflow + +Standard sandbox datasets are listed in the +[spatialdata datasets docs](https://spatialdata.scverse.org/en/stable/tutorials/notebooks/datasets/README.html): + +| Dataset | URL | +|---------|-----| +| `xenium_rep1_io.zarr` | `https://s3.embl.de/spatialdata/spatialdata-sandbox/xenium_rep1_io.zarr/` | +| `xenium_rep2_io.zarr` | `https://s3.embl.de/spatialdata/spatialdata-sandbox/xenium_rep2_io.zarr/` | +| `visium_associated_xenium_io.zarr` | `https://s3.embl.de/spatialdata/spatialdata-sandbox/visium_associated_xenium_io.zarr/` | + +After downloading a store locally: + +```bash +uv run spatialdata-experimental-writer morton-points-from-zarr \ + ~/data/spatialdata/sdata_inputs/xenium_rep1_io.zarr \ + --points-key transcripts +``` + +This replaces `points/transcripts/points.parquet` in place (single-file output; +multipart source directories are replaced). Open the store in `@spatialdata/vis` +with `experimentalOptimizations="auto"` to use TileLayer row-group reads. + +For sort-strategy benchmarks on a **copy** of the store: + +```bash +uv run spatialdata-experimental-writer write-index-permutations \ + ~/data/spatialdata/sdata_inputs/xenium_rep1_io.zarr \ + ~/data/spatialdata/sdata_inputs/xenium_rep1_index-permutations.zarr \ + --max-rows 500000 + +uv run python scripts/benchmark_points_index.py \ + ~/data/spatialdata/sdata_inputs/xenium_rep1_index-permutations.zarr +``` + +## Multiscale hook + +The package also includes a Padua-style multiscale Parquet writer that stores +`spatialdata_multiscale` JSON metadata in the Parquet schema. That layout is +non-standard for morton-points v1 and belongs under `points.experimental/` if +persisted. diff --git a/python/spatialdata-experimental-writer/pyproject.toml b/python/spatialdata-experimental-writer/pyproject.toml new file mode 100644 index 00000000..d9303edb --- /dev/null +++ b/python/spatialdata-experimental-writer/pyproject.toml @@ -0,0 +1,37 @@ +[project] +name = "spatialdata-experimental-writer" +version = "0.1.0" +description = "Experimental SpatialData vector optimization writers" +requires-python = ">=3.12" +readme = "README.md" +license = { text = "MIT" } +authors = [{ name = "SpatialData.js contributors" }] +dependencies = [ + "numpy>=2.0", + "pandas>=2.2", + "pyarrow>=18", +] + +[project.scripts] +spatialdata-experimental-writer = "spatialdata_experimental_writer.cli:main" + +[build-system] +requires = ["setuptools>=68"] +build-backend = "setuptools.build_meta" + +[tool.setuptools.packages.find] +where = ["src"] + +[tool.uv] +package = true + +[tool.pytest.ini_options] +testpaths = ["tests"] + +[dependency-groups] +dev = [ + "pytest>=8.0", +] +tui = [ + "textual>=1.0", +] diff --git a/python/spatialdata-experimental-writer/scripts/benchmark_points_index.py b/python/spatialdata-experimental-writer/scripts/benchmark_points_index.py new file mode 100644 index 00000000..57b08aaa --- /dev/null +++ b/python/spatialdata-experimental-writer/scripts/benchmark_points_index.py @@ -0,0 +1,155 @@ +#!/usr/bin/env python3 +"""Benchmark points index permutations using index-manifest.json.""" + +from __future__ import annotations + +import argparse +import json +import time +from pathlib import Path + +import pandas as pd +import pyarrow.parquet as pq + + +def _load_bounds(manifest: dict, scenario_id: str | None) -> dict[str, float]: + scenarios = manifest.get("benchmark_scenarios") or [] + if scenario_id: + for scenario in scenarios: + if scenario.get("id") == scenario_id: + return scenario["bounds"] + raise SystemExit(f"Unknown scenario id: {scenario_id}") + if scenarios: + return scenarios[0]["bounds"] + raise SystemExit("Manifest has no benchmark_scenarios") + + +def _feature_codes(manifest: dict, scenario_id: str | None) -> list[int] | None: + scenarios = manifest.get("benchmark_scenarios") or [] + if not scenario_id: + return None + for scenario in scenarios: + if scenario.get("id") == scenario_id: + codes = scenario.get("feature_codes") + return list(codes) if codes is not None else None + return None + + +def _parquet_path(store: Path, element_path: str) -> Path: + return store / element_path / "points.parquet" + + +def _read_rows_in_bounds( + parquet_path: Path, + bounds: dict[str, float], + feature_codes: list[int] | None, + feature_key: str | None, +) -> tuple[int, int]: + if parquet_path.is_dir(): + parts = sorted(parquet_path.glob("part.*.parquet")) + if not parts: + raise FileNotFoundError(f"No parquet parts under {parquet_path}") + frames = [pd.read_parquet(part) for part in parts] + df = pd.concat(frames, ignore_index=True) + bytes_read = sum(part.stat().st_size for part in parts) + else: + bytes_read = parquet_path.stat().st_size + df = pd.read_parquet(parquet_path) + + mask = ( + (df["x"] >= bounds["minX"]) + & (df["x"] <= bounds["maxX"]) + & (df["y"] >= bounds["minY"]) + & (df["y"] <= bounds["maxY"]) + ) + if feature_codes is not None: + code_column = f"{feature_key}_codes" if feature_key else "feature_name_codes" + if code_column not in df.columns: + raise KeyError(f"Missing feature code column {code_column!r}") + mask &= df[code_column].isin(feature_codes) + return int(mask.sum()), int(bytes_read) + + +def _estimate_row_group_bytes(parquet_path: Path, bounds: dict[str, float]) -> int | None: + if not parquet_path.is_file(): + return None + if "morton_code_2d" not in pq.ParquetFile(parquet_path).schema_arrow.names: + return None + # Upper bound only: full file size when row-group APIs are unavailable in this script. + return parquet_path.stat().st_size + + +def benchmark_store( + store: Path, + *, + scenario_id: str | None, + conditions: list[str] | None, +) -> list[dict]: + manifest_path = store / "index-manifest.json" + if not manifest_path.exists(): + raise FileNotFoundError(f"Missing index-manifest.json under {store}") + manifest = json.loads(manifest_path.read_text()) + bounds = _load_bounds(manifest, scenario_id) + feature_codes = _feature_codes(manifest, scenario_id) + feature_key = manifest.get("feature_key") + selected = conditions or [entry["id"] for entry in manifest.get("conditions", [])] + + results: list[dict] = [] + for condition in manifest.get("conditions", []): + condition_id = condition["id"] + if condition_id not in selected: + continue + element_path = condition["element_path"] + parquet_path = _parquet_path(store, element_path) + started = time.perf_counter() + try: + rows, bytes_read = _read_rows_in_bounds( + parquet_path, bounds, feature_codes, feature_key + ) + row_group_hint = _estimate_row_group_bytes(parquet_path, bounds) + except Exception as error: # noqa: BLE001 - report per condition + results.append( + { + "condition": condition_id, + "element_path": element_path, + "error": str(error), + } + ) + continue + elapsed_ms = (time.perf_counter() - started) * 1000 + results.append( + { + "condition": condition_id, + "element_path": element_path, + "sort_order": condition.get("sort_order"), + "tiling_kind": condition.get("tiling_kind"), + "rows_in_bounds": rows, + "bytes_read_estimate": bytes_read, + "morton_row_group_bytes_upper_bound": row_group_hint, + "latency_ms": round(elapsed_ms, 2), + "bounds": bounds, + "feature_codes": feature_codes, + } + ) + return results + + +def main() -> None: + parser = argparse.ArgumentParser( + description="Benchmark points index permutations from index-manifest.json" + ) + parser.add_argument("store", type=Path, help="Derivative Zarr store path") + parser.add_argument("--scenario", metavar="ID", help="benchmark_scenarios id") + parser.add_argument( + "--conditions", + metavar="IDS", + help="Comma-separated condition ids (default: all in manifest)", + ) + args = parser.parse_args() + condition_ids = args.conditions.split(",") if args.conditions else None + results = benchmark_store(args.store, scenario_id=args.scenario, conditions=condition_ids) + print(json.dumps({"store": str(args.store), "results": results}, indent=2, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/__init__.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/__init__.py new file mode 100644 index 00000000..99eaa590 --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/__init__.py @@ -0,0 +1,17 @@ +from .points import ( + MORTON_CODE_2D_COLUMN, + MORTON_CODE_EXTREME_VALUE_INDICATOR, + build_spatialdata_multiscale_metadata, + morton_sort_points, + write_morton_points_parquet, + write_multiscale_points_parquet, +) + +__all__ = [ + "MORTON_CODE_2D_COLUMN", + "MORTON_CODE_EXTREME_VALUE_INDICATOR", + "build_spatialdata_multiscale_metadata", + "morton_sort_points", + "write_morton_points_parquet", + "write_multiscale_points_parquet", +] diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/cli.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/cli.py new file mode 100644 index 00000000..75347ece --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/cli.py @@ -0,0 +1,348 @@ +from __future__ import annotations + +import argparse +import json +from collections.abc import Callable +from typing import Any + +from .errors import WriterCommandError +from .runners import ( + run_list_points, + run_morton_points, + run_morton_points_from_zarr, + run_multiscale_points, + run_write_index_permutations, +) + +_EPILOG = """\ +examples: + # List Points elements in a SpatialData Zarr store + spatialdata-experimental-writer list-points ~/data/xenium.zarr + + # Morton-sort transcripts in-place on the canonical points element + spatialdata-experimental-writer morton-points-from-zarr \\ + ~/data/xenium.zarr --points-key transcripts + + # Build a derivative store with transcript index permutations + spatialdata-experimental-writer write-index-permutations \\ + ~/data/xenium_rep1_io.zarr ~/data/xenium_rep1_index-permutations.zarr + + # Morton-sort a CSV or single Parquet file + spatialdata-experimental-writer morton-points input.csv output.parquet \\ + --feature-key feature_name + + # Write multiscale Parquet with embedded spatialdata_multiscale metadata + spatialdata-experimental-writer multiscale-points input.parquet output.parquet + + # Interactive workflow TUI + uv sync --group tui + spatialdata-experimental-writer tui ~/data/xenium.zarr +""" + + +def _positive_int(value: str) -> int: + parsed = int(value) + if parsed <= 0: + raise argparse.ArgumentTypeError("value must be positive") + return parsed + + +def _print_json(payload: dict[str, Any]) -> None: + print(json.dumps(payload, indent=2, sort_keys=True)) + + +def _run_command(command: Callable[[], dict[str, Any]]) -> None: + try: + _print_json(command()) + except WriterCommandError as exc: + raise SystemExit(str(exc)) from exc + + +def _list_points(args: argparse.Namespace) -> None: + _run_command(lambda: run_list_points(args.zarr)) + + +def _morton_points(args: argparse.Namespace) -> None: + _run_command( + lambda: run_morton_points( + args.input, + args.output, + feature_key=args.feature_key, + row_group_size=args.row_group_size, + compression=args.compression, + ) + ) + + +def _multiscale_points(args: argparse.Namespace) -> None: + _run_command( + lambda: run_multiscale_points( + args.input, + args.output, + metadata_json=args.metadata_json, + row_group_size=args.row_group_size, + compression=args.compression, + ) + ) + + +def _morton_points_from_zarr(args: argparse.Namespace) -> None: + _run_command( + lambda: run_morton_points_from_zarr( + args.zarr, + points_key=args.points_key, + experimental=args.experimental, + output=args.output, + output_points_key=args.output_points_key, + feature_key=args.feature_key, + overwrite=args.overwrite, + row_group_size=args.row_group_size, + compression=args.compression, + ) + ) + + +def _write_index_permutations(args: argparse.Namespace) -> None: + condition_ids = args.conditions.split(",") if args.conditions else None + _run_command( + lambda: run_write_index_permutations( + args.source_zarr, + args.dest_zarr, + points_key=args.points_key, + max_rows=args.max_rows, + condition_ids=condition_ids, + overwrite=args.overwrite, + row_group_size=args.row_group_size, + compression=args.compression, + ) + ) + + +def _tui(args: argparse.Namespace) -> None: + try: + from .tui.app import run_tui + except ImportError as exc: + raise SystemExit( + "TUI dependencies are not installed. Run: uv sync --group tui" + ) from exc + run_tui(initial_zarr=args.zarr) + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser( + description=( + "Write browser-oriented SpatialData vector optimization artifacts " + "(Morton-sorted Points Parquet and multiscale metadata)." + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=_EPILOG, + ) + subparsers = parser.add_subparsers(dest="command", required=True, metavar="command") + + list_points = subparsers.add_parser( + "list-points", + help="list Points element keys in a SpatialData Zarr store", + description="List Points element keys under /points/.", + ) + list_points.add_argument( + "zarr", + metavar="ZARR", + help="Path to a SpatialData Zarr store (directory containing points/)", + ) + list_points.set_defaults(func=_list_points) + + morton_from_zarr = subparsers.add_parser( + "morton-points-from-zarr", + help="Morton-sort a Points element from a SpatialData Zarr store", + description=( + "Read points//points.parquet from a SpatialData Zarr store, " + "add morton_code_2d sentinel rows, and write Vitessce-compatible Parquet. " + "Defaults to in-place replacement of points//points.parquet." + ), + ) + morton_from_zarr.add_argument( + "zarr", + metavar="ZARR", + help="Path to a SpatialData Zarr store", + ) + morton_from_zarr.add_argument( + "--experimental", + action="store_true", + help="Write to points.experimental//points.parquet instead of canonical path", + ) + morton_from_zarr.add_argument( + "--points-key", + metavar="KEY", + help=( + "Points element name under points/ (for example transcripts). " + "Required when the store has more than one Points element." + ), + ) + morton_from_zarr.add_argument( + "--output", + metavar="PATH", + help=( + "Output Parquet path (default: in-place on points//points.parquet, " + "or points.experimental//points.parquet with --experimental)" + ), + ) + morton_from_zarr.add_argument( + "--output-points-key", + metavar="KEY", + help=( + "Output Points element name under points/ (for example transcripts_morton). " + "Cannot be combined with --output." + ), + ) + morton_from_zarr.add_argument( + "--overwrite", + action="store_true", + help="Allow overwriting an existing explicit output path or output Points element.", + ) + morton_from_zarr.add_argument( + "--feature-key", + metavar="COLUMN", + help=( + "Column used to derive _codes (default: spatialdata_attrs.feature_key " + "from the element zarr.json)" + ), + ) + morton_from_zarr.add_argument( + "--row-group-size", + type=_positive_int, + default=50_000, + metavar="N", + help="Target row-group size after sentinel rows (default: 50000)", + ) + morton_from_zarr.add_argument( + "--compression", + default="zstd", + help="Parquet compression codec (default: zstd)", + ) + morton_from_zarr.set_defaults(func=_morton_points_from_zarr) + + morton = subparsers.add_parser( + "morton-points", + help="Morton-sort points from CSV or Parquet", + description=( + "Sort x/y points by 2D Morton order, prepend sentinel bbox rows, " + "and write Vitessce-compatible Parquet." + ), + ) + morton.add_argument( + "input", + metavar="INPUT", + help="Input .csv, .parquet file, or directory of Parquet parts", + ) + morton.add_argument( + "output", + metavar="OUTPUT", + help="Output .parquet file", + ) + morton.add_argument( + "--feature-key", + metavar="COLUMN", + help="Column used to derive _codes for categorical features", + ) + morton.add_argument( + "--row-group-size", + type=_positive_int, + default=50_000, + metavar="N", + help="Target row-group size after sentinel rows (default: 50000)", + ) + morton.add_argument( + "--compression", + default="zstd", + help="Parquet compression codec (default: zstd)", + ) + morton.set_defaults(func=_morton_points) + + multiscale = subparsers.add_parser( + "multiscale-points", + help="write multiscale Points Parquet with spatialdata_multiscale metadata", + description=( + "Write Points Parquet with Padua-style spatialdata_multiscale JSON " + "stored in the file schema metadata." + ), + ) + multiscale.add_argument( + "input", + metavar="INPUT", + help="Input .csv, .parquet file, or directory of Parquet parts", + ) + multiscale.add_argument( + "output", + metavar="OUTPUT", + help="Output .parquet file", + ) + multiscale.add_argument( + "--metadata-json", + metavar="PATH", + help="Optional spatialdata_multiscale metadata JSON (default: inferred from input)", + ) + multiscale.add_argument( + "--row-group-size", + type=_positive_int, + default=50_000, + metavar="N", + help="Target row-group size (default: 50000)", + ) + multiscale.add_argument( + "--compression", + default="zstd", + help="Parquet compression codec (default: zstd)", + ) + multiscale.set_defaults(func=_multiscale_points) + + index_permutations = subparsers.add_parser( + "write-index-permutations", + help="write derivative Zarr with transcript index sort permutations", + description=( + "Copy a SpatialData Zarr store and add sibling points elements with " + "different transcript sort/index layouts plus index-manifest.json." + ), + ) + index_permutations.add_argument("source_zarr", metavar="SOURCE_ZARR") + index_permutations.add_argument("dest_zarr", metavar="DEST_ZARR") + index_permutations.add_argument("--points-key", metavar="KEY") + index_permutations.add_argument("--max-rows", type=_positive_int, metavar="N") + index_permutations.add_argument( + "--conditions", + metavar="IDS", + help="Comma-separated condition ids (default: all)", + ) + index_permutations.add_argument("--overwrite", action="store_true") + index_permutations.add_argument( + "--row-group-size", + type=_positive_int, + default=50_000, + metavar="N", + ) + index_permutations.add_argument("--compression", default="zstd") + index_permutations.set_defaults(func=_write_index_permutations) + + tui = subparsers.add_parser( + "tui", + help="interactive terminal workflow for writer commands", + description="Launch a Textual workflow UI for SpatialData experimental writer commands.", + ) + tui.add_argument( + "zarr", + nargs="?", + metavar="ZARR", + help="Optional SpatialData Zarr store path (skips initial store picker)", + ) + tui.set_defaults(func=_tui) + + return parser + + +def main(argv: list[str] | None = None) -> None: + parser = build_parser() + args = parser.parse_args(argv) + args.func(args) + + +if __name__ == "__main__": + main() diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/errors.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/errors.py new file mode 100644 index 00000000..6844e51d --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/errors.py @@ -0,0 +1,5 @@ +from __future__ import annotations + + +class WriterCommandError(RuntimeError): + """Expected user-facing failure from a writer command.""" diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/index_permutations.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/index_permutations.py new file mode 100644 index 00000000..9463044b --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/index_permutations.py @@ -0,0 +1,210 @@ +from __future__ import annotations + +import json +import shutil +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Sequence + +import pandas as pd + +from .points import MORTON_CODE_2D_COLUMN, write_morton_points_parquet +from .zarr import ( + list_points_keys, + points_parquet_path, + read_points_dataframe, + read_points_element_attrs, + register_points_elements_in_consolidated_metadata, +) + + +@dataclass(frozen=True) +class IndexCondition: + id: str + element_suffix: str + sort_order: tuple[str, ...] | None + tiling_kind: str | None + + +DEFAULT_CONDITIONS: tuple[IndexCondition, ...] = ( + IndexCondition("canonical", "", None, None), + IndexCondition("morton", "_morton", (MORTON_CODE_2D_COLUMN,), "morton-points"), + IndexCondition( + "morton-then-feature", + "_morton_then_feature", + (MORTON_CODE_2D_COLUMN, "feature_name_codes"), + "morton-points", + ), + IndexCondition( + "feature-then-morton", + "_feature_then_morton", + ("feature_name_codes", MORTON_CODE_2D_COLUMN), + "experimental", + ), +) + + +def _resolve_feature_code_column(feature_key: str | None) -> str: + if feature_key: + return f"{feature_key}_codes" + return "feature_name_codes" + + +def _condition_sort_order( + condition: IndexCondition, feature_key: str | None +) -> list[str] | None: + if condition.sort_order is None: + return None + feature_code_column = _resolve_feature_code_column(feature_key) + return [ + feature_code_column if column == "feature_name_codes" else column + for column in condition.sort_order + ] + + +def _copy_store_shell(source: Path, dest: Path, *, overwrite: bool) -> None: + if dest.exists(): + if not overwrite: + raise FileExistsError(f"Destination already exists: {dest}") + shutil.rmtree(dest) + + def ignore_points(directory: str, names: list[str]) -> set[str]: + if Path(directory) == source: + return {"points"} if "points" in names else set() + return set() + + shutil.copytree(source, dest, ignore=ignore_points) + + +def _write_element_zarr_json(source_element_dir: Path, dest_element_dir: Path) -> None: + source_json = source_element_dir / "zarr.json" + dest_element_dir.mkdir(parents=True, exist_ok=True) + shutil.copy2(source_json, dest_element_dir / "zarr.json") + + +def _copy_canonical_parquet(source_parquet: Path, dest_parquet: Path) -> None: + dest_parquet.parent.mkdir(parents=True, exist_ok=True) + if source_parquet.is_dir(): + if dest_parquet.exists(): + shutil.rmtree(dest_parquet) + shutil.copytree(source_parquet, dest_parquet) + else: + shutil.copy2(source_parquet, dest_parquet) + + +def write_index_permutations( + source_zarr: str | Path, + dest_zarr: str | Path, + *, + points_key: str | None = None, + max_rows: int | None = None, + conditions: Sequence[IndexCondition] | None = None, + overwrite: bool = False, + row_group_size: int = 50_000, + compression: str = "zstd", +) -> dict[str, Any]: + source_path = Path(source_zarr) + dest_path = Path(dest_zarr) + keys = list_points_keys(source_path) + if not keys: + raise FileNotFoundError(f"No Points elements found under {source_path / 'points'}") + + resolved_key = points_key or (keys[0] if len(keys) == 1 else None) + if resolved_key is None: + raise ValueError( + "Multiple Points elements found; pass points_key. " + f"Available keys: {', '.join(keys)}" + ) + if resolved_key not in keys: + raise ValueError(f"Unknown points key {resolved_key!r}. Available: {', '.join(keys)}") + + attrs = read_points_element_attrs(source_path, resolved_key) + feature_key = attrs.get("feature_key") + source_element_dir = source_path / "points" / resolved_key + source_parquet = points_parquet_path(source_path, resolved_key) + + _copy_store_shell(source_path, dest_path, overwrite=overwrite) + + df = read_points_dataframe(source_parquet) + if max_rows is not None and len(df) > max_rows: + df = df.sample(n=max_rows, random_state=0).reset_index(drop=True) + + selected = tuple(conditions or DEFAULT_CONDITIONS) + manifest_conditions: list[dict[str, Any]] = [] + + for condition in selected: + element_key = ( + resolved_key if condition.id == "canonical" else f"{resolved_key}{condition.element_suffix}" + ) + element_dir = dest_path / "points" / element_key + output_parquet = element_dir / "points.parquet" + _write_element_zarr_json(source_element_dir, element_dir) + + if condition.sort_order is None: + if max_rows is not None: + output_parquet.parent.mkdir(parents=True, exist_ok=True) + if output_parquet.exists(): + if output_parquet.is_dir(): + shutil.rmtree(output_parquet) + else: + output_parquet.unlink() + df.to_parquet(output_parquet, index=False) + else: + _copy_canonical_parquet(source_parquet, output_parquet) + else: + sort_order = _condition_sort_order(condition, feature_key) + write_morton_points_parquet( + df, + output_parquet, + feature_key=feature_key, + sort_order=sort_order, + row_group_size=row_group_size, + compression=compression, + ) + + manifest_conditions.append( + { + "id": condition.id, + "element_path": f"points/{element_key}", + "sort_order": list(condition.sort_order) if condition.sort_order else None, + "tiling_kind": condition.tiling_kind, + } + ) + + manifest = { + "version": "0.1", + "store_path": str(dest_path), + "source_store": str(source_path), + "source_element": f"points/{resolved_key}", + "feature_key": feature_key, + "n_points": int(len(df)), + "conditions": manifest_conditions, + "benchmark_scenarios": [ + { + "id": "center-tile", + "bounds": { + "minX": float(df["x"].quantile(0.25)), + "maxX": float(df["x"].quantile(0.75)), + "minY": float(df["y"].quantile(0.25)), + "maxY": float(df["y"].quantile(0.75)), + }, + } + ], + } + element_keys = [ + ( + resolved_key + if condition.id == "canonical" + else f"{resolved_key}{condition.element_suffix}" + ) + for condition in selected + ] + register_points_elements_in_consolidated_metadata( + dest_path, + element_keys, + template_key=resolved_key, + ) + + manifest_path = dest_path / "index-manifest.json" + manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n") + return manifest diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/points.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/points.py new file mode 100644 index 00000000..30515459 --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/points.py @@ -0,0 +1,239 @@ +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Sequence + +import numpy as np +import pandas as pd +import pyarrow as pa +import pyarrow.compute as pc +import pyarrow.parquet as pq + +MORTON_CODE_2D_COLUMN = "morton_code_2d" +MORTON_CODE_EXTREME_VALUE_INDICATOR = np.uint32(0) +MORTON_CODE_BITS_PER_AXIS = 16 +MORTON_CODE_VALUE_MAX = np.uint32((2**MORTON_CODE_BITS_PER_AXIS) - 1) +MORTON_SENTINEL_COUNT_ATTR = "spatialdata_experimental_morton_sentinel_count" + + +def _norm_series_to_uint(series: pd.Series, v_min: float, v_max: float) -> pd.Series: + if v_max == v_min: + return pd.Series(np.zeros(len(series), dtype=np.uint32), index=series.index) + normalized = (series.astype("float64") - v_min) / (v_max - v_min) + clipped = normalized.clip(0.0, 1.0).fillna(0.0) + return (clipped * int(MORTON_CODE_VALUE_MAX)).astype(np.uint32) + + +def _part1by1_16(values: np.ndarray) -> np.ndarray: + x = values.astype(np.uint32) & np.uint32(0x0000FFFF) + x = (x | np.left_shift(x, 8)) & np.uint32(0x00FF00FF) + x = (x | np.left_shift(x, 4)) & np.uint32(0x0F0F0F0F) + x = (x | np.left_shift(x, 2)) & np.uint32(0x33333333) + x = (x | np.left_shift(x, 1)) & np.uint32(0x55555555) + return x + + +def morton_code_2d(x_uint: pd.Series, y_uint: pd.Series) -> np.ndarray: + xs = _part1by1_16(x_uint.to_numpy(np.uint32)) + ys = _part1by1_16(y_uint.to_numpy(np.uint32)) + return (np.left_shift(ys.astype(np.uint64), 1) | xs.astype(np.uint64)).astype(np.uint32) + + +def _extreme_positions(df: pd.DataFrame) -> list[int]: + extreme_values = [ + ("x", df["x"].min()), + ("x", df["x"].max()), + ("y", df["y"].min()), + ("y", df["y"].max()), + ] + result: list[int] = [] + for column, value in extreme_values: + matches = np.flatnonzero((df[column] == value).to_numpy()) + if len(matches) == 0: + continue + position = int(matches[0]) + if position not in result: + result.append(position) + return result + + +def _append_feature_codes(df: pd.DataFrame, feature_key: str | None) -> pd.DataFrame: + if not feature_key or feature_key not in df.columns: + return df + code_column = f"{feature_key}_codes" + if code_column in df.columns: + return df + out = df.copy() + values = out[feature_key] + if isinstance(values.dtype, pd.CategoricalDtype): + out[code_column] = values.cat.codes.astype("int32") + else: + categories = pd.Categorical(values) + out[code_column] = categories.codes.astype("int32") + return out + + +def _move_string_like_columns_right(df: pd.DataFrame) -> pd.DataFrame: + string_like: list[str] = [] + other: list[str] = [] + for column in df.columns: + dtype = df[column].dtype + if isinstance(dtype, pd.CategoricalDtype) or pd.api.types.is_string_dtype(dtype): + string_like.append(column) + else: + other.append(column) + return df[[*other, *string_like]] + + +def morton_sort_points( + df: pd.DataFrame, + *, + feature_key: str | None = None, + sort_order: Sequence[str] | None = None, +) -> pd.DataFrame: + missing = [column for column in ("x", "y") if column not in df.columns] + if missing: + raise ValueError("Points dataframe is missing required columns: " + ", ".join(missing)) + + out = _append_feature_codes(df.copy(), feature_key) + x_min = float(out["x"].min()) + x_max = float(out["x"].max()) + y_min = float(out["y"].min()) + y_max = float(out["y"].max()) + x_uint = _norm_series_to_uint(out["x"], x_min, x_max) + y_uint = _norm_series_to_uint(out["y"], y_min, y_max) + out[MORTON_CODE_2D_COLUMN] = morton_code_2d(x_uint, y_uint) + + sentinel_positions = _extreme_positions(out) + sentinel = out.iloc[sentinel_positions].copy().reset_index(drop=True) + sentinel[MORTON_CODE_2D_COLUMN] = MORTON_CODE_EXTREME_VALUE_INDICATOR + + rest_mask = np.ones(len(out), dtype=bool) + rest_mask[sentinel_positions] = False + rest = out.iloc[rest_mask] + if sort_order is None: + sort_columns: list[str] = [MORTON_CODE_2D_COLUMN] + if "z" in rest.columns and rest["z"].nunique(dropna=False) < 100: + sort_columns = ["z", MORTON_CODE_2D_COLUMN] + else: + sort_columns = list(sort_order) + rest = rest.sort_values(sort_columns, kind="mergesort").reset_index(drop=True) + + combined = pd.concat([sentinel, rest], ignore_index=True) + combined = _move_string_like_columns_right(combined) + combined.attrs[MORTON_SENTINEL_COUNT_ATTR] = len(sentinel) + return combined + + +def _write_arrow_table_in_row_groups( + table: pa.Table, + output_path: Path, + *, + row_group_size: int, + sentinel_count: int | None = None, + metadata: dict[str, Any] | None = None, + compression: str = "zstd", +) -> None: + if row_group_size <= 0: + raise ValueError("row_group_size must be positive") + output_path.parent.mkdir(parents=True, exist_ok=True) + schema = table.schema + if metadata: + merged = dict(schema.metadata or {}) + merged[b"spatialdata_multiscale"] = json.dumps(metadata).encode() + schema = schema.with_metadata(merged) + + writer = pq.ParquetWriter(output_path, schema, compression=compression, write_statistics=True) + try: + if sentinel_count is None: + sentinel_count = 0 + if sentinel_count == 0 and MORTON_CODE_2D_COLUMN in table.column_names: + morton_column = table.column(MORTON_CODE_2D_COLUMN).combine_chunks() + for i in range(min(4, table.num_rows)): + if morton_column[i].as_py() != 0: + break + sentinel_count += 1 + if sentinel_count: + writer.write_table(table.slice(0, sentinel_count), row_group_size=sentinel_count) + for start in range(sentinel_count, table.num_rows, row_group_size): + chunk = table.slice(start, min(row_group_size, table.num_rows - start)) + writer.write_table(chunk, row_group_size=chunk.num_rows) + finally: + writer.close() + + +def write_morton_points_parquet( + df: pd.DataFrame, + output_path: str | Path, + *, + feature_key: str | None = None, + sort_order: Sequence[str] | None = None, + row_group_size: int = 50_000, + compression: str = "zstd", +) -> pd.DataFrame: + sorted_df = morton_sort_points(df, feature_key=feature_key, sort_order=sort_order) + indexed = sorted_df.copy() + indexed.index.name = "__index_level_0__" + table = pa.Table.from_pandas(indexed, preserve_index=True) + sentinel_count = sorted_df.attrs.get(MORTON_SENTINEL_COUNT_ATTR) + if not isinstance(sentinel_count, int): + sentinel_count = None + _write_arrow_table_in_row_groups( + table, + Path(output_path), + row_group_size=row_group_size, + sentinel_count=sentinel_count, + compression=compression, + ) + return sorted_df + + +def build_spatialdata_multiscale_metadata( + df: pd.DataFrame, + *, + axes: tuple[str, ...] = ("x", "y", "z"), + coordinate_space: str = "raw", + version: str = "1.0", + levels: list[dict[str, Any]] | None = None, + limit: int | None = None, +) -> dict[str, Any]: + available_axes = [axis for axis in axes if axis in df.columns] + if not available_axes: + raise ValueError("No requested coordinate axes are present in the dataframe.") + return { + "version": version, + "format": "spatialdata_multiscale_points", + "axes": available_axes, + "bounding_box": { + "min": [float(df[axis].min()) for axis in available_axes], + "max": [float(df[axis].max()) for axis in available_axes], + }, + "coordinate_space": coordinate_space, + "limit": limit, + "levels": levels or [], + "n_points_total": int(len(df)), + } + + +def write_multiscale_points_parquet( + df: pd.DataFrame, + output_path: str | Path, + *, + metadata: dict[str, Any], + row_group_size: int = 50_000, + compression: str = "zstd", +) -> None: + table = pa.Table.from_pandas(df, preserve_index=False) + if {"__spatial_index__", "__morton__"}.issubset(df.columns): + sort_keys = [("__spatial_index__", "ascending"), ("__morton__", "ascending")] + if "gene" in df.columns: + sort_keys.insert(0, ("gene", "ascending")) + table = table.take(pc.sort_indices(table, sort_keys=sort_keys)) + _write_arrow_table_in_row_groups( + table, + Path(output_path), + row_group_size=row_group_size, + metadata=metadata, + compression=compression, + ) diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/runners.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/runners.py new file mode 100644 index 00000000..3f2477ce --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/runners.py @@ -0,0 +1,315 @@ +from __future__ import annotations + +import json +import shutil +from collections.abc import Callable +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Sequence, TypeVar + +import pandas as pd + +from .errors import WriterCommandError +from .index_permutations import DEFAULT_CONDITIONS, IndexCondition, write_index_permutations +from .points import ( + build_spatialdata_multiscale_metadata, + write_morton_points_parquet, + write_multiscale_points_parquet, +) +from .zarr import ( + copy_points_element_metadata, + experimental_points_output_path, + list_points_keys, + points_parquet_path, + read_points_dataframe, + read_points_element_attrs, + register_points_elements_in_consolidated_metadata, + validate_points_key, +) + +_T = TypeVar("_T") + + +@dataclass(frozen=True) +class MortonZarrOutput: + source_parquet: Path + output_parquet: Path + in_place: bool + output_points_key: str | None + collection: str + + +def _as_command_error(action: Callable[[], _T]) -> _T: + try: + return action() + except WriterCommandError: + raise + except (FileNotFoundError, ValueError) as exc: + raise WriterCommandError(str(exc)) from exc + + +def read_input_dataframe(path: str | Path) -> pd.DataFrame: + input_path = Path(path) + if input_path.is_dir(): + return read_points_dataframe(input_path) + suffix = input_path.suffix.lower() + if suffix == ".csv": + return pd.read_csv(input_path) + if suffix in {".parquet", ".pq"}: + return pd.read_parquet(input_path) + raise WriterCommandError( + f"Unsupported input: {path}\n" + "Expected a .csv file, .parquet file, or a directory of Parquet parts." + ) + + +def run_list_points(zarr: str | Path) -> dict[str, Any]: + keys = list_points_keys(zarr) + if not keys: + raise WriterCommandError(f"No Points elements found under {Path(zarr) / 'points'}") + return {"zarr": str(zarr), "points_keys": keys} + + +def run_morton_points( + input_path: str | Path, + output_path: str | Path, + *, + feature_key: str | None = None, + row_group_size: int = 50_000, + compression: str = "zstd", +) -> dict[str, Any]: + df = _as_command_error(lambda: read_input_dataframe(input_path)) + sorted_df = _as_command_error( + lambda: write_morton_points_parquet( + df, + output_path, + feature_key=feature_key, + row_group_size=row_group_size, + compression=compression, + ) + ) + return { + "format": "morton-points", + "rows": int(len(sorted_df)), + "output": str(output_path), + "row_group_size": row_group_size, + } + + +def run_multiscale_points( + input_path: str | Path, + output_path: str | Path, + *, + metadata_json: str | Path | None = None, + row_group_size: int = 50_000, + compression: str = "zstd", +) -> dict[str, Any]: + df = _as_command_error(lambda: read_input_dataframe(input_path)) + if metadata_json: + metadata = _as_command_error(lambda: json.loads(Path(metadata_json).read_text())) + else: + metadata = _as_command_error(lambda: build_spatialdata_multiscale_metadata(df)) + _as_command_error( + lambda: write_multiscale_points_parquet( + df, + output_path, + metadata=metadata, + row_group_size=row_group_size, + compression=compression, + ) + ) + return { + "format": "spatialdata_multiscale_points", + "rows": int(len(df)), + "output": str(output_path), + "row_group_size": row_group_size, + } + + +def resolve_morton_from_zarr_output( + zarr_path: Path, + points_key: str, + *, + output: str | Path | None = None, + output_points_key: str | None = None, + experimental: bool = False, +) -> MortonZarrOutput: + if output is not None and output_points_key is not None: + raise WriterCommandError("Pass either output or output_points_key, not both.") + source_parquet = points_parquet_path(zarr_path, points_key) + if output: + resolved_output = Path(output) + return MortonZarrOutput( + source_parquet=source_parquet, + output_parquet=resolved_output, + in_place=resolved_output == source_parquet, + output_points_key=None, + collection="path", + ) + if experimental: + resolved_output_key = validate_points_key(output_points_key or points_key) + resolved_output = experimental_points_output_path(zarr_path, resolved_output_key) + return MortonZarrOutput( + source_parquet=source_parquet, + output_parquet=resolved_output, + in_place=False, + output_points_key=resolved_output_key, + collection="points.experimental", + ) + resolved_output_key = validate_points_key(output_points_key or points_key) + resolved_output = points_parquet_path(zarr_path, resolved_output_key) + return MortonZarrOutput( + source_parquet=source_parquet, + output_parquet=resolved_output, + in_place=resolved_output == source_parquet, + output_points_key=resolved_output_key, + collection="points", + ) + + +def run_morton_points_from_zarr( + zarr: str | Path, + *, + points_key: str | None = None, + experimental: bool = False, + output: str | Path | None = None, + output_points_key: str | None = None, + feature_key: str | None = None, + overwrite: bool = False, + row_group_size: int = 50_000, + compression: str = "zstd", +) -> dict[str, Any]: + zarr_path = Path(zarr) + keys = list_points_keys(zarr_path) + if not keys: + raise WriterCommandError(f"No Points elements found under {zarr_path / 'points'}") + + resolved_key = points_key + if resolved_key is None: + if len(keys) == 1: + resolved_key = keys[0] + else: + raise WriterCommandError( + "Multiple Points elements found; pass points_key. " + f"Available keys: {', '.join(keys)}" + ) + if resolved_key not in keys: + raise WriterCommandError( + f"Unknown Points element {resolved_key!r}. Available: {', '.join(keys)}" + ) + + attrs = _as_command_error(lambda: read_points_element_attrs(zarr_path, resolved_key)) + resolved_feature_key = feature_key or attrs.get("feature_key") + output_spec = resolve_morton_from_zarr_output( + zarr_path, + resolved_key, + output=output, + output_points_key=output_points_key, + experimental=experimental, + ) + if ( + output_spec.output_parquet.exists() + and not output_spec.in_place + and not overwrite + ): + raise WriterCommandError( + f"Output already exists: {output_spec.output_parquet}\n" + "Choose another element name/path or enable overwrite." + ) + if ( + output_spec.collection == "points" + and output_spec.output_points_key is not None + and output_spec.output_points_key != resolved_key + ): + target_element_dir = zarr_path / "points" / output_spec.output_points_key + if target_element_dir.exists() and not overwrite: + raise WriterCommandError( + f"Points element already exists: points/{output_spec.output_points_key}\n" + "Choose another element name or enable overwrite." + ) + + df = _as_command_error(lambda: read_points_dataframe(output_spec.source_parquet)) + if output_spec.output_parquet.exists(): + if output_spec.output_parquet.is_dir(): + shutil.rmtree(output_spec.output_parquet) + else: + output_spec.output_parquet.unlink() + if ( + output_spec.collection == "points" + and output_spec.output_points_key is not None + and output_spec.output_points_key != resolved_key + ): + _as_command_error( + lambda: copy_points_element_metadata( + zarr_path, + source_key=resolved_key, + dest_key=output_spec.output_points_key, + ) + ) + sorted_df = _as_command_error( + lambda: write_morton_points_parquet( + df, + output_spec.output_parquet, + feature_key=resolved_feature_key, + row_group_size=row_group_size, + compression=compression, + ) + ) + if ( + output_spec.collection == "points" + and output_spec.output_points_key is not None + and output_spec.output_points_key != resolved_key + ): + _as_command_error( + lambda: register_points_elements_in_consolidated_metadata( + zarr_path, + [resolved_key, output_spec.output_points_key], + template_key=resolved_key, + ) + ) + return { + "format": "morton-points", + "zarr": str(zarr_path), + "points_key": resolved_key, + "output_points_key": output_spec.output_points_key, + "output_collection": output_spec.collection, + "source": str(output_spec.source_parquet), + "output": str(output_spec.output_parquet), + "in_place": output_spec.in_place, + "feature_key": resolved_feature_key, + "rows": int(len(sorted_df)), + "row_group_size": row_group_size, + } + + +def run_write_index_permutations( + source_zarr: str | Path, + dest_zarr: str | Path, + *, + points_key: str | None = None, + max_rows: int | None = None, + condition_ids: Sequence[str] | None = None, + overwrite: bool = False, + row_group_size: int = 50_000, + compression: str = "zstd", +) -> dict[str, Any]: + selected: tuple[IndexCondition, ...] | None = None + if condition_ids: + by_id = {condition.id: condition for condition in DEFAULT_CONDITIONS} + missing = [value for value in condition_ids if value not in by_id] + if missing: + raise WriterCommandError(f"Unknown conditions: {', '.join(missing)}") + selected = tuple(by_id[value] for value in condition_ids) + + return _as_command_error( + lambda: write_index_permutations( + source_zarr, + dest_zarr, + points_key=points_key, + max_rows=max_rows, + conditions=selected, + overwrite=overwrite, + row_group_size=row_group_size, + compression=compression, + ) + ) diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/__init__.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/app.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/app.py new file mode 100644 index 00000000..fd4754c1 --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/app.py @@ -0,0 +1,112 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +from textual.app import App + +from .models import WriterContext + +if TYPE_CHECKING: + from .screens import HomeScreen + + +class WriterApp(App[None]): + CSS = """ + Screen { + align: center middle; + } + + .screen-title { + width: 100%; + content-align: center middle; + padding: 1 0; + text-style: bold; + } + + .section-label { + padding-top: 1; + text-style: bold; + } + + #command-list { + width: 70; + height: auto; + max-height: 16; + border: solid $accent; + margin: 1 0; + } + + #points-key-list { + width: 70; + height: auto; + max-height: 12; + border: solid $accent; + margin: 1 0; + } + + VerticalScroll { + width: 80; + height: 1fr; + border: solid $primary; + padding: 1 2; + } + + Input { + margin-bottom: 1; + } + + #run-log { + width: 90; + height: 1fr; + border: solid $accent; + margin: 1 0; + } + + #verify-table { + width: 100%; + height: auto; + max-height: 14; + margin: 1 0; + } + + #confirm-message { + width: 80; + padding: 1 2; + border: solid $warning; + margin: 1 0; + } + + Horizontal { + width: auto; + height: auto; + align: center middle; + } + + Button { + margin: 0 1; + } + """ + + BINDINGS = [("q", "quit", "Quit")] + + def __init__(self, *, initial_zarr: str | None = None) -> None: + super().__init__() + self.context = WriterContext(zarr_path=initial_zarr) + self.home_screen: HomeScreen | None = None + + def on_mount(self) -> None: + from .screens import HomeScreen + + self.home_screen = HomeScreen() + self.install_screen(self.home_screen, "home") + self.push_screen("home") + + def go_home(self) -> None: + if self.home_screen is None: + return + self.switch_screen("home") + + +def run_tui(*, initial_zarr: str | None = None) -> None: + app = WriterApp(initial_zarr=initial_zarr) + app.run() diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/models.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/models.py new file mode 100644 index 00000000..c2042c49 --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/models.py @@ -0,0 +1,35 @@ +from __future__ import annotations + +from dataclasses import dataclass, field +from enum import Enum +from pathlib import Path +from typing import Any, Callable, Literal + + +class CommandId(str, Enum): + LIST_POINTS = "list-points" + MORTON_FROM_ZARR = "morton-points-from-zarr" + MORTON_POINTS = "morton-points" + MULTISCALE_POINTS = "multiscale-points" + INDEX_PERMUTATIONS = "write-index-permutations" + + +VerifyKind = Literal["none", "morton", "multiscale", "manifest"] + + +@dataclass +class TaskSpec: + command: CommandId + title: str + runner: Callable[[], dict[str, Any]] + verify_kind: VerifyKind = "none" + verify_paths: list[Path] = field(default_factory=list) + requires_confirm: bool = False + confirm_message: str = "" + log_lines: list[str] = field(default_factory=list) + + +@dataclass +class WriterContext: + zarr_path: str | None = None + points_key: str | None = None diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/screens.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/screens.py new file mode 100644 index 00000000..91088dc2 --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/tui/screens.py @@ -0,0 +1,873 @@ +from __future__ import annotations + +import json +import traceback +from pathlib import Path +from typing import Any + +from textual import getters, work +from textual.app import ComposeResult +from textual.binding import Binding +from textual.events import ScreenResume +from textual.containers import Horizontal, Vertical, VerticalScroll +from textual.screen import Screen +from textual.widgets import ( + Button, + Checkbox, + DataTable, + Footer, + Header, + Input, + Label, + ListItem, + ListView, + RichLog, + Static, +) + +from ..errors import WriterCommandError +from ..index_permutations import DEFAULT_CONDITIONS +from ..runners import ( + resolve_morton_from_zarr_output, + run_list_points, + run_morton_points, + run_morton_points_from_zarr, + run_multiscale_points, + run_write_index_permutations, +) +from ..verify import ( + VerifyCheck, + all_passed, + verify_index_permutations_manifest, + verify_morton_parquet, + verify_multiscale_parquet, +) +from ..zarr import list_points_keys, read_points_element_attrs +from .app import WriterApp +from .models import CommandId, TaskSpec + + +def _positive_int(value: str, default: int) -> int: + stripped = value.strip() + if not stripped: + return default + parsed = int(stripped) + if parsed <= 0: + raise ValueError("value must be positive") + return parsed + + +class WriterScreen(Screen[None]): + app = getters.app(WriterApp) + + +class InputFormScreen(WriterScreen): + """Form screen with Enter-to-advance/submit and Escape-to-back.""" + + INPUT_ORDER: tuple[str, ...] = () + PRIMARY_BUTTON_ID: str = "run" + + BINDINGS = [ + Binding("escape", "go_back", "Back"), + ] + + def on_mount(self) -> None: + if self.INPUT_ORDER: + self.query_one(f"#{self.INPUT_ORDER[0]}", Input).focus() + + def on_input_submitted(self, event: Input.Submitted) -> None: + input_id = event.input.id + if input_id is None: + self._press_primary() + return + if not self.INPUT_ORDER or input_id not in self.INPUT_ORDER: + self._press_primary() + return + if input_id == self.INPUT_ORDER[-1]: + self._press_primary() + return + next_index = self.INPUT_ORDER.index(input_id) + 1 + self.query_one(f"#{self.INPUT_ORDER[next_index]}", Input).focus() + + def action_go_back(self) -> None: + self.app.pop_screen() + + def _press_primary(self) -> None: + self.query_one(f"#{self.PRIMARY_BUTTON_ID}", Button).press() + + +class HomeScreen(WriterScreen): + BINDINGS = [("q", "quit", "Quit")] + + def on_mount(self) -> None: + self.query_one("#command-list", ListView).focus() + + def on_screen_resume(self, event: ScreenResume) -> None: + self.refresh() + self.query_one("#command-list", ListView).focus() + + def compose(self) -> ComposeResult: + yield Header() + yield Static( + "SpatialData experimental writer — pick a command.", + id="home-title", + ) + yield ListView( + ListItem(Label("List Points elements in a Zarr store"), id="cmd-list-points"), + ListItem(Label("Morton-sort Points from Zarr"), id="cmd-morton-from-zarr"), + ListItem(Label("Morton-sort CSV/Parquet file"), id="cmd-morton-points"), + ListItem(Label("Write multiscale Points Parquet"), id="cmd-multiscale-points"), + ListItem( + Label("Write index permutations derivative store"), + id="cmd-index-permutations", + ), + id="command-list", + ) + yield Footer() + + def on_list_view_selected(self, event: ListView.Selected) -> None: + item_id = event.item.id or "" + if item_id == "cmd-list-points": + self._start_zarr_command(CommandId.LIST_POINTS) + elif item_id == "cmd-morton-from-zarr": + self._start_zarr_command(CommandId.MORTON_FROM_ZARR) + elif item_id == "cmd-morton-points": + self.app.push_screen(MortonFileScreen()) + elif item_id == "cmd-multiscale-points": + self.app.push_screen(MultiscaleScreen()) + elif item_id == "cmd-index-permutations": + self.app.push_screen(IndexPermutationsScreen()) + + def _start_zarr_command(self, command: CommandId) -> None: + if command == CommandId.LIST_POINTS: + if self.app.context.zarr_path: + self._run_list_points(self.app.context.zarr_path) + else: + self.app.push_screen(ZarrPathScreen(command)) + return + if self.app.context.zarr_path: + self.app.push_screen(PointsKeyScreen(command)) + else: + self.app.push_screen(ZarrPathScreen(command)) + + def _run_list_points(self, zarr: str) -> None: + def runner() -> dict[str, Any]: + return run_list_points(zarr) + + self.app.push_screen( + RunScreen( + TaskSpec( + command=CommandId.LIST_POINTS, + title="List Points", + runner=runner, + verify_kind="none", + ) + ) + ) + + +class ZarrPathScreen(InputFormScreen): + INPUT_ORDER = ("zarr-path",) + PRIMARY_BUTTON_ID = "continue" + + def __init__(self, command: CommandId) -> None: + super().__init__() + self.command = command + + def compose(self) -> ComposeResult: + yield Header() + yield Static("SpatialData Zarr store path", classes="screen-title") + yield Input(placeholder="/path/to/store.zarr", id="zarr-path") + with Horizontal(): + yield Button("Continue", variant="primary", id="continue") + yield Button("Back", id="back") + yield Footer() + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id == "back": + self.action_go_back() + return + self._continue() + + def _continue(self) -> None: + path = self.query_one("#zarr-path", Input).value.strip() + if not path: + self.notify("Enter a Zarr store path.", severity="error") + return + resolved = Path(path) + if not resolved.is_dir(): + self.notify(f"Not a directory: {path}", severity="error") + return + self.app.context.zarr_path = str(resolved) + if self.command == CommandId.LIST_POINTS: + self._run_list_points(path) + return + self.app.push_screen(PointsKeyScreen(self.command)) + + def _run_list_points(self, zarr: str) -> None: + def runner() -> dict[str, Any]: + return run_list_points(zarr) + + self.app.push_screen( + RunScreen( + TaskSpec( + command=CommandId.LIST_POINTS, + title="List Points", + runner=runner, + verify_kind="none", + ) + ) + ) + + +class PointsKeyScreen(WriterScreen): + BINDINGS = [ + Binding("escape", "go_back", "Back"), + ] + + def __init__(self, command: CommandId) -> None: + super().__init__() + self.command = command + + def compose(self) -> ComposeResult: + yield Header() + yield Static("Select Points element", classes="screen-title") + yield ListView(id="points-key-list") + with Horizontal(): + yield Button("Continue", variant="primary", id="continue") + yield Button("Back", id="back") + yield Footer() + + def on_mount(self) -> None: + zarr = self.app.context.zarr_path + list_view = self.query_one("#points-key-list", ListView) + if not zarr: + return + keys = list_points_keys(zarr) + if not keys: + list_view.mount(Static("No Points elements found.")) + return + for key in keys: + list_view.mount(ListItem(Label(key), id=f"key-{key}")) + if len(keys) == 1: + self.app.context.points_key = keys[0] + list_view.index = 0 + list_view.focus() + + def on_list_view_selected(self, event: ListView.Selected) -> None: + item_id = event.item.id or "" + if item_id.startswith("key-"): + self.app.context.points_key = item_id.removeprefix("key-") + self._continue() + + def action_go_back(self) -> None: + self.app.pop_screen() + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id == "back": + self.action_go_back() + return + self._continue() + + def _continue(self) -> None: + list_view = self.query_one("#points-key-list", ListView) + if list_view.index is None: + self.notify("Select a Points element.", severity="error") + return + item = list_view.children[list_view.index] + item_id = item.id or "" + if not item_id.startswith("key-"): + self.notify("Select a Points element.", severity="error") + return + self.app.context.points_key = item_id.removeprefix("key-") + if self.command == CommandId.MORTON_FROM_ZARR: + self.app.push_screen(MortonFromZarrScreen()) + elif self.command == CommandId.INDEX_PERMUTATIONS: + self.app.push_screen(IndexPermutationsScreen(from_zarr_context=True)) + + +class MortonFromZarrScreen(InputFormScreen): + INPUT_ORDER = ("feature-key", "row-group-size", "compression", "output-element") + + def compose(self) -> ComposeResult: + yield Header() + yield Static("Morton-sort Points from Zarr", classes="screen-title") + with VerticalScroll(): + yield Label("Feature key column (optional)") + yield Input(placeholder="feature_name", id="feature-key") + yield Label("Row group size") + yield Input(value="50000", id="row-group-size") + yield Label("Compression") + yield Input(value="zstd", id="compression") + yield Label("Output Points element name (optional)") + yield Input(placeholder="leave empty to overwrite selected element", id="output-element") + yield Checkbox("Write to points.experimental/", id="experimental") + with Horizontal(): + yield Button("Run", variant="primary", id="run") + yield Button("Back", id="back") + yield Footer() + + def on_mount(self) -> None: + zarr = self.app.context.zarr_path + key = self.app.context.points_key + if not zarr or not key: + return + try: + attrs = read_points_element_attrs(zarr, key) + feature_key = attrs.get("feature_key") + if feature_key: + self.query_one("#feature-key", Input).value = str(feature_key) + except OSError: + pass + super().on_mount() + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id == "back": + self.action_go_back() + return + self._submit() + + def _submit(self) -> None: + zarr = self.app.context.zarr_path + key = self.app.context.points_key + if not zarr or not key: + self.notify("Missing Zarr context.", severity="error") + return + + feature_key = self.query_one("#feature-key", Input).value.strip() or None + output_key = self.query_one("#output-element", Input).value.strip() or None + experimental = self.query_one("#experimental", Checkbox).value + try: + row_group_size = _positive_int( + self.query_one("#row-group-size", Input).value, 50_000 + ) + except ValueError as exc: + self.notify(str(exc), severity="error") + return + compression = self.query_one("#compression", Input).value.strip() or "zstd" + + try: + output_spec = resolve_morton_from_zarr_output( + Path(zarr), + key, + output_points_key=output_key, + experimental=experimental, + ) + except WriterCommandError as exc: + self.notify(str(exc), severity="error") + return + + target_exists = output_spec.output_parquet.exists() + if ( + output_spec.collection == "points" + and output_spec.output_points_key is not None + ): + target_exists = target_exists or ( + Path(zarr) / "points" / output_spec.output_points_key + ).exists() + requires_confirm = output_spec.in_place or target_exists + + def runner() -> dict[str, Any]: + return run_morton_points_from_zarr( + zarr, + points_key=key, + experimental=experimental, + output_points_key=output_key, + feature_key=feature_key, + overwrite=requires_confirm, + row_group_size=row_group_size, + compression=compression, + ) + + if output_spec.in_place: + confirm_message = ( + f"Overwrite selected Points element:\npoints/{key}\n\n" + f"Parquet path:\n{output_spec.output_parquet}\n\nProceed?" + ) + elif target_exists and output_spec.collection == "points": + confirm_message = ( + f"Overwrite existing Points element:\n" + f"points/{output_spec.output_points_key}\n\n" + f"Parquet path:\n{output_spec.output_parquet}\n\nProceed?" + ) + elif target_exists and output_spec.collection == "points.experimental": + confirm_message = ( + f"Overwrite existing experimental Points artifact:\n" + f"points.experimental/{output_spec.output_points_key}\n\n" + f"Parquet path:\n{output_spec.output_parquet}\n\nProceed?" + ) + else: + confirm_message = "" + + task = TaskSpec( + command=CommandId.MORTON_FROM_ZARR, + title="Morton-sort from Zarr", + runner=runner, + verify_kind="morton", + verify_paths=[output_spec.output_parquet], + requires_confirm=requires_confirm, + confirm_message=confirm_message, + ) + self._launch_task(task) + + def _launch_task(self, task_spec: TaskSpec) -> None: + if task_spec.requires_confirm: + self.app.push_screen(ConfirmScreen(task_spec)) + else: + self.app.push_screen(RunScreen(task_spec)) + + +class MortonFileScreen(InputFormScreen): + INPUT_ORDER = ( + "input-path", + "output-path", + "feature-key", + "row-group-size", + "compression", + ) + + def compose(self) -> ComposeResult: + yield Header() + yield Static("Morton-sort CSV/Parquet", classes="screen-title") + with VerticalScroll(): + yield Label("Input path") + yield Input(placeholder="input.csv or input.parquet", id="input-path") + yield Label("Output path") + yield Input(placeholder="output.parquet", id="output-path") + yield Label("Feature key column (optional)") + yield Input(placeholder="feature_name", id="feature-key") + yield Label("Row group size") + yield Input(value="50000", id="row-group-size") + yield Label("Compression") + yield Input(value="zstd", id="compression") + with Horizontal(): + yield Button("Run", variant="primary", id="run") + yield Button("Back", id="back") + yield Footer() + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id == "back": + self.action_go_back() + return + self._submit() + + def _submit(self) -> None: + input_path = self.query_one("#input-path", Input).value.strip() + output_path = self.query_one("#output-path", Input).value.strip() + if not input_path or not output_path: + self.notify("Input and output paths are required.", severity="error") + return + feature_key = self.query_one("#feature-key", Input).value.strip() or None + try: + row_group_size = _positive_int( + self.query_one("#row-group-size", Input).value, 50_000 + ) + except ValueError as exc: + self.notify(str(exc), severity="error") + return + compression = self.query_one("#compression", Input).value.strip() or "zstd" + + def runner() -> dict[str, Any]: + return run_morton_points( + input_path, + output_path, + feature_key=feature_key, + row_group_size=row_group_size, + compression=compression, + ) + + self.app.push_screen( + RunScreen( + TaskSpec( + command=CommandId.MORTON_POINTS, + title="Morton-sort file", + runner=runner, + verify_kind="morton", + verify_paths=[Path(output_path)], + ) + ) + ) + + +class MultiscaleScreen(InputFormScreen): + INPUT_ORDER = ( + "input-path", + "output-path", + "metadata-json", + "row-group-size", + "compression", + ) + + def compose(self) -> ComposeResult: + yield Header() + yield Static("Multiscale Points Parquet", classes="screen-title") + with VerticalScroll(): + yield Label("Input path") + yield Input(placeholder="input.parquet", id="input-path") + yield Label("Output path") + yield Input(placeholder="output.parquet", id="output-path") + yield Label("Metadata JSON path (optional)") + yield Input(placeholder="metadata.json", id="metadata-json") + yield Label("Row group size") + yield Input(value="50000", id="row-group-size") + yield Label("Compression") + yield Input(value="zstd", id="compression") + with Horizontal(): + yield Button("Run", variant="primary", id="run") + yield Button("Back", id="back") + yield Footer() + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id == "back": + self.action_go_back() + return + self._submit() + + def _submit(self) -> None: + input_path = self.query_one("#input-path", Input).value.strip() + output_path = self.query_one("#output-path", Input).value.strip() + if not input_path or not output_path: + self.notify("Input and output paths are required.", severity="error") + return + metadata_json = self.query_one("#metadata-json", Input).value.strip() or None + try: + row_group_size = _positive_int( + self.query_one("#row-group-size", Input).value, 50_000 + ) + except ValueError as exc: + self.notify(str(exc), severity="error") + return + compression = self.query_one("#compression", Input).value.strip() or "zstd" + + def runner() -> dict[str, Any]: + return run_multiscale_points( + input_path, + output_path, + metadata_json=metadata_json, + row_group_size=row_group_size, + compression=compression, + ) + + self.app.push_screen( + RunScreen( + TaskSpec( + command=CommandId.MULTISCALE_POINTS, + title="Multiscale Points", + runner=runner, + verify_kind="multiscale", + verify_paths=[Path(output_path)], + ) + ) + ) + + +class IndexPermutationsScreen(InputFormScreen): + INPUT_ORDER = ( + "source-zarr", + "dest-zarr", + "points-key", + "max-rows", + "row-group-size", + "compression", + ) + + def __init__(self, *, from_zarr_context: bool = False) -> None: + super().__init__() + self.from_zarr_context = from_zarr_context + + def compose(self) -> ComposeResult: + yield Header() + yield Static("Write index permutations", classes="screen-title") + with VerticalScroll(): + yield Label("Source Zarr") + yield Input(id="source-zarr") + yield Label("Destination Zarr") + yield Input(id="dest-zarr") + yield Label("Points key (optional if single element)") + yield Input(id="points-key") + yield Label("Max rows (optional)") + yield Input(id="max-rows") + yield Label("Row group size") + yield Input(value="50000", id="row-group-size") + yield Label("Compression") + yield Input(value="zstd", id="compression") + yield Checkbox("Overwrite destination if it exists", id="overwrite") + yield Static("Conditions (default: all)", classes="section-label") + with Vertical(id="conditions"): + for condition in DEFAULT_CONDITIONS: + yield Checkbox(condition.id, value=True, id=f"cond-{condition.id}") + with Horizontal(): + yield Button("Run", variant="primary", id="run") + yield Button("Back", id="back") + yield Footer() + + def on_mount(self) -> None: + if self.app.context.zarr_path: + self.query_one("#source-zarr", Input).value = self.app.context.zarr_path + if self.from_zarr_context and self.app.context.points_key: + self.query_one("#points-key", Input).value = self.app.context.points_key + super().on_mount() + + def on_button_pressed(self, event: Button.Pressed) -> None: + if event.button.id == "back": + self.action_go_back() + return + self._submit() + + def _submit(self) -> None: + source = self.query_one("#source-zarr", Input).value.strip() + dest = self.query_one("#dest-zarr", Input).value.strip() + if not source or not dest: + self.notify("Source and destination Zarr paths are required.", severity="error") + return + points_key = self.query_one("#points-key", Input).value.strip() or None + max_rows_text = self.query_one("#max-rows", Input).value.strip() + max_rows = None + if max_rows_text: + try: + max_rows = _positive_int(max_rows_text, 0) + except ValueError as exc: + self.notify(str(exc), severity="error") + return + try: + row_group_size = _positive_int( + self.query_one("#row-group-size", Input).value, 50_000 + ) + except ValueError as exc: + self.notify(str(exc), severity="error") + return + compression = self.query_one("#compression", Input).value.strip() or "zstd" + overwrite = self.query_one("#overwrite", Checkbox).value + selected = [ + condition.id + for condition in DEFAULT_CONDITIONS + if self.query_one(f"#cond-{condition.id}", Checkbox).value + ] + if not selected: + self.notify("Select at least one condition.", severity="error") + return + all_selected = len(selected) == len(DEFAULT_CONDITIONS) + condition_ids = None if all_selected else selected + + def runner() -> dict[str, Any]: + return run_write_index_permutations( + source, + dest, + points_key=points_key, + max_rows=max_rows, + condition_ids=condition_ids, + overwrite=overwrite, + row_group_size=row_group_size, + compression=compression, + ) + + self.app.push_screen( + RunScreen( + TaskSpec( + command=CommandId.INDEX_PERMUTATIONS, + title="Index permutations", + runner=runner, + verify_kind="manifest", + verify_paths=[Path(dest)], + ) + ) + ) + + +class ConfirmScreen(WriterScreen): + BINDINGS = [ + Binding("enter", "confirm", "Confirm overwrite"), + Binding("escape", "cancel", "Cancel"), + ] + + def __init__(self, task_spec: TaskSpec) -> None: + super().__init__() + self.task_spec = task_spec + self._handled = False + + def compose(self) -> ComposeResult: + yield Header() + yield Static("Confirm in-place write", classes="screen-title") + yield Static(self.task_spec.confirm_message, id="confirm-message") + with Horizontal(): + yield Button("Confirm overwrite", variant="error", id="confirm") + yield Button("Cancel", id="cancel") + yield Footer() + + def on_mount(self) -> None: + self.query_one("#confirm", Button).focus() + + def action_confirm(self) -> None: + self._confirm() + + def action_cancel(self) -> None: + self._cancel() + + def on_button_pressed(self, event: Button.Pressed) -> None: + event.stop() + if event.button.id == "cancel": + self._cancel() + return + self._confirm() + + def _confirm(self) -> None: + if self._handled: + return + self._handled = True + self.app.pop_screen() + self.app.push_screen(RunScreen(self.task_spec)) + + def _cancel(self) -> None: + if self._handled: + return + self._handled = True + self.app.pop_screen() + + +class RunScreen(WriterScreen): + def __init__(self, task_spec: TaskSpec) -> None: + super().__init__() + self.task_spec = task_spec + + def compose(self) -> ComposeResult: + yield Header() + yield Static(self.task_spec.title, classes="screen-title") + yield RichLog(id="run-log", highlight=True, markup=False) + yield Footer() + + def on_mount(self) -> None: + self.run_task() + + @work(thread=True) + def run_task(self) -> None: + log = self.query_one("#run-log", RichLog) + result: dict[str, Any] | None = None + error: str | None = None + checks: list[VerifyCheck] = [] + try: + self.app.call_from_thread( + log.write, f"Running {self.task_spec.command.value}..." + ) + result = self.task_spec.runner() + self.app.call_from_thread( + log.write, json.dumps(result, indent=2, sort_keys=True) + ) + checks = self._verify() + except WriterCommandError as exc: + error = str(exc) + self.app.call_from_thread(log.write, error) + except Exception as exc: + error = "".join(traceback.format_exception_only(exc)).strip() + self.app.call_from_thread(log.write, error) + + self.app.call_from_thread( + self.app.push_screen, + VerifyReportScreen( + task_spec=self.task_spec, + result=result, + checks=checks, + error=error, + ), + ) + + def _verify(self) -> list[VerifyCheck]: + if self.task_spec.verify_kind == "none": + return [] + if self.task_spec.verify_kind == "morton": + checks: list[VerifyCheck] = [] + for path in self.task_spec.verify_paths: + checks.extend(verify_morton_parquet(path)) + return checks + if self.task_spec.verify_kind == "multiscale": + checks = [] + for path in self.task_spec.verify_paths: + checks.extend(verify_multiscale_parquet(path)) + return checks + if self.task_spec.verify_kind == "manifest": + checks = [] + for path in self.task_spec.verify_paths: + checks.extend(verify_index_permutations_manifest(path)) + return checks + return [] + + +class VerifyReportScreen(WriterScreen): + BINDINGS = [ + Binding("enter", "go_home", "Home"), + Binding("escape", "go_home", "Home"), + ] + + def __init__( + self, + *, + task_spec: TaskSpec, + result: dict[str, Any] | None, + checks: list[VerifyCheck], + error: str | None, + ) -> None: + super().__init__() + self.task_spec = task_spec + self.result = result + self.checks = checks + self.error = error + self._handled = False + + def compose(self) -> ComposeResult: + yield Header() + yield Static("Run complete", classes="screen-title") + with VerticalScroll(): + if self.error: + yield Static(f"Error: {self.error}", id="error-text") + elif self.result: + if self.task_spec.command == CommandId.LIST_POINTS: + keys = self.result.get("points_keys", []) + yield Static(f"Points keys: {', '.join(keys) or '(none)'}") + else: + rows = self.result.get("rows") + output = self.result.get("output") + if rows is not None: + yield Static(f"Rows: {rows}") + if output: + yield Static(f"Output: {output}") + if self.checks: + passed = all_passed(self.checks) + status = "All checks passed" if passed else "Some checks failed" + yield Static(f"Verification: {status}", id="verify-summary") + table = DataTable(id="verify-table") + table.add_columns("Check", "Status", "Detail") + for check in self.checks: + table.add_row( + check.id, + "PASS" if check.passed else "FAIL", + check.detail, + ) + yield table + with Horizontal(): + yield Button("Home", variant="primary", id="home") + yield Button("Quit", id="quit") + yield Footer() + + def on_mount(self) -> None: + self.query_one("#home", Button).focus() + + def action_go_home(self) -> None: + self._go_home() + + def on_button_pressed(self, event: Button.Pressed) -> None: + event.stop() + if event.button.id == "home": + self._go_home() + return + self._exit() + + def _go_home(self) -> None: + if self._handled: + return + self._handled = True + self.app.go_home() + + def _exit(self) -> None: + if self._handled: + return + self._handled = True + self.app.exit() diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/verify.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/verify.py new file mode 100644 index 00000000..1bde1e37 --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/verify.py @@ -0,0 +1,319 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import pandas as pd +import pyarrow.parquet as pq + +from .points import MORTON_CODE_2D_COLUMN + +_UINT_INTERMEDIATE_SUFFIXES = ("_uint",) + + +def _dataframe_column(df: pd.DataFrame, name: str) -> pd.Series: + column = df[name] + if isinstance(column, pd.DataFrame): + raise TypeError(f"expected column {name!r} to be a Series, got DataFrame") + return column + + +def _series_extrema(series: pd.Series) -> tuple[float, float]: + return float(series.min()), float(series.max()) + + +@dataclass(frozen=True) +class VerifyCheck: + id: str + passed: bool + detail: str + + +def _check(id: str, passed: bool, detail: str) -> VerifyCheck: + return VerifyCheck(id=id, passed=passed, detail=detail) + + +def _count_sentinel_prefix(morton_values: pd.Series) -> int: + count = 0 + for value in morton_values.head(4): + if int(value) != 0: + break + count += 1 + return count + + +def _sentinel_count_from_row_group( + parquet: pq.ParquetFile, + morton_values: pd.Series, +) -> int | None: + if parquet.num_row_groups == 0: + return None + first_group_rows = parquet.metadata.row_group(0).num_rows + if not 2 <= first_group_rows <= 4: + return None + first_values = morton_values.iloc[:first_group_rows] + if first_values.astype("int64").eq(0).all(): + return int(first_group_rows) + return None + + +def verify_morton_parquet(path: str | Path) -> list[VerifyCheck]: + parquet_path = Path(path) + checks: list[VerifyCheck] = [] + + if not parquet_path.is_file(): + return [_check("file_exists", False, f"Parquet file not found: {parquet_path}")] + + checks.append(_check("file_exists", True, str(parquet_path))) + + try: + parquet = pq.ParquetFile(parquet_path) + except Exception as exc: + checks.append( + _check("parquet_readable", False, f"failed to read Parquet metadata: {exc}") + ) + return checks + + columns = parquet.schema_arrow.names + checks.append( + _check( + "column_present", + MORTON_CODE_2D_COLUMN in columns, + f"expected column {MORTON_CODE_2D_COLUMN!r} in schema", + ) + ) + checks.append( + _check("x_column_present", "x" in columns, "expected column 'x' in schema") + ) + checks.append( + _check("y_column_present", "y" in columns, "expected column 'y' in schema") + ) + + uint_columns = [ + name + for name in columns + if any(name.endswith(suffix) for suffix in _UINT_INTERMEDIATE_SUFFIXES) + ] + checks.append( + _check( + "no_uint_intermediates", + not uint_columns, + "no uint staging columns" + if not uint_columns + else f"unexpected uint columns: {', '.join(uint_columns)}", + ) + ) + + if ( + MORTON_CODE_2D_COLUMN not in columns + or "x" not in columns + or "y" not in columns + ): + return checks + + try: + df = pd.read_parquet(parquet_path, columns=[MORTON_CODE_2D_COLUMN, "x", "y"]) + except Exception as exc: + checks.append( + _check( + "required_columns_readable", + False, + f"failed to read required columns: {exc}", + ) + ) + return checks + + morton_column = _dataframe_column(df, MORTON_CODE_2D_COLUMN) + x_column = _dataframe_column(df, "x") + y_column = _dataframe_column(df, "y") + sentinel_count = _sentinel_count_from_row_group(parquet, morton_column) + if sentinel_count is None: + sentinel_count = _count_sentinel_prefix(morton_column) + checks.append( + _check( + "sentinel_prefix", + 2 <= sentinel_count <= 4, + f"sentinel prefix rows: {sentinel_count} (expected 2–4)", + ) + ) + + if sentinel_count > 0: + sentinel_x = x_column.iloc[:sentinel_count] + sentinel_y = y_column.iloc[:sentinel_count] + sentinel_x_min, sentinel_x_max = _series_extrema(sentinel_x) + dataset_x_min, dataset_x_max = _series_extrema(x_column) + sentinel_y_min, sentinel_y_max = _series_extrema(sentinel_y) + dataset_y_min, dataset_y_max = _series_extrema(y_column) + bbox_match = ( + sentinel_x_min == dataset_x_min + and sentinel_x_max == dataset_x_max + and sentinel_y_min == dataset_y_min + and sentinel_y_max == dataset_y_max + ) + checks.append( + _check( + "sentinel_bbox", + bbox_match, + "sentinel rows encode full x/y bounds" + if bbox_match + else "sentinel x/y extrema do not match dataset bounds", + ) + ) + + if sentinel_count < len(df): + tail = morton_column.iloc[sentinel_count:].astype("int64") + monotonic = bool((tail.diff().dropna() >= 0).all()) + checks.append( + _check( + "morton_monotonic", + monotonic, + "morton_code_2d non-decreasing after sentinels" + if monotonic + else "morton_code_2d decreases after sentinel prefix", + ) + ) + else: + checks.append( + _check( + "morton_monotonic", + True, + "all rows are sentinel prefix (small dataset)", + ) + ) + + if parquet.num_row_groups > 0: + first_group_rows = parquet.metadata.row_group(0).num_rows + sentinel_only = first_group_rows <= 4 and first_group_rows == sentinel_count + checks.append( + _check( + "row_group_sentinels", + sentinel_only, + f"row group 0 has {first_group_rows} rows (sentinel count {sentinel_count})", + ) + ) + else: + checks.append(_check("row_group_sentinels", False, "parquet has no row groups")) + + return checks + + +def verify_multiscale_parquet(path: str | Path) -> list[VerifyCheck]: + parquet_path = Path(path) + checks: list[VerifyCheck] = [] + + if not parquet_path.is_file(): + return [_check("file_exists", False, f"Parquet file not found: {parquet_path}")] + + checks.append(_check("file_exists", True, str(parquet_path))) + + schema_metadata = pq.ParquetFile(parquet_path).schema_arrow.metadata + if schema_metadata is None or b"spatialdata_multiscale" not in schema_metadata: + checks.append( + _check( + "multiscale_metadata", + False, + "missing spatialdata_multiscale schema metadata", + ) + ) + return checks + + stored = json.loads(schema_metadata[b"spatialdata_multiscale"]) + checks.append( + _check( + "multiscale_metadata", + stored.get("format") == "spatialdata_multiscale_points", + f"format={stored.get('format')!r}", + ) + ) + + bbox = stored.get("bounding_box") + has_bbox = ( + isinstance(bbox, dict) + and isinstance(bbox.get("min"), list) + and isinstance(bbox.get("max"), list) + and len(bbox.get("min", [])) > 0 + and len(bbox.get("max", [])) > 0 + ) + checks.append( + _check( + "multiscale_bbox", + has_bbox, + "bounding_box min/max present" + if has_bbox + else "bounding_box missing or incomplete", + ) + ) + return checks + + +def verify_index_permutations_manifest(dest_zarr: str | Path) -> list[VerifyCheck]: + dest_path = Path(dest_zarr) + manifest_path = dest_path / "index-manifest.json" + checks: list[VerifyCheck] = [] + + if not manifest_path.is_file(): + return [ + _check( + "manifest_exists", + False, + f"index-manifest.json not found under {dest_path}", + ) + ] + + checks.append(_check("manifest_exists", True, str(manifest_path))) + manifest: dict[str, Any] = json.loads(manifest_path.read_text()) + conditions = manifest.get("conditions", []) + if not isinstance(conditions, list) or not conditions: + checks.append(_check("manifest_conditions", False, "no conditions in manifest")) + return checks + + checks.append( + _check("manifest_conditions", True, f"{len(conditions)} condition(s) listed") + ) + + for condition in conditions: + if not isinstance(condition, dict): + continue + condition_id = condition.get("id", "unknown") + element_path = condition.get("element_path") + if not isinstance(element_path, str): + checks.append( + _check( + f"path_{condition_id}", + False, + "condition missing element_path", + ) + ) + continue + + parquet_path = dest_path / element_path / "points.parquet" + if not parquet_path.is_file() and not parquet_path.is_dir(): + checks.append( + _check( + f"path_{condition_id}", + False, + f"missing output: {parquet_path}", + ) + ) + continue + + checks.append(_check(f"path_{condition_id}", True, str(parquet_path))) + + tiling_kind = condition.get("tiling_kind") + if tiling_kind == "morton-points" and parquet_path.is_file(): + for morton_check in verify_morton_parquet(parquet_path): + checks.append( + VerifyCheck( + id=f"{condition_id}_{morton_check.id}", + passed=morton_check.passed, + detail=morton_check.detail, + ) + ) + + return checks + + +def all_passed(checks: list[VerifyCheck]) -> bool: + return bool(checks) and all(check.passed for check in checks) diff --git a/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/zarr.py b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/zarr.py new file mode 100644 index 00000000..894e46e0 --- /dev/null +++ b/python/spatialdata-experimental-writer/src/spatialdata_experimental_writer/zarr.py @@ -0,0 +1,150 @@ +from __future__ import annotations + +import json +import shutil +from pathlib import Path +from typing import Any, Sequence + +import pandas as pd +import pyarrow.dataset as ds + + +def _points_root(zarr_path: Path) -> Path: + return zarr_path / "points" + + +def validate_points_key(points_key: str) -> str: + key = points_key.strip() + if not key: + raise ValueError("Points element name cannot be empty.") + if Path(key).name != key or key in {".", ".."}: + raise ValueError( + "Points element name must be a single name under points/, not a path." + ) + return key + + +def list_points_keys(zarr_path: str | Path) -> list[str]: + root = _points_root(Path(zarr_path)) + if not root.is_dir(): + return [] + return sorted( + child.name + for child in root.iterdir() + if child.is_dir() and (child / "zarr.json").is_file() + ) + + +def _read_zarr_json(path: Path) -> dict[str, Any]: + return json.loads(path.read_text()) + + +def read_points_element_attrs(zarr_path: str | Path, points_key: str) -> dict[str, Any]: + element_json = _points_root(Path(zarr_path)) / points_key / "zarr.json" + if not element_json.is_file(): + raise FileNotFoundError(f"Points element not found: points/{points_key}") + attrs = _read_zarr_json(element_json).get("attributes", {}) + spatialdata_attrs = attrs.get("spatialdata_attrs", {}) + if not isinstance(spatialdata_attrs, dict): + spatialdata_attrs = {} + return { + "axes": attrs.get("axes", []), + "feature_key": spatialdata_attrs.get("feature_key"), + "instance_key": spatialdata_attrs.get("instance_key"), + "version": spatialdata_attrs.get("version"), + } + + +def points_parquet_path(zarr_path: str | Path, points_key: str) -> Path: + return _points_root(Path(zarr_path)) / validate_points_key(points_key) / "points.parquet" + + +def experimental_points_output_path(zarr_path: str | Path, points_key: str) -> Path: + return Path(zarr_path) / "points.experimental" / validate_points_key(points_key) / "points.parquet" + + +def copy_points_element_metadata( + zarr_path: str | Path, + *, + source_key: str, + dest_key: str, +) -> None: + source = _points_root(Path(zarr_path)) / validate_points_key(source_key) / "zarr.json" + dest = _points_root(Path(zarr_path)) / validate_points_key(dest_key) / "zarr.json" + if not source.is_file(): + raise FileNotFoundError(f"Points element metadata not found: {source}") + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(source, dest) + + +def _points_element_consolidated_entry( + zarr_path: Path, + points_key: str, + *, + metadata: dict[str, Any] | None = None, +) -> dict[str, Any]: + if metadata is not None: + entry = metadata.get(f"points/{points_key}") + if isinstance(entry, dict): + return json.loads(json.dumps(entry)) + element_json = _points_root(zarr_path) / points_key / "zarr.json" + element_doc = _read_zarr_json(element_json) + return { + "attributes": element_doc.get("attributes", {}), + "node_type": element_doc.get("node_type", "group"), + "zarr_format": element_doc.get("zarr_format", 3), + } + + +def read_store_consolidated_metadata(zarr_path: str | Path) -> dict[str, Any]: + root_json = Path(zarr_path) / "zarr.json" + if not root_json.is_file(): + raise FileNotFoundError(f"Missing store metadata: {root_json}") + doc = _read_zarr_json(root_json) + consolidated = doc.get("consolidated_metadata") + if not isinstance(consolidated, dict): + raise ValueError(f"Store has no consolidated metadata: {root_json}") + metadata = consolidated.get("metadata") + if not isinstance(metadata, dict): + raise ValueError(f"Store consolidated metadata has no metadata map: {root_json}") + return metadata + + +def register_points_elements_in_consolidated_metadata( + zarr_path: str | Path, + element_keys: Sequence[str], + *, + template_key: str, +) -> None: + """Register sibling points elements in the store root consolidated metadata.""" + store_path = Path(zarr_path) + root_json = store_path / "zarr.json" + doc = _read_zarr_json(root_json) + consolidated = doc.get("consolidated_metadata") + if not isinstance(consolidated, dict): + consolidated = {"kind": "inline", "metadata": {}} + doc["consolidated_metadata"] = consolidated + metadata = consolidated.get("metadata") + if not isinstance(metadata, dict): + metadata = {} + consolidated["metadata"] = metadata + + template_entry = _points_element_consolidated_entry( + store_path, + template_key, + metadata=metadata, + ) + for key in element_keys: + metadata[f"points/{key}"] = json.loads(json.dumps(template_entry)) + root_json.write_text(json.dumps(doc, indent=2) + "\n") + + +def read_points_dataframe(parquet_path: str | Path) -> pd.DataFrame: + path = Path(parquet_path) + if not path.exists(): + raise FileNotFoundError(f"Points Parquet not found: {path}") + if path.is_dir(): + table = ds.dataset(path, format="parquet").to_table() + else: + table = ds.dataset(path, format="parquet").to_table() + return table.to_pandas() diff --git a/python/spatialdata-experimental-writer/tests/test_integration.py b/python/spatialdata-experimental-writer/tests/test_integration.py new file mode 100644 index 00000000..1495c6cb --- /dev/null +++ b/python/spatialdata-experimental-writer/tests/test_integration.py @@ -0,0 +1,234 @@ +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq + +from spatialdata_experimental_writer.index_permutations import write_index_permutations +from spatialdata_experimental_writer.errors import WriterCommandError +from spatialdata_experimental_writer.points import MORTON_CODE_2D_COLUMN, write_morton_points_parquet +from spatialdata_experimental_writer.runners import run_morton_points_from_zarr +from spatialdata_experimental_writer.zarr import list_points_keys + + +def _write_points_element( + zarr_root: Path, + key: str, + *, + feature_key: str = "feature_name", + rows: int = 200, +) -> None: + element_dir = zarr_root / "points" / key + parquet_path = element_dir / "points.parquet" + element_dir.mkdir(parents=True) + rng = pd.Series(range(rows)) + table = pa.Table.from_pandas( + pd.DataFrame( + { + "x": (rng.astype("float64") % 100).tolist(), + "y": ((rng * 3).astype("float64") % 100).tolist(), + "feature_name": (["gene_a", "gene_b", "gene_c"] * rows)[:rows], + } + ), + preserve_index=False, + ) + pq.write_table(table, parquet_path) + element_dir.joinpath("zarr.json").write_text( + json.dumps( + { + "attributes": { + "encoding-type": "ngff:points", + "axes": ["x", "y"], + "spatialdata_attrs": { + "feature_key": feature_key, + "version": "0.2", + }, + }, + "zarr_format": 3, + "node_type": "group", + } + ) + ) + zarr_root.joinpath("zarr.json").write_text( + json.dumps({"zarr_format": 3, "node_type": "group"}) + ) + + +def test_morton_parquet_does_not_persist_uint_columns(tmp_path: Path) -> None: + df = pd.DataFrame( + { + "x": [0.0, 10.0, 5.0, 2.0, 8.0], + "y": [3.0, 4.0, 20.0, 0.0, 9.0], + "feature_name": ["b", "a", "b", "c", "a"], + } + ) + output = tmp_path / "points.parquet" + write_morton_points_parquet(df, output, feature_key="feature_name", row_group_size=2) + columns = pq.ParquetFile(output).schema_arrow.names + assert "x_uint" not in columns + assert "y_uint" not in columns + assert MORTON_CODE_2D_COLUMN in columns + assert "feature_name_codes" in columns + + +def test_morton_points_from_zarr_defaults_to_canonical_path(tmp_path: Path) -> None: + zarr_root = tmp_path / "store.zarr" + _write_points_element(zarr_root, "transcripts") + canonical = zarr_root / "points" / "transcripts" / "points.parquet" + + subprocess.run( + [ + sys.executable, + "-m", + "spatialdata_experimental_writer.cli", + "morton-points-from-zarr", + str(zarr_root), + "--points-key", + "transcripts", + "--row-group-size", + "50", + ], + check=True, + cwd=Path(__file__).resolve().parents[1], + ) + + assert canonical.is_file() + columns = pq.ParquetFile(canonical).schema_arrow.names + assert MORTON_CODE_2D_COLUMN in columns + assert "feature_name_codes" in columns + + +def test_cli_expected_error_has_no_traceback(tmp_path: Path) -> None: + missing = tmp_path / "missing.zarr" + + result = subprocess.run( + [ + sys.executable, + "-m", + "spatialdata_experimental_writer.cli", + "list-points", + str(missing), + ], + check=False, + cwd=Path(__file__).resolve().parents[1], + capture_output=True, + text=True, + ) + + assert result.returncode != 0 + assert "No Points elements found" in result.stderr + assert "Traceback" not in result.stderr + + +def test_morton_points_from_zarr_output_points_key_writes_element(tmp_path: Path) -> None: + zarr_root = tmp_path / "store.zarr" + _write_points_element(zarr_root, "transcripts") + + result = run_morton_points_from_zarr( + zarr_root, + points_key="transcripts", + output_points_key="transcripts_morton", + row_group_size=50, + ) + + output = zarr_root / "points" / "transcripts_morton" / "points.parquet" + assert result["output_points_key"] == "transcripts_morton" + assert result["output"] == str(output) + assert output.is_file() + assert (zarr_root / "points" / "transcripts_morton" / "zarr.json").is_file() + assert list_points_keys(zarr_root) == ["transcripts", "transcripts_morton"] + metadata = json.loads(zarr_root.joinpath("zarr.json").read_text())[ + "consolidated_metadata" + ]["metadata"] + assert "points/transcripts_morton" in metadata + + +def test_morton_points_from_zarr_output_points_key_can_be_experimental( + tmp_path: Path, +) -> None: + zarr_root = tmp_path / "store.zarr" + _write_points_element(zarr_root, "transcripts") + + result = run_morton_points_from_zarr( + zarr_root, + points_key="transcripts", + experimental=True, + output_points_key="transcripts_morton", + row_group_size=50, + ) + + output = zarr_root / "points.experimental" / "transcripts_morton" / "points.parquet" + assert result["output_collection"] == "points.experimental" + assert result["output_points_key"] == "transcripts_morton" + assert result["output"] == str(output) + assert output.is_file() + assert not (zarr_root / "points" / "transcripts_morton").exists() + + +def test_morton_points_from_zarr_requires_overwrite_for_existing_output_element( + tmp_path: Path, +) -> None: + zarr_root = tmp_path / "store.zarr" + _write_points_element(zarr_root, "transcripts") + _write_points_element(zarr_root, "transcripts_morton") + + try: + run_morton_points_from_zarr( + zarr_root, + points_key="transcripts", + output_points_key="transcripts_morton", + row_group_size=50, + ) + except WriterCommandError as exc: + assert "already exists" in str(exc) + else: + raise AssertionError("expected WriterCommandError") + + run_morton_points_from_zarr( + zarr_root, + points_key="transcripts", + output_points_key="transcripts_morton", + overwrite=True, + row_group_size=50, + ) + assert (zarr_root / "points" / "transcripts_morton" / "points.parquet").is_file() + + +def test_write_index_permutations_writes_manifest(tmp_path: Path) -> None: + source = tmp_path / "source.zarr" + dest = tmp_path / "dest.zarr" + _write_points_element(source, "transcripts", rows=120) + + manifest = write_index_permutations( + source, + dest, + points_key="transcripts", + row_group_size=40, + conditions=tuple( + condition + for condition in __import__( + "spatialdata_experimental_writer.index_permutations", + fromlist=["DEFAULT_CONDITIONS"], + ).DEFAULT_CONDITIONS + if condition.id in {"canonical", "morton"} + ), + ) + + assert (dest / "index-manifest.json").exists() + assert manifest["source_element"] == "points/transcripts" + assert (dest / "points" / "transcripts" / "points.parquet").exists() + assert (dest / "points" / "transcripts_morton" / "points.parquet").exists() + assert list_points_keys(dest) == ["transcripts", "transcripts_morton"] + consolidated = json.loads((dest / "zarr.json").read_text())["consolidated_metadata"][ + "metadata" + ] + assert "points/transcripts_morton" in consolidated + morton_columns = pq.ParquetFile( + dest / "points" / "transcripts_morton" / "points.parquet" + ).schema_arrow.names + assert MORTON_CODE_2D_COLUMN in morton_columns diff --git a/python/spatialdata-experimental-writer/tests/test_points.py b/python/spatialdata-experimental-writer/tests/test_points.py new file mode 100644 index 00000000..f1f40b6a --- /dev/null +++ b/python/spatialdata-experimental-writer/tests/test_points.py @@ -0,0 +1,121 @@ +from __future__ import annotations + +import json + +import pandas as pd +import pyarrow.parquet as pq + +from spatialdata_experimental_writer import ( + MORTON_CODE_2D_COLUMN, + build_spatialdata_multiscale_metadata, + morton_sort_points, + write_morton_points_parquet, + write_multiscale_points_parquet, +) + + +def test_morton_sort_points_adds_sentinel_rows_and_feature_codes() -> None: + df = pd.DataFrame( + { + "x": [0.0, 10.0, 5.0, 2.0], + "y": [3.0, 4.0, 20.0, 0.0], + "feature_name": ["b", "a", "b", "c"], + } + ) + + sorted_df = morton_sort_points(df, feature_key="feature_name") + + assert MORTON_CODE_2D_COLUMN in sorted_df.columns + assert "feature_name_codes" in sorted_df.columns + assert sorted_df[MORTON_CODE_2D_COLUMN].iloc[:4].eq(0).all() + assert sorted_df.columns[-1] == "feature_name" + + +def test_morton_sort_points_uses_extreme_row_positions_not_duplicate_index_labels() -> None: + df = pd.DataFrame( + { + "x": [0.0, 50.0, 100.0, 25.0, 75.0, 10.0], + "y": [50.0, 100.0, 25.0, 0.0, 75.0, 10.0], + "feature_name": ["x_min", "y_max", "x_max", "y_min", "other", "near_min"], + }, + index=[7, 7, 8, 8, 9, 9], + ) + + sorted_df = morton_sort_points(df, feature_key="feature_name") + + sentinel = sorted_df.iloc[:4] + assert sentinel[MORTON_CODE_2D_COLUMN].eq(0).all() + assert sentinel["x"].min() == 0.0 + assert sentinel["x"].max() == 100.0 + assert sentinel["y"].min() == 0.0 + assert sentinel["y"].max() == 100.0 + assert sorted_df["feature_name"].value_counts().to_dict() == { + "x_min": 1, + "y_max": 1, + "x_max": 1, + "y_min": 1, + "other": 1, + "near_min": 1, + } + + +def test_write_morton_points_parquet_uses_small_sentinel_row_group(tmp_path) -> None: + df = pd.DataFrame( + { + "x": [0.0, 10.0, 5.0, 2.0, 8.0], + "y": [3.0, 4.0, 20.0, 0.0, 9.0], + "feature_name": ["b", "a", "b", "c", "a"], + } + ) + output = tmp_path / "points.parquet" + + write_morton_points_parquet(df, output, feature_key="feature_name", row_group_size=2) + + parquet = pq.ParquetFile(output) + assert parquet.num_row_groups >= 2 + assert parquet.metadata.row_group(0).num_rows <= 4 + + +def test_write_morton_points_parquet_keeps_quantized_zero_points_out_of_sentinel_row_group( + tmp_path, +) -> None: + df = pd.DataFrame( + { + "x": [0.0, 10.0, 0.00001, 5.0, 2.0], + "y": [0.0, 20.0, 0.00001, 10.0, 7.0], + "feature_name": ["min", "max", "near_min", "mid", "other"], + } + ) + output = tmp_path / "points.parquet" + + sorted_df = write_morton_points_parquet( + df, + output, + feature_key="feature_name", + row_group_size=2, + ) + + assert sorted_df[MORTON_CODE_2D_COLUMN].iloc[:3].eq(0).all() + parquet = pq.ParquetFile(output) + assert parquet.metadata.row_group(0).num_rows == 2 + + +def test_write_multiscale_points_parquet_stores_metadata(tmp_path) -> None: + df = pd.DataFrame( + { + "x": [0.0, 10.0], + "y": [3.0, 4.0], + "__spatial_index__": [0, 1], + "__morton__": [0, 1], + } + ) + output = tmp_path / "points.parquet" + metadata = build_spatialdata_multiscale_metadata(df, axes=("x", "y")) + + write_multiscale_points_parquet(df, output, metadata=metadata, row_group_size=2) + + schema_metadata = pq.ParquetFile(output).schema_arrow.metadata + assert schema_metadata is not None + stored = json.loads(schema_metadata[b"spatialdata_multiscale"]) + assert stored["format"] == "spatialdata_multiscale_points" + assert stored["bounding_box"]["min"] == [0.0, 3.0] diff --git a/python/spatialdata-experimental-writer/tests/test_verify.py b/python/spatialdata-experimental-writer/tests/test_verify.py new file mode 100644 index 00000000..796e9cba --- /dev/null +++ b/python/spatialdata-experimental-writer/tests/test_verify.py @@ -0,0 +1,166 @@ +from __future__ import annotations + +import json + +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq + +from spatialdata_experimental_writer.index_permutations import write_index_permutations +from spatialdata_experimental_writer.points import ( + MORTON_CODE_2D_COLUMN, + build_spatialdata_multiscale_metadata, + write_morton_points_parquet, + write_multiscale_points_parquet, +) +from spatialdata_experimental_writer.verify import ( + all_passed, + verify_index_permutations_manifest, + verify_morton_parquet, + verify_multiscale_parquet, +) + + +def test_verify_morton_parquet_passes_for_writer_output(tmp_path) -> None: + df = pd.DataFrame( + { + "x": [0.0, 10.0, 5.0, 2.0, 8.0], + "y": [3.0, 4.0, 20.0, 0.0, 9.0], + "feature_name": ["b", "a", "b", "c", "a"], + } + ) + output = tmp_path / "points.parquet" + write_morton_points_parquet(df, output, feature_key="feature_name", row_group_size=2) + + checks = verify_morton_parquet(output) + assert all_passed(checks) + + +def test_verify_morton_parquet_uses_sentinel_row_group_boundary(tmp_path) -> None: + df = pd.DataFrame( + { + "x": [0.0, 10.0, 0.00001, 5.0, 2.0], + "y": [0.0, 20.0, 0.00001, 10.0, 7.0], + "feature_name": ["min", "max", "near_min", "mid", "other"], + } + ) + output = tmp_path / "points.parquet" + write_morton_points_parquet(df, output, feature_key="feature_name", row_group_size=2) + + checks = verify_morton_parquet(output) + + assert all_passed(checks) + sentinel = next(check for check in checks if check.id == "sentinel_prefix") + row_group = next(check for check in checks if check.id == "row_group_sentinels") + assert "sentinel prefix rows: 2" in sentinel.detail + assert "row group 0 has 2 rows" in row_group.detail + + +def test_verify_morton_parquet_fails_for_unsorted_tail(tmp_path) -> None: + rows = 40 + rng = pd.Series(range(rows)) + df = pd.DataFrame( + { + "x": (rng.astype("float64") % 20).tolist(), + "y": ((rng * 3).astype("float64") % 20).tolist(), + "feature_name": (["a", "b", "c"] * rows)[:rows], + } + ) + output = tmp_path / "points.parquet" + write_morton_points_parquet(df, output, feature_key="feature_name", row_group_size=8) + + table = pq.read_table(output) + pdf = table.to_pandas() + sentinel_count = int((pdf[MORTON_CODE_2D_COLUMN].head(4) == 0).sum()) + assert sentinel_count < len(pdf) - 1 + later = sentinel_count + 1 + pdf.at[pdf.index[later], MORTON_CODE_2D_COLUMN] = int( + pdf[MORTON_CODE_2D_COLUMN].iloc[sentinel_count] + ) - 1 + pq.write_table(pa.Table.from_pandas(pdf, preserve_index=False), output) + + checks = verify_morton_parquet(output) + monotonic = next(check for check in checks if check.id == "morton_monotonic") + assert not monotonic.passed + + +def test_verify_morton_parquet_reports_missing_coordinate_columns(tmp_path) -> None: + output = tmp_path / "points.parquet" + table = pa.table({MORTON_CODE_2D_COLUMN: pa.array([0, 1], type=pa.uint64())}) + pq.write_table(table, output) + + checks = verify_morton_parquet(output) + + x_check = next(check for check in checks if check.id == "x_column_present") + y_check = next(check for check in checks if check.id == "y_column_present") + assert not x_check.passed + assert not y_check.passed + assert not all_passed(checks) + + +def test_verify_multiscale_parquet(tmp_path) -> None: + df = pd.DataFrame( + { + "x": [0.0, 10.0], + "y": [3.0, 4.0], + "__spatial_index__": [0, 1], + "__morton__": [0, 1], + } + ) + output = tmp_path / "points.parquet" + metadata = build_spatialdata_multiscale_metadata(df, axes=("x", "y")) + write_multiscale_points_parquet(df, output, metadata=metadata, row_group_size=2) + + checks = verify_multiscale_parquet(output) + assert all_passed(checks) + + +def test_verify_index_permutations_manifest(tmp_path) -> None: + source = tmp_path / "source.zarr" + dest = tmp_path / "dest.zarr" + element_dir = source / "points" / "transcripts" + element_dir.mkdir(parents=True) + df = pd.DataFrame( + { + "x": [0.0, 10.0, 5.0, 2.0], + "y": [3.0, 4.0, 20.0, 0.0], + "feature_name": ["b", "a", "b", "c"], + } + ) + pq.write_table(pa.Table.from_pandas(df, preserve_index=False), element_dir / "points.parquet") + element_dir.joinpath("zarr.json").write_text( + json.dumps( + { + "attributes": { + "encoding-type": "ngff:points", + "axes": ["x", "y"], + "spatialdata_attrs": {"feature_key": "feature_name", "version": "0.2"}, + }, + "zarr_format": 3, + "node_type": "group", + } + ) + ) + source.joinpath("zarr.json").write_text( + json.dumps({"zarr_format": 3, "node_type": "group"}) + ) + + write_index_permutations( + source, + dest, + points_key="transcripts", + row_group_size=2, + conditions=tuple( + condition + for condition in __import__( + "spatialdata_experimental_writer.index_permutations", + fromlist=["DEFAULT_CONDITIONS"], + ).DEFAULT_CONDITIONS + if condition.id in {"canonical", "morton"} + ), + ) + + checks = verify_index_permutations_manifest(dest) + morton_checks = [check for check in checks if check.id.endswith("morton_monotonic")] + assert morton_checks + assert all(check.passed for check in morton_checks) diff --git a/python/spatialdata-experimental-writer/tests/test_zarr.py b/python/spatialdata-experimental-writer/tests/test_zarr.py new file mode 100644 index 00000000..42d979ff --- /dev/null +++ b/python/spatialdata-experimental-writer/tests/test_zarr.py @@ -0,0 +1,113 @@ +from __future__ import annotations + +import json +from pathlib import Path + +import pandas as pd +import pyarrow as pa +import pyarrow.parquet as pq +import pytest + +from spatialdata_experimental_writer.zarr import ( + experimental_points_output_path, + list_points_keys, + points_parquet_path, + read_points_dataframe, + read_points_element_attrs, + register_points_elements_in_consolidated_metadata, + validate_points_key, +) + + +def _write_points_element( + zarr_root: Path, + key: str, + *, + feature_key: str = "feature_name", +) -> None: + element_dir = zarr_root / "points" / key + parquet_dir = element_dir / "points.parquet" + parquet_dir.mkdir(parents=True) + table = pa.Table.from_pandas( + pd.DataFrame( + { + "x": [0.0, 1.0], + "y": [2.0, 3.0], + "feature_name": ["a", "b"], + } + ), + preserve_index=False, + ) + pq.write_table(table, parquet_dir / "part.0.parquet") + element_dir.joinpath("zarr.json").write_text( + json.dumps( + { + "attributes": { + "encoding-type": "ngff:points", + "axes": ["x", "y"], + "spatialdata_attrs": { + "feature_key": feature_key, + "version": "0.2", + }, + }, + "zarr_format": 3, + "node_type": "group", + } + ) + ) + + +def test_register_points_elements_in_consolidated_metadata(tmp_path: Path) -> None: + _write_points_element(tmp_path, "transcripts") + root_json = tmp_path / "zarr.json" + root_json.write_text( + json.dumps( + { + "zarr_format": 3, + "node_type": "group", + "consolidated_metadata": { + "kind": "inline", + "metadata": { + "points/transcripts": { + "attributes": { + "encoding-type": "ngff:points", + "axes": ["x", "y"], + "spatialdata_attrs": { + "feature_key": "feature_name", + "version": "0.2", + }, + }, + "node_type": "group", + "zarr_format": 3, + } + }, + }, + } + ) + ) + register_points_elements_in_consolidated_metadata( + tmp_path, + ["transcripts", "transcripts_morton"], + template_key="transcripts", + ) + metadata = json.loads(root_json.read_text())["consolidated_metadata"]["metadata"] + assert "points/transcripts_morton" in metadata + assert metadata["points/transcripts_morton"]["attributes"]["encoding-type"] == "ngff:points" + + +def test_list_points_keys_and_read_element(tmp_path: Path) -> None: + _write_points_element(tmp_path, "transcripts") + assert list_points_keys(tmp_path) == ["transcripts"] + attrs = read_points_element_attrs(tmp_path, "transcripts") + assert attrs["feature_key"] == "feature_name" + df = read_points_dataframe(points_parquet_path(tmp_path, "transcripts")) + assert list(df.columns) == ["x", "y", "feature_name"] + assert experimental_points_output_path(tmp_path, "transcripts") == ( + tmp_path / "points.experimental" / "transcripts" / "points.parquet" + ) + + +def test_validate_points_key_rejects_paths() -> None: + assert validate_points_key(" transcripts_morton ") == "transcripts_morton" + with pytest.raises(ValueError, match="single name"): + validate_points_key("nested/transcripts_morton") diff --git a/python/spatialdata-experimental-writer/uv.lock b/python/spatialdata-experimental-writer/uv.lock new file mode 100644 index 00000000..17728657 --- /dev/null +++ b/python/spatialdata-experimental-writer/uv.lock @@ -0,0 +1,394 @@ +version = 1 +revision = 3 +requires-python = ">=3.12" +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'win32'", + "python_full_version >= '3.14' and sys_platform == 'emscripten'", + "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", + "python_full_version < '3.14' and sys_platform == 'win32'", + "python_full_version < '3.14' and sys_platform == 'emscripten'", + "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32'", +] + +[[package]] +name = "colorama" +version = "0.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d8/53/6f443c9a4a8358a93a6792e2acffb9d9d5cb0a5cfd8802644b7b1c9a02e4/colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44", size = 27697, upload-time = "2022-10-25T02:36:22.414Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, +] + +[[package]] +name = "iniconfig" +version = "2.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/34/14ca021ce8e5dfedc35312d08ba8bf51fdd999c576889fc2c24cb97f4f10/iniconfig-2.3.0.tar.gz", hash = "sha256:c76315c77db068650d49c5b56314774a7804df16fee4402c1f19d6d15d8c4730", size = 20503, upload-time = "2025-10-18T21:55:43.219Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, +] + +[[package]] +name = "linkify-it-py" +version = "2.1.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "uc-micro-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/2e/c9/06ea13676ef354f0af6169587ae292d3e2406e212876a413bf9eece4eb23/linkify_it_py-2.1.0.tar.gz", hash = "sha256:43360231720999c10e9328dc3691160e27a718e280673d444c38d7d3aaa3b98b", size = 29158, upload-time = "2026-03-01T07:48:47.683Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b4/de/88b3be5c31b22333b3ca2f6ff1de4e863d8fe45aaea7485f591970ec1d3e/linkify_it_py-2.1.0-py3-none-any.whl", hash = "sha256:0d252c1594ecba2ecedc444053db5d3a9b7ec1b0dd929c8f1d74dce89f86c05e", size = 19878, upload-time = "2026-03-01T07:48:46.098Z" }, +] + +[[package]] +name = "markdown-it-py" +version = "4.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mdurl" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/06/ff/7841249c247aa650a76b9ee4bbaeae59370dc8bfd2f6c01f3630c35eb134/markdown_it_py-4.2.0.tar.gz", hash = "sha256:04a21681d6fbb623de53f6f364d352309d4094dd4194040a10fd51833e418d49", size = 82454, upload-time = "2026-05-07T12:08:28.36Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/81/4da04ced5a082363ecfa159c010d200ecbd959ae410c10c0264a38cac0f5/markdown_it_py-4.2.0-py3-none-any.whl", hash = "sha256:9f7ebbcd14fe59494226453aed97c1070d83f8d24b6fc3a3bcf9a38092641c4a", size = 91687, upload-time = "2026-05-07T12:08:27.182Z" }, +] + +[package.optional-dependencies] +linkify = [ + { name = "linkify-it-py" }, +] + +[[package]] +name = "mdit-py-plugins" +version = "0.6.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/59/fc/f8d0863f8862f25602c0404d75568e89fb6b4109804645e5cdfb1be5cf56/mdit_py_plugins-0.6.1.tar.gz", hash = "sha256:a2bca0f039f39dbd35fb74ae1b5f998608c437463371f0ff7f49a19a17a114d0", size = 56114, upload-time = "2026-05-13T09:03:38.91Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a5/69/6da5581c6a7fede7dc261bf4e67d6adca4196f176b43288b55b3db395b6e/mdit_py_plugins-0.6.1-py3-none-any.whl", hash = "sha256:214c82fb2ac524472ab6a5bcab1de80f73b50443e187f401bfd77efbc7c6481d", size = 66663, upload-time = "2026-05-13T09:03:37.76Z" }, +] + +[[package]] +name = "mdurl" +version = "0.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d6/54/cfe61301667036ec958cb99bd3efefba235e65cdeb9c84d24a8293ba1d90/mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba", size = 8729, upload-time = "2022-08-14T12:40:10.846Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b3/38/89ba8ad64ae25be8de66a6d463314cf1eb366222074cfda9ee839c56a4b4/mdurl-0.1.2-py3-none-any.whl", hash = "sha256:84008a41e51615a49fc9966191ff91509e3c40b939176e643fd50a5c2196b8f8", size = 9979, upload-time = "2022-08-14T12:40:09.779Z" }, +] + +[[package]] +name = "numpy" +version = "2.4.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d0/ad/fed0499ce6a338d2a03ebae59cd15093910c8875328855781952abf6c2fe/numpy-2.4.6.tar.gz", hash = "sha256:f3a3570c4a2a16746ac2c31a7c7c7b0c186b95ce902e33db6f28094ed7387dda", size = 20735807, upload-time = "2026-05-18T23:37:14.07Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/95/2a/3d7b5ac8aac24feaf9ad7ed58f45b0bbc06d37e4338ae84c9f2298b570f9/numpy-2.4.6-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:001fbb8e08d942dd57599e781f2472269ee7f2755fae407b4f67b2f0b17da3f1", size = 16689119, upload-time = "2026-05-18T23:33:54.065Z" }, + { url = "https://files.pythonhosted.org/packages/ea/12/92c4c131527599e8288d6918e888d88726f84d805d784b771f32408aeaef/numpy-2.4.6-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:ebfb099f8dcf083deef3ac1ca4c1503f387cf76296fcb3816b66f5ecb5f54fdb", size = 14699246, upload-time = "2026-05-18T23:33:57.621Z" }, + { url = "https://files.pythonhosted.org/packages/ad/fe/c0a6b7b2ca128a8fb228575147073b660656734b8ebe4d76c8fd748dcc79/numpy-2.4.6-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:3213d622a0283a39a93d188f3cf72b26862df52fbb4ca3697f51705016523d41", size = 5204410, upload-time = "2026-05-18T23:34:00.302Z" }, + { url = "https://files.pythonhosted.org/packages/f3/d4/9770d14ba719432bb90a421bfd443872ed0f70f7264b64bec12ea363d5fd/numpy-2.4.6-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:357cc07a6d7b0b182ff02249616a03742827ebb1277546b5c7cd7f7620a45698", size = 6551240, upload-time = "2026-05-18T23:34:02.852Z" }, + { url = "https://files.pythonhosted.org/packages/c9/c6/50a46a6205feba2343f1d6d17438107c5dc491ed1c736e6ea68689fd906b/numpy-2.4.6-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f9fb9157b4ce2971008323afe46053787b526ef624fea915b261468a8421a0f", size = 15671012, upload-time = "2026-05-18T23:34:05.485Z" }, + { url = "https://files.pythonhosted.org/packages/99/60/14115e6364fa676c5397c2ad3004e527e9aa487abf5d0706ec81bbd08529/numpy-2.4.6-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:90f9849678c75fe7afa2d348ac842c168b0a4d3d61919687216dfc547976d853", size = 16645538, upload-time = "2026-05-18T23:34:09.265Z" }, + { url = "https://files.pythonhosted.org/packages/ae/c5/693cbe59e57db94d2231fa519ca3978dc9e19da5a8f088588f5c6e947ff2/numpy-2.4.6-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c1a2af6c6ef86344a6b0db6b97834208bf598db514f2b155042439b62605601a", size = 17020706, upload-time = "2026-05-18T23:34:13.053Z" }, + { url = "https://files.pythonhosted.org/packages/ef/fc/85b7c4eff9b4966ade25c2273cf7e7012e92366c032058653934b37de044/numpy-2.4.6-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e5805d5a22fd19c8ccff10a9561f9df94436b0545619ea579db2d3c35294bce2", size = 18368541, upload-time = "2026-05-18T23:34:17.024Z" }, + { url = "https://files.pythonhosted.org/packages/f6/81/e1b27545deedce7f4a0b348618c6b62d74e36a4dc9ccd42f3eb2f85eee32/numpy-2.4.6-cp312-cp312-win32.whl", hash = "sha256:e3eeb0aabd6bd5ce64faae67e9935203a6991b4bc2a485a767fbafb2c5125f45", size = 5962825, upload-time = "2026-05-18T23:34:20.3Z" }, + { url = "https://files.pythonhosted.org/packages/ab/ca/feab00bd44aa5fe1ad2c18f08b4d3bb92e26484b0b1d1443897809ed528c/numpy-2.4.6-cp312-cp312-win_amd64.whl", hash = "sha256:d8e8286dd7cea7895157318d1b91cdacac64c479f3cbc8dce548331728484751", size = 12321687, upload-time = "2026-05-18T23:34:23.095Z" }, + { url = "https://files.pythonhosted.org/packages/63/cf/5a6d34850a39d1093558564f77ee8e8e0bee5061151b8f05a55711001ec7/numpy-2.4.6-cp312-cp312-win_arm64.whl", hash = "sha256:4081eb135ac24158bd51cdfbef16f1c64df7063b1143f24731387137c092bec8", size = 10221482, upload-time = "2026-05-18T23:34:25.876Z" }, + { url = "https://files.pythonhosted.org/packages/fb/82/bdab26d7438c6791ca31b7c024ca37c1eab8b726ba236129005cd4a06e45/numpy-2.4.6-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:511dbaf848decaaaf4b4ca48032619fb3138710c4bf7da7617765edad1ef96b0", size = 16684648, upload-time = "2026-05-18T23:34:29.41Z" }, + { url = "https://files.pythonhosted.org/packages/1b/30/a80189bcc7f5e4258b3fbc3968d909d1756f54d023299ecc39ad6fdb9ef8/numpy-2.4.6-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:bf162abab1c1a736333192707cef898e735a5ca00f38f27eeedf44b39d9e85eb", size = 14693902, upload-time = "2026-05-18T23:34:33.013Z" }, + { url = "https://files.pythonhosted.org/packages/97/12/70b5d0d7c15e1ebb8a6a84a8caa1d19e181d84fb58bb6d70aca29099dec1/numpy-2.4.6-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:043191bfa8eab18c776647b62723ac9dddece59743b13f49b2016094129c2b3f", size = 5198992, upload-time = "2026-05-18T23:34:36.132Z" }, + { url = "https://files.pythonhosted.org/packages/ba/8c/ebd2a8f8a83541f8d38cc5667e8c2b69cecfd30da6e45693e8158857d44b/numpy-2.4.6-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:6180d8b35af935aed8ece3a85e0a43f87393ae0ac87c8d2c8bd2c993f7270ef3", size = 6546944, upload-time = "2026-05-18T23:34:38.484Z" }, + { url = "https://files.pythonhosted.org/packages/bb/c5/7b863a97a91671a0338f4253bd3b5a3d3852f0692dae91711c9f4a10e787/numpy-2.4.6-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:72fbe16c6fac95aedf5937fa873445cec2110be35d8a4e9433d7501fd98dae6b", size = 15669392, upload-time = "2026-05-18T23:34:41.257Z" }, + { url = "https://files.pythonhosted.org/packages/a5/9d/3584b9984ca4c047aea75214ce1a4c4c73d849bd71b604264b7f5653f8a8/numpy-2.4.6-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a7830bab239b79cda9c08c2da014761cafb48da6150e1da17ac06283f43b6089", size = 16633220, upload-time = "2026-05-18T23:34:45.075Z" }, + { url = "https://files.pythonhosted.org/packages/05/ae/7c67fba23bd98caec7c99261f3a16072ade14813486b0282cb29846de832/numpy-2.4.6-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ef4aea96ce4d3b074422cb4f2f64e216bf9e213004bb58ecfdf50ea02ea8eb9a", size = 17020800, upload-time = "2026-05-18T23:34:49.065Z" }, + { url = "https://files.pythonhosted.org/packages/d9/5d/3b6725cb31d983c5e66916f5d36f6d7e5521129e4c4404d64f918292a5b6/numpy-2.4.6-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:dfa20cc6ca228e6b155b11da03825975ce66aea520985dbbddf0f2a5a495c605", size = 18357600, upload-time = "2026-05-18T23:34:52.709Z" }, + { url = "https://files.pythonhosted.org/packages/f7/da/2ccc6c2fe8898dee01d90c75c5f5f914a23daf99e3e0f59516a08760c8b5/numpy-2.4.6-cp313-cp313-win32.whl", hash = "sha256:56b39e5e0622a09a25bf5baf62f4bcf0cb8a41ae6e2819cf49bbc5a74c083f91", size = 5961134, upload-time = "2026-05-18T23:34:55.618Z" }, + { url = "https://files.pythonhosted.org/packages/b5/cd/9cc4dc876fb065d5c220aae4d5e14826b2715331bb7618ce1fb07a679d99/numpy-2.4.6-cp313-cp313-win_amd64.whl", hash = "sha256:c4fc99836233ea196540b17ab0983aff60ed07941751930f5f4d05bc3b3b7359", size = 12318598, upload-time = "2026-05-18T23:34:58.928Z" }, + { url = "https://files.pythonhosted.org/packages/39/1e/c0bcba1f8694116485fe28fd1be698c278fcda4141c5b0e53a2aed8b12a8/numpy-2.4.6-cp313-cp313-win_arm64.whl", hash = "sha256:a7c711e21628b52034bb5ab8d1bce291f752fcc5e92accc615778acee1ff4778", size = 10222272, upload-time = "2026-05-18T23:35:02.167Z" }, + { url = "https://files.pythonhosted.org/packages/63/6d/cc5619247c8f4204e507f5883528372e4ac4bb189e579fb859a12e480b1f/numpy-2.4.6-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:112b06a867b235ef466ed3508ddf0238050df9c727cafb5301ac385b899189a1", size = 14821197, upload-time = "2026-05-18T23:35:05.468Z" }, + { url = "https://files.pythonhosted.org/packages/00/58/f1c39161c87d9e9bed660f1ed4bafc0e403d5ec9650b6dd77aead07d489b/numpy-2.4.6-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:eaf7fa2de5c0be8ae6ff8e9bea2ccd725e980541244521d8d4b5f3354a27babe", size = 5326287, upload-time = "2026-05-18T23:35:08.693Z" }, + { url = "https://files.pythonhosted.org/packages/af/57/3917ab0fd97f271a8694513581b8a36c655f111c446852c302f04ccdb6fc/numpy-2.4.6-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:7265a2f3d436e54ef9f2b52b5c937e6be778781bd97a590319d7348f1c1ca997", size = 6646763, upload-time = "2026-05-18T23:35:11.459Z" }, + { url = "https://files.pythonhosted.org/packages/eb/0f/037e64c494b67581ae18193d770adef354c41f3f2c8ebf865602d949bf8f/numpy-2.4.6-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f74a575920ab21fe304421a3fc28793d82e299cae9eccb37084e9fc7f3617c20", size = 15728070, upload-time = "2026-05-18T23:35:14.79Z" }, + { url = "https://files.pythonhosted.org/packages/21/a6/5d2bae9c9542eb4df16dc9c46dc79c186e9bad53805dfa5399a6023c6db0/numpy-2.4.6-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ede83e07a75dd06bc501566c1eca2afc0d61677c1472ac9ad93fdee6e638a48d", size = 16681752, upload-time = "2026-05-18T23:35:18.836Z" }, + { url = "https://files.pythonhosted.org/packages/92/14/23d1dfb410ae362cd59ce53e936b1513d545eb40db3949ced632e19a459e/numpy-2.4.6-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:68bb27509ac1b9a3443094260f6326150663b06abe40b73a2f81160623da5b67", size = 17086024, upload-time = "2026-05-18T23:35:22.52Z" }, + { url = "https://files.pythonhosted.org/packages/4b/6e/23595a2c642cdf3bc567877064bdd7f91c8b0038a4453cf2daf7248eafe9/numpy-2.4.6-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:a0df0043bdb289bde1f62da130d20df23d58b45429f752bc7a8fc5325a225ecd", size = 18403398, upload-time = "2026-05-18T23:35:26.398Z" }, + { url = "https://files.pythonhosted.org/packages/8a/90/0ac3bc947217e66dec77e7cbc6a1979d1af70b6461b82f620d3bccd5e4c8/numpy-2.4.6-cp313-cp313t-win32.whl", hash = "sha256:29a287e0cf63ff528da061de6b9f64a4618da591ca1046aafc54062e40ca7eab", size = 6084971, upload-time = "2026-05-18T23:35:29.387Z" }, + { url = "https://files.pythonhosted.org/packages/77/71/5673e351671a1d2bd6063b91b44f70c0affea7d1516fa7a6572941ba4aa1/numpy-2.4.6-cp313-cp313t-win_amd64.whl", hash = "sha256:25c692919ac5a01f170a3bfcd62d745b24fd095c353d50812637d6fcab442e75", size = 12458532, upload-time = "2026-05-18T23:35:32.175Z" }, + { url = "https://files.pythonhosted.org/packages/3f/88/19d3503c5046e688f049274b27a3ef3d771152fa80d3ba3d01a3dff61abe/numpy-2.4.6-cp313-cp313t-win_arm64.whl", hash = "sha256:1e978ec1e8bd0e0e4de6bb75de9d30cbb74db6b6a2bb727618613703ca0167dd", size = 10291881, upload-time = "2026-05-18T23:35:35.465Z" }, + { url = "https://files.pythonhosted.org/packages/f8/91/3ab2044d05fd16d343c5ac2e69b127f1b2854040dd20b193257c78028bd3/numpy-2.4.6-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:06ca2f61ec4385a07a6977c55ba998a4466c123642b4a32694d3128fce18c079", size = 16683458, upload-time = "2026-05-18T23:35:38.353Z" }, + { url = "https://files.pythonhosted.org/packages/8e/62/764ce66fa4147ae6d73071a3abf804ffe606f174618697c571acdf26a7c9/numpy-2.4.6-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:38efbc8de75c7a0fc1ac190162d892787f3f47b57cc291231aafee36b80982b7", size = 14704559, upload-time = "2026-05-18T23:35:42.14Z" }, + { url = "https://files.pythonhosted.org/packages/60/61/23f27c172f022e04025b7dc2367f4d63c1a398120607ec896228649a6f48/numpy-2.4.6-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:d581b735e177fdcdce6fed8e7e8880a3fb6ee4e3653a3ac6af01c6f4c03effc5", size = 5209716, upload-time = "2026-05-18T23:35:45.377Z" }, + { url = "https://files.pythonhosted.org/packages/03/71/21cf70dc6ea3e3acb95fc53a265b2fc248b981f0194ceb5b475271b8809d/numpy-2.4.6-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:0a041d3d761dc3c35cc56ce0351506a02bcbc25f7b169f652435141a17db9096", size = 6543947, upload-time = "2026-05-18T23:35:47.926Z" }, + { url = "https://files.pythonhosted.org/packages/d5/91/64288395ee1799bd2e0b04a305dce9666da90c961e1f3fe982a05ee1c036/numpy-2.4.6-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:40fdc1ae7125e518ea98e53e69a4ebc27e1fd50510c47b7ea130cf21e5e1d42b", size = 15685197, upload-time = "2026-05-18T23:35:50.863Z" }, + { url = "https://files.pythonhosted.org/packages/f3/eb/ebffaa97dc55502df69584a8f0dcf07f69a3e0b3e2323670a2722db9aa39/numpy-2.4.6-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a2c306dea656c12c68f51f4cea133cbe78ca7435eb28c735eac1d3ebe73be6e8", size = 16638245, upload-time = "2026-05-18T23:35:54.752Z" }, + { url = "https://files.pythonhosted.org/packages/b8/0b/54f9da33128d7e350fab89c7455902eeae70349ee52bddb448dc4a576f45/numpy-2.4.6-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:33111801a01c12a8a1e3721f0a9232f8cfc8ae2c6b7098167e6f623c6073f402", size = 17036587, upload-time = "2026-05-18T23:35:58.355Z" }, + { url = "https://files.pythonhosted.org/packages/b6/f0/fdebc1052db1cc37c64beb22072d67cd6d1c71adca1299f53dec2b5e20d3/numpy-2.4.6-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ae506e6902902557576a26ff33eda8695e7ecb3cb36c3b573a0765dee114ebdb", size = 18363226, upload-time = "2026-05-18T23:36:02.845Z" }, + { url = "https://files.pythonhosted.org/packages/aa/b4/298628d98c72b57e57f7165ae6a481a1deaf6f3c28262a6e4c739c275930/numpy-2.4.6-cp314-cp314-win32.whl", hash = "sha256:aaf159caa35993cb1f56fb9b8e4610d35758e7ca005412eb1daa856a78c9c4b1", size = 6010196, upload-time = "2026-05-18T23:36:05.92Z" }, + { url = "https://files.pythonhosted.org/packages/df/ac/46de6dda46478f7942f839e094970be2d4a861e005c4b3bf07c92e291a09/numpy-2.4.6-cp314-cp314-win_amd64.whl", hash = "sha256:b507f5c4c1d508876d1819b6bf9a49d365b96320b5d4993426b33a23ca4b8261", size = 12450334, upload-time = "2026-05-18T23:36:09.107Z" }, + { url = "https://files.pythonhosted.org/packages/78/92/b8b798ac784102c0da830d2257d59358e3d3d90d1e2b3f2575dad976c5cf/numpy-2.4.6-cp314-cp314-win_arm64.whl", hash = "sha256:6f41ae150c4e32db4f3310cdaf64b1593a03dbabe29eec77fc9b50fe64061df6", size = 10495678, upload-time = "2026-05-18T23:36:12.766Z" }, + { url = "https://files.pythonhosted.org/packages/30/34/ec28d1aa8115971537c01469ab2011ee96827930f0a124de1000cc2a7ed7/numpy-2.4.6-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ece3d2cfe132e7d51f44a832b303895e6f2d499c5e74dfbdb06ee246147a304a", size = 14823672, upload-time = "2026-05-18T23:36:16.473Z" }, + { url = "https://files.pythonhosted.org/packages/16/bd/f6d1fede4e54e8042a7ff97bb495510f3c220f94bcd9e8b228e87c92cc0d/numpy-2.4.6-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:e3e5193ef5a3dc73bceee50f7fdc2c90dbb76c42df8d8fae3d1067a583df579e", size = 5328731, upload-time = "2026-05-18T23:36:19.767Z" }, + { url = "https://files.pythonhosted.org/packages/f4/f0/e105b9e2fd728a9910103884decd6951d9dd73896b914a98d9a231de02ee/numpy-2.4.6-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:17f9ade344e7d9b464a084d69bcf18fc691cb1db67c62ed80820bf4926d78f0e", size = 6649805, upload-time = "2026-05-18T23:36:22.266Z" }, + { url = "https://files.pythonhosted.org/packages/82/dd/1206a7ca6ab15e3f02069707ca96222e202af681bb73756da7527f3cb837/numpy-2.4.6-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9cd5ffd25db4e7ba6a375693b3fc0fc1791ec636c17db3720da19bde7180ec43", size = 15730496, upload-time = "2026-05-18T23:36:25.713Z" }, + { url = "https://files.pythonhosted.org/packages/51/e7/38d3ea825dcab85a591734decb2f6c67caa7c8367d374df1a1c3842f9b07/numpy-2.4.6-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7d92c3819208a60205a12a245c91ad70cb0a85336659b19b834205573ac8456e", size = 16679616, upload-time = "2026-05-18T23:36:29.652Z" }, + { url = "https://files.pythonhosted.org/packages/93/b7/caabfdf53edf663e0b4eb74d7d405d83baef09eb5e83bcd32d601d72b93e/numpy-2.4.6-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e85b752a1e912b70eaad4fafbd4d1238007ab221de2009b9a2f5ae7461239895", size = 17085145, upload-time = "2026-05-18T23:36:33.449Z" }, + { url = "https://files.pythonhosted.org/packages/f9/45/68d7c33a6bcf3e5aa3bdbd57a367e6f615286dfd6482f97e8ffeb734306e/numpy-2.4.6-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:29cb7f67d10b479ff07c17d33e39f78c07f71c40ef30d63c153d340e96cd3fb4", size = 18403813, upload-time = "2026-05-18T23:36:37.369Z" }, + { url = "https://files.pythonhosted.org/packages/9c/50/0753655aa844c99cd9e018aacf76f130f1bd81d881bb74bc0aef5d73a8ba/numpy-2.4.6-cp314-cp314t-win32.whl", hash = "sha256:260a5d70215b61ab4fadf5c7baacd64821842975eea312125ed3c39a6391b063", size = 6156982, upload-time = "2026-05-18T23:36:40.817Z" }, + { url = "https://files.pythonhosted.org/packages/b2/d4/7c67becf668f973cb490cec3e98dfd799d866f9c989a54d355672cfa0db6/numpy-2.4.6-cp314-cp314t-win_amd64.whl", hash = "sha256:81a1cca95ed5bb92aa8b10dd2cdc9a0d3853a50fad926c28b5d7e8ea54389627", size = 12638908, upload-time = "2026-05-18T23:36:43.996Z" }, + { url = "https://files.pythonhosted.org/packages/43/bb/e1c71a4295b1b1d1393d50dbb4f2a36283c6859d9d3892e84f00ec5a91d5/numpy-2.4.6-cp314-cp314t-win_arm64.whl", hash = "sha256:0c9136e14ed34a9e343a31c533d78a9813a69a3148332bce5e9821cb2f996e66", size = 10565867, upload-time = "2026-05-18T23:36:47.114Z" }, +] + +[[package]] +name = "packaging" +version = "26.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/f1/e7a6dd94a8d4a5626c03e4e99c87f241ba9e350cd9e6d75123f992427270/packaging-26.2.tar.gz", hash = "sha256:ff452ff5a3e828ce110190feff1178bb1f2ea2281fa2075aadb987c2fb221661", size = 228134, upload-time = "2026-04-24T20:15:23.917Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/df/b2/87e62e8c3e2f4b32e5fe99e0b86d576da1312593b39f47d8ceef365e95ed/packaging-26.2-py3-none-any.whl", hash = "sha256:5fc45236b9446107ff2415ce77c807cee2862cb6fac22b8a73826d0693b0980e", size = 100195, upload-time = "2026-04-24T20:15:22.081Z" }, +] + +[[package]] +name = "pandas" +version = "3.0.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy" }, + { name = "python-dateutil" }, + { name = "tzdata", marker = "sys_platform == 'emscripten' or sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f8/87/4341c6252d1c47b08768c3d25ac487362bf403f0313ddae4a2a26c9b1b4c/pandas-3.0.3.tar.gz", hash = "sha256:696a4a00a2a2a35d4e5deb3fc946641b96c944f02230e4f76137fe35d806c4fc", size = 4651414, upload-time = "2026-05-11T18:54:29.21Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/f1/392f8c5bfc16f66a0d2d41561c01627c228fe7ed2a0d056ef11315042570/pandas-3.0.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fed2ff7fd9779120e388e285fc029bd5cf9490cdd2e4166a9ee22c0e49a9ab09", size = 10357846, upload-time = "2026-05-11T18:52:36.143Z" }, + { url = "https://files.pythonhosted.org/packages/cf/3d/b16412745651e855f357e5e66930248688378853a6e2698a214e331fba1f/pandas-3.0.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:b168fc218fd80a6cbdbdbc1a97ddc7889ed057d7eb45f50d866ceab5f39904c4", size = 9899550, upload-time = "2026-05-11T18:52:38.976Z" }, + { url = "https://files.pythonhosted.org/packages/31/a8/fa2535168fffcedf67f4f6de28d2dd903a747ca7c8ea6989451aaeb3a92f/pandas-3.0.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0383c72c75cdcca61a9e116e611143902dbfd08bff356829c2f6d1cf40a9ca8c", size = 10412965, upload-time = "2026-05-11T18:52:41.915Z" }, + { url = "https://files.pythonhosted.org/packages/65/b6/09b01cdbc15224e2850365192d17b7bdebb8bdbd8780ed221fcdf0d9a515/pandas-3.0.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6dc0b3fd2169c9157deed50b4d519553a3655c8c6a96027136d654592be973a9", size = 10894600, upload-time = "2026-05-11T18:52:45.02Z" }, + { url = "https://files.pythonhosted.org/packages/c9/a4/2eb28f2fccb4ced4a2c79ab2a5dee9ade1ebf44922ebad6fea158c9f95d4/pandas-3.0.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:7e65d5407dc0b394f509699650e4a2ec01c0514f21850f453fa60f3be79a5dbf", size = 11422824, upload-time = "2026-05-11T18:52:48.058Z" }, + { url = "https://files.pythonhosted.org/packages/f8/45/830bb57f533a4604b355e07edcb8ea18cf88b5f94e5fca92f27052d7c597/pandas-3.0.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:f8894dc474d648fe7b6ff0ca9b0bd73950d19952bc1a6534540762c5d79d305c", size = 11950889, upload-time = "2026-05-11T18:52:50.905Z" }, + { url = "https://files.pythonhosted.org/packages/b9/c5/fc1b368f303087d20e8c9bf3d6ceb186263cfac0ade735cd938538bea839/pandas-3.0.3-cp312-cp312-win_amd64.whl", hash = "sha256:c7be265b62cef88e253a941e4698604973736dcfe242fdb5198f0f7bc473cdcc", size = 9755463, upload-time = "2026-05-11T18:52:53.386Z" }, + { url = "https://files.pythonhosted.org/packages/86/bd/fda8f9705b1b09c6ebe14bfc0fa0e4ec8584d54ea673628f157ff55131af/pandas-3.0.3-cp312-cp312-win_arm64.whl", hash = "sha256:557409bc4178e70ee8d9ddb494798e51ebf6ea59330f6be22c51bab2a7db6c49", size = 9066158, upload-time = "2026-05-11T18:52:56.038Z" }, + { url = "https://files.pythonhosted.org/packages/c5/90/62d8302883c44308c477e222c3daf7c813a34c8e96985882fbd53d964352/pandas-3.0.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:67b3b64c11910cfa29f4e94a14d3bff9ee693b6fc76055e7cad549cee0aec5fa", size = 10331071, upload-time = "2026-05-11T18:52:58.838Z" }, + { url = "https://files.pythonhosted.org/packages/7f/ae/6a6493c783a101f165e4356953ba3c74d6f77f0042fa7d753da9dfbb640c/pandas-3.0.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:39436b377d56d2a2e52d0395bdbee171f01068e99af5250509aceeb929f765c7", size = 9875690, upload-time = "2026-05-11T18:53:01.431Z" }, + { url = "https://files.pythonhosted.org/packages/62/7c/5df8e9f56c69a2769fbe9382a5ef8f2658c007e376434e1e2cbb57ad895f/pandas-3.0.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d4be06d68f9ddcfc645b87534911da79a8fbffc7573c80e0edcf42a5020624d8", size = 10381634, upload-time = "2026-05-11T18:53:04.393Z" }, + { url = "https://files.pythonhosted.org/packages/99/68/1237369725aa617bb358263d535803e3053fdbc593513ec5ed9c9896b5b6/pandas-3.0.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a4eeb6830daf35a71cc09649bd823e2b542dac246cdee9614c6e4bd65028cd6a", size = 10891243, upload-time = "2026-05-11T18:53:07.643Z" }, + { url = "https://files.pythonhosted.org/packages/25/93/77d108e8af7222b4a503ebde0e30215b1c2e4f8e53a526431890f22d5586/pandas-3.0.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1928e07221f82db493cd4af1e23c1bfca524a19a4699887975bff68f49a72bfb", size = 11388659, upload-time = "2026-05-11T18:53:10.634Z" }, + { url = "https://files.pythonhosted.org/packages/d0/bd/eff5b4399f332ac386c853f6cd2bd3fa2ca0061b9f36ecd9c4d7c4265649/pandas-3.0.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:51b1fe551acb77dac643c6fda86084d8d446c10fe64b06a9cc29c4cc8540e7f2", size = 11942880, upload-time = "2026-05-11T18:53:13.536Z" }, + { url = "https://files.pythonhosted.org/packages/2c/20/559ace4200982c3887d0b86bfd0d856a2143ef8ddab63cc07934951a964c/pandas-3.0.3-cp313-cp313-win_amd64.whl", hash = "sha256:a82d532a3351d435432cd913edbccaf8b8e01d4dd0e5ced5a8d2e8ecd94c7e44", size = 9757091, upload-time = "2026-05-11T18:53:16.306Z" }, + { url = "https://files.pythonhosted.org/packages/3a/66/69055a09fe200f29f922a3eeec4804611900b95f52d932ece3393c3c0c19/pandas-3.0.3-cp313-cp313-win_arm64.whl", hash = "sha256:275c14e0fce14a2ec20eee474aecd305478ea3c1e6f6a9d8fe219a165542717e", size = 9057282, upload-time = "2026-05-11T18:53:18.768Z" }, + { url = "https://files.pythonhosted.org/packages/57/0e/efe801b0e6811e8e650cd21b7f2608e30f08a7067e2bf6e8752b0d56ee3c/pandas-3.0.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:46997386d528eb40376ecd6b033cf4a8a1e5282580f68f43de875b78cba2199d", size = 10767016, upload-time = "2026-05-11T18:53:21.227Z" }, + { url = "https://files.pythonhosted.org/packages/ea/dc/eb55135a1d5f0f0519f28da1f609a206d2cad1f9c35c32d51e38dd7261ae/pandas-3.0.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:261e308dfb22448384b7580cf719d2f998fe2966c92893c3e77d14008af1f066", size = 10420210, upload-time = "2026-05-11T18:53:23.982Z" }, + { url = "https://files.pythonhosted.org/packages/c6/3e/b1d5d955ce33ffecb407465a60bc32769d74fcf68224b7ae67ae11d4dea4/pandas-3.0.3-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dd1a5d1def6a46002e964510bdc67c368aa0951df5d1d9f8365336f5a1f490cd", size = 10336126, upload-time = "2026-05-11T18:53:26.731Z" }, + { url = "https://files.pythonhosted.org/packages/f5/76/a01261711ab60a22d71b862f0de20e4c504bf80457270ad8cb42110f6abc/pandas-3.0.3-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d72828c20c6d6e83e1e22a6a3b47b326b71664112fa9705dcbccfd7a39b62085", size = 10728051, upload-time = "2026-05-11T18:53:29.125Z" }, + { url = "https://files.pythonhosted.org/packages/e9/21/ea191195e587b18cf682e97f433f81b2d0fbe341380e80a3e0d6e4403c8e/pandas-3.0.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:d26cbe1fcfc12e8fd900e2454163e466b2d3af84f7c75481df7683ffc073d870", size = 11350796, upload-time = "2026-05-11T18:53:32.056Z" }, + { url = "https://files.pythonhosted.org/packages/64/69/f0eaaf54939f0e8c6768fd06be9af2cef9b36048b96dfb9e1b2c685a807e/pandas-3.0.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:3e91cec1879ada0624fc3dc9953c5cbd60208e59c0db28f540c5d6d47502422f", size = 11799741, upload-time = "2026-05-11T18:53:34.985Z" }, + { url = "https://files.pythonhosted.org/packages/45/a4/865e0e510cae5fc2194de4db28be638952de942571ba9125934fd9c01d47/pandas-3.0.3-cp313-cp313t-win_amd64.whl", hash = "sha256:08d789b41f87e0905880e293cedf6197ce71fe67cc081358b1e148a491b9bd13", size = 10499958, upload-time = "2026-05-11T18:53:37.857Z" }, + { url = "https://files.pythonhosted.org/packages/86/54/effdcc3c0ff7a08037889200e148ebe94c16c4f653be078c7b3675955df1/pandas-3.0.3-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:3650109c0f22879df8bd6179ab9ee3d7f1d1d4e7e0094a3f0032d9f51e2e64ac", size = 10336065, upload-time = "2026-05-11T18:53:41.099Z" }, + { url = "https://files.pythonhosted.org/packages/68/10/bf2d6738d72748b961a3751ab89522d58c54efc36a8e1a12161216cd45cf/pandas-3.0.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:bab900348131a7db1f69a7309ef141fd5680f1487094193bcbbb61791573bf8f", size = 9926101, upload-time = "2026-05-11T18:53:43.515Z" }, + { url = "https://files.pythonhosted.org/packages/ae/e9/e35cf11c8a136e757b956f5f0efdcaa50aecde85ea055f1898dfc68262f3/pandas-3.0.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ba7e08b9ac1d54569cd1e256e3668975ed624d6826f7b68df0342b012007bddb", size = 10457553, upload-time = "2026-05-11T18:53:46.394Z" }, + { url = "https://files.pythonhosted.org/packages/58/3b/1cdec6772bdbaf7b25dab360c59f03cadf05492dd724c6540af905389b07/pandas-3.0.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9d71c63ae4ebdbf70209742096f1fc46a83a0613c99d4b23766cced9ff8cd62a", size = 10914065, upload-time = "2026-05-11T18:53:49.134Z" }, + { url = "https://files.pythonhosted.org/packages/c4/c2/1ef644445fcd72e3627bceec77e3560636f87ddce4ed841afe76b83b5bf9/pandas-3.0.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:e3a2ec42c98ffa2565a67e08e218d06d72576d758d90facb7c00805194d8f360", size = 11459188, upload-time = "2026-05-11T18:53:52.527Z" }, + { url = "https://files.pythonhosted.org/packages/7e/49/4d8d4f42cbc9c4adc7a1870f269c02cbd6cd40d059622c06fb298addcbad/pandas-3.0.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:335f62418ed562cfc3c49e9e196375c28b729dcef8543abf4f9438e381bf3c76", size = 11982966, upload-time = "2026-05-11T18:53:55.043Z" }, + { url = "https://files.pythonhosted.org/packages/38/55/792619469bab9882d8bbd5865d45a72f6478762d04a9af4bf0d08c503e95/pandas-3.0.3-cp314-cp314-win_amd64.whl", hash = "sha256:3c20a521bbb85902f79f7270c80a59e1b5452d96d170c034f207181870f97ac5", size = 9876755, upload-time = "2026-05-11T18:53:58.067Z" }, + { url = "https://files.pythonhosted.org/packages/2a/af/33c469653b0ba03b50c3a98192d4c07f0c75c66b263ceb097fce0ee97d31/pandas-3.0.3-cp314-cp314-win_arm64.whl", hash = "sha256:a2d2dff8a04f3917b55ab3910c32990f8ddf7eceba114947838cefa976a68977", size = 9198658, upload-time = "2026-05-11T18:54:00.733Z" }, + { url = "https://files.pythonhosted.org/packages/a2/fa/b8c257bd76b8bd060c3a9151c1fca05e9b9c5e3af5d0f549c0356f6d143d/pandas-3.0.3-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:0d589105b3c14645af1738ff279b2995102d8f7a03b0a66dc8d95550eb513e04", size = 10787242, upload-time = "2026-05-11T18:54:03.564Z" }, + { url = "https://files.pythonhosted.org/packages/54/eb/f19206ffb0bf1919002969aa448b4702c6594845156a6f8050674855aac3/pandas-3.0.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:13fc1e853d9e04743d11ba75a985ccbc2a317fe07d8af61e445a6fd24dacd6a6", size = 10436369, upload-time = "2026-05-11T18:54:06.311Z" }, + { url = "https://files.pythonhosted.org/packages/fd/24/c7c39fb4fe22b71a0c2d78bf0c585c600092d85f94f086d2b3b2f6ca27e2/pandas-3.0.3-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:819959dab7bbd0049c15623fbac4e29a191b9528160a61fb1032242d8ced2d9c", size = 10358306, upload-time = "2026-05-11T18:54:09.085Z" }, + { url = "https://files.pythonhosted.org/packages/16/ec/dd2a9eb7fa1204df88c0864164e35b228ac581062ac612ba0a67fd812e4c/pandas-3.0.3-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:60ae316d3fd75d1858d450d0db0103ea2be3e7d4a95ec2f064f7e2ae63f7b028", size = 10758394, upload-time = "2026-05-11T18:54:11.956Z" }, + { url = "https://files.pythonhosted.org/packages/95/6e/00c61ea8e85b4f6d8d35e11852a1a4998fc7fafc91c6a602d1cc9c972d64/pandas-3.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:bd3a518890b400d32f9023722dc9a9a5c969f00b415419a3c06c043f09bb5d7d", size = 11375717, upload-time = "2026-05-11T18:54:14.539Z" }, + { url = "https://files.pythonhosted.org/packages/31/89/8fc1c268969fac43688d65fd92e67df24bd128d53cb4d2eee534cd307399/pandas-3.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9c39be2d709d01fa972a0cabc522389fceca4f3969332ba25a7d6c5802cf976a", size = 11828897, upload-time = "2026-05-11T18:54:17.146Z" }, + { url = "https://files.pythonhosted.org/packages/56/3b/e7d20dea247a3e6dc0bd8a6953854afbedc03951def4e7371e05e7263e25/pandas-3.0.3-cp314-cp314t-win_amd64.whl", hash = "sha256:4db8c527972a821cf5286b40ccc57642a39bc62e62022b42f99f8a67fca8c3a1", size = 10900855, upload-time = "2026-05-11T18:54:19.72Z" }, + { url = "https://files.pythonhosted.org/packages/0f/54/68a0978d1ef8502b8492099beaa6e7a0c1b32e3b5d4f677f5810cb08711c/pandas-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:b2c95f8bfc1ee412bf482605d7bfd30c12d1d26bd59fdd91efeef1d4718decb1", size = 9466464, upload-time = "2026-05-11T18:54:22.754Z" }, +] + +[[package]] +name = "platformdirs" +version = "4.10.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d7/47/e4501f49c178ae1d9f4a75073fda4204f52647993f075a9db4d14930e0c5/platformdirs-4.10.0.tar.gz", hash = "sha256:31e761a6a0ca04faf7353ea759bdba55652be214725111e5aac52dfa29d4bef7", size = 31224, upload-time = "2026-05-28T03:32:53.587Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/81/e6/cd9575ac904136b3cbf7aa7ee819ef86eedb7274e46f230e94ea4342e729/platformdirs-4.10.0-py3-none-any.whl", hash = "sha256:fb516cdb12eb0d857d0cd85a7c57cea4d060bee4578d6cf5a14dfdf8cbf8784a", size = 22743, upload-time = "2026-05-28T03:32:52.175Z" }, +] + +[[package]] +name = "pluggy" +version = "1.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f9/e2/3e91f31a7d2b083fe6ef3fa267035b518369d9511ffab804f839851d2779/pluggy-1.6.0.tar.gz", hash = "sha256:7dcc130b76258d33b90f61b658791dede3486c3e6bfb003ee5c9bfb396dd22f3", size = 69412, upload-time = "2025-05-15T12:30:07.975Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, +] + +[[package]] +name = "pyarrow" +version = "24.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/91/13/13e1069b351bdc3881266e11147ffccf687505dbb0ea74036237f5d454a5/pyarrow-24.0.0.tar.gz", hash = "sha256:85fe721a14dd823aca09127acbb06c3ca723efbd436c004f16bca601b04dcc83", size = 1180261, upload-time = "2026-04-21T10:51:25.837Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b4/a9/9686d9f07837f91f775e8932659192e02c74f9d8920524b480b85212cc68/pyarrow-24.0.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:6233c9ed9ab9d1db47de57d9753256d9dcffbf42db341576099f0fd9f6bf4810", size = 34981559, upload-time = "2026-04-21T10:47:22.17Z" }, + { url = "https://files.pythonhosted.org/packages/80/b6/0ddf0e9b6ead3474ab087ae598c76b031fc45532bf6a63f3a553440fb258/pyarrow-24.0.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:f7616236ec1bc2b15bfdec22a71ab38851c86f8f05ff64f379e1278cf20c634a", size = 36663654, upload-time = "2026-04-21T10:47:28.315Z" }, + { url = "https://files.pythonhosted.org/packages/7c/3b/926382efe8ce27ba729071d3566ade6dfb86bdf112f366000196b2f5780a/pyarrow-24.0.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:1617043b99bd33e5318ae18eb2919af09c71322ef1ca46566cdafc6e6712fb66", size = 45679394, upload-time = "2026-04-21T10:47:34.821Z" }, + { url = "https://files.pythonhosted.org/packages/b3/7a/829f7d9dfd37c207206081d6dad474d81dde29952401f07f2ba507814818/pyarrow-24.0.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:6165461f55ef6314f026de6638d661188e3455d3ec49834556a0ebbdbace18bb", size = 48863122, upload-time = "2026-04-21T10:47:42.056Z" }, + { url = "https://files.pythonhosted.org/packages/5f/e8/f88ce625fe8babaae64e8db2d417c7653adb3019b08aae85c5ed787dc816/pyarrow-24.0.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3b13dedfe76a0ad2d1d859b0811b53827a4e9d93a0bcb05cf59333ab4980cc7e", size = 49376032, upload-time = "2026-04-21T10:47:48.967Z" }, + { url = "https://files.pythonhosted.org/packages/36/7a/82c363caa145fff88fb475da50d3bf52bb024f61917be5424c3392eaf878/pyarrow-24.0.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:25ea65d868eb04015cd18e6df2fbe98f07e5bda2abefabcb88fce39a947716f6", size = 51929490, upload-time = "2026-04-21T10:47:55.981Z" }, + { url = "https://files.pythonhosted.org/packages/66/1c/e3e72c8014ad2743ca64a701652c733cc5cbcee15c0463a32a8c55518d9e/pyarrow-24.0.0-cp312-cp312-win_amd64.whl", hash = "sha256:295f0a7f2e242dabd513737cf076007dc5b2d59237e3eca37b05c0c6446f3826", size = 27355660, upload-time = "2026-04-21T10:48:01.718Z" }, + { url = "https://files.pythonhosted.org/packages/6f/d3/a1abf004482026ddc17f4503db227787fa3cfe41ec5091ff20e4fea55e57/pyarrow-24.0.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:02b001b3ed4723caa44f6cd1af2d5c86aa2cf9971dacc2ffa55b21237713dfba", size = 34976759, upload-time = "2026-04-21T10:48:07.258Z" }, + { url = "https://files.pythonhosted.org/packages/4f/4a/34f0a36d28a2dd32225301b79daad44e243dc1a2bb77d43b60749be255c4/pyarrow-24.0.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:04920d6a71aabd08a0417709efce97d45ea8e6fb733d9ca9ecffb13c67839f68", size = 36658471, upload-time = "2026-04-21T10:48:13.347Z" }, + { url = "https://files.pythonhosted.org/packages/1f/78/543b94712ae8bb1a6023bcc1acf1a740fbff8286747c289cd9468fced2a5/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:a964266397740257f16f7bb2e4f08a0c81454004beab8ff59dd531b73610e9f2", size = 45675981, upload-time = "2026-04-21T10:48:20.201Z" }, + { url = "https://files.pythonhosted.org/packages/84/9f/8fb7c222b100d314137fa40ec050de56cd8c6d957d1cfff685ce72f15b17/pyarrow-24.0.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:6f066b179d68c413374294bc1735f68475457c933258df594443bb9d88ddc2a0", size = 48859172, upload-time = "2026-04-21T10:48:27.541Z" }, + { url = "https://files.pythonhosted.org/packages/a7/d3/1ea72538e6c8b3b475ed78d1049a2c518e655761ea50fe1171fc855fcab7/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:1183baeb14c5f587b1ec52831e665718ce632caab84b7cd6b85fd44f96114495", size = 49385733, upload-time = "2026-04-21T10:48:34.7Z" }, + { url = "https://files.pythonhosted.org/packages/c3/be/c3d8b06a1ba35f2260f8e1f771abbee7d5e345c0937aab90675706b1690a/pyarrow-24.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:806f24b4085453c197a5078218d1ee08783ebbba271badd153d1ae22a3ee804f", size = 51934335, upload-time = "2026-04-21T10:48:42.099Z" }, + { url = "https://files.pythonhosted.org/packages/9c/62/89e07a1e7329d2cde3e3c6994ba0839a24977a2beda8be6005ea3d860b99/pyarrow-24.0.0-cp313-cp313-win_amd64.whl", hash = "sha256:e4505fc6583f7b05ab854934896bcac8253b04ac1171a77dfb73efef92076d91", size = 27271748, upload-time = "2026-04-21T10:49:42.532Z" }, + { url = "https://files.pythonhosted.org/packages/17/1a/cff3a59f80b5b1658549d46611b67163f65e0664431c076ad728bf9d5af4/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:1a4e45017efbf115032e4475ee876d525e0e36c742214fbe405332480ecd6275", size = 35238554, upload-time = "2026-04-21T10:48:48.526Z" }, + { url = "https://files.pythonhosted.org/packages/a8/99/cce0f42a327bfef2c420fb6078a3eb834826e5d6697bf3009fe11d2ad051/pyarrow-24.0.0-cp313-cp313t-macosx_12_0_x86_64.whl", hash = "sha256:7986f1fa71cee060ad00758bcc79d3a93bab8559bf978fab9e53472a2e25a17b", size = 36782301, upload-time = "2026-04-21T10:48:55.181Z" }, + { url = "https://files.pythonhosted.org/packages/2a/66/8e560d5ff6793ca29aca213c53eec0dd482dd46cb93b2819e5aab52e4252/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:d3e0b61e8efb24ed38898e5cdc5fffa9124be480008d401a1f8071500494ae42", size = 45721929, upload-time = "2026-04-21T10:49:03.676Z" }, + { url = "https://files.pythonhosted.org/packages/27/0c/a26e25505d030716e078d9f16eb74973cbf0b33b672884e9f9da1c83b871/pyarrow-24.0.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:55a3bc1e3df3b5567b7d27ef551b2283f0c68a5e86f1cd56abc569da4f31335b", size = 48825365, upload-time = "2026-04-21T10:49:11.714Z" }, + { url = "https://files.pythonhosted.org/packages/5f/eb/771f9ecb0c65e73fe9dccdd1717901b9594f08c4515d000c7c62df573811/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:641f795b361874ac9da5294f8f443dfdbee355cf2bd9e3b8d97aaac2306b9b37", size = 49451819, upload-time = "2026-04-21T10:49:21.474Z" }, + { url = "https://files.pythonhosted.org/packages/48/da/61ae89a88732f5a785646f3ec6125dbb640fa98a540eb2b9889caa561403/pyarrow-24.0.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:8adc8e6ce5fccf5dc707046ae4914fd537def529709cc0d285d37a7f9cd442ca", size = 51909252, upload-time = "2026-04-21T10:49:31.164Z" }, + { url = "https://files.pythonhosted.org/packages/cb/1a/8dd5cafab7b66573fa91c03d06d213356ad4edd71813aa75e08ce2b3a844/pyarrow-24.0.0-cp313-cp313t-win_amd64.whl", hash = "sha256:9b18371ad2f44044b81a8d23bc2d8a9b6a6226dca775e8e16cfee640473d6c5d", size = 27388127, upload-time = "2026-04-21T10:49:37.334Z" }, + { url = "https://files.pythonhosted.org/packages/ad/80/d022a34ff05d2cbedd8ccf841fc1f532ecfa9eb5ed1711b56d0e0ea71fc9/pyarrow-24.0.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:1cc9057f0319e26333b357e17f3c2c022f1a83739b48a88b25bfd5fa2dc18838", size = 35007997, upload-time = "2026-04-21T10:49:48.796Z" }, + { url = "https://files.pythonhosted.org/packages/1a/ff/f01485fda6f4e5d441afb8dd5e7681e4db18826c1e271852f5d3957d6a80/pyarrow-24.0.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:e6f1278ee4785b6db21229374a1c9e54ec7c549de5d1efc9630b6207de7e170b", size = 36678720, upload-time = "2026-04-21T10:49:55.858Z" }, + { url = "https://files.pythonhosted.org/packages/9e/c2/2d2d5fea814237923f71b36495211f20b43a1576f9a4d6da7e751a64ec6f/pyarrow-24.0.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:adbbedc55506cbdabb830890444fb856bfb0060c46c6f8026c6c2f2cf86ae795", size = 45741852, upload-time = "2026-04-21T10:50:04.624Z" }, + { url = "https://files.pythonhosted.org/packages/8e/3a/28ba9c1c1ebdbb5f1b94dfebb46f207e52e6a554b7fe4132540fde29a3a0/pyarrow-24.0.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:ae8a1145af31d903fa9bb166824d7abe9b4681a000b0159c9fb99c11bc11ad26", size = 48889852, upload-time = "2026-04-21T10:50:12.293Z" }, + { url = "https://files.pythonhosted.org/packages/df/51/4a389acfd31dca009f8fb82d7f510bb4130f2b3a8e18cf00194d0687d8ac/pyarrow-24.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d7027eba1df3b2069e2e8d80f644fa0918b68c46432af3d088ddd390d063ecde", size = 49445207, upload-time = "2026-04-21T10:50:20.677Z" }, + { url = "https://files.pythonhosted.org/packages/19/4b/0bab2b23d2ae901b1b9a03c0efd4b2d070256f8ce3fc43f6e58c167b2081/pyarrow-24.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:e56a1ffe9bf7b727432b89104cc0849c21582949dd7bdcb34f17b2001a351a76", size = 51954117, upload-time = "2026-04-21T10:50:29.14Z" }, + { url = "https://files.pythonhosted.org/packages/29/88/f4e9145da0417b3d2c12035a8492b35ff4a3dbc653e614fcfb51d9dedb38/pyarrow-24.0.0-cp314-cp314-win_amd64.whl", hash = "sha256:38be1808cdd068605b787e6ca9119b27eb275a0234e50212c3492331680c3b1e", size = 28001155, upload-time = "2026-04-21T10:51:22.337Z" }, + { url = "https://files.pythonhosted.org/packages/79/4f/46a49a63f43526da895b1a45bbb51d5baf8e4d77159f8528fc3e5490007f/pyarrow-24.0.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:418e48ce50a45a6a6c73c454677203a9c75c966cb1e92ca3370959185f197a05", size = 35250387, upload-time = "2026-04-21T10:50:35.552Z" }, + { url = "https://files.pythonhosted.org/packages/a0/da/d5e0cd5ef00796922404806d5f00325cdadc3441ce2c13fe7115f2df9a64/pyarrow-24.0.0-cp314-cp314t-macosx_12_0_x86_64.whl", hash = "sha256:2f16197705a230a78270cdd4ea8a1d57e86b2fdcbc34a1f6aebc72e65c986f9a", size = 36797102, upload-time = "2026-04-21T10:50:42.417Z" }, + { url = "https://files.pythonhosted.org/packages/34/c7/5904145b0a593a05236c882933d439b5720f0a145381179063722fbfc123/pyarrow-24.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:fb24ac194bfc5e86839d7dcd52092ee31e5fe6733fe11f5e3b06ef0812b20072", size = 45745118, upload-time = "2026-04-21T10:50:49.324Z" }, + { url = "https://files.pythonhosted.org/packages/13/d3/cca42fe166d1c6e4d5b80e530b7949104d10e17508a90ae202dac205ce2a/pyarrow-24.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:9700ebd9a51f5895ce75ff4ac4b3c47a7d4b42bc618be8e713e5d56bacf5f931", size = 48844765, upload-time = "2026-04-21T10:50:55.579Z" }, + { url = "https://files.pythonhosted.org/packages/b0/49/942c3b79878ba928324d1e17c274ed84581db8c0a749b24bcf4cbdf15bd3/pyarrow-24.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:d8ddd2768da81d3ee08cfea9b597f4abb4e8e1dc8ae7e204b608d23a0d3ab699", size = 49471890, upload-time = "2026-04-21T10:51:02.439Z" }, + { url = "https://files.pythonhosted.org/packages/76/97/ff71431000a75d84135a1ace5ca4ba11726a231a8007bbb320a4c54075d5/pyarrow-24.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:61a3d7eaa97a14768b542f3d284dc6400dd2470d9f080708b13cd46b6ae18136", size = 51932250, upload-time = "2026-04-21T10:51:10.576Z" }, + { url = "https://files.pythonhosted.org/packages/51/be/6f79d55816d5c22557cf27533543d5d70dfe692adfbee4b99f2760674f38/pyarrow-24.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:c91d00057f23b8d353039520dc3a6c09d8608164c692e9f59a175a42b2ae0c19", size = 28131282, upload-time = "2026-04-21T10:51:16.815Z" }, +] + +[[package]] +name = "pygments" +version = "2.20.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c3/b2/bc9c9196916376152d655522fdcebac55e66de6603a76a02bca1b6414f6c/pygments-2.20.0.tar.gz", hash = "sha256:6757cd03768053ff99f3039c1a36d6c0aa0b263438fcab17520b30a303a82b5f", size = 4955991, upload-time = "2026-03-29T13:29:33.898Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f4/7e/a72dd26f3b0f4f2bf1dd8923c85f7ceb43172af56d63c7383eb62b332364/pygments-2.20.0-py3-none-any.whl", hash = "sha256:81a9e26dd42fd28a23a2d169d86d7ac03b46e2f8b59ed4698fb4785f946d0176", size = 1231151, upload-time = "2026-03-29T13:29:30.038Z" }, +] + +[[package]] +name = "pytest" +version = "9.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, + { name = "iniconfig" }, + { name = "packaging" }, + { name = "pluggy" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e4/47/b9efed96c114afcfa3c9d3fe98a76a1d14c74a9e266d397cf6eb64be5e01/pytest-9.1.1.tar.gz", hash = "sha256:1088fbde8f2b49d95a549a195707afa7a76a3ce9bcadc26b6d71f0ffda5fe313", size = 1636369, upload-time = "2026-06-19T10:58:32.857Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/24/25/1de2678b631f5a49215c6c96fff41ba892b0a34df68d6d80292b1b48aa7f/pytest-9.1.1-py3-none-any.whl", hash = "sha256:37a86b45efb9a47a61a36449063e8e18d0cab3161329fc099eb21783169c4f0c", size = 386536, upload-time = "2026-06-19T10:58:31.347Z" }, +] + +[[package]] +name = "python-dateutil" +version = "2.9.0.post0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/66/c0/0c8b6ad9f17a802ee498c46e004a0eb49bc148f2fd230864601a86dcf6db/python-dateutil-2.9.0.post0.tar.gz", hash = "sha256:37dd54208da7e1cd875388217d5e00ebd4179249f90fb72437e91a35459a0ad3", size = 342432, upload-time = "2024-03-01T18:36:20.211Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, +] + +[[package]] +name = "rich" +version = "15.0.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py" }, + { name = "pygments" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/c0/8f/0722ca900cc807c13a6a0c696dacf35430f72e0ec571c4275d2371fca3e9/rich-15.0.0.tar.gz", hash = "sha256:edd07a4824c6b40189fb7ac9bc4c52536e9780fbbfbddf6f1e2502c31b068c36", size = 230680, upload-time = "2026-04-12T08:24:00.75Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, +] + +[[package]] +name = "six" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/94/e7/b2c673351809dca68a0e064b6af791aa332cf192da575fd474ed7d6f16a2/six-1.17.0.tar.gz", hash = "sha256:ff70335d468e7eb6ec65b95b99d3a2836546063f63acc5171de367e834932a81", size = 34031, upload-time = "2024-12-04T17:35:28.174Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, +] + +[[package]] +name = "spatialdata-experimental-writer" +version = "0.1.0" +source = { editable = "." } +dependencies = [ + { name = "numpy" }, + { name = "pandas" }, + { name = "pyarrow" }, +] + +[package.dev-dependencies] +dev = [ + { name = "pytest" }, +] +tui = [ + { name = "textual" }, +] + +[package.metadata] +requires-dist = [ + { name = "numpy", specifier = ">=2.0" }, + { name = "pandas", specifier = ">=2.2" }, + { name = "pyarrow", specifier = ">=18" }, +] + +[package.metadata.requires-dev] +dev = [{ name = "pytest", specifier = ">=8.0" }] +tui = [{ name = "textual", specifier = ">=1.0" }] + +[[package]] +name = "textual" +version = "8.2.7" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markdown-it-py", extra = ["linkify"] }, + { name = "mdit-py-plugins" }, + { name = "platformdirs" }, + { name = "pygments" }, + { name = "rich" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/9b/7a/c519db0aba5024f86e71e9631810bfdd6866ed2c8695bd7fa34b90e7ef59/textual-8.2.7.tar.gz", hash = "sha256:658f568ff81e30ed43890c3e07520390e5cf1b4763822006e060656b0a88f105", size = 1859249, upload-time = "2026-05-19T10:52:49.531Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a8/f5/c1e18bc0707300a0e90204343abbf7d7acd6fb7ebe03a6d4893b99a234b8/textual-8.2.7-py3-none-any.whl", hash = "sha256:4caaa13a90bc4cf9c6c862c067ccd34fe84e9c161710a2a907a8026313b6bd73", size = 731129, upload-time = "2026-05-19T10:52:51.773Z" }, +] + +[[package]] +name = "typing-extensions" +version = "4.15.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" }, +] + +[[package]] +name = "tzdata" +version = "2026.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ba/19/1b9b0e29f30c6d35cb345486df41110984ea67ae69dddbc0e8a100999493/tzdata-2026.2.tar.gz", hash = "sha256:9173fde7d80d9018e02a662e168e5a2d04f87c41ea174b139fbef642eda62d10", size = 198254, upload-time = "2026-04-24T15:22:08.651Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ce/e4/dccd7f47c4b64213ac01ef921a1337ee6e30e8c6466046018326977efd95/tzdata-2026.2-py2.py3-none-any.whl", hash = "sha256:bbe9af844f658da81a5f95019480da3a89415801f6cc966806612cc7169bffe7", size = 349321, upload-time = "2026-04-24T15:22:05.876Z" }, +] + +[[package]] +name = "uc-micro-py" +version = "2.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/78/67/9a363818028526e2d4579334460df777115bdec1bb77c08f9db88f6389f2/uc_micro_py-2.0.0.tar.gz", hash = "sha256:c53691e495c8db60e16ffc4861a35469b0ba0821fe409a8a7a0a71864d33a811", size = 6611, upload-time = "2026-03-01T06:31:27.526Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/61/73/d21edf5b204d1467e06500080a50f79d49ef2b997c79123a536d4a17d97c/uc_micro_py-2.0.0-py3-none-any.whl", hash = "sha256:3603a3859af53e5a39bc7677713c78ea6589ff188d70f4fee165db88e22b242c", size = 6383, upload-time = "2026-03-01T06:31:26.257Z" }, +]