[codex] SQLite node store read scaling and UI materialized views by crs48 · Pull Request #24 · crs48/xNet

crs48 · 2026-06-01T22:45:25Z

Summary

Adds SQLite-backed NodeStore query planning for scalar filters, adaptive indexes, spatial/FTS candidates, and materialized query views.
Wires existing Electron and web database table/board views to stable materialized row-list cache keys.
Marks the SQLite read-scaling exploration complete.

Validation

pnpm typecheck
pnpm test (389 files, 5588 passed, 3 skipped)
pnpm --filter xnet-desktop build
pnpm --filter xnet-web build
Pre-commit hooks on the final commits

Summary by CodeRabbit

New Features

Added full-text search capabilities with tokenized filtering across document content and titles.
Introduced materialized view support for optimized database query performance.
Enabled sorting nodes by creation or update time in ascending/descending order.
Added offset-based pagination alongside existing cursor-based pagination.
Database views now scope row queries to the active view mode (table/board).

Documentation

Updated implementation guide documenting SQLite query scaling, property indexing strategies, and query planning architecture.

- add shared NodeStore query descriptors with plan metadata - add storage pushdown hooks for NodeStore-backed readers - add SQLite scalar property sidecar, rebuild support, and batch hydration - add conservative SQL candidate planning with JS verification - route data bridges and Electron IPC storage through safe paginated paths - update exploration checklist and focused coverage for indexes and IPC storage

- record stable NodeQuery descriptor stats for SQLite query planning - add opt-in bounded partial scalar indexes for hot property filters - expose adaptive index options through SQLite adapter factory exports - document Phase 3 status and migration checklist updates

- add bounded parity metadata for SQLite-backed NodeQuery plans - compare SQL results against shared JS descriptor semantics for small scopes - log high-severity diagnostics when SQL candidates miss expected nodes - mark the Phase 2 verification checklist item complete

- time SQLite candidate queries through shared diagnostics helpers - attach used indexes, plan details, and index inventory to query plans - run ANALYZE after adaptive index creation before PRAGMA optimize - update the exploration checklist for diagnostics integration

- track adaptive index estimated bytes and indexed rows in schema metadata - drop stale or least-recently-used adaptive indexes when budgets are exceeded - skip adaptive index creation when disk or write-row budgets cannot fit - add opt-in query and adaptive-index debug logging - update exploration checklists for Phase 3 budget enforcement

- filter NodeStore get, list, and query reads through read authorization - disable storage query pushdown when a read policy evaluator is active - apply pagination after read filtering so hidden rows cannot affect pages - redact query plan counts to visible nodes for auth-sensitive reads - add regression coverage and update the exploration checklist

- add cached FTS5 and R-Tree virtual-table probes to sqlite diagnostics - expose storage capability metadata on SQLite-backed NodeStore query plans - guard existing FTS helpers with runtime FTS5 detection - check off the capability-detection validation item in the exploration

- add synthetic SQLite NodeStore benchmark fixtures for 1k, 10k, 100k, and 1M node scales - add Vitest benchmark coverage for SQL pagination and scalar candidate reads - cover JS-verified property sorts in the benchmark harness - measure scalar sidecar writes, adaptive indexes, and write amplification - check off the performance and mutation benchmark validation items

- create SQLite R-Tree side tables lazily when runtime support is detected - compile spatial query descriptors into R-Tree candidate joins - keep exact spatial semantics in the shared JavaScript verifier - maintain spatial rows after registered mappings see later node writes - cover unsupported sql.js fallback and native R-Tree acceleration

- formalize token-prefix search descriptors for node queries - use SQLite FTS5 as a candidate accelerator when available - keep exact search filtering in the shared JS descriptor matcher - disable unsafe fallback pagination before search filtering - cover sql.js fallback and native FTS maintenance

- add stable materializedView query options for database views - store disposable node ID and ordinal result caches in SQLite - hydrate current NodeState pages from materialized IDs - invalidate cached view results after node writes and deletes - cover refresh, cache-hit pagination, and invalidation behavior

- Route canonical row pages through NodeStore.query descriptors. - Add offset and materializedView options to database row queries. - Opt Electron and web table/board views into stable cache keys. - Cover descriptor propagation and visible-view switching in tests. - Document the completed UI integration.

- Rename the SQLite read-scaling exploration from [_] to [x]. - Preserve the completed implementation notes and checklist state.

greptile-apps

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

coderabbitai · 2026-06-01T22:45:37Z

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

This PR implements a comprehensive SQLite-backed query descriptor system for xNet's NodeStore, enabling efficient read scaling through SQL pushdown, scalar property indexing, adaptive index optimization, and materialized view caching. The work centralizes query semantics in a new @xnetjs/data query module, extends storage adapter contracts with conditional indexing and optional query support, refactors DataBridge to delegate to NodeStore.query, and integrates view-scoped materialized caching into React hooks and UI components.

Changes

SQLite Read Scaling & Query System

Layer / File(s)	Summary
Query descriptor types and normalization utilities `packages/data/src/store/query.ts`	Exports query descriptor types for spatial/search/materialized view filtering; implements normalization for user inputs, predicate matching, multi-key sorting with null handling, and result pagination.
DataBridge query delegation to data package `packages/data-bridge/src/query-descriptor.ts`, `packages/data-bridge/src/__tests__/query-descriptor.test.ts`	Replaces hand-rolled DataBridge query logic by delegating to `@xnetjs/data` query utilities, removing JSON serialization and manual spatial/sort handling.
Storage adapter contract: SetNodeOptions and query support `packages/data/src/store/types.ts`, `packages/data/src/store/index.ts`	Extends `NodeStorageAdapter.setNode(node, options?)` with `SetNodeOptions` for conditional indexing; adds optional `queryNodes` method; introduces `orderBy` to `ListNodesOptions`.
SQLite schema v5 with scalar indexing and query tracking tables `packages/sqlite/src/schema.ts`	Bumps schema to v5 and adds DDL for scalar property sidecar, query descriptor statistics, index candidates, and materialized view cache tables with supporting indexes and migrations.
SQLite FTS5 and R-Tree runtime capability detection `packages/sqlite/src/diagnostics.ts`, `packages/sqlite/src/fts.ts`, `packages/sqlite/src/index.ts`, `packages/sqlite/src/adapter.test.ts`	Adds `detectSQLiteCapabilities` to probe FTS5/R-Tree support via virtual table creation, caches results per adapter, and integrates into FTS support checking.
Memory adapter orderBy support `packages/data/src/store/memory-adapter.ts`	Updates `listNodes()` to use `options.orderBy` for dynamic multi-field sorting by `createdAt`/`updatedAt` with configurable direction.
SQLite scalar property index synchronization and maintenance `packages/data/src/store/sqlite-adapter.ts` (partial)	Implements `syncScalarRowsForNode` and `rebuildScalarIndex` to maintain scalar sidecar; integrates conditional `indexProperties` control into node write/delete paths.
SQLite query compilation with spatial/FTS planning and parity auditing `packages/data/src/store/sqlite-adapter.ts` (partial)	Replaces `queryNodes` with compiled-query pipeline: prepares plans, compiles candidate SQL, executes with diagnostics, hydrates nodes, applies post-filtering, and optionally audits parity.
NodeStore query integration with auth-aware fallback `packages/data/src/store/store.ts`	Adds `NodeStore.query(descriptor)` that prefers storage-level `queryNodes` when available; otherwise loads fallback candidates, decrypts, filters for readability, applies `applyNodeQueryDescriptor`.
DataBridge query loading via NodeStore.query `packages/data-bridge/src/main-thread-bridge.ts`, `packages/data-bridge/src/native-bridge.ts`, `packages/data-bridge/src/worker/data-worker.ts`	Refactors `loadQuery` to use `store.query(descriptor)` instead of manual `get`/`list` branching and `applyQueryDescriptor` post-processing.
Public re-exports of query descriptors and storage adapters from data package `packages/data/src/index.ts`, `packages/data/src/store/index.ts`	Expands `@xnetjs/data` public API by re-exporting query descriptor types/utilities and SQLite adapter options.
Database row querying with offset pagination and materialized views `packages/data/src/database/row-operations.ts`, `packages/data/src/database/row-operations.test.ts`	Updates `queryRows` to use `store.query(descriptor)`, adds offset-based pagination, integrates `materializedView` selection.
Electron data process: SetNodeOptions forwarding through IPC `apps/electron/src/data-process/...`, `apps/electron/src/main/...`, `apps/electron/src/preload/...`, `apps/electron/src/renderer/lib/ipc-node-storage.ts`	Updates Electron IPC chain to accept and forward optional `options` parameter from renderer to storage adapter.
useDatabase hook: materialized view option and offset pagination `packages/react/src/hooks/useDatabase.ts`	Adds `UseDatabaseOptions.materializedView` support, introduces `resolveDatabaseMaterializedView`, refactors pagination to offset-based for canonical queries.
DatabaseView components: view-scoped materialized row fetching `apps/electron/src/renderer/components/DatabaseView.tsx`, `apps/web/src/components/DatabaseView.tsx`, `apps/electron/src/renderer/components/DatabaseView.test.tsx`	Computes active view ID from view mode, builds `rowQueryOptions` with view-specific `materializedView` ID, scopes row fetching per surface view.
DataBridge types: search filters and materialized view options `packages/data-bridge/src/types.ts`	Introduces `QuerySearchField`/`QuerySearchFilter` and `QueryMaterializedViewOptions`; extends `QueryOptions` and `QueryDescriptor` with optional `search` and `materializedView`.
SQLite benchmark fixtures and write amplification estimation `packages/data/src/store/sqlite-benchmarks.ts`, `packages/data/src/store/sqlite-benchmarks.test.ts`	Introduces deterministic benchmark node generation and write amplification estimation helpers.
SQLite adapter test suite: indexing, query planning, adaptive indexing `packages/data/src/store/sqlite-adapter.test.ts`	Adds comprehensive coverage for scalar indexing, query planning with pushdown/fallback/parity, FTS/R-Tree candidates, and adaptive index lifecycle.
SQLite Node Store Read Scaling architecture documentation `docs/explorations/0123_[x]_SQLITE_NODE_STORE_READ_SCALING_AND_AUTOMATIC_INDEXING.md`	Comprehensive documentation of SQLite read-scaling architecture, implementation status, validation checklist, benchmarks, and rollout sequence.

🎯 4 (Complex) | ⏱️ ~60 minutes

🐰 Through queries and indexes we spring,
Sqlite's power on leafy wing,
From scattered rows to ordered view,
Fast reads hop through, a Postgres too!

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch codex/sqlite-node-store-read-scaling

github-actions · 2026-06-01T22:50:40Z

Preview: https://xnet.fyi/pr/24/app/

crs48 added 13 commits June 1, 2026 12:53

docs(exploration): mark sqlite read scaling complete

cee0ab7

- Rename the SQLite read-scaling exploration from [_] to [x]. - Preserve the completed implementation notes and checklist state.

greptile-apps Bot reviewed Jun 1, 2026

View reviewed changes

github-actions Bot added a commit that referenced this pull request Jun 1, 2026

deploy(preview): publish PR #24 preview

86d349a

crs48 merged commit 80c841b into main Jun 1, 2026
7 of 8 checks passed

crs48 deleted the codex/sqlite-node-store-read-scaling branch June 1, 2026 22:51

coderabbitai Bot mentioned this pull request Aug 1, 2026

feat: the social graph atlas — exploration 0419 #674

Merged

Navigation Menu

Uh oh!

[codex] SQLite node store read scaling and UI materialized views - #24

[codex] SQLite node store read scaling and UI materialized views#24
crs48 merged 13 commits into
mainfrom
codex/sqlite-node-store-read-scaling

crs48 commented Jun 1, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

greptile-apps Bot left a comment

Uh oh!

coderabbitai Bot commented Jun 1, 2026 •

edited

Loading

Review failed

Walkthrough

Changes

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

crs48 commented Jun 1, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Summary by CodeRabbit

New Features

Documentation

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Uh oh!

github-actions Bot commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

crs48 commented Jun 1, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 1, 2026 •

edited

Loading