Add loadSubset State Tracking and On-Demand Sync Mode #669

samwillis · 2025-10-12T18:19:30Z

Still to do:

Framework hook intigration

Overview

This PR adds comprehensive loading status tracking to collections and live queries, enabling UIs to display loading indicators when more data is being fetched. It also introduces a new syncMode configuration option to control when collections load data - either eagerly on initial sync or on-demand as queries request it.

Problem

No Loading Indicators: When live queries pushed predicates down to source collections via syncMore, there was no indication that data was being loaded. This made it impossible to show proper loading states in the UI.
Always-Eager Loading: Collections always loaded all data immediately during initial sync, even when using predicate pushdown. There was no way to configure collections to only load data as it was requested by queries.

Solution

This PR implements a multi-layered loading state tracking system with configurable sync modes:

Loading State Tracking

Collection Subscriptions: Track their own loading state when requesting snapshots, with a public Subscription interface for external consumers
Collections: Track pending load promises and expose an isLoadingMore property
Live Queries: Automatically reflect the loading state of their source collection subscriptions

Key Isolation Property: Each live query maintains its own loading state based on its own subscriptions. When a live query triggers loading via predicate pushdown, only that live query's isLoadingMore becomes true. Other live queries that share the same source collection but don't need that specific snapshot remain unaffected.

Sync Modes

Collections can now be configured with two sync modes:

eager (default): Loads all data immediately during initial sync. loadSubset calls are bypassed.
on-demand: Only loads data as it's requested via loadSubset calls. Requires a loadSubset handler.

Changes

1. Renamed `syncMore`/`loadMore` to `loadSubset`

Files: packages/db/src/collection/sync.ts, packages/db/src/types.ts, packages/db/src/collection/subscription.ts

Renamed syncMore → loadSubset (collection method)
Renamed onLoadMore → loadSubset (sync config property)
Renamed OnLoadMoreOptions → LoadSubsetOptions (type)
Renamed syncOnLoadMoreFn → syncLoadSubsetFn (internal)
Renamed pendingLoadMorePromises → pendingLoadSubsetPromises (internal)

Rationale: "loadSubset" better reflects that this operation loads a filtered/limited subset of data, not just "more" data.

2. Added `syncMode` Configuration

Files: packages/db/src/types.ts, packages/db/src/collection/sync.ts

New syncMode property on collection config:

type SyncMode = 'eager' | 'on-demand'

interface CollectionConfig {
  syncMode?: SyncMode // defaults to 'eager'
  // ... other properties
}

Behavior:

eager (default): loadSubset is bypassed and returns undefined. All data is loaded during initial sync.
on-demand: loadSubset calls proceed normally. Data is only loaded as requested.

Validation:

If syncMode: 'on-demand' is set but no loadSubset handler is provided, throws CollectionConfigurationError with a helpful message
Validation occurs when the sync function is first called

3. Standardized `loadSubset` Return Type

Files: packages/db/src/collection/sync.ts, packages/db/src/types.ts

Changed to consistently return Promise<void> | undefined
Returns undefined when syncMode is 'eager' or no sync implementation is configured
Wraps synchronous loadSubset results in Promise.resolve()
Updated SyncConfigRes['loadSubset'] type signature

4. CollectionSubscription Status Tracking & Events

File: packages/db/src/collection/subscription.ts

Added comprehensive status tracking:

Status Property: status: 'ready' | 'loadingMore' (readonly getter with private mutable field)
Concurrent Tracking: pendingLoadSubsetPromises: Set<Promise<void>>
Events:
- status:change - Emitted when status transitions
- status:ready - Emitted when entering ready state
- status:loadingMore - Emitted when entering loading state
- unsubscribed - New event emitted when subscription is destroyed
Error Handling: Status returns to ready even on promise rejection
Cleanup: unsubscribe() emits unsubscribed event before clearing listeners

5. Generic `isLoadingMore` for All Collections

Files: packages/db/src/collection/sync.ts, packages/db/src/collection/index.ts, packages/db/src/collection/events.ts

All collections now track their loading state:

Property: isLoadingMore (boolean getter, not a method)
Tracking: pendingLoadSubsetPromises: Set<Promise<void>> in CollectionSyncManager
Events: loadingMore:change event when state transitions
API: trackLoadPromise(promise: Promise<void>) method for internal coordination
Access: Made _sync public on Collection for internal use

6. Live Query Integration

Files: packages/db/src/query/live/collection-subscriber.ts, packages/db/src/query/live/collection-config-builder.ts

Live queries reflect loading state from their subscriptions:

CollectionSubscriber subscribes to subscription status:change events
Creates deferred promises when subscriptions enter loadingMore state
Passes promises to result collection via liveQueryCollection.trackLoadPromise()
Result collection's isLoadingMore reflects subscription loading states
Proper cleanup on unsubscribe (resolves pending promises)

Loading State Isolation: Each live query's subscriptions are independent. Query A triggering loadSubset doesn't affect Query B's isLoadingMore status.

7. Subscription Interface for External Consumers

File: packages/db/src/types.ts

New public Subscription interface:

export interface Subscription extends EventEmitter<SubscriptionEvents> {
  readonly status: SubscriptionStatus
}

export type LoadSubsetOptions = {
  where?: BasicExpression<boolean>
  orderBy?: OrderBy
  limit?: number
  subscription?: Subscription // Optional, for sync implementation use
}

Purpose: Allows sync implementations to:

Track which subscription triggered a loadSubset call
Subscribe to subscription lifecycle events (e.g., unsubscribed)
Implement advanced caching/ref-counting based on subscription lifecycle

8. Reusable Event Emitter

File: packages/db/src/event-emitter.ts (new)

Extracted event emitter logic into a reusable base class:

Type-Safe: Generic EventEmitter<TEvents> with full type safety
Methods: on, once, off, waitFor
Protected: emitInner (for subclass use), clearListeners
Error Handling: Re-throws listener errors via queueMicrotask
Usage:
- CollectionEventsManager extends it and wraps emitInner with public emit
- CollectionSubscription extends it and uses emitInner internally

9. Fixed Local-Only Collection Types

File: packages/db/src/local-only.ts

Fixed mutation function typing issues:

Wrapper functions now accept broader UtilsRecord type parameters
Properly handles contravariance in function parameter types
Removed unnecessary type casts in return statement

10. Comprehensive Test Coverage

Files: packages/db/tests/collection-subscription.test.ts, packages/db/tests/collection.test.ts, packages/db/tests/query/live-query-collection.test.ts

Added extensive test suites:

CollectionSubscription: Status tracking, event emission, concurrent promises, error handling, cleanup
Collection.isLoadingMore: Property tracking, event emission, concurrent loads, error handling
Live Query Integration: Result collection reflects subscription states, isolation between queries
All tests using loadSubset: Updated to use syncMode: 'on-demand'

API

Sync Mode Configuration

// Eager mode (default) - loads all data immediately
const eagerCollection = createCollection({
  getKey: (item) => item.id,
  syncMode: 'eager', // optional, this is the default
  sync: {
    sync: ({ begin, write, commit, markReady }) => {
      // Load all data here
      begin()
      allData.forEach(item => write({ type: 'insert', value: item }))
      commit()
      markReady()
    }
  }
})

// On-demand mode - only loads data as requested
const onDemandCollection = createCollection({
  getKey: (item) => item.id,
  syncMode: 'on-demand',
  sync: {
    sync: ({ markReady }) => {
      markReady() // Don't load data initially
      
      return {
        // Required for on-demand mode
        loadSubset: async (options) => {
          const { where, limit, orderBy, subscription } = options
          // Load only the requested subset
          const data = await fetchDataSubset(where, limit, orderBy)
          // ... apply data to collection
        }
      }
    }
  }
})

Collection Subscription

const subscription = collection.subscribeChanges(callback, options)

// Status property
console.log(subscription.status) // 'ready' | 'loadingMore'

// Event listeners
subscription.on('status:change', (event) => {
  console.log(`Status: ${event.previousStatus} → ${event.status}`)
})

subscription.on('status:loadingMore', (event) => {
  console.log('Loading more data...')
})

subscription.on('status:ready', (event) => {
  console.log('Data loaded')
})

subscription.on('unsubscribed', (event) => {
  console.log('Subscription destroyed')
})

// Cleanup
subscription.unsubscribe()

Collection

// All collections have isLoadingMore property
console.log(collection.isLoadingMore) // boolean

// Listen for loading state changes
collection.on('loadingMore:change', (event) => {
  console.log(`Loading: ${event.isLoadingMore}`)
})

Live Query

const liveQuery = createLiveQueryCollection({
  query: (q) => q.from({ users: userCollection })
    .where(({ users }) => users.active)
})

// Result collection automatically tracks loading from subscriptions
console.log(liveQuery.isLoadingMore) // boolean

liveQuery.on('loadingMore:change', (event) => {
  if (event.isLoadingMore) {
    showLoadingSpinner()
  } else {
    hideLoadingSpinner()
  }
})

Breaking Changes

Renamed Methods and Types

syncMore → loadSubset (collection method)
onLoadMore → loadSubset (sync config property)
OnLoadMoreOptions → LoadSubsetOptions (type)

Migration:

// Before
const result = await collection.syncMore({ where: expr })
sync: {
  sync: () => ({
    onLoadMore: (options) => { /* ... */ }
  })
}

// After
const result = await collection._sync.loadSubset({ where: expr })
sync: {
  sync: () => ({
    loadSubset: (options) => { /* ... */ }
  })
}

Note: loadSubset is now called via collection._sync.loadSubset() as it's an internal coordination API, not for general public use.

Migration Guide

For Collection Users

No changes required if you're just using collections - isLoadingMore is automatically available.

For Sync Implementers

Rename onLoadMore to loadSubset:

// Before
return { onLoadMore: (options) => { /* ... */ } }

// After  
return { loadSubset: (options) => { /* ... */ } }

(Optional) Use the new subscription parameter for advanced use cases:

return {
  loadSubset: (options) => {
    // Track which subscription triggered this
    const sub = options.subscription
    
    // Can subscribe to unsubscribe event for cleanup
    sub?.on('unsubscribed', () => {
      // Clean up resources for this subscription
    })
  }
}

(Optional) Configure syncMode for on-demand loading:

createCollection({
  syncMode: 'on-demand', // Only load data as requested
  sync: {
    sync: ({ markReady }) => {
      markReady()
      return {
        loadSubset: async (options) => { /* load subset */ }
      }
    }
  }
})

…ubclass the same event emiiter implimetation

changeset-bot · 2025-10-12T18:19:35Z

🦋 Changeset detected

Latest commit: 53fa027

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
@tanstack/db	Patch
@tanstack/angular-db	Patch
@tanstack/electric-db-collection	Patch
@tanstack/query-db-collection	Patch
@tanstack/react-db	Patch
@tanstack/rxdb-db-collection	Patch
@tanstack/solid-db	Patch
@tanstack/svelte-db	Patch
@tanstack/trailbase-db-collection	Patch
@tanstack/vue-db	Patch
todos	Patch
@tanstack/db-example-react-todo	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

pkg-pr-new · 2025-10-12T18:21:02Z

More templates

@tanstack/angular-db

npm i https://pkg.pr.new/@tanstack/angular-db@669

@tanstack/db

npm i https://pkg.pr.new/@tanstack/db@669

@tanstack/db-ivm

npm i https://pkg.pr.new/@tanstack/db-ivm@669

@tanstack/electric-db-collection

npm i https://pkg.pr.new/@tanstack/electric-db-collection@669

@tanstack/query-db-collection

npm i https://pkg.pr.new/@tanstack/query-db-collection@669

@tanstack/react-db

npm i https://pkg.pr.new/@tanstack/react-db@669

@tanstack/rxdb-db-collection

npm i https://pkg.pr.new/@tanstack/rxdb-db-collection@669

@tanstack/solid-db

npm i https://pkg.pr.new/@tanstack/solid-db@669

@tanstack/svelte-db

npm i https://pkg.pr.new/@tanstack/svelte-db@669

@tanstack/trailbase-db-collection

npm i https://pkg.pr.new/@tanstack/trailbase-db-collection@669

@tanstack/vue-db

npm i https://pkg.pr.new/@tanstack/vue-db@669

commit: 53fa027

github-actions · 2025-10-12T18:24:11Z

Size Change: +1.66 kB (+2.04%)

Total Size: 83.2 kB

Filename	Size	Change
`./packages/db/dist/esm/collection/events.js`	413 B	-247 B (-37.42%)	🎉
`./packages/db/dist/esm/collection/index.js`	3.32 kB	+13 B (+0.39%)
`./packages/db/dist/esm/collection/subscription.js`	2.16 kB	+328 B (+17.93%)	⚠️
`./packages/db/dist/esm/collection/sync.js`	2.21 kB	+526 B (+31.25%)	🚨
`./packages/db/dist/esm/query/live/collection-config-builder.js`	5.31 kB	+5 B (+0.09%)
`./packages/db/dist/esm/query/live/collection-subscriber.js`	2.1 kB	+241 B (+12.94%)	⚠️
`./packages/db/dist/esm/event-emitter.js`	798 B	+798 B (new file)	🆕

ℹ️ View Unchanged

Filename	Size
`./packages/db/dist/esm/collection/change-events.js`	963 B
`./packages/db/dist/esm/collection/changes.js`	1.01 kB
`./packages/db/dist/esm/collection/indexes.js`	1.16 kB
`./packages/db/dist/esm/collection/lifecycle.js`	1.8 kB
`./packages/db/dist/esm/collection/mutations.js`	2.52 kB
`./packages/db/dist/esm/collection/state.js`	3.79 kB
`./packages/db/dist/esm/deferred.js`	230 B
`./packages/db/dist/esm/errors.js`	3.5 kB
`./packages/db/dist/esm/index.js`	1.63 kB
`./packages/db/dist/esm/indexes/auto-index.js`	794 B
`./packages/db/dist/esm/indexes/base-index.js`	835 B
`./packages/db/dist/esm/indexes/btree-index.js`	2 kB
`./packages/db/dist/esm/indexes/lazy-index.js`	1.21 kB
`./packages/db/dist/esm/indexes/reverse-index.js`	577 B
`./packages/db/dist/esm/local-only.js`	967 B
`./packages/db/dist/esm/local-storage.js`	2.33 kB
`./packages/db/dist/esm/optimistic-action.js`	294 B
`./packages/db/dist/esm/proxy.js`	3.86 kB
`./packages/db/dist/esm/query/builder/functions.js`	615 B
`./packages/db/dist/esm/query/builder/index.js`	4.04 kB
`./packages/db/dist/esm/query/builder/ref-proxy.js`	938 B
`./packages/db/dist/esm/query/compiler/evaluators.js`	1.55 kB
`./packages/db/dist/esm/query/compiler/expressions.js`	760 B
`./packages/db/dist/esm/query/compiler/group-by.js`	2.04 kB
`./packages/db/dist/esm/query/compiler/index.js`	2.19 kB
`./packages/db/dist/esm/query/compiler/joins.js`	2.63 kB
`./packages/db/dist/esm/query/compiler/order-by.js`	1.26 kB
`./packages/db/dist/esm/query/compiler/select.js`	1.28 kB
`./packages/db/dist/esm/query/ir.js`	785 B
`./packages/db/dist/esm/query/live-query-collection.js`	404 B
`./packages/db/dist/esm/query/live/collection-registry.js`	233 B
`./packages/db/dist/esm/query/optimizer.js`	3.26 kB
`./packages/db/dist/esm/scheduler.js`	1.29 kB
`./packages/db/dist/esm/SortedMap.js`	1.24 kB
`./packages/db/dist/esm/transactions.js`	3.05 kB
`./packages/db/dist/esm/utils.js`	1.01 kB
`./packages/db/dist/esm/utils/browser-polyfills.js`	365 B
`./packages/db/dist/esm/utils/btree.js`	6.01 kB
`./packages/db/dist/esm/utils/comparison.js`	754 B
`./packages/db/dist/esm/utils/index-optimization.js`	1.73 kB

_{compressed-size-action::db-package-size}

github-actions · 2025-10-12T18:25:35Z

Size Change: 0 B

Total Size: 1.46 kB

ℹ️ View Unchanged

Filename	Size
`./packages/react-db/dist/esm/index.js`	152 B
`./packages/react-db/dist/esm/useLiveQuery.js`	1.31 kB

_{compressed-size-action::react-db-package-size}

…an unsunbscribed event to it

… event to the subscription, fix types

KyleAMathews

🚀

KyleAMathews · 2025-10-13T20:35:05Z

packages/db/src/collection/sync.ts

+    if (this.syncLoadSubsetFn) {
+      const result = this.syncLoadSubsetFn(options)
+
+      // If the result is void (synchronous), wrap in Promise.resolve()


We talked about collections needing to be able to return true or a promise — true if the data is loaded already. We need a sync response as otherwise useLiveQuery can't run sync

hmm need to check why it's wrapping it there...

Returning void/undefined (synchronicity) makes it easy to check if a promise was returned as it's thruthy. Returning undefined if there is noting to do makes the code simpler than true - no typeof checks - but happy to change.

KyleAMathews · 2025-10-13T20:38:48Z

packages/db/tests/query/live-query-collection.test.ts

+
+      await liveQuery.preload()
+
+      // Calling loadSubset directly on source collection sets its own isLoadingMore


I thought the plan here was isLoadingMore is true only if a live query calls updatePredicate? Otherwise a live query would be see there isLoadingMore set to true when e.g. a joined collection needs to grab an object, etc. So any UI set to this would be flickering on and off seemingly randomly w/o any way to control it by the dev.

This bit of the code all dates to when I was working over the weekend. Will update...

I do wander if we need to expose that it is grabbing data from the server though, joins lazy appearing without the dev being able to put a placeholder/spinner does feel messy.

Maybe we need to separate flags?

Also not that this version from the weekend made the isLoadingMore prop a standard on all collections. For base collections it's true when their own loadSubset is pending, for live query collections it's when a subscription trigger a loadSubset that returns a promise.

Maybe we need an isLoadingMore and a isLoadingSubset, the latter whenever any subset triggered by the collection is loading, and isLoadingMore for when it's triggered by an offset/limit change (which do not isn't complete yet, and not in this PR)

samwillis added 2 commits October 12, 2025 18:48

wip

f329fb1

refactor so both CollectionEventsManager and CollectionSubscription s…

dc34f5b

…ubclass the same event emiiter implimetation

changeset

b7f45c8

samwillis added 3 commits October 13, 2025 18:12

rename loadMore to loadSubset

d6c6e7e

feed the subscription object through to the loadSubset call, and add …

7b584b1

…an unsunbscribed event to it

feed subscription through to the loadSubset callback, add unsubscribe…

5636c46

… event to the subscription, fix types

samwillis force-pushed the samwillis/load-more-tracking branch from 4d186d9 to 68dc938 Compare October 13, 2025 19:37

fix types

b1aeceb

samwillis force-pushed the samwillis/load-more-tracking branch from 68dc938 to b1aeceb Compare October 13, 2025 19:52

add sync mode to base colleciton

53fa027

samwillis changed the title ~~Add "Load More" Status Tracking to Collections and Live Queries~~ Add loadSubset State Tracking and On-Demand Sync Mode Oct 13, 2025

KyleAMathews reviewed Oct 13, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add loadSubset State Tracking and On-Demand Sync Mode #669

Add loadSubset State Tracking and On-Demand Sync Mode #669

samwillis commented Oct 12, 2025 •

edited

Loading

Uh oh!

changeset-bot bot commented Oct 12, 2025 •

edited

Loading

Uh oh!

pkg-pr-new bot commented Oct 12, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 12, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Oct 12, 2025 •

edited

Loading

Uh oh!

KyleAMathews left a comment

Uh oh!

KyleAMathews Oct 13, 2025

Uh oh!

samwillis Oct 13, 2025

Uh oh!

KyleAMathews Oct 13, 2025

Uh oh!

samwillis Oct 13, 2025 •

edited

Loading

Uh oh!

samwillis Oct 13, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		await liveQuery.preload()

		// Calling loadSubset directly on source collection sets its own isLoadingMore

Add loadSubset State Tracking and On-Demand Sync Mode #669

Are you sure you want to change the base?

Add loadSubset State Tracking and On-Demand Sync Mode #669

Conversation

samwillis commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Problem

Solution

Loading State Tracking

Sync Modes

Changes

1. Renamed syncMore/loadMore to loadSubset

2. Added syncMode Configuration

3. Standardized loadSubset Return Type

4. CollectionSubscription Status Tracking & Events

5. Generic isLoadingMore for All Collections

6. Live Query Integration

7. Subscription Interface for External Consumers

8. Reusable Event Emitter

9. Fixed Local-Only Collection Types

10. Comprehensive Test Coverage

API

Sync Mode Configuration

Collection Subscription

Collection

Live Query

Breaking Changes

Renamed Methods and Types

Migration Guide

For Collection Users

For Sync Implementers

Uh oh!

changeset-bot bot commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

pkg-pr-new bot commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KyleAMathews left a comment

Choose a reason for hiding this comment

Uh oh!

KyleAMathews Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

samwillis Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

KyleAMathews Oct 13, 2025

Choose a reason for hiding this comment

Uh oh!

samwillis Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samwillis Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

samwillis commented Oct 12, 2025 •

edited

Loading

1. Renamed `syncMore`/`loadMore` to `loadSubset`

2. Added `syncMode` Configuration

3. Standardized `loadSubset` Return Type

5. Generic `isLoadingMore` for All Collections

changeset-bot bot commented Oct 12, 2025 •

edited

Loading

pkg-pr-new bot commented Oct 12, 2025 •

edited

Loading

github-actions bot commented Oct 12, 2025 •

edited

Loading

github-actions bot commented Oct 12, 2025 •

edited

Loading

samwillis Oct 13, 2025 •

edited

Loading

samwillis Oct 13, 2025 •

edited

Loading