Skip to content

feat: StreamPaged<T> - stream a paged JSON envelope in a single round-trip (fixes #5009) - #5014

Merged
jeremydmiller merged 3 commits into
JasperFx:masterfrom
erdtsieck:erdtsieck-streampaged-issue-5009
Jul 22, 2026
Merged

feat: StreamPaged<T> - stream a paged JSON envelope in a single round-trip (fixes #5009)#5014
jeremydmiller merged 3 commits into
JasperFx:masterfrom
erdtsieck:erdtsieck-streampaged-issue-5009

Conversation

@erdtsieck

Copy link
Copy Markdown
Contributor

Summary

Adds a new StreamPaged<T> result type to Marten.AspNetCore that streams a paged list of documents plus paging metadata as a single JSON envelope directly to the HTTP response, in one database round trip:

{"pageNumber":3,"pageSize":25,"totalItemCount":1207,"pageCount":49,"hasNextPage":true,"hasPreviousPage":true,"items":[...]}

Closes #5009.

Implementation

  • Marten.Linq.MartenLinqQueryable<T>.StreamPagedJsonArray - sets Statistics, applies Skip/Take, delegates to the provider.
  • MartenLinqQueryProvider.StreamPagedMany - builds the LINQ statement/command (which includes count(*) OVER() as total_rows via LinqConstants.StatsColumn when Statistics is set), then executes via the session.
  • QuerySession.StreamPagedMany + JsonStreamingExtensions.StreamPagedMany - reads total_rows from the first row of the DbDataReader, computes pageCount/hasNextPage/hasPreviousPage (mirroring PagedList.ProcessResults), writes the envelope prefix, then streams each row's raw document JSON straight into the items array without deserializing/reserializing documents.
  • QueryableExtensions.StreamPagedJsonArray<T> (Marten core) - public entry point exposing the above.
  • Marten.AspNetCore.QueryableExtensions.WritePaged<T> + StreamPaged<T> (IResult + IEndpointMetadataProvider, mirroring StreamMany<T>) - the minimal-API/Wolverine.Http-facing wrapper.

Zero-row result sets produce totalItemCount: 0, pageCount: 0, and an empty items array (count(*) OVER() is not present on any row when nothing matches).

Tests

  • src/DocumentDbTests/Reading/Json/streaming_json_results.cs - 8 new [Fact]s covering first/middle/last/single page, no-hits, and argument validation (pageNumber < 1, pageSize < 1). All passing.
  • src/Marten.AspNetCore.Testing/stream_paged_tests.cs (new) - Alba HTTP-level tests against a new /minimal/issues/paged/{pageNumber}/{pageSize} endpoint added to IssueService.

Docs

  • docs/documents/querying/linq/paging.md - new "Streaming a Paged JSON Envelope" section describing StreamPagedJsonArray<T>().
  • docs/documents/aspnetcore.md - added StreamPaged<T> to the typed streaming result types table and a full usage subsection.
  • Verified clean with markdownlint-cli (--disable MD009) and cspell against docs/**/*.md.

Test status

  • DocumentDbTests (both net9.0 and net10.0, run individually): all 8 new stream_paged_json_array_* tests pass.
  • Marten.AspNetCore.Testing: the whole suite (including pre-existing streaming_result_types_tests) currently fails to boot the Alba test host on this branch's base commit due to an unrelated, pre-existing environment issue (InvalidProjectionException: No source-generated dispatcher found for SingleStreamProjection<Order, Guid>). Verified this failure is not caused by this change - it reproduces identically on a clean, unmodified checkout of master.

Notes

  • Running the two test TFMs (net9.0/net10.0) together via dotnet test in parallel against the same shared Postgres schema can cause cross-run data contention/deadlocks unrelated to this feature; running each TFM individually avoids this and both pass cleanly.

…-trip (fixes JasperFx#5009)

Adds a new StreamPaged<T> result type to Marten.AspNetCore that streams a
paged list of documents plus paging metadata as a single JSON envelope
directly to the HTTP response, in a single database round trip:

  {"pageNumber":3,"pageSize":25,"totalItemCount":1207,"pageCount":49,
   "hasNextPage":true,"hasPreviousPage":true,"items":[...]}

Implementation:
- Marten.Linq.MartenLinqQueryable<T>.StreamPagedJsonArray: sets Statistics,
  applies Skip/Take, delegates to the provider.
- MartenLinqQueryProvider.StreamPagedMany: builds the Linq statement/command
  (which includes count(*) OVER() as total_rows via LinqConstants.StatsColumn
  when Statistics is set), then executes via the session.
- QuerySession.StreamPagedMany + JsonStreamingExtensions.StreamPagedMany:
  reads total_rows from the first row of the DbDataReader, computes
  pageCount/hasNextPage/hasPreviousPage (mirroring PagedList.ProcessResults),
  writes the envelope prefix, then streams each row's raw document JSON
  into the items array without deserializing/reserializing documents.
- QueryableExtensions.StreamPagedJsonArray<T> (Marten core): public entry
  point exposing the above.
- Marten.AspNetCore.QueryableExtensions.WritePaged<T> + StreamPaged<T>
  (IResult + IEndpointMetadataProvider, mirroring StreamMany<T>): the
  minimal-API/Wolverine.Http-facing wrapper.

Zero-row result sets produce totalItemCount: 0, pageCount: 0, and an empty
items array (count(*) OVER() is not present when no rows match).

Tests:
- src/DocumentDbTests/Reading/Json/streaming_json_results.cs: 8 new Facts
  covering first/middle/last/single page, no-hits, and argument validation.
- src/Marten.AspNetCore.Testing/stream_paged_tests.cs (new): Alba HTTP-level
  tests against a new /minimal/issues/paged/{pageNumber}/{pageSize} endpoint.
- Adds a new 'Streaming a Paged JSON Envelope' section to
  docs/documents/querying/linq/paging.md describing
  StreamPagedJsonArray<T>() with a code example.
- Adds a new StreamPaged<T> subsection (with a Wolverine.Http-style
  Minimal API example) to the 'Typed Streaming Result Types' table
  and section in docs/documents/aspnetcore.md, cross-linking to the
  paging.md section.
- Verified clean with markdownlint-cli (--disable MD009) and cspell
  against docs/**/*.md.
@erdtsieck
erdtsieck force-pushed the erdtsieck-streampaged-issue-5009 branch 2 times, most recently from af5cc07 to a3126fe Compare July 22, 2026 10:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StreamPaged<T>: stream a paged JSON envelope in a single round-trip

2 participants