[#4409] JSON-encode scalar projections when streaming to a response body#4464
Merged
jeremydmiller merged 3 commits intoMay 18, 2026
Merged
Conversation
WriteArray / StreamJsonArray / WriteOne / StreamJsonOne against a scalar projection (`.Select(x => x.SomeEnum)` under `EnumStorage.AsString`, or any `.Select(x => x.StringProperty)`) used to emit raw bytes from Postgres between the array brackets and commas. Postgres returns `data->>'X'` as text (no JSON quoting), so the body landed as `[FooValue,BarValue]` — invalid JSON, and downstream `System.Text.Json` reads blew up with "'F' is an invalid start of a value". Numeric / boolean projections happened to look like valid JSON, masking the bug. Centralize per-row writes through a new `WriteJsonValueAsync` on `NpgsqlDataReader` that branches on `GetDataTypeName`: - `jsonb` / `json` columns — copy the field stream byte-for-byte with the existing SOH-skip (unchanged document-streaming behavior). - DBNull — write the JSON `null` literal. - Everything else — materialize the .NET value via `reader.GetFieldValueAsync<object>` and round-trip through `JsonSerializer.SerializeAsync` so strings get JSON-quoted and escaped (handling embedded `"`, `\`, control characters), numerics / booleans / dates get their JSON literal representation, and enums-as-string come out as `"FooValue"` instead of `FooValue`. Routes the existing `StreamMany` / `StreamOne` extensions and the inline copy in `OneResultHandler.StreamJson` through the new helper. Document streaming paths (whole-doc jsonb column) keep the previous fast-path verbatim — `WriteJsonValueAsync`'s first branch is the same SOH-skip copy that was there before. Closes #4409. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nc's cancellation guard #4463's OCE-rethrow guard only matched bare OperationCanceledException. JasperFxAsyncDaemon.CatchUpAsync bundles per-shard cancellation exceptions into an AggregateException before throwing, so a CI run where the test's CTS fires partway through the catch-up still surfaces as a misleading "exceptions should be empty but had 1 item" assertion failure — the AggregateException wraps a single OperationCanceledException (wrapping a Postgres 57014 'canceling statement due to user request') but the outer type didn't match the guard. Tighten the guard with an IsCallerCancellation helper that recursively treats AggregateException-of-cancellations as caller cancellation too. On the cancelled-CT path the helper paves over both the bare OCE and the AggregateException cases, replaces both with a single OperationCanceledException re-throw, and leaves genuine daemon exceptions flowing into the list as before. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#4463 added an OCE-rethrow guard, and `4f715820` extended it to unwrap AggregateException-of-OCEs. Neither helped the latest CI flake: https://github.com/JasperFx/marten/actions/runs/26035890428/job/76533776264?pr=4464 The new evidence: failure lands in 176 ms with the test's CTS set to 45s, so `cancellation.IsCancellationRequested` is false in the catch handler — the OCE isn't caller cancellation. It originates at GroupedProjectionExecution.processRangeAsync line 192 → range.Agent.ReportCriticalFailureAsync(e), where the per-shard internal CTS gets cancelled during the batch build. The daemon's Recorder captures it and JasperFxAsyncDaemon.CatchUpAsync line 716 re-throws as AggregateException — by which point ForceAllMartenDaemonActivityToCatchUpAsync's guards can't tell apart "user cancelled" from "shard's own state got cancelled by its own lifecycle." The real fix lives in the JasperFx.Events daemon's internal cancellation contract (StopAllAsync followed by CatchUpAsync leaking a cancelled per-shard CTS into the new agents under timing pressure). Tracking that in #4462. The AggregateException unwrap stays in place as defense in depth for the caller-cancellation path — it's still correct, just insufficient on its own. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #4409.
WriteArray/StreamJsonArray/WriteOne/StreamJsonOneagainst a scalar projection —.Select(x => x.SomeEnum)underEnumStorage.AsString, or any.Select(x => x.StringProperty)— used to emit raw Postgres bytes between the array brackets and commas.data->>'X'comes back astextwith no JSON quoting, so the body landed as— invalid JSON, with downstream
System.Text.Jsonblowing up:Numeric / boolean projections happened to look like valid JSON literals (
[1,2,3]), which is why the bug only surfaced for string-valued scalars.Fix
Marten.Services.JsonStreamingExtensionsgets a newWriteJsonValueAsync(NpgsqlDataReader, int ordinal, Stream, CancellationToken)helper that branches onreader.GetDataTypeName(ordinal):jsonb/json→ copy the field stream byte-for-byte with the existing SOH-skip (unchanged document-streaming behavior).DBNull→ write the JSONnullliteral.reader.GetFieldValueAsync<object>and round-trip throughJsonSerializer.SerializeAsyncwith the column's CLR type. Strings get JSON-quoted and escaped (handles embedded",\, control characters); numerics, booleans, dates, and enum-as-string come out as their JSON literal / quoted-string representation.Routes the existing
StreamMany/StreamOnereader extensions and the inline copy inOneResultHandler.StreamJsonthrough the new helper. Document-streaming paths (whole-docjsonbcolumn) keep the previous fast-path verbatim —WriteJsonValueAsync's first branch is the same SOH-skip copy that was there before.Test plan
Five new regression tests in
Bug_4409_streaming_scalar_string_projection:stream_array_of_enum_as_string_projection_emits_valid_json— primary repro from the issue (enum-as-string).stream_array_of_string_projection_emits_valid_json—Select(x => x.Name)case the issue also calls out.stream_array_of_string_projection_escapes_embedded_special_characters— pins the JSON-escaping path against quotes, backslashes, and newlines in the projected text.stream_array_of_int_projection_still_emits_valid_json— guards against regressing the numeric path that already produced valid JSON.stream_array_of_jsonb_documents_still_emits_valid_json— guards the whole-document jsonb path (must take the fast-path branch).Pre-fix: 4/5 fail with
JsonReaderExceptionon body parse. Post-fix: 5/5 green.Broader runs:
DocumentDbTests— 987 pass, 1 pre-existing skip, 0 fail (net10.0)LinqTests— 1257 pass, 1 pre-existing skip, 0 fail (net10.0)🤖 Generated with Claude Code