Skip to content

[#4409] JSON-encode scalar projections when streaming to a response body#4464

Merged
jeremydmiller merged 3 commits into
masterfrom
fix/4409-scalar-string-streaming-json-quoting
May 18, 2026
Merged

[#4409] JSON-encode scalar projections when streaming to a response body#4464
jeremydmiller merged 3 commits into
masterfrom
fix/4409-scalar-string-streaming-json-quoting

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Summary

Closes #4409.

WriteArray / StreamJsonArray / WriteOne / StreamJsonOne against a scalar projection.Select(x => x.SomeEnum) under EnumStorage.AsString, or any .Select(x => x.StringProperty) — used to emit raw Postgres bytes between the array brackets and commas. data->>'X' comes back as text with no JSON quoting, so the body landed as

[FooValue,BarValue]

— invalid JSON, with downstream System.Text.Json blowing up:

'F' is an invalid start of a value. Path: $[0] | LineNumber: 0 | BytePositionInLine: 1.

Numeric / boolean projections happened to look like valid JSON literals ([1,2,3]), which is why the bug only surfaced for string-valued scalars.

Fix

Marten.Services.JsonStreamingExtensions gets a new WriteJsonValueAsync(NpgsqlDataReader, int ordinal, Stream, CancellationToken) helper that branches on reader.GetDataTypeName(ordinal):

  • jsonb / json → copy the field stream byte-for-byte with the existing SOH-skip (unchanged document-streaming behavior).
  • DBNull → write the JSON null literal.
  • Everything else → materialize the .NET value via reader.GetFieldValueAsync<object> and round-trip through JsonSerializer.SerializeAsync with the column's CLR type. Strings get JSON-quoted and escaped (handles embedded ", \, control characters); numerics, booleans, dates, and enum-as-string come out as their JSON literal / quoted-string representation.

Routes the existing StreamMany / StreamOne reader extensions and the inline copy in OneResultHandler.StreamJson through the new helper. Document-streaming paths (whole-doc jsonb column) keep the previous fast-path verbatim — WriteJsonValueAsync's first branch is the same SOH-skip copy that was there before.

Test plan

Five new regression tests in Bug_4409_streaming_scalar_string_projection:

  • stream_array_of_enum_as_string_projection_emits_valid_json — primary repro from the issue (enum-as-string).
  • stream_array_of_string_projection_emits_valid_jsonSelect(x => x.Name) case the issue also calls out.
  • stream_array_of_string_projection_escapes_embedded_special_characters — pins the JSON-escaping path against quotes, backslashes, and newlines in the projected text.
  • stream_array_of_int_projection_still_emits_valid_json — guards against regressing the numeric path that already produced valid JSON.
  • stream_array_of_jsonb_documents_still_emits_valid_json — guards the whole-document jsonb path (must take the fast-path branch).

Pre-fix: 4/5 fail with JsonReaderException on body parse. Post-fix: 5/5 green.

Broader runs:

  • Build clean (0 errors)
  • DocumentDbTests — 987 pass, 1 pre-existing skip, 0 fail (net10.0)
  • LinqTests — 1257 pass, 1 pre-existing skip, 0 fail (net10.0)
  • CI matrix

🤖 Generated with Claude Code

jeremydmiller and others added 2 commits May 18, 2026 07:14
WriteArray / StreamJsonArray / WriteOne / StreamJsonOne against a scalar
projection (`.Select(x => x.SomeEnum)` under `EnumStorage.AsString`, or any
`.Select(x => x.StringProperty)`) used to emit raw bytes from Postgres
between the array brackets and commas. Postgres returns `data->>'X'` as
text (no JSON quoting), so the body landed as `[FooValue,BarValue]` —
invalid JSON, and downstream `System.Text.Json` reads blew up with
"'F' is an invalid start of a value". Numeric / boolean projections
happened to look like valid JSON, masking the bug.

Centralize per-row writes through a new `WriteJsonValueAsync` on
`NpgsqlDataReader` that branches on `GetDataTypeName`:
- `jsonb` / `json` columns — copy the field stream byte-for-byte with the
  existing SOH-skip (unchanged document-streaming behavior).
- DBNull — write the JSON `null` literal.
- Everything else — materialize the .NET value via
  `reader.GetFieldValueAsync<object>` and round-trip through
  `JsonSerializer.SerializeAsync` so strings get JSON-quoted and escaped
  (handling embedded `"`, `\`, control characters), numerics / booleans /
  dates get their JSON literal representation, and enums-as-string come
  out as `"FooValue"` instead of `FooValue`.

Routes the existing `StreamMany` / `StreamOne` extensions and the inline
copy in `OneResultHandler.StreamJson` through the new helper. Document
streaming paths (whole-doc jsonb column) keep the previous fast-path
verbatim — `WriteJsonValueAsync`'s first branch is the same SOH-skip
copy that was there before.

Closes #4409.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nc's cancellation guard

#4463's OCE-rethrow guard only matched bare OperationCanceledException.
JasperFxAsyncDaemon.CatchUpAsync bundles per-shard cancellation
exceptions into an AggregateException before throwing, so a CI run
where the test's CTS fires partway through the catch-up still surfaces
as a misleading "exceptions should be empty but had 1 item" assertion
failure — the AggregateException wraps a single OperationCanceledException
(wrapping a Postgres 57014 'canceling statement due to user request')
but the outer type didn't match the guard.

Tighten the guard with an IsCallerCancellation helper that recursively
treats AggregateException-of-cancellations as caller cancellation too.
On the cancelled-CT path the helper paves over both the bare OCE and
the AggregateException cases, replaces both with a single
OperationCanceledException re-throw, and leaves genuine daemon
exceptions flowing into the list as before.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
#4463 added an OCE-rethrow guard, and `4f715820` extended it to unwrap
AggregateException-of-OCEs. Neither helped the latest CI flake:
https://github.com/JasperFx/marten/actions/runs/26035890428/job/76533776264?pr=4464

The new evidence: failure lands in 176 ms with the test's CTS set to 45s,
so `cancellation.IsCancellationRequested` is false in the catch handler
— the OCE isn't caller cancellation. It originates at
GroupedProjectionExecution.processRangeAsync line 192 →
range.Agent.ReportCriticalFailureAsync(e), where the per-shard internal
CTS gets cancelled during the batch build. The daemon's Recorder
captures it and JasperFxAsyncDaemon.CatchUpAsync line 716 re-throws as
AggregateException — by which point ForceAllMartenDaemonActivityToCatchUpAsync's
guards can't tell apart "user cancelled" from "shard's own state got
cancelled by its own lifecycle."

The real fix lives in the JasperFx.Events daemon's internal cancellation
contract (StopAllAsync followed by CatchUpAsync leaking a cancelled
per-shard CTS into the new agents under timing pressure). Tracking that
in #4462. The AggregateException unwrap stays in place as defense in
depth for the caller-cancellation path — it's still correct, just
insufficient on its own.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@jeremydmiller jeremydmiller merged commit e3aca30 into master May 18, 2026
6 checks passed
@jeremydmiller jeremydmiller deleted the fix/4409-scalar-string-streaming-json-quoting branch May 18, 2026 13:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WriteArray/StreamMany produces unquoted JSON for scalar enum projections (EnumStorage.AsString)

1 participant