Fix #5461 — an absent key must not project as an explicit JSON null

[jeremydmiller]

Closes #5461

The bug

A Select() projection is translated to jsonb_build_object(), and a stored document that simply does not have the key — a property added to the document type after those documents were written — locates as SQL NULL. jsonb_build_object() has no way to leave a pair out, so the projection gains a key the stored document never had, holding an explicit null. Deserializing that null into a non-nullable value type (bool, int, Guid, an enum) throws, while the very same document loads fine as a whole document and projects fine as a bare scalar.

Reproduced with the default Newtonsoft serializer as well as System.Text.Json, so it is not an STJ quirk.

Not a #5011 regression

The issue attributes this to #5011. It predates it: the jsonb_build_object translation for simple flat member-access projections dates to 14068e075 (Dec 2023, GH-1926) and is present unchanged on 8.0. #5011 only added the client-side fallback for projections that are not simple (method calls, arithmetic, ternaries). The reported shape — new { x.CareTypes, x.ExecuteWlzCheck } — took the jsonb_build_object path both before and after it. 7.x/8.x/9.x are all affected, so 8.0 is a valid backport target.

The fix

jsonb_strip_nulls() puts the absence back, but it also recurses into object and array values, which would quietly drop meaningful nulls out of a projected Dictionary<string, T?> or nested document. So NewObject now partitions its pairs and sends only the ones locating a non-nullable value-type scalar through the strip:

(jsonb_build_object('CareTypes', CAST(d.data ->> 'CareTypes' as jsonb))
 || jsonb_strip_nulls(jsonb_build_object('ExecuteWlzCheck', CAST(d.data ->> 'ExecuteWlzCheck' as boolean))))

When no pair qualifies — the common case — the emitted SQL is byte-identical to before. Collections, child documents and nested NewObjects never enter the strip, so their values cannot be reached into.

Omitting the pair rather than coalesce-ing to a CLR default is the better of the issue's directions: it defers to the target type's own initializer, needs no per-type default literal (Guid, DateTime, enum-as-string, decimal…), and does not write a value into streamed JSON that the document does not contain. For anonymous types — the overwhelming case — the two are equivalent anyway.

ChildCollectionJsonPathCount carries the same latent bug (Count(predicate) over an absent collection is jsonb_array_length(NULL) → null → throws for int) and opts in through the new IOmittedWhenNull marker. Synthetic members (an array index) carry no MemberType at all and keep the behavior they already had rather than guessing.

Verification

• LinqTests 1492/1492 green under both Newtonsoft and System.Text.Json
• DocumentDbTests 1113/1113 green
• New Bug_5461_absent_key_in_select_projection.cs: the repro, the three control rows from the issue's table, the emitted SQL shape, raw JSON streaming, and a stored dictionary null that a blanket jsonb_strip_nulls() would have dropped

Deliberately out of scope

• Absent collection / child-document members still project as explicit null. Omitting them would not help: an anonymous type's constructor parameter defaults to null either way.
• GroupBy aggregate fragments (Sum over all-absent keys) can still emit null into a non-nullable target — same class, separate fix.
• Neither omission nor coalesce can reproduce a source type's non-default initializer (= true); only the client-side path knows it. Omission at least defers to the target type.

🤖 Generated with Claude Code

https://claude.ai/code/session_01481eiEZg1Bv6pu4DrgPRg3