You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Cosmos membership reads can combine a table version with rows from a different membership update. Heartbeat document etags can invalidate membership writes, cleanup can delete a row which changed after selection, and failed transactional responses can be reported as ordinary contention.
Solution and rationale
Inherit the configured client/account consistency for membership point reads and immutable-ID-ordered query pages. Match opening/closing version etags to fence reads which straddle a canonical membership update. Retry changed views and report failure when a stable view cannot be obtained.
Expose the table-version etag as each canonical row token. Full-row writes validate the supplied logical row/table tokens, then atomically compare-and-swap the table version and replace the existing silo row. Heartbeat-only changes preserve these tokens. Canonical conflicts and missing rows reject the whole transaction.
Publish the single-owner periodic heartbeat with one native no-content patch setting only /IAmAliveTime. This path performs zero reads and one unconditional column write; native cancellation and storage failures, including missing rows, remain visible. Full-row writes may overwrite the unversioned heartbeat timestamp, as permitted by the membership contract.
Prune only Dead rows whose maximum start, heartbeat, and suspicion-vote timestamp is strictly before the UTC cutoff. Delete using each captured physical row etag while leaving the membership version intact, consistent with fix(membership): preserve canonical membership views #11296.
Scope administrative deletion to the configured cluster, capture a stable snapshot, and condition deletion on the captured physical row etags and canonical version etag. Respect native batch limits and preserve concurrent changes with visible conflicts; completed chunks remain committed if a later chunk conflicts.
Preserve cancellation, shared-client ownership, and visible infrastructure failures. Reuse one feed iterator per snapshot attempt, suppress unused canonical write response bodies, and rely on the SDK's successful create-if-absent completion contract.
Configuration compatibility
Production membership uses a Cosmos account with a single writable region and Strong consistency. Configure clients to inherit the account default or use Strong consistency. Read requests inherit that effective configuration, as requested during review; the provider performs no account-policy validation. Clients selecting weaker consistency are outside the documented coherent-read guarantee.
Document schemas, partition keys, IDs, serialization contracts, and public signatures remain unchanged. Logical row tokens remain opaque strings and track canonical membership independently of heartbeat-modified document etags.
Native CI backend and coverage
The net8.0/net10.0 Cosmos jobs exercise the full native provider suite on Ubuntu against pinned vNext Linux emulator vnext-EN20260907@sha256:2db1f9e74c506bcf6fc347aa937aea1c00fa756061296a5a9efba530ce86ec02, using its documented readiness endpoint. The three transactional-etag membership tests formerly skipped on the preview emulator remain enabled. A native regression verifies a canonical update can use original row/table tokens after an owner heartbeat, and rejects subsequent stale canonical tokens.
The full suite passes with the emulator's inherited Eventual default; this establishes functional compatibility, not distributed Strong consistency. Production configuration requirements remain documented. Classic Linux emulator crashes remain tracked separately in #11323.
The existing Linux/net10.0 coverage policy automatically includes Cosmos. Shared coverage actions, coverage regression scripts, and the required artifact matrix are unchanged from the PR baseline. Trusted coverage-publication input checks remain intact; the remaining emulator workflow changes are subject to that review gate.
Extraction provenance and scope
Adapted the Cosmos portion of ReubenBond/orleans:rb-test-membership-provider-conformance at 1dfe8b54cda3582ddf3795113eaa2f4e9d1b6254, originally based on ad71848f8495b9b87c7891f4ddc0c1d8759d21d6, onto main d515d75eaaa3a0390a74cb11c96aa758358b5a67 (the #11296 merge).
Separate prerequisite c9eaf2781f58c3c8490ff24197e1fa87f4a695fe carries only shared legacy cleanup-test preparation from bede893d290a9035f196f4c3ed58213d102efc6c, also used by #11308. Initial provider extraction 9e922275130d18241529bb839ec05af688daad24 is followed by review changes for administrative deletion, configured read consistency, a single-owner blind heartbeat, heartbeat-independent canonical tokens, and reduced redundant storage work.
Omitted source requirements include account-metadata startup validation, versioned cleanup transactions/reserved slots, multi-writer heartbeat maxima, retired-row heartbeat no-ops, and full-row heartbeat preservation. The full conformance kit/adapters, project/solution wiring, core/other-provider changes, and broad documentation remain separate. There is no new schema or sidecar.
Extract the Cosmos data-plane changes from ReubenBond/orleans at 1dfe8b5 (original base ad71848), adapted to main d515d75 after dotnet#11296.
Fence partition reads with session tokens and version etags, preserve heartbeat maxima, and distinguish contention from infrastructure failures. Prune captured Dead rows conditionally without version transactions. Use native SDK query fakes for focused regressions.
Omit source account-startup validation, cleanup version reservations, conformance adapters/testkit wiring, and changes to core or other providers. The preceding commit carries only the shared legacy cleanup-test prep from bede893 (dotnet#11308).
Capture a session-fenced membership snapshot and condition row and version deletion on its etags. Concurrent row changes and advanced or recreated versions survive with a visible conflict. Add deterministic interleaving regressions and correct test-helper whitespace reported by CI.
This loop has no retry bound or backoff: if membership updates continuously change the row etag, every replace can fail with PreconditionFailed and the heartbeat task will run indefinitely until its caller cancels it. Unlike the snapshot path, this operation has no defined failure outcome under sustained contention, which can leave liveness work permanently occupied. Limit the retries (and surface a wrapped failure or retryable result) while preserving cancellation.
The newest successful coverage run tested 53b0307, not current main f278e80.
Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.
The comparison remains report-only while normal line and branch variance is calibrated.
Request Strong consistency on initialization, membership snapshots, heartbeat/full-row reads, cleanup selection, and version-history checks. Retain version-etag fences and immutable-ID pagination, and remove session-token propagation. Cover every read entry point and visible consistency rejection in focused mocked tests.
BREAKING CHANGE: Cosmos membership now requires an account configured for Strong consistency.
Use the pinned classic emulator 2.14.26 with AZURE_COSMOS_EMULATOR_ARGS=/Consistency=Strong instead of vNext's Eventual-only account. Verify account metadata in native tests and enable transactional-etag membership tests for the actual backend. Preserve the net8/net10 Cosmos matrix and all provider cases.
Cover heartbeat CAS cancellation and convergence after a newer heartbeat without imposing an arbitrary contention cap. Native Strong configuration uses the pass-through documented in Azure/azure-cosmos-db-emulator-docker#141 and the emulator's supported /Consistency option.
Addressing the heartbeat-loop concern in #11312 (review): the loop retries only an etag precondition failure (412), with an awaited Strong reread before each conditional replacement. A newer stored heartbeat completes the operation; cancellation is checked on every iteration and passed to both SDK calls. Authorization, transport, missing-history and other storage failures propagate to the caller. Sustained valid contention can continue until cancellation, so this is cancellation-bounded rather than attempt-bounded; an arbitrary cap would turn valid membership contention into a heartbeat failure. The native SDK requests provide I/O suspension, and a 412 indicates a changed row etag rather than a malformed local-state spin.
Commit 75bdbba adds HeartbeatContentionObservesCancellation and HeartbeatContentionCompletesWhenConcurrentHeartbeatAdvances; all 71 focused cases pass locally. Production Strong consistency is unchanged.
The same commit addresses the native CI failure: the preview emulator advertises Eventual consistency and rejects Strong requests. The existing Cosmos jobs now use pinned classic emulator 2.14.26 with its native /Consistency=Strong option (Azure/azure-cosmos-db-emulator-docker#141; https://learn.microsoft.com/en-us/azure/cosmos-db/emulator-windows-arguments). A native test verifies actual account metadata reports Strong, and the three existing transactional-etag membership tests are enabled. Both net8.0 and net10.0 CI must pass on this head before readiness.
Use the current 2.14.28 classic emulator with its canonical OCI digest. The previous 2.14.26 image exits during native startup because its evaluation period expired. Preserve native Strong configuration and all Cosmos test cases.
Print the container exit/OOM state and native emulator log before cleanup so CI distinguishes provider contention from backend process failures. Keep the same Strong backend and test matrix.
Honor the single-owner periodic heartbeat contract with one no-content PatchItemStreamAsync setting only /IAmAliveTime. Remove heartbeat reads, etag/filter conditions, history probes, retry loops, and retired-row success conversion. Preserve native cancellation and visible failures, Strong membership reads, and full-row write protection.
Replace obsolete multi-writer heartbeat-max and retired-noop expectations with exact zero-read/one-patch, timestamp-path/value, native response failure, transport failure, and cancellation regressions.
The direct single-owner heartbeat requirement supersedes my earlier response to #11312 (review). Commit dc19e0e removes the heartbeat retry loop entirely: one PatchItemStreamAsync sets only /IAmAliveTime, with no read, etag/filter condition, history probe, or application retry. The SDK request disables write-response content. Native cancellation and storage errors, including missing rows, propagate. Strong consistency remains on membership reads and full-row membership writes retain their conditional protection.
HeartbeatUsesOneBlindTimestampPatch asserts exactly one SDK call, the exact timestamp path/value, no conditions, and no response body. NativeHeartbeatPatchCancellationRetainsToken and HeartbeatPatchFailuresRemainVisibleWithoutAdditionalRequests cover native cancellation and 404/412/403/503 outcomes. All 64 focused mocked cases pass locally. Current-head native CI is still required; the classic backend has already passed actual Strong metadata and basic transactional-etag tests on the prior head, but parallel membership/backend stability diagnostics remain under investigation.
Audited final head e177034a5b8e99b3ccd046a4e419c05868b0914d against main/base d515d75eaaa3a0390a74cb11c96aa758358b5a67, including every membership operation, resource/client initialization, and unchanged gateway discovery.
Concrete reductions
CosmosMembershipTable.ReadMembershipSnapshot: reuse one SDK iterator per attempt, rather than reconstructing it for each continuation page. Explicit session-token propagation was already removed when Strong reads were selected. The native iterator carries continuation state and is drained through empty pages using HasMoreResults; immutable-ID order and both version fences remain.
InsertRowAsync / UpdateRowAsync: retain one transactional request with two item operations, while suppressing both unused returned document bodies. The table CAS, existing-row replacement, and heartbeat-independent logical tokens are unchanged. Request serialization still contains the two required input documents; the reduction is unused response serialization/payload, not a claim that two typed deserializations previously occurred.
TryCreateCosmosResources: use one CreateContainerIfNotExistsAsync invocation rather than up to four calls and three one-second delays after a successful 200. The SDK reads the container, creates it when missing, and rereads after a concurrent-create 409. Both 200 and 201 are successful resource completion; native failures remain visible.
Operation-cost comparison
Counts below describe provider-issued data operations. SDK metadata/query-plan traffic and automatic transport retries are excluded; these are not latency or RU measurements. P is the number of query pages, N the selected row count, and D the eligible Dead candidates. Constructing a container/batch/iterator object is not counted as a data request.
Path
Main
Final head
Why retained
Initialize
One version read, optional one version create; optional resource creation
Same version operations; one database-CIFNE and one container-CIFNE invocation
Native CIFNE performs its own existence read/optional create/race reread; shared factory task retains ownership across canceled waits
ReadRow
Two parallel point reads
Three ordered Strong point reads per attempt, at most five attempts
Before/member/after etags prove the canonical row/version view, including absence
ReadAll
P pages plus one parallel version read
P pages plus two ordered version reads per attempt, at most five attempts
Strong applies per operation; separate pages are not one transactional snapshot; one iterator now serves all P pages
Insert / Update
One batch, two item operations
One batch, two item operations, zero returned document bodies
Same-partition ACID transaction combines canonical version CAS with create/existing-row replacement; stale logical-token disagreement rejects locally with zero requests
Heartbeat
One cached conditional full-row replace, plus a read when cache is empty
One blind no-content timestamp patch; zero reads/provider retries
Single-owner unversioned liveness publication; full-row writes accept the supplied liveness value
Defunct cleanup
Non-Active query pages plus one unchecked, unconditional batch when candidates exist; at most 100 operations
Dead-only pages plus D conditional point deletes; one history read only for an item404
Captured physical etags guard eligibility; 412 retains a changed candidate, real failures propagate; no rescan/version write
Administrative delete
P pages plus one unconditional N+1-item batch, valid only through N=99
Stable P-page snapshot plus two version reads, ceil(N/100) row batches, and one conditional version delete
Supports the native batch limit and the reviewed changed-row/recreated-version race; earlier chunks remain committed on later failure
Gateway discovery
P gateway query pages
Unchanged
Refresh/staleness contract returns endpoints rather than a canonical versioned snapshot
Defunct cleanup intentionally retains independent deletes: this costs D requests but isolates concurrent-candidate failures without a new batch rollback/reselection protocol. It is a cold maintenance path, not the heartbeat/canonical-write path. Physical etags remain private to conditional deletion; canonical membership tokens remain independent of heartbeat document etags.
ReadRow atomic-batch alternative
A same-partition Strong TransactionalBatch.ReadItem(version).ReadItem(member) is a valid one-request/two-operation atomic pair when both items exist. It was evaluated, not ruled out merely because separate Strong reads need fencing. On a missing member, the batch aborts and returns 424 for the version read and 404 for the member; the contract does not promise a usable version resource on the failed dependency. The SDK emulator tests explicitly assert those outcomes (BatchSinglePartitionKeyTests.cs, BatchErrorSessionToken*, existing-plus-missing reads).
A version-only fallback is unsound: a missing row at V7 can be inserted at V8 before that fallback returns V8, incorrectly pairing absence with a version where the row exists. Correct recovery adds a fenced absence/retry path. The uniform three-read implementation is retained rather than introducing a branchy successful-read fast path plus four-request missing-row fallback.
Native guarantees and unchanged boundaries
Cosmos transactional batches are ACID/snapshot-isolated within one logical partition and report the causal operation failure with dependent 424s. This justifies atomic canonical updates and prevents classifying an independent authorization failure as a competing batch conflict. Strong reads remain explicit. Immutable-ID ordering prevents unversioned heartbeat changes moving rows across the continuation order. Dead-only pruning without a version increment remains permitted by #11296. Cancellation and infrastructure failures propagate; there is no heartbeat/history fallback, marker schema, sidecar, account-metadata startup policy, or newly introduced legacy mode. The existing tryInitTableVersion behavior and obsolete forwarding methods are unchanged from main.
Evidence and review disposition
ReadAllUsesStrongConsistencyAcrossEmptyPagesInImmutableIdOrder now asserts one iterator/three page fetches/two version reads. MembershipWritesUseAtomicCanonicalVersionConditions asserts two native batch items with no returned resource bodies and unchanged conditions. ResourceInitializationUsesOneNativeCreateCall covers database201 with container200 and container201. Original-token-after-heartbeat, canonical conflicts, missing rows, cleanup, and cancellation cases remain covered.
Focused net10 build/format and all64 mocked cases pass. Same-head native Windows Strong Cosmos jobs pass on net8.0 and net10.0: each213 passed,0failed,2pre-existing persistence-format skips, with all11 native membership cases executed. Main CI35374074919, CodeQL35374074819, Documentation35374074710 and static audit35374074955 succeeded:71 successful checks and only the expected Pages skip. All three inline threads are resolved; the resource-creation and body-only review notes have concrete guarantee-based dispositions. Readiness is unchanged; no merge or auto-merge action was taken.
Critical-only API scan checklist: culture-sensitive IndexOf/StartsWith/EndsWith/Contains recipes0; Substring0; sync-over-async, stackalloc/buffer, regex, per-call HttpClient/JsonOptions hazards0. One factory ValueTask is converted once to a retained Task. Four LINQ call sites and two unsealed internal classes were reviewed without speculative micro-optimization.
Severity
Count
Outcome
Critical
0
No critical API-pattern finding in the bounded scan
This is an AI-assisted static/native-contract audit, not a latency or RU benchmark. Native regressions verify behavior; performance estimates still require workload measurement and human review.
Add an explicit coverage selection to the shared test action while preserving Linux/net10 as the automatic default. Opt the Windows Cosmos net10 partition into setup, initial collection, retry collection, and metadata/report upload. Keep the required coverage artifact matrix and trusted-input validation intact.
Extend the existing coverage workflow regression checks for all four override boundaries, the Windows framework selection, and the required Cosmos artifact identity.
Coverage collection is restored in 132900d21bfacad0b394d525da4780397bd0a56f.
The shared test action now accepts an explicit coverage selection; its default remains Linux/net10.0. The Windows Cosmos job enables coverage only for net10.0. Setup, initial collection, retry collection and report/metadata upload share the same selection.
The required 23-partition artifact manifest and trusted-input checks are unchanged. All54 local coverage-pipeline regression cases passed, including the four override boundaries.
Actual current-head Windows Cosmos/net10 job https://github.com/dotnet/orleans/actions/runs/35386554870/job/105734723542 passed213 tests,0failed,2existing persistence skips; it installed dotnet-coverage18.11.2, collected a27,860,213-byte Cobertura report, and uploaded coverage_test_output_test-azure-cosmosdb_net10.0-attempt-1 (artifact10564577242) with the required metadata. There are executed lines for Orleans.Clustering.Cosmos in this real report.
All23 artifacts validate with a single tested merge commit 1de2805b673c488bf8c0a042df00dd4c55630f9a, whose parents include this PR head. The checked-in coverage reader successfully normalized and aggregated the Linux and Windows reports against that exact merge's sources. Windows entries use canonical /_/src/... paths.
Collection success and publication acceptance are distinct: the trusted reporting run https://github.com/dotnet/orleans/actions/runs/35388415319 validated the23-artifact matrix and tested commit, then stopped at Verify trusted coverage inputs because this PR changes reviewed collection infrastructure. That gate remains intact. The infrastructure changes require human-reviewed landing on the trusted branch before an updated aggregate comparison can be published under the current policy; the old ff9967ce86 coverage comment is not current-head evidence.
Test/build CI is green on132900d21b; the separate coverage publication workflow is blocked as described. Ready state is unchanged and no merge or trust bypass was performed. The latest review's body-only tryInitTableVersion observation concerns unchanged baseline behavior already discussed in issuecomment5733910640; this coverage-only change does not alter initialization.
Use the client/account read consistency without per-request overrides. Keep version fencing, canonical batch predicates and blind heartbeats intact, and document effective Strong consistency as the production requirement. Adapt request-shape tests and remove the test-backend-specific Strong metadata requirement.
Exercise the full native Cosmos suite against pinned Linux vNext EN20260907 on both frameworks, retaining all transactional-etag cases and net10 coverage collection. This tests functional emulator compatibility rather than proving distributed Strong semantics.
Emit actual account-default consistency and client connection mode in the existing native initialization case, and assert the test client leaves its consistency override unset. Keep configuration observation separate from assertions of distributed Strong semantics.
Always inspect whether the named emulator container was created before capturing state/logs and removing it. Preserve genuine Docker failures and the original startup failure while tolerating a missing container, including partial docker run failures.
Actual Init output on both frameworks: Account default consistency: Eventual; client consistency override: inherited; connection mode: Gateway. Provider request-shape tests verify point requests have no ItemRequestOptions consistency override and queries leave ConsistencyLevel unset. This establishes functional compatibility with this pinned single-instance emulator, not distributed Strong-consistency guarantees. Production still requires effective Strong client/account configuration as documented.
Coverage remains complete: all23 expected artifacts validate for tested merge dd2f846, which has this PR head as a parent. The Cosmos Cobertura/metadata artifact10566573712 is present and includes executed provider lines. The separate trusted reporter35395544927 still stops at Verify trusted coverage inputs, preserving the previously documented human-reviewed infrastructure landing gate; the historical coverage comparison comment is not current-head evidence.
All76 current-head PR checks succeeded with only the expected Pages deployment skip; all4inline threads are resolved. The PR remains ready, with no merge or auto-merge action.
Restore the shared test action and coverage regression tests to the existing Linux/net10 policy. Remove the Windows-era Cosmos coverage selection and unused start-step ID now that native Cosmos CI runs on Linux. Keep the verified emulator pin, readiness and cleanup fixes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Cosmos membership reads can combine a table version with rows from a different membership update. Heartbeat document etags can invalidate membership writes, cleanup can delete a row which changed after selection, and failed transactional responses can be reported as ordinary contention.
Solution and rationale
/IAmAliveTime. This path performs zero reads and one unconditional column write; native cancellation and storage failures, including missing rows, remain visible. Full-row writes may overwrite the unversioned heartbeat timestamp, as permitted by the membership contract.Configuration compatibility
Production membership uses a Cosmos account with a single writable region and Strong consistency. Configure clients to inherit the account default or use Strong consistency. Read requests inherit that effective configuration, as requested during review; the provider performs no account-policy validation. Clients selecting weaker consistency are outside the documented coherent-read guarantee.
Document schemas, partition keys, IDs, serialization contracts, and public signatures remain unchanged. Logical row tokens remain opaque strings and track canonical membership independently of heartbeat-modified document etags.
Native CI backend and coverage
The net8.0/net10.0 Cosmos jobs exercise the full native provider suite on Ubuntu against pinned vNext Linux emulator
vnext-EN20260907@sha256:2db1f9e74c506bcf6fc347aa937aea1c00fa756061296a5a9efba530ce86ec02, using its documented readiness endpoint. The three transactional-etag membership tests formerly skipped on the preview emulator remain enabled. A native regression verifies a canonical update can use original row/table tokens after an owner heartbeat, and rejects subsequent stale canonical tokens.The full suite passes with the emulator's inherited Eventual default; this establishes functional compatibility, not distributed Strong consistency. Production configuration requirements remain documented. Classic Linux emulator crashes remain tracked separately in #11323.
The existing Linux/net10.0 coverage policy automatically includes Cosmos. Shared coverage actions, coverage regression scripts, and the required artifact matrix are unchanged from the PR baseline. Trusted coverage-publication input checks remain intact; the remaining emulator workflow changes are subject to that review gate.
Extraction provenance and scope
Adapted the Cosmos portion of
ReubenBond/orleans:rb-test-membership-provider-conformanceat1dfe8b54cda3582ddf3795113eaa2f4e9d1b6254, originally based onad71848f8495b9b87c7891f4ddc0c1d8759d21d6, onto maind515d75eaaa3a0390a74cb11c96aa758358b5a67(the #11296 merge).Separate prerequisite
c9eaf2781f58c3c8490ff24197e1fa87f4a695fecarries only shared legacy cleanup-test preparation frombede893d290a9035f196f4c3ed58213d102efc6c, also used by #11308. Initial provider extraction9e922275130d18241529bb839ec05af688daad24is followed by review changes for administrative deletion, configured read consistency, a single-owner blind heartbeat, heartbeat-independent canonical tokens, and reduced redundant storage work.Omitted source requirements include account-metadata startup validation, versioned cleanup transactions/reserved slots, multi-writer heartbeat maxima, retired-row heartbeat no-ops, and full-row heartbeat preservation. The full conformance kit/adapters, project/solution wiring, core/other-provider changes, and broad documentation remain separate. There is no new schema or sidecar.
Microsoft Reviewers: Open in CodeFlow