fix(azcosmos): GetChangeFeed split-resilient via overlap match, 410/Gone retry, multi-range continuation queue (#26767)

[tvaron3]

Closes #26767.

Purpose

ContainerClient.GetChangeFeed does not survive partition splits today. Three independent failure modes:

1. Silent skip when the customer's FeedRange straddles new children. The legacy header builder exact-matched against the cached PK ranges via findPartitionKeyRangeID. After a split, the parent range no longer exactly equals any current child, so the request was issued without x-ms-documentdb-partitionkeyrangeid.
2. Unrecoverable 410/Gone surfaces to the caller. GetChangeFeed propagated *azcore.ResponseError directly. The partitionKeyRangeCache.forceRefresh and maxPKRangeGoneRetries plumbing added in [Cosmos] add container cache and pk range cache #26723 was never wired into the change feed.
3. Single-range continuation token. PopulateCompositeContinuationToken wrote one changeFeedRange. If the original FeedRange covered a parent that has since split, resume routed against a no-longer-physical EPK boundary instead of fanning out — events between calls were lost at the boundary.

There was also no cross-container guard: a token's ResourceID was never checked against the current container's _rid, so a wrong-container token would silently misroute against the wrong routing map.

What this change does

Wires GetChangeFeed to a queue-driven drain loop that mirrors the Java/.NET SDK behavior:

• Overlap-match resolution. New overlappingPartitionKeyRanges + rangesOverlap helpers in cosmos_feed_range.go replace exact-match for routing decisions. Empty MaxExclusive is normalized to "FF".
• Multi-range composite continuation token. New head/advance/replaceHeadWithChildren/dropHeadContinuation queue helpers on compositeContinuationToken. The composite token now describes the full set of unfinished sub-ranges; each entry is rotated FIFO with its own ETag.
• Split expansion. When a single inbound range overlaps N children (post-split), the head is replaced with N entries, each inheriting the parent's ETag so no events are skipped at the boundary. If a queued sub-range splits mid-drain, the rotation budget is bumped so newly-inserted children are queried in the same call. The 410/Gone budget is reset on both split-expansion and head-advance so each newly-inserted physical head gets its own retry allowance.
• 410/Gone retry with cache refresh. Bounded at maxPKRangeGoneRetries. The PK-range snapshot is fetched once per call (through the cache when present, falling back to a direct fetch) and reused across iterations; the 410 path re-fetches.
• Cross-container ResourceID guard. A continuation token whose ResourceID differs from the current container's _rid is rejected loudly. The token's ResourceID is populated from the response body's _rid on first success so the guard remains meaningful across resume.
• ErrFeedRangeUnresolved is returned (wrapped) when a customer-supplied FeedRange/token doesn't overlap any current physical range even after a forced refresh — a signal callers can errors.Is against and recover from rather than seeing an opaque error.
• Pure buildRequestHeaders builder. New (head changeFeedRange, resolvedPKRangeID string) (map[string]string, error) builder. Returns an error when PartitionKey serialization fails so the caller can surface the cause instead of silently dropping the request (the original-bug pattern). The drain loop calls this with a pre-resolved head; the loop owns overlap-resolution upstream.
• Cache-nil degraded fallback. When the PK-range cache is unavailable AND the direct fetch returns no ranges (test scaffolding only — production always wires up caches via acquireCaches), the loop logs a warning via log.Writef(azlog.EventResponse, ...), drops the head's continuation token, and reads fresh with a passthrough range so the request goes out without an x-ms-documentdb-partitionkeyrangeid header. Strict feedRangeUnresolvedError is preserved for genuine "FeedRange doesn't apply" cases.

The legacy ChangeFeedOptions.toHeaders and findPartitionKeyRangeID (the original-bug source) have been deleted. They were unexported and had no production callers after the rewrite. Coverage of the field-to-header mapping is preserved via cosmos_change_feed_request_options_test.go against the new buildRequestHeaders.

File organization

• cosmos_container.go — public GetChangeFeed entrypoint stays here so the canonical container file continues to host the public API surface.
• cosmos_container_change_feed.go (new) — buildChangeFeedInitialQueue, resolveFeedRangeToChildren, buildChildQueueEntries, getChangeFeedForQueue (drain loop), serializeCompositeContinuationToken. Mirrors the cosmos_container_read_many.go split.
• cosmos_change_feed_composite_continuation_token.go — queue-mutation helpers.
• cosmos_change_feed_request_options.go — buildRequestHeaders (legacy toHeaders deleted).
• cosmos_change_feed_response.go — defer resp.Body.Close() ordering fix (above the 304 short-circuit).
• cosmos_feed_range.go — overlappingPartitionKeyRanges, rangesOverlap, normalizeMaxBoundary.

Tests

Mock-based unit tests — cosmos_change_feed_split_test.go:

• TestGetChangeFeed_410Gone_TriggersCacheRefreshAndRetry — 410 + cache-refresh + retry.
• TestGetChangeFeed_RetryCapAt3_ReturnsLastErrorOnRepeated410 — surface final 410 after cap.
• TestGetChangeFeed_TokenResourceIDMismatch_Rejected — cross-container guard.
• TestGetChangeFeed_NoOverlapAfterRefresh_ReturnsErrFeedRangeUnresolved — loud-fail on unresolvable FeedRange.
• TestGetChangeFeed_SplitExpansion_RoutesToFirstChild — child routing + persisted multi-range token.
• TestGetChangeFeed_MultiRangeContinuation_RoundTrips — resume from a multi-element queue.
• TestGetChangeFeed_SplitDuringDrain_QueriesEveryNewlyInsertedChild — regression guard for the rotation-budget bump.

Header-builder unit tests — cosmos_change_feed_request_options_test.go:

• _BuildRequestHeaders_Defaults — empty options yields only the AIM header.
• _BuildRequestHeaders_AllFields — every field maps to the right header.
• _BuildRequestHeaders_EmptyContinuationOmitsIfNoneMatch — explicit empty ETag does not emit a stray IfNoneMatch.
• _BuildRequestHeaders_PartitionKeySerializationError — verifies the new error-returning contract.

Emulator integration tests — emulator_cosmos_change_feed_split_test.go (skipped when EMULATOR is unset):

• _F1_SubRangeOfPhysicalRange — direct repro of the original bug (strict sub-range FeedRange).
• _F1_WideRangeMultiDrain — paginate via composite token until all items across all physical ranges are drained.
• _F1_CrossContainerTokenRejected — token from container A handed to container B is rejected.

The 3 pre-existing change-feed tests (TestContainerGetChangeFeedWithStartFrom, TestContainerGetChangeFeedWithStartFromFiltering, TestContainerGetChangeFeedForEPKRange) continue to pass.

go test -race -short -count=1 ./sdk/data/azcosmos/... is green. CI lint is green.

Three deep-reviewer rounds completed; the latest pass found no critical or should-fix issues.

Notes for reviewers

• Token forward-compat: tokens issued by the old single-range writer still resume correctly (one queue entry → one head → drain loop runs once).
• Token grows linearly with split count. A FeedRange that has straddled K splits will produce K queue entries in the persisted token. Acceptable for typical workloads; bounded by the physical-range fan-out.
• response.FeedRange after split: when a customer FeedRange straddles a split, ChangeFeedResponse.FeedRange reports the head being drained for that page (not the original parent). Callers that previously relied on it round-tripping should use the persisted continuation token instead — noted in CHANGELOG.
• Symbols used from [Cosmos] add container cache and pk range cache #26723: partitionKeyRangeCache, forceRefresh, maxPKRangeGoneRetries, isPKRangeGoneResponseError, ContainerClient.refreshPKRangeCache, ContainerClient.getContainerRID.

Checklist

• The purpose of this PR is explained in this or a referenced issue ([Cosmos] azcosmos: GetChangeFeed silently loses data on partition split — no overlap-match, no 410/Gone retry, single-range continuation #26767).
• The PR does not update generated files.
• Tests are included and/or updated for code changes.
• Updates to module CHANGELOG.md are included.
• MIT license headers are included in each file.

[Copilot]

Pull request overview

This PR updates sdk/data/azcosmos change feed paging/routing to be resilient to partition splits by resolving FeedRanges via overlap (instead of exact boundary match), expanding split parents into per-child work items, and retrying 410/Gone PK-range errors via cache refresh.

Changes:

• Reworked ContainerClient.GetChangeFeed into a queue-driven drain loop that overlap-resolves ranges, expands splits, and performs bounded 410/Gone retries with PK-range cache refresh.
• Added new helpers for overlap detection (overlappingPartitionKeyRanges, rangesOverlap, max-boundary normalization) and composite continuation queue operations (head/advance/replaceHeadWithChildren).
• Added extensive unit tests for split/retry/token behavior plus new emulator coverage; updated request-header building tests and changelog.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
sdk/data/azcosmos/emulator_cosmos_change_feed_split_test.go	Adds emulator tests validating overlap routing, multi-drain continuation, and cross-container token rejection.
sdk/data/azcosmos/cosmos_feed_range.go	Introduces overlap-based PK range resolution helpers and ErrFeedRangeUnresolved.
sdk/data/azcosmos/cosmos_container.go	Routes GetChangeFeed through the new queue-based implementation.
sdk/data/azcosmos/cosmos_container_test.go	Removes a now-obsolete continuation/header assertion tied to the old toHeaders behavior.
sdk/data/azcosmos/cosmos_container_change_feed.go	New change-feed drain loop implementation (queue, split expansion, 410 retry, token guard).
sdk/data/azcosmos/cosmos_change_feed_split_test.go	New unit tests covering 410 retry, retry cap, split expansion, multi-range resume, and rotation budget bump.
sdk/data/azcosmos/cosmos_change_feed_response.go	Ensures response bodies are closed even on 304 to avoid leaks during drain.
sdk/data/azcosmos/cosmos_change_feed_request_options.go	Replaces legacy header builder with buildRequestHeaders for queue-head-based requests.
sdk/data/azcosmos/cosmos_change_feed_request_options_test.go	Updates/expands tests to validate buildRequestHeaders behavior and error surfacing.
sdk/data/azcosmos/cosmos_change_feed_composite_continuation_token.go	Adds composite continuation token queue helpers (head/advance/replaceHeadWithChildren).
sdk/data/azcosmos/CHANGELOG.md	Documents the split-resilient GetChangeFeed fix.

[kushagraThapar]

Hi @tvaron3, thank you for putting this together — I really enjoyed reading through the queue-driven drain loop and the way the 410/Gone retry budget resets across split-expansion. The three deep-reviewer rounds clearly show in the structural quality.

A few items below for your consideration. None of them are hard blockers IMO; the heaviest one is M-2 (cross-container _rid guard behavior on first-call 304) which I think is worth addressing in this PR. The rest are smaller correctness, testing, or documentation gaps.

A couple of items I'm planning to file as follow-up issues rather than block this PR:

• The ChangeFeedWithStartTimePostMerge capability bit (pre-existing gap, not introduced here, but the new "split-resilient" framing makes it more customer-visible after a merge).
• Two design-doc extension proposals for cosmosdb-design-docs (continuation-token container-identity invariant, and the FeedRange-becomes-unresolvable contract).

All comments are in the spirit of "could you please help me understand" rather than "must change" — happy to defer any of them if you have context I'm missing.

[xinlian12]

@sdkReviewAgent

[xinlian12]

✅ Review complete (50:22)

Posted 3 inline comment(s).

_{Steps: ✓ context, correctness, cross-sdk, design, history, past-prs, synthesis, test-coverage}

[kushagraThapar]

LGTM, thanks @tvaron3

[FabianMeiswinkel]

LGTM