Change Feed Processor: Fixes duplicate lease docs by using id as partition-key value

[NaluTripician]

Change Feed Processor: Fixes duplicate lease docs by using id as partition-key value

Closes IcM 768856224.

---

The bug

What the customer saw

The customer's Change Feed Processor (CFP) silently stopped processing changes. Restarting hosts did not help. Their lease container contained documents like:

id	LeaseToken	partitionKey	Owner	_ts
MyProcessorhost1_abc_xyz..0	0	d4e2… (Guid)	null	frozen
MyProcessorhost1_abc_xyz..0	0	7a9f… (different Guid)	null	frozen

Every lease had a duplicate entry: same id, same LeaseToken, different Guid partitionKey, Owner: null, timestamps frozen at the moment the processor died.

Root cause

The CFP lease container was partitioned by /partitionKey. On a partition split, PartitionSynchronizerCore.HandlePartitionGoneAsync delegates child-lease creation to DocumentServiceLeaseManagerCosmos.CreateLeaseIfNotExistAsync, which generated the partition-key value with Guid.NewGuid().ToString():

// DocumentServiceLeaseManagerCosmos.cs (before)
this.requestOptionsFactory.AddPartitionKeyIfNeeded(
    (string pk) => documentServiceLease.LeasePartitionKey = pk,
    Guid.NewGuid().ToString());

TryCreateItemAsync relies on Cosmos's per-partition-key id uniqueness check to turn concurrent creates into a 409 Conflict. That check only catches duplicates with the same partition key. Every retry of split handling — host restart mid-split, transient error retry, or two hosts racing on the same parent lease — rolled a new random Guid, bypassing the uniqueness check and silently persisting a second document with identical id/LeaseToken but a different partitionKey.

Once duplicates exist, every balance tick (~13 s) lands here in EqualPartitionsBalancingStrategy.CategorizeLeases:

allPartitions.Add(lease.CurrentLeaseToken, lease); // ArgumentException on duplicate

The ArgumentException propagates out of CalculateLeasesToTake, the balancer catches it, logs, and returns zero leases — every tick, forever. No host ever claims a lease; Owner stays null; the feed permanently stalls. There is no automatic recovery: the duplicate documents must be deleted manually.

---

The fix

Set the partition-key value on each new lease document to the lease's own deterministic id (already computed as this.GetDocumentId(leaseToken)), in both overloads of DocumentServiceLeaseManagerCosmos.CreateLeaseIfNotExistAsync:

// DocumentServiceLeaseManagerCosmos.cs (after)
// Use the lease document id as the partition-key value so that retries / concurrent
// creates for the same lease resolve to the same (id, partitionKey) tuple. This lets the
// Cosmos per-partition-key id-uniqueness check turn duplicates into a 409 Conflict
// instead of silently persisting cross-partition-key duplicates.
this.requestOptionsFactory.AddPartitionKeyIfNeeded(
    (string pk) => documentServiceLease.LeasePartitionKey = pk,
    leaseDocId);

With this change, concurrent or retried creates of the same lease resolve to the same (id, partitionKey) tuple, so Cosmos's per-partition-key id-uniqueness check fires normally and returns 409 Conflict. TryCreateItemAsync turns that into a benign "already exists" outcome — the duplicate is prevented at the source. This is the pattern already used by DocumentServiceLeaseStoreCosmos for its marker and lock documents (which use markerDocId / lockId as both id and pk).

Backward compatibility

Fully compatible with existing lease containers. The read path (TryGetLeaseAsync, ReleaseAsync, UpdateLeaseAsync, CheckpointAsync, DeleteAsync) pulls the partition-key value off the deserialized lease document via requestOptionsFactory.GetPartitionKey(lease.Id, lease.PartitionKey), which for /partitionKey-partitioned containers returns new PartitionKey(partitionKey) — i.e. whatever was stored. Pre-existing lease documents that have Guid-based partitionKey values continue to load, refresh, and be released normally. New lease documents created after this change are written with partitionKey == id; once all parent leases have been split / replaced, the container naturally converges to the new scheme with no manual migration.

Affected containers

Only /partitionKey-partitioned lease containers were exposed to the bug. Lease containers partitioned by /id or on single-partition (fixed) collections never exercised the affected code path (AddPartitionKeyIfNeeded is a no-op on those request-options factories — see PartitionedByIdCollectionRequestOptionsFactory and SinglePartitionRequestOptionsFactory) and require no action.

---

Testing

Unit tests

Extended the existing ValidateRequestOptionsFactory helper used by DocumentServiceLeaseManagerCosmosTests.CreatesEPKBasedLease and CreatesPartitionKeyBasedLease (both [DataTestMethod] with rows for each of the three RequestOptionsFactory implementations). For the PartitionedByPartitionKeyCollectionRequestOptionsFactory case the helper now asserts lease.PartitionKey == lease.Id, proving the new deterministic wiring for both the PK-range overload and the EPK overload.

Passed!  - Failed: 0, Passed: 15, Skipped: 0, Total: 15

Full ChangeFeed unit-test slice: 261 / 261 pass.

Manual verification

• dotnet build Microsoft.Azure.Cosmos.sln -c Debug — 0 warnings, 0 errors.
• ChangeFeed unit-test slice passes locally.

---

Performance impact

None. The change replaces one Guid.NewGuid().ToString() call with a variable reference (leaseDocId, already computed one line earlier in the same method). No extra IO.

---

Customer remediation (for customers already in this state)

Delete the duplicate lease documents (same id, different partitionKey) from the lease container, then restart the processors. A one-time query is sufficient:

SELECT c.id, c.partitionKey, c.LeaseToken FROM c

Group by id; for each group with more than one document, keep one and delete the rest (using the matching partitionKey as the partition-key value on the delete).

Once this fix rolls out, new child leases created during a split/merge are immune to the duplicate-creation race — concurrent or retried creates for the same lease collide on (id, partitionKey) and Cosmos enforces uniqueness.

---

Type of change

• Bug fix (non-breaking)
• New feature
• Breaking change
• Documentation

Checklist

• My code follows the code style of this project
• I have added tests to cover my changes (extended ValidateRequestOptionsFactory to assert deterministic pk == id on /partitionKey-partitioned containers)
• All new and existing tests pass locally (261 / 261 ChangeFeed unit tests)
• I have verified the fix resolves the customer incident scenario (IcM 768856224)

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[xinlian12]

@sdkReviewAgent

[xinlian12]

✅ Review complete (38:33)

Posted 3 inline comment(s).

_{Steps: ✓ context, correctness, cross-sdk, design, history, past-prs, synthesis, test-coverage}

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[kirankumarkolli]

Duplicate #5807