Backport GetPreExistingChunksIdsAsync pagination fix

[Copilot]

Backport GetPreExistingChunksIdsAsync fix to data-ingestion-preview2 branch ✅

Changes Completed:

• Add Skip parameter to GetAsync call in GetPreExistingChunksIdsAsync method to fix infinite loop bug
• Add test IncrementalIngestion_WithManyRecords_DeletesAllPreExistingChunks to verify pagination works correctly
• Update test to use TestChunkFactory.CreateChunk for proper token counting instead of hardcoded values
• Build and test the changes - all tests passing
• Code review completed - no issues found
• Security scan completed - no vulnerabilities detected

Summary:

Successfully backported the fix from PR #7311 to the data-ingestion-preview2 branch. The fix resolves an infinite loop bug where GetPreExistingChunksIdsAsync would repeatedly fetch the same records when a document had more than 1,000 pre-existing chunks.

Technical Details:

Bug Fix: Added options: new() { Skip = keys.Count } to the GetAsync call to properly paginate through results.

Test Implementation: Uses TestChunkFactory.CreateChunk to create chunks with properly calculated token counts using the tokenizer, rather than hardcoded values.

Test Results:

✅ All 14 VectorStoreWriter tests passed on net8.0, net9.0, and net10.0
✅ New test IncrementalIngestion_WithManyRecords_DeletesAllPreExistingChunks verifies pagination with 2500 chunks
✅ Code review found no issues
✅ Security scan found no vulnerabilities

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Backport GetPreExistingChunksIdsAsync fix to data-ingestion-preview2 branch</issue_title>
<issue_description>Backport the fix from #7311 into data-ingestion-preview2 branch

This is going to require updating the test, as IngestionChunk was extended with TokenCount property recently (#7264)</issue_description>

Comments on the Issue (you are @copilot in this section)

• Fixes Backport GetPreExistingChunksIdsAsync fix to data-ingestion-preview2 branch #7312

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Microsoft Reviewers: Open in CodeFlow

[adamsitnik]

@copilot please address my feedback

[Copilot]

@copilot please address my feedback

Addressed in 2e592d7. Updated both locations to use TestChunkFactory.CreateChunk instead of hardcoded token counts.