[Pull-based ingestion] Use writer tracked shard pointer for recovery

[varunbharadwaj]

Description

This PR fixes a bug in the pull-based ingestion flow where some messages can be skipped in some scenarios on shard recovery. The fix is to track the shard pointer on the writer thread and use that as the start point for recovery.

Additionally, this PR also avoids failing entire batch if one record fails and lets the error policy decide when to pause ingestion.

Related Issues

Resolves #17853

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for c4bdc7d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 9e0b808: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 9e0b808: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 9e0b808: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 00cf587: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 00cf587: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 677b3d9: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 85.71429% with 5 lines in your changes missing coverage. Please review.

Project coverage is 72.43%. Comparing base (1628152) to head (677b3d9).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
...rch/indices/pollingingest/DefaultStreamPoller.java	84.84%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17868      +/-   ##
============================================
- Coverage     72.43%   72.43%   -0.01%     
- Complexity    66789    66815      +26     
============================================
  Files          5449     5452       +3     
  Lines        309085   309198     +113     
  Branches      44979    44988       +9     
============================================
+ Hits         223899   223957      +58     
- Misses        66906    66912       +6     
- Partials      18280    18329      +49     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.