[Pull-based Ingestion] Add time based periodic flush support

[varunbharadwaj]

Description

Add a time based periodic flush task similar to the periodic refresh task at the shard level. This will default to 30 minutes for pull-based ingestion index. For regular indices, this will be disabled by default.

This will help ensure the batchStartPointer is persisted in the commit periodically, and prevent a large lag build up on replicas when promoted. It also ensures the batchStartPointer is not stale and older than the streaming source retention period given that there are incoming events.

Related Issues

Resolves #19860

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 6a36a46: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6a36a46: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 7b55d32: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.19%. Comparing base (46164b5) to head (b5073cc).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...in/java/org/opensearch/index/shard/IndexShard.java	78.26%	0 Missing and 5 partials ⚠️
...a/org/opensearch/index/engine/IngestionEngine.java	84.61%	1 Missing and 1 partial ⚠️
.../main/java/org/opensearch/index/IndexSettings.java	91.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19878      +/-   ##
============================================
- Coverage     73.21%   73.19%   -0.03%     
+ Complexity    71254    71248       -6     
============================================
  Files          5766     5766              
  Lines        325470   325514      +44     
  Branches      47084    47094      +10     
============================================
- Hits         238296   238257      -39     
- Misses        68043    68157     +114     
+ Partials      19131    19100      -31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Gradle check result for f531f16: SUCCESS

[github-actions]

❌ Gradle check result for 9abade3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for b5073cc: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.