[Pull-based Ingestion] Support all-active mode in pull-based ingestion

[varunbharadwaj]

Description

Pull-based ingestion today only supports segment replication. This PR adds an all-active mode where the replica also indexes the documents. This is done by having the replica shards ingest from the streaming source independently and process the messages when the all-active ingestion mode is enabled. The replicas will continue to use the peer recovery mode during shard reinitialization. Note that there is no replication or coordination between the primary and replicas in this mode, as each of them will ingest and index independently.

This mode can be expected to support better throughput in comparison to traditional doc-rep, and also provide better freshness as there is no replication delays.

The GetIngestionState API is enhanced to include primary and node name in the response to differentiate in the response.

Related Issues

Resolves #19287

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 202c295: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 99993fd: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for c613264: SUCCESS

[codecov]

Codecov Report

❌ Patch coverage is 70.96774% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.86%. Comparing base (1f9eb87) to head (9b0bdf8).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
.../streamingingestion/state/ShardIngestionState.java	81.08%	5 Missing and 2 partials ⚠️
.../indices/pollingingest/IngestionEngineFactory.java	16.66%	3 Missing and 2 partials ⚠️
...estion/state/TransportGetIngestionStateAction.java	73.33%	1 Missing and 3 partials ⚠️
...ion/state/TransportUpdateIngestionStateAction.java	63.63%	1 Missing and 3 partials ⚠️
...org/opensearch/cluster/metadata/IndexMetadata.java	69.23%	3 Missing and 1 partial ⚠️
...g/opensearch/cluster/metadata/IngestionSource.java	80.00%	0 Missing and 2 partials ⚠️
...mingingestion/state/GetIngestionStateResponse.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #19316      +/-   ##
============================================
- Coverage     72.89%   72.86%   -0.04%     
+ Complexity    69870    69809      -61     
============================================
  Files          5673     5674       +1     
  Lines        320754   320831      +77     
  Branches      46367    46383      +16     
============================================
- Hits         233824   233758      -66     
- Misses        68012    68093      +81     
- Partials      18918    18980      +62     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

❌ Gradle check result for 6152478: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 6152478: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 9b0bdf8: SUCCESS

[yupeng9]

LGTM