Skip to content

Conversation

@q-andy
Copy link
Contributor

@q-andy q-andy commented May 13, 2025

Description

Support system ingest pipelines for bulk update operations

Update Request Type Classification

  • Introduce a method to expose all child index requests associated with an update operation

Pipeline Resolution Enhancement

  • Use resolveSystemIngestPipeline to enable resolving only the system ingest pipeline while setting the others to NOOP
  • Based on update request fields, we extract the update request children and conditionally resolve ALL pipelines, resolve ONLY system ingest pipelines, or no pipelines at all.

Slot Management

  • Introduce childSlot to track individual child index requests within anupdate operation
  • Use innerslot to map pipeline execution results back to the correct child request using (slot, innerSlot) pairs
  • Maintain proper error handling and response mapping for both parent and child operations to their original bulk request slot

Related Issues

Resolves #18276
Partially addresses #17742

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@q-andy q-andy requested a review from a team as a code owner May 13, 2025 16:48
@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing labels May 13, 2025
@github-actions
Copy link
Contributor

❌ Gradle check result for f8169f3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

❌ Gradle check result for 94653c1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

❌ Gradle check result for 959b59f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

❌ Gradle check result for b2c9db0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@q-andy q-andy force-pushed the system-pipeline-bulk-update-2 branch from b2c9db0 to fbafb82 Compare May 14, 2025 20:21
@github-actions
Copy link
Contributor

❌ Gradle check result for fbafb82: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@q-andy q-andy force-pushed the system-pipeline-bulk-update-2 branch from fbafb82 to 2341708 Compare May 14, 2025 21:04
@github-actions
Copy link
Contributor

❌ Gradle check result for 2341708: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@q-andy q-andy force-pushed the system-pipeline-bulk-update-2 branch from 2341708 to f5aa904 Compare May 14, 2025 21:39
@github-actions
Copy link
Contributor

✅ Gradle check result for f5aa904: SUCCESS

@codecov
Copy link

codecov bot commented May 14, 2025

Codecov Report

Attention: Patch coverage is 94.69697% with 7 lines in your changes missing coverage. Please review.

Project coverage is 72.74%. Comparing base (5255f5b) to head (d37361f).
Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...main/java/org/opensearch/ingest/IngestService.java 94.52% 1 Missing and 3 partials ⚠️
...rg/opensearch/action/bulk/TransportBulkAction.java 93.18% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #18277      +/-   ##
============================================
+ Coverage     72.71%   72.74%   +0.02%     
- Complexity    68094    68136      +42     
============================================
  Files          5537     5537              
  Lines        313312   313375      +63     
  Branches      45460    45473      +13     
============================================
+ Hits         227813   227951     +138     
+ Misses        66967    66956      -11     
+ Partials      18532    18468      -64     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 2, 2025

❌ Gradle check result for 915a5a2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@github-actions
Copy link
Contributor

github-actions bot commented Jun 4, 2025

❌ Gradle check result for 82d3c3f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@q-andy q-andy force-pushed the system-pipeline-bulk-update-2 branch from 82d3c3f to 1d83672 Compare June 4, 2025 22:46
@q-andy q-andy closed this Jun 11, 2025
@q-andy q-andy reopened this Jun 11, 2025
@github-actions
Copy link
Contributor

❌ Gradle check result for ff5d40d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@q-andy q-andy closed this Jun 11, 2025
@q-andy q-andy reopened this Jun 11, 2025
@github-actions
Copy link
Contributor

❌ Gradle check result for 3d53411: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@q-andy q-andy closed this Jun 11, 2025
@q-andy q-andy reopened this Jun 11, 2025
@q-andy
Copy link
Contributor Author

q-andy commented Jun 11, 2025

flakey tests
#18485
#15840
#14407
#16658

@github-actions
Copy link
Contributor

❌ Gradle check result for 3d53411: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@q-andy q-andy force-pushed the system-pipeline-bulk-update-2 branch from 3d53411 to d37361f Compare June 11, 2025 15:53
@q-andy
Copy link
Contributor Author

q-andy commented Jun 11, 2025

Added same fix for example mapping plugin yaml tests related to legacy templates

@github-actions
Copy link
Contributor

✅ Gradle check result for d37361f: SUCCESS

@owaiskazi19 owaiskazi19 merged commit 8a59ca1 into opensearch-project:main Jun 11, 2025
30 checks passed
abhita pushed a commit to abhita/OpenSearch that referenced this pull request Jun 17, 2025
…h-project#18277)

* Support system ingest pipelines for bulk update operations

Signed-off-by: Andy Qin <[email protected]>

* Spotless

Signed-off-by: Andy Qin <[email protected]>

# Conflicts:
#	CHANGELOG.md

* Refactor UpdateRequest type checking

Signed-off-by: Andy Qin <[email protected]>

* Fix integ tests after rebase

Signed-off-by: Andy Qin <[email protected]>

* Rename innerSlot to childSlot

Signed-off-by: Andy Qin <[email protected]>

* Update yaml tests

Signed-off-by: Andy Qin <[email protected]>

* Update comments

Signed-off-by: Andy Qin <[email protected]>

* Add allowed warning header for yaml tests

Signed-off-by: Andy Qin <[email protected]>

* Update warning headers

Signed-off-by: Andy Qin <[email protected]>

---------

Signed-off-by: Andy Qin <[email protected]>
Signed-off-by: Sarat Vemulapalli <[email protected]>
Co-authored-by: Sarat Vemulapalli <[email protected]>
neuenfeldttj added a commit to neuenfeldttj/OpenSearch that referenced this pull request Jun 26, 2025
…h-project#18277)

* Support system ingest pipelines for bulk update operations

Signed-off-by: Andy Qin <[email protected]>

* Spotless

Signed-off-by: Andy Qin <[email protected]>

# Conflicts:
#	CHANGELOG.md

* Refactor UpdateRequest type checking

Signed-off-by: Andy Qin <[email protected]>

* Fix integ tests after rebase

Signed-off-by: Andy Qin <[email protected]>

* Rename innerSlot to childSlot

Signed-off-by: Andy Qin <[email protected]>

* Update yaml tests

Signed-off-by: Andy Qin <[email protected]>

* Update comments

Signed-off-by: Andy Qin <[email protected]>

* Add allowed warning header for yaml tests

Signed-off-by: Andy Qin <[email protected]>

* Update warning headers

Signed-off-by: Andy Qin <[email protected]>

---------

Signed-off-by: Andy Qin <[email protected]>
Signed-off-by: Sarat Vemulapalli <[email protected]>
Co-authored-by: Sarat Vemulapalli <[email protected]>Signed-off-by: TJ Neuenfeldt <[email protected]>
neuenfeldttj pushed a commit to neuenfeldttj/OpenSearch that referenced this pull request Jun 26, 2025
…h-project#18277)

* Support system ingest pipelines for bulk update operations

Signed-off-by: Andy Qin <[email protected]>

* Spotless

Signed-off-by: Andy Qin <[email protected]>

# Conflicts:
#	CHANGELOG.md

* Refactor UpdateRequest type checking

Signed-off-by: Andy Qin <[email protected]>

* Fix integ tests after rebase

Signed-off-by: Andy Qin <[email protected]>

* Rename innerSlot to childSlot

Signed-off-by: Andy Qin <[email protected]>

* Update yaml tests

Signed-off-by: Andy Qin <[email protected]>

* Update comments

Signed-off-by: Andy Qin <[email protected]>

* Add allowed warning header for yaml tests

Signed-off-by: Andy Qin <[email protected]>

* Update warning headers

Signed-off-by: Andy Qin <[email protected]>

---------

Signed-off-by: Andy Qin <[email protected]>
Signed-off-by: Sarat Vemulapalli <[email protected]>
Co-authored-by: Sarat Vemulapalli <[email protected]>
tandonks pushed a commit to tandonks/OpenSearch that referenced this pull request Aug 5, 2025
…h-project#18277)

* Support system ingest pipelines for bulk update operations

Signed-off-by: Andy Qin <[email protected]>

* Spotless

Signed-off-by: Andy Qin <[email protected]>

# Conflicts:
#	CHANGELOG.md

* Refactor UpdateRequest type checking

Signed-off-by: Andy Qin <[email protected]>

* Fix integ tests after rebase

Signed-off-by: Andy Qin <[email protected]>

* Rename innerSlot to childSlot

Signed-off-by: Andy Qin <[email protected]>

* Update yaml tests

Signed-off-by: Andy Qin <[email protected]>

* Update comments

Signed-off-by: Andy Qin <[email protected]>

* Add allowed warning header for yaml tests

Signed-off-by: Andy Qin <[email protected]>

* Update warning headers

Signed-off-by: Andy Qin <[email protected]>

---------

Signed-off-by: Andy Qin <[email protected]>
Signed-off-by: Sarat Vemulapalli <[email protected]>
Co-authored-by: Sarat Vemulapalli <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request] Support system generated ingest pipelines for bulk update operations

4 participants