Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Onboard search req / search resp ML processors #256

Merged
merged 4 commits into from
Aug 2, 2024

Conversation

ohltyler
Copy link
Member

@ohltyler ohltyler commented Aug 2, 2024

Description

This PR adds ML processor as options in the context of search request and search response. Additionally, it finishes all of the logic in the input transform / output transform modals to dynamically fetch input / generate output. There are a few places where this preview / advanced transform view is gated:

  1. Chaining multiple search request processors - in this case, since we don't have per-processor logging available, we cannot generate the interim transformed request. This is already tracked here: [Feature Request] Support a verbose/debugging param in search pipelines OpenSearch#14745
  2. Advanced output transform for search request processors - we cannot do partial simulation of the search request processor in the context of output transformations, since there is no way to configure a working, simulate-able search request processor with an empty output_map - it is always required, as both input_map and output_map are optional in the context of ingest and search response. We need an empty output_map in order to isolate the response up to the point of configuring it.

More implementation details:

  • finishes handling the different contexts in InputTransformModal and the logic needed to fetch inputs
  • finishes handling the different contexts in OutputTransformModal and the logic needed to fetch inputs
  • refactors formikToIngestPipeline -> formikToPartialPipeline to be generic and handle for all contexts
  • adds gating in MLProcessorInputs to enable/disable the Preview button based on the different scenarios mentioned above ^
  • extra limitations on search request processor contexts: the input maps and output maps are required: warning modal added when these are empty
  • minor change in formatting of search responses to just show the nested _source document info. This is also to keep consistent with how the ML inference processors handle this input JSON, so we can replicate it correctly & consistently in the transform modals.

Testing:

  • Both added ML processors compile and work as expected, and generate valid search template resources
  • Preview works for both
  • Preview is disabled based on 2 scenarios above
  • Existing ingest processor & preview works as expected

Demo video, showing the 2 new processors, and preview functionality for both.

screen-capture.17.webm

Issues Resolved

Resolves #217
Makes progress on #23
Makes progress on #219

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ohltyler ohltyler marked this pull request as ready for review August 2, 2024 16:02
@ohltyler ohltyler merged commit 9aaf67a into opensearch-project:main Aug 2, 2024
6 checks passed
@ohltyler ohltyler deleted the ml-search-processors branch August 2, 2024 19:49
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 2, 2024
Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit 9aaf67a)
ohltyler added a commit that referenced this pull request Aug 2, 2024
Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit 9aaf67a)

Co-authored-by: Tyler Ohlsen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Workflow editor] Integrate with ML inference search processors
2 participants