Added release notes for 1.3.7 - #5418
Merged
andrross merged 4 commits intoDec 1, 2022
Merged
Conversation
Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com>
Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com>
andrross
reviewed
Dec 1, 2022
| - [Test] Add IAE test for deprecated edgeNGram analyzer name ([#5040](https://github.com/opensearch-project/OpenSearch/pull/5040)) | ||
| - Allow mmap to use new JDK-19 preview APIs in Apache Lucene 9.4+ ([#5151](https://github.com/opensearch-project/OpenSearch/pull/5151)) | ||
| - Add feature flag for extensions ([#5211](https://github.com/opensearch-project/OpenSearch/pull/5211)) | ||
| - Added release notes for 1.3.7 ([#5418](https://github.com/opensearch-project/OpenSearch/pull/5418)) |
Member
There was a problem hiding this comment.
You can omit a changelog entry for this, just apply the skip-changelog label to the PR to make the workflow pass. See details about when to add changelog entries in CONTRIBUTING.md#Changelog
|
|
||
|
|
||
| ### Enhancements | ||
| * copy build.sh over from opensearch-build ([#4992](https://github.com/opensearch-project/OpenSearch/pull/4992)) |
Member
There was a problem hiding this comment.
I think we can apply the same criteria here as is documented for the changelog. All these test and build-related changes can be omitted. I think that will only leave the dependency upgrades above and PR 5149 below.
Member
Author
There was a problem hiding this comment.
Thanks for the pointer. Pushed the change.
owaiskazi19
force-pushed
the
release-notes-1.3.7
branch
from
December 1, 2022 01:03
26d5e63 to
fcd26f6
Compare
Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com>
owaiskazi19
force-pushed
the
release-notes-1.3.7
branch
from
December 1, 2022 01:04
fcd26f6 to
c985c91
Compare
Rishikesh1159
approved these changes
Dec 1, 2022
Contributor
Gradle Check (Jenkins) Run Completed with:
|
Contributor
Gradle Check (Jenkins) Run Completed with:
|
Contributor
Gradle Check (Jenkins) Run Completed with:
|
Contributor
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5418 +/- ##
============================================
+ Coverage 70.98% 71.07% +0.09%
- Complexity 58185 58208 +23
============================================
Files 4711 4711
Lines 277573 277573
Branches 40180 40180
============================================
+ Hits 197031 197292 +261
+ Misses 64386 64175 -211
+ Partials 16156 16106 -50 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
andrross
reviewed
Dec 1, 2022
Comment on lines
+8
to
+9
| * Updated Jackson to 2.14.0 ([#5113](https://github.com/opensearch-project/OpenSearch/pull/5113)) | ||
| * Updated Jackson to 2.14.1 ([#5356](https://github.com/opensearch-project/OpenSearch/pull/5356)) |
Member
There was a problem hiding this comment.
Sorry, another nitpick, but there is no need to include the version 2.14.0 because it was never released. Same thing with the first line "Updated Jackson Databind to 2.13.4.2"...that has also been superseded by PR 5356.
Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com>
andrross
approved these changes
Dec 1, 2022
Contributor
Gradle Check (Jenkins) Run Completed with:
|
opensearch-trigger-bot Bot
pushed a commit
that referenced
this pull request
Dec 1, 2022
Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com> (cherry picked from commit 31bcda6) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
xuezhou25
pushed a commit
that referenced
this pull request
Dec 9, 2022
Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com> (cherry picked from commit 31bcda6) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: owaiskazi19 <owaiskazi19@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 7, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
## Why an adapter extension is necessary
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
opensearch-project#21527, is extended here to recognize 3 OR 4 operands. Pattern is at position 1
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
## Out of scope (deferred to Part 2)
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
## Test results
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
## Companion PR
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 8, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
## Why an adapter extension is necessary
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
opensearch-project#21527, is extended here to recognize 3 OR 4 operands. Pattern is at position 1
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
## Out of scope (deferred to Part 2)
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
## Test results
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
## Companion PR
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 8, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 8, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 8, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 8, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 9, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 11, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 14, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
RyanL1997
added a commit
to RyanL1997/OpenSearch
that referenced
this pull request
May 14, 2026
…dge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from opensearch-project#21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (opensearch-project#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
* `RegexpReplaceAdapterTests` — 21/21 (19 from opensearch-project#21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume opensearch-project#5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
mch2
pushed a commit
that referenced
this pull request
May 14, 2026
* [Analytics Backend / DataFusion] Wire PPL rex sed-mode (Part 1) — bridge-only
Onboards the PPL `rex` command's `mode=sed` surface — the part that lowers to
standard Calcite library operators and bridges through Substrait to DataFusion's
native UDFs. Three sed sub-variants covered:
* `rex field=f mode=sed "s/old/new/"` (no flags) → SqlLibraryOperators.REGEXP_REPLACE_3
(already mapped via the PPL `replace` onboarding from #21527 — no-op here).
* `rex field=f mode=sed "s/old/new/g"` / `/i` / `/gi` → SqlLibraryOperators.REGEXP_REPLACE_PG_4
(4-arg with flags string). New bridge in this PR. DataFusion's regexp_replace
natively accepts 4-arg `(str, pat, repl, flags)` per its substrait UDF binding.
* `rex field=f mode=sed "y/from/to/"` (transliteration) → SqlLibraryOperators.TRANSLATE3.
New bridge in this PR. Resolves to DataFusion's `translate` UDF
(datafusion-functions/src/unicode/translate.rs).
The 4-arg `REGEXP_REPLACE_PG_4` carries the same Java-regex syntax baggage as the
3-arg form: `\Q…\E` quoted-literal blocks (Rust regex rejects them) and bare `$N`
backreferences in the replacement (Rust's identifier-greedy parser
mis-resolves them). RegexpReplaceAdapter, introduced for the 3-arg form in
and replacement at position 2 in both signatures — the rewrite logic doesn't
change. Operands beyond position 2 (the flags string in the 4-arg form) pass
through verbatim. Two new RegexpReplaceAdapterTests cover the 4-arg path.
`TRANSLATE3` doesn't need an adapter — its arguments are character classes, not
regex syntax.
* Rex extract mode (`rex field=f "(?<g>...)"`) — uses the SQL plugin's custom
Java UDFs `REX_EXTRACT`, `REX_EXTRACT_MULTI`, `REX_OFFSET`, which have no
native DataFusion equivalent. Slated for a follow-up PR that adds Rust-side
UDF implementations, similar to the convert_tz precedent (#21476).
* Sed with occurrence flag (`s/.../.../<N>`) — emits 5-arg
`REGEXP_REPLACE_5`, which DataFusion's native `regexp_replace` does not
support (max 4 args). Also Part 2.
* `RegexpReplaceAdapterTests` — 21/21 (19 from #21527 + 2 new for the 4-arg path).
* `RexCommandIT` (new self-contained QA IT, calcs dataset) — 9/9. Covers all sed
sub-variants: literal (no flags), `/g` global, `/i` case-insensitive, `/gi`
combined, backreferences via `$N`, transliteration `y/from/to/` and
no-match passthrough.
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green.
The unified-path NPE caused by a missing PPL_REX_MAX_MATCH_LIMIT default is fixed
in opensearch-project/sql#5418 — required for any rex query (sed or extract) to
reach the planner via /_analytics/ppl. This PR's Test results assume #5418 is
applied. Pre-fix: every query NPEs in `AstBuilder.visitRexCommand`. Post-fix:
9/9 RexCommandIT pass.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
* [Analytics Backend / DataFusion] Wire PPL rex extract-mode (Part 2) — Rust UDFs + array result type
Completes the PPL `rex` onboarding started in Part 1 (#21550). The sed-mode forms
were already covered by bridges to existing Calcite/DataFusion operators. The
extract-mode form has no native DataFusion equivalent and needs three custom
Rust UDFs, three Java SqlOperator adapters, and a small handful of analytics-
framework / engine plumbing changes to model array result types end-to-end.
* `rex_extract(input, pattern_lit, group_lit) -> varchar` — single named or
numbered group capture. Compiles the regex once at plan time, runs per row.
* `rex_extract_multi(input, pattern_lit, group_lit, max_match) -> list<varchar>`
— multi-match. `max_match=0` means unbounded; otherwise caps the result at
the requested element count. Returns NULL (not an empty list) when there
are no matches, matching the SQL plugin's Java implementation.
* `rex_offset(input, pattern_lit) -> varchar` — emits the named-group offsets
formatted as `"name1=s1-e1&name2=s2-e2"`, alphabetically sorted; end is
inclusive, matching the SQL plugin's `RexOffsetFunction.end - 1` convention.
Each UDF has 5 unit tests covering the contract above.
* `RexExtractAdapter`, `RexExtractMultiAdapter`, `RexOffsetAdapter` — keyed on
the SQL plugin's PPL builtin operator names (`REX_EXTRACT`,
`REX_EXTRACT_MULTI`, `REX_OFFSET`) via the analytics-framework
ScalarFunction enum. Each adapter rewrites the incoming RexCall to a local
target SqlOperator (`LOCAL_REX_EXTRACT_OP`, etc.) that
`DataFusionFragmentConvertor`'s `ADDITIONAL_SCALAR_SIGS` maps to the
corresponding `rex_extract` / `rex_extract_multi` / `rex_offset` Substrait
extension declared in `opensearch_scalar_functions.yaml`.
* Pattern operands (and the group operand for the extract variants) are
validated as RexLiterals at plan time. Column-valued patterns would force
per-row regex compilation on the Rust side and are rejected with an
IllegalArgumentException — same contract as the precedent set by
RegexpReplaceAdapter in Part 1. `RexExtractAdapterTests` covers this.
* `FieldType.ARRAY` enum value + `fromSqlTypeName(ARRAY) -> FieldType.ARRAY`
in analytics-framework. Without this, `OpenSearchProjectRule.resolveScalar
ViableBackends` returns `null` for any scalar with an array return type
and the planner emits "No backend supports scalar function [REX_EXTRACT_
MULTI] among [datafusion]". `REX_EXTRACT_MULTI`'s ProjectCapability.Scalar
declaration is now `Set.of(FieldType.ARRAY)` rather than the broad scalar
set used by every other op (UPPER, ABS, ...) — those genuinely don't return
arrays.
* `ListVector` handling in three call sites that previously triggered Arrow's
`JsonStringArrayList.<clinit>`, which references `JavaTimeModule` from
`jackson-datatype-jsr310` (not on the `arrow-flight-rpc` parent plugin's
classloader). Bypassing `getObject()` and reading offset buffer + inner
data vector directly:
- `DatafusionResultStream.getFieldValue` (shard-side row materialization)
- `ArrowValues.toJavaValue` (coordinator post-execution row reading)
- `RowResponseCodec` (`inferArrowField` + `setVectorValue`) — the
Object[]-row → Arrow VectorSchemaRoot wire codec needed an explicit
list<utf8> Field with proper child Field, plus a ListVector setter
using `startNewValue`/`endValue` + the inner VarCharVector's `setSafe`.
* `RexCommandIT` extended from 9 sed tests to 16 — adds 7 extract-mode cases:
single named group, multiple named groups in one row, missing-group
NULL handling, multi-match capturing all, `max_match` cap, offset_field
output, and no-match passthrough as NULL.
* Rust UDFs — 15/15 unit tests (5 per UDF).
* `RexExtractAdapterTests` — 4/4.
* `RexCommandIT` — 16/16 (9 sed from Part 1 + 7 new extract).
* `./gradlew check -p sandbox -Dsandbox.enabled=true` — green (678 tasks,
all sandbox module unit tests + spotless + license + forbidden API).
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
* [Analytics Backend / DataFusion] Onboard array_length scalar function (Part 3)
Wires Calcite's `SqlLibraryOperators.ARRAY_LENGTH` to DataFusion's native
`array_length`, completing the end-to-end story for PPL `rex` extract-mode
multi-match: queries can now size the list returned by `rex_extract_multi`
(`eval count = array_length(g)`).
* `ScalarFunction.ARRAY_LENGTH` enum value (resolves via the `valueOf()`
fallback on the Calcite operator name).
* Registered in `STANDARD_PROJECT_OPS`. Returns `bigint`, so the existing
`SUPPORTED_FIELD_TYPES` (numeric ∪ keyword ∪ date ∪ {BOOLEAN, TEXT})
covers the capability lookup — no special-case needed.
* `FunctionMappings.s(SqlLibraryOperators.ARRAY_LENGTH, "array_length")` in
`DataFusionFragmentConvertor.ADDITIONAL_SCALAR_SIGS`. Library operators
don't auto-resolve through the substrait default catalog — the same
explicit pinning pattern used for `ILIKE`, `DATE_PART`, and the
`REGEXP_REPLACE_*` family.
* `array_length` extension declaration in `opensearch_scalar_functions.yaml`
with `list<varchar<L1>>` → `i64` and `list<string>` → `i64` impls. Without
a custom YAML extension that matches the actual list type, isthmus emits
"Unable to convert call ARRAY_LENGTH(list<varchar<...>>)" for the
`rex_extract_multi` output.
Lifts CalciteRexCommandIT (SQL plugin's standard rex IT class) through the
analytics-engine route from 14/18 → 17/18. The remaining failure
(testRexMaxMatchConfigurableLimit) is a unified-query architectural gap —
`UnifiedQueryContext` ignores cluster-setting overrides and uses the static
default — unrelated to rex or array_length.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
* Collapse array_length impls to single list<any1>
CI surfaced this on the post-rebase rex run:
Duplicate key FunctionAnchor{urn=extension:org.opensearch:scalar_functions,
key=array_length:list} (attempted merging values
array_length:list and array_length:list)
The Part 3 commit declared two impls — `list<varchar<L1>>` and `list<string>`
— with the intent of covering both element-type families produced by
`rex_extract_multi`'s pair of impls. But substrait's compound function key
drops the inner parametric element type at the key level, so both impls
collapse to the same key `array_length:list`. The YAML loader rejects the
collision when the analytics-backend-datafusion plugin's
`SimpleExtension.ExtensionCollection` merges the file in.
Replace the two impls with a single `list<any1>` polymorphic impl. The
`any1` type variable matches any element type at planning, so a call site
that produces `list<varchar<L1>>` (rex_extract_multi varchar overload) and
a call site that produces `list<string>` (rex_extract_multi string
overload) both bind to the one impl. Net effect on planning is equivalent
and the duplicate-key collision goes away.
The duplicate didn't surface on the original rex CI run because the prior
PPL_REX_MAX_MATCH_LIMIT NPE failed every query at plan time before the
function-extension merge was reached. Once the mavenLocal pin fix landed
the prior commit and queries actually reached the planner, this older
latent collision was unmasked.
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
---------
Signed-off-by: Jialiang Liang <jiallian@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Signed-off-by: owaiskazi19 owaiskazi19@gmail.com
Description
Add release notes for patch release 1.3.7
Issues Resolved
[List any issues this PR will resolve]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.