Skip to content

Add JSON_EXTRACT_ALL internal function for Calcite PPL #4489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ykmr1224 merged 3 commits into from
Oct 14, 2025
Merged

Add JSON_EXTRACT_ALL internal function for Calcite PPL #4489

ykmr1224 merged 3 commits into from
Oct 14, 2025

Conversation

@ykmr1224
Copy link
Collaborator

@ykmr1224 ykmr1224 commented Oct 10, 2025

Description

  • Add JSON_EXTRACT_ALL internal function for Calcite PPL
    • This function will be used by spath command without path param implementation.
    • Extracts all the values from input JSON string, and store them into MAP with JSON path as the key.
    • Example: input={"a": 1, "b": [2, 3, 4], "c": [{"e": 5}, {"e": 6}, {"f": 7}]} output=Map("a"=> 1, "b{}"=> [2,3,4], "c{}.e"=> [5, 6], "c{}.f"=> 7)
      • {} is a special notation to indicate array in the path.
      • A value could be array or single value regardless the key ends with or contains {}. (see c{}.e and c{}.f in above example)
    • Please check unit tests for detailed spec.
  • We might extend it to allow specifying paths to collect only needed values for optimization.

Related Issues

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ykmr1224 ykmr1224 added PPL Piped processing language feature calcite calcite migration releated backport 2.19-dev labels Oct 10, 2025
@ykmr1224 ykmr1224 marked this pull request as ready for review October 10, 2025 01:07
@ykmr1224
Address comments
59c9ff7 
Signed-off-by: Tomoyuki Morita <[email protected]>
@ykmr1224
Minor fix
e472adf 
Signed-off-by: Tomoyuki Morita <[email protected]>
@ykmr1224 ykmr1224 merged commit 9c97cfb into opensearch-project:main Oct 14, 2025
33 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 14, 2025
@github-actions
Add JSON_EXTRACT_ALL internal function for Calcite PPL (#4489)
80992bd 
* Add JSON_EXTRACT_ALL internal function for Calcite PPL

Signed-off-by: Tomoyuki Morita <[email protected]>

* Address comments

Signed-off-by: Tomoyuki Morita <[email protected]>

* Minor fix

Signed-off-by: Tomoyuki Morita <[email protected]>

---------

Signed-off-by: Tomoyuki Morita <[email protected]>
(cherry picked from commit 9c97cfb)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Oct 14, 2025
Merged
ykmr1224 added a commit to ykmr1224/sql that referenced this pull request Oct 15, 2025
@ykmr1224
Merge main to map-remove:
6acef56 
commit cba8d02
Author: Tomoyuki MORITA <[email protected]>
Date:   Wed Oct 15 13:08:05 2025 -0700

    Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515)

    * Add MAP_APPEND internal function to Calcite PPL

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Minor fix

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Address comment

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Rebase and fix IT issue

    Signed-off-by: Tomoyuki Morita <[email protected]>

    ---------

    Signed-off-by: Tomoyuki Morita <[email protected]>

commit 3388dc7
Author: Lantao Jin <[email protected]>
Date:   Thu Oct 16 01:45:29 2025 +0800

    Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569)

    * Use _shard_doc as sort tiebreaker

    Signed-off-by: Lantao Jin <[email protected]>

    * _doc as a part of tie-breaker have better performance

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

commit 5630119
Author: qianheng <[email protected]>
Date:   Wed Oct 15 16:40:41 2025 +0800

    Fix sort push down into agg after project already pushed (opensearch-project#4546)

    * Fix sort push down into agg

    Signed-off-by: Heng Qian <[email protected]>

    * Change some json files to yaml format

    Signed-off-by: Heng Qian <[email protected]>

    ---------

    Signed-off-by: Heng Qian <[email protected]>

commit 1e62fba
Author: Tomoyuki MORITA <[email protected]>
Date:   Tue Oct 14 17:20:38 2025 -0700

    Fix JsonExtractAllFunctionIT failure (opensearch-project#4556)

    Signed-off-by: Tomoyuki Morita <[email protected]>

commit 02ee33e
Author: Kai Huang <[email protected]>
Date:   Tue Oct 14 14:28:53 2025 -0700

    Add more examples to the `where` command doc (opensearch-project#4457)

    Co-authored-by: Manasvini B S <[email protected]>

commit 0b7e86c
Author: Jialiang Liang <[email protected]>
Date:   Tue Oct 14 10:46:01 2025 -0700

    [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434)

    Co-authored-by: Simeon Widdis <[email protected]>

commit 9c97cfb
Author: Tomoyuki MORITA <[email protected]>
Date:   Tue Oct 14 08:36:43 2025 -0700

    Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489)

    * Add JSON_EXTRACT_ALL internal function for Calcite PPL

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Address comments

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Minor fix

    Signed-off-by: Tomoyuki Morita <[email protected]>

    ---------

    Signed-off-by: Tomoyuki Morita <[email protected]>

commit 89dbc31
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 18:24:52 2025 +0800

    Check server status before starting Prometheus (opensearch-project#4537)

    * Check server status before starting Prometheus

    Signed-off-by: Lantao Jin <[email protected]>

    * Change to func call

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix doc

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

commit fe62472
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 18:10:27 2025 +0800

    Update request builder after pushdown sort into agg buckets (opensearch-project#4541)

    Signed-off-by: Lantao Jin <[email protected]>

commit 42a415f
Author: qianheng <[email protected]>
Date:   Tue Oct 14 17:42:45 2025 +0800

    Including metadata fields type when doing agg/filter script push down (opensearch-project#4522)

    * Including metadata fields type when doing agg/filter script push down

    Signed-off-by: Heng Qian <[email protected]>

    * Fix IT

    Signed-off-by: Heng Qian <[email protected]>

    ---------

    Signed-off-by: Heng Qian <[email protected]>

commit 8de0386
Author: Xinyuan Lu <[email protected]>
Date:   Tue Oct 14 16:41:08 2025 +0800

    Fix percentile bug (opensearch-project#4539)

    * fix percentile bug

    Signed-off-by: xinyual <[email protected]>

    * add IT

    Signed-off-by: xinyual <[email protected]>

    * optimize it

    Signed-off-by: xinyual <[email protected]>

    ---------

    Signed-off-by: xinyual <[email protected]>

commit de2fdc8
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 12:29:03 2025 +0800

    [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534)

    * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited

    Signed-off-by: Lantao Jin <[email protected]>

    * fix doctest

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix conflicts

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

commit 977b7ab
Author: Simeon Widdis <[email protected]>
Date:   Mon Oct 13 20:23:10 2025 -0700

    Update stalled action (opensearch-project#4485)

commit fddbb70
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 10:23:12 2025 +0800

    Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501)

    * Add configurable sytem limitations for subsearch and join command

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix IT

    Signed-off-by: Lantao Jin <[email protected]>

    * typo

    Signed-off-by: Lantao Jin <[email protected]>

    * fix IT

    Signed-off-by: Lantao Jin <[email protected]>

    * remove rollback in doc

    Signed-off-by: Lantao Jin <[email protected]>

    * address comments

    Signed-off-by: Lantao Jin <[email protected]>

    * fix typo

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix IT

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

Signed-off-by: Tomoyuki Morita <[email protected]>
ykmr1224 added a commit that referenced this pull request Oct 15, 2025
@opensearch-trigger-bot @github-actions
@LantaoJin @ykmr1224
[Backport 2.19-dev] Add JSON_EXTRACT_ALL internal function for Calcit…
fd1082a 
…e PPL (#4548)

* Add JSON_EXTRACT_ALL internal function for Calcite PPL (#4489)

* Add JSON_EXTRACT_ALL internal function for Calcite PPL

Signed-off-by: Tomoyuki Morita <[email protected]>

* Address comments

Signed-off-by: Tomoyuki Morita <[email protected]>

* Minor fix

Signed-off-by: Tomoyuki Morita <[email protected]>

---------

Signed-off-by: Tomoyuki Morita <[email protected]>
(cherry picked from commit 9c97cfb)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Fix build failure

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix build failure

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix IT failure

Signed-off-by: Tomoyuki Morita <[email protected]>

---------

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Signed-off-by: Lantao Jin <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Lantao Jin <[email protected]>
Co-authored-by: Tomoyuki Morita <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@penghuo penghuo penghuo approved these changes

@dai-chen dai-chen dai-chen approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar vamsimanohar is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@LantaoJin LantaoJin Awaiting requested review from LantaoJin LantaoJin is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

@qianheng-aws qianheng-aws Awaiting requested review from qianheng-aws qianheng-aws is a code owner

@yuancu yuancu Awaiting requested review from yuancu yuancu is a code owner

@RyanL1997 RyanL1997 Awaiting requested review from RyanL1997 RyanL1997 is a code owner

Assignees

No one assigned

Labels

backport 2.19-dev calcite calcite migration releated feature PPL Piped processing language

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ykmr1224 @penghuo @dai-chen