Add EARLIEST/LATEST aggregate functions for PPL #4100

ykmr1224 · 2025-08-21T20:32:14Z

Description

Implements EARLIEST and LATEST aggregate functions for PPL using a ARG_MIN/ARG_MAX.
Uses @timestamp field to decide the earliest/latest by default.
It accepts optional argument to specify timestamp field to be used.
Pushdown for OpenSearch will be implemented in a separate PR.
Fixed doctest to enable Calcite.
Refactored test_docs.py and enabled Calcite enabled tests separately from existing Calcite disabled cases.

Usage

# Basic usage
source=logs | stats earliest(message), latest(response) by host

# Custom time field
source=metrics | stats latest(cpu_usage, event_time) by server

Related Issues

[FEATURE] PPL earliest/latest aggregation function support #3639

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

ykmr1224 · 2025-08-25T15:36:10Z

Checking doctest failure.

docs/user/ppl/cmd/stats.rst

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstExpressionBuilder.java

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLEarliestLatestTest.java

integ-test/src/test/java/org/opensearch/sql/ppl/ExplainIT.java

integ-test/src/test/java/org/opensearch/sql/ppl/NewAddedCommandsIT.java

core/src/main/java/org/opensearch/sql/calcite/udf/udaf/MaxByAggFunction.java

dai-chen

Thanks for the changes!

dai-chen · 2025-08-28T22:42:53Z

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLEarliestLatestTest.java

+    verifyResult(root, expectedResult);
+
+    String expectedSparkSql =
+        "SELECT ARG_MIN(`message`, `@timestamp`) `earliest_message`\n" + "FROM `POST`.`LOGS`";


Not sure if this is bug. I only see min/max_by in SparkSQL.

Ah, I will address this in the next PR.

opensearch-trigger-bot · 2025-08-28T22:43:07Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4100-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e2a1132018b260017e0c387122ef6d8523726c94
# Push it to GitHub
git push --set-upstream origin backport/backport-4100-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4100-to-2.19-dev.

opensearch-trigger-bot · 2025-08-28T23:22:31Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4100-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e2a1132018b260017e0c387122ef6d8523726c94
# Push it to GitHub
git push --set-upstream origin backport/backport-4100-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4100-to-2.19-dev.

opensearch-trigger-bot · 2025-08-29T20:03:09Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4100-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e2a1132018b260017e0c387122ef6d8523726c94
# Push it to GitHub
git push --set-upstream origin backport/backport-4100-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4100-to-2.19-dev.

ykmr1224 · 2025-08-29T20:47:24Z

backport blocker: #4094

opensearch-trigger-bot · 2025-09-03T15:35:37Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4100-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e2a1132018b260017e0c387122ef6d8523726c94
# Push it to GitHub
git push --set-upstream origin backport/backport-4100-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4100-to-2.19-dev.

opensearch-trigger-bot · 2025-09-03T20:56:40Z

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4100-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e2a1132018b260017e0c387122ef6d8523726c94
# Push it to GitHub
git push --set-upstream origin backport/backport-4100-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4100-to-2.19-dev.

) * Add earliest/latest aggregate function to PPL Signed-off-by: Tomoyuki Morita <[email protected]> * Reformat Signed-off-by: Tomoyuki Morita <[email protected]> * Add IT Signed-off-by: Tomoyuki Morita <[email protected]> * Fix integ test Signed-off-by: Tomoyuki Morita <[email protected]> * Use ARG_MAX and ARG_MIN instead Signed-off-by: Tomoyuki Morita <[email protected]> * Revert comment deletion Signed-off-by: Tomoyuki Morita <[email protected]> * Remove min_by and max_by Signed-off-by: Tomoyuki Morita <[email protected]> * Update doc Signed-off-by: Tomoyuki Morita <[email protected]> * Fix tests Signed-off-by: Tomoyuki Morita <[email protected]> * Fix tests Signed-off-by: Tomoyuki Morita <[email protected]> * Fix tests Signed-off-by: Tomoyuki Morita <[email protected]> * Fit IT Signed-off-by: Tomoyuki Morita <[email protected]> * Fix doctest Signed-off-by: Tomoyuki Morita <[email protected]> * Fix doctest Signed-off-by: Tomoyuki Morita <[email protected]> * Remove unneeded test Signed-off-by: Tomoyuki Morita <[email protected]> * Delete unneeded files Signed-off-by: Tomoyuki Morita <[email protected]> * Minor fix Signed-off-by: Tomoyuki Morita <[email protected]> * Fix issue caused by conflict Signed-off-by: Tomoyuki Morita <[email protected]> * reformat Signed-off-by: Tomoyuki Morita <[email protected]> * Fix doctest issue Signed-off-by: Tomoyuki Morita <[email protected]> * Minor fix Signed-off-by: Tomoyuki Morita <[email protected]> * Fix tests Signed-off-by: Tomoyuki Morita <[email protected]> * Fix test_docs Signed-off-by: Tomoyuki Morita <[email protected]> * Update test_docs.py Signed-off-by: Tomoyuki Morita <[email protected]> * Refactor test_docs.py and fix Signed-off-by: Tomoyuki Morita <[email protected]> * Fix issues caused by merge Signed-off-by: Tomoyuki Morita <[email protected]> --------- Signed-off-by: Tomoyuki Morita <[email protected]> Signed-off-by: Tomoyuki MORITA <[email protected]> Signed-off-by: Tomoyuki Morita <[email protected]>

* Add earliest/latest aggregate function to PPL * Reformat * Add IT * Fix integ test * Use ARG_MAX and ARG_MIN instead * Revert comment deletion * Remove min_by and max_by * Update doc * Fix tests * Fix tests * Fix tests * Fit IT * Fix doctest * Fix doctest * Remove unneeded test * Delete unneeded files * Minor fix * Fix issue caused by conflict * reformat * Fix doctest issue * Minor fix * Fix tests * Fix test_docs * Update test_docs.py * Refactor test_docs.py and fix * Fix issues caused by merge --------- Signed-off-by: Tomoyuki Morita <[email protected]> Signed-off-by: Tomoyuki MORITA <[email protected]>

ykmr1224 added PPL Piped processing language feature calcite calcite migration releated labels Aug 22, 2025

ykmr1224 force-pushed the earliest-agg branch 2 times, most recently from f3299c4 to a820a9c Compare August 22, 2025 17:15

ykmr1224 marked this pull request as ready for review August 25, 2025 15:35

ykmr1224 requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, qianheng-aws, seankao-az, vamsimanohar and yuancu as code owners August 25, 2025 15:35

dai-chen mentioned this pull request Aug 18, 2025

[FEATURE] PPL eventstats command enhancement #4047

Open

penghuo reviewed Aug 25, 2025

View reviewed changes

dai-chen approved these changes Aug 28, 2025

View reviewed changes

ykmr1224 merged commit e2a1132 into opensearch-project:main Aug 28, 2025
23 checks passed

dai-chen reviewed Aug 28, 2025

View reviewed changes

opensearch-trigger-bot bot added the backport-failed label Aug 28, 2025

ykmr1224 added backport 2.19-dev and removed backport-failed backport 2.19-dev labels Aug 28, 2025

opensearch-trigger-bot bot added the backport-failed label Aug 28, 2025

ykmr1224 mentioned this pull request Aug 29, 2025

Support bin command with Calcite #3878

Merged

ykmr1224 added backport 2.19-dev and removed backport 2.19-dev labels Aug 29, 2025

ykmr1224 mentioned this pull request Aug 29, 2025

Add documents on how to develop a UDF / UDAF #4094

Merged

8 tasks

ykmr1224 mentioned this pull request Sep 2, 2025

[Backport 2.19-dev] Add documents on how to develop a UDF / UDAF (#4094) #4191

Merged

ykmr1224 added backport 2.19-dev and removed backport 2.19-dev labels Sep 3, 2025

ykmr1224 mentioned this pull request Sep 3, 2025

Pushdown earliest/latest aggregate functions #4166

Merged

8 tasks

ykmr1224 added backport 2.19-dev and removed backport 2.19-dev labels Sep 3, 2025

ykmr1224 mentioned this pull request Sep 3, 2025

[Backport 2.19-dev] Add EARLIEST/LATEST aggregate functions for PPL (#4100) #4221

Merged

ykmr1224 added the backport-manually Filed a PR to backport manually. label Sep 3, 2025

ykmr1224 deleted the earliest-agg branch September 4, 2025 16:46

Add EARLIEST/LATEST aggregate functions for PPL #4100

Add EARLIEST/LATEST aggregate functions for PPL #4100

Uh oh!

Conversation

ykmr1224 commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Usage

Related Issues

Check List

Uh oh!

ykmr1224 commented Aug 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dai-chen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dai-chen Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

ykmr1224 Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

opensearch-trigger-bot bot commented Aug 28, 2025

Uh oh!

opensearch-trigger-bot bot commented Aug 28, 2025

Uh oh!

opensearch-trigger-bot bot commented Aug 29, 2025

Uh oh!

ykmr1224 commented Aug 29, 2025

Uh oh!

opensearch-trigger-bot bot commented Sep 3, 2025

Uh oh!

opensearch-trigger-bot bot commented Sep 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ykmr1224 commented Aug 21, 2025 •

edited

Loading