Support refering to implicit `@timestamp` field in span #4138

yuancu · 2025-08-27T05:19:54Z

Description

This PR allows makes the query source=my-index | stats count() by span(1h) equivalent to source=my-index | stats count() by span(@timestamp, 1h)

Update documentations to reflect this change
Resolve by span(1d) as by span(@timestamp, 1d)
Throw exception for zero span

Related Issues

Resolves #4136, resolves #4527

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Yuanchun Shen <[email protected]>

…t specifying a field Signed-off-by: Yuanchun Shen <[email protected]>

LantaoJin

@yuancu can you fix the conflicts

Signed-off-by: Yuanchun Shen <[email protected]>

qianheng-aws · 2025-09-17T08:46:18Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRexNodeVisitor.java

  public RexNode visitSpan(Span node, CalcitePlanContext context) {
-    RexNode field = analyze(node.getField(), context);
+    RexNode field;
+    if (node.getField() != null) {


Will it be better to set field in span to be new Field(new QualifiedName(OpenSearchConstants.IMPLICIT_FIELD_TIMESTAMP)) if field is null? Then we don't need any change here.

And I'm wondering if v2 works well since we only make change for calcite visitor.

I referred it here because I thought the AST level should be consistent with users' input. E.g. BY span(1h) should be parsed to Span(value=1, unit=hour, field=null). Subsequent implicit reference can be added later because this semantic of how absent field should be interpreted is in a higher level than syntax tree construction.

But it's also not bizarre to refer it while building AST tree. I'll modify the implementation accordingly.

- Additionally refactored visitTimechartCommand to reuse spanLiteral definition Signed-off-by: Yuanchun Shen <[email protected]>

Signed-off-by: Yuanchun Shen <[email protected]>

LantaoJin · 2025-09-22T09:13:08Z

ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java

+      if (field != null) {
+        return StringUtils.format("span(%s, %s %s)", field, value, node.getUnit().getName());
+      } else {
+        return StringUtils.format("span(%s %s)", value, node.getUnit().getName());


no test for verification?

Reverted this change to Span as span will always come with a field with latest implementation -- I inject @timestamp to spans without a field specified in the AST layer now.

…ementation Signed-off-by: Yuanchun Shen <[email protected]>

Signed-off-by: Yuanchun Shen <[email protected]>

opensearch-trigger-bot · 2025-10-14T15:22:21Z

This PR is stalled because it has been open for 2 weeks with no activity.

RyanL1997 · 2025-10-14T17:35:21Z

This PR is stalled because it has been open for 2 weeks with no activity.

Hi @yuancu , is this implementation still in track?

Signed-off-by: Yuanchun Shen <[email protected]>

@timestamp

* Support refering to implicit @timestamp field in time-based aggregations Signed-off-by: Yuanchun Shen <[email protected]> * Update documentation of stats to reflect that span can be used without specifying a field Signed-off-by: Yuanchun Shen <[email protected]> * Move @timestamp reference to AST layer - Additionally refactored visitTimechartCommand to reuse spanLiteral definition Signed-off-by: Yuanchun Shen <[email protected]> * Unit test visitSpanLiteral, vistSpanClause, and visitTimechartParamter Signed-off-by: Yuanchun Shen <[email protected]> * Revert changes to Span will always have a field with the current implementation Signed-off-by: Yuanchun Shen <[email protected]> * Throw exception for zero span Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> (cherry picked from commit c30d5d0) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

@timestamp

…n span (#4610) * Support refering to implicit `@timestamp` field in span (#4138) * Support refering to implicit @timestamp field in time-based aggregations Signed-off-by: Yuanchun Shen <[email protected]> * Update documentation of stats to reflect that span can be used without specifying a field Signed-off-by: Yuanchun Shen <[email protected]> * Move @timestamp reference to AST layer - Additionally refactored visitTimechartCommand to reuse spanLiteral definition Signed-off-by: Yuanchun Shen <[email protected]> * Unit test visitSpanLiteral, vistSpanClause, and visitTimechartParamter Signed-off-by: Yuanchun Shen <[email protected]> * Revert changes to Span will always have a field with the current implementation Signed-off-by: Yuanchun Shen <[email protected]> * Throw exception for zero span Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> (cherry picked from commit c30d5d0) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Change pattern variable to cast Signed-off-by: Yuanchun Shen <[email protected]> --------- Signed-off-by: Yuanchun Shen <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Yuanchun Shen <[email protected]>

Support refering to implicit @timestamp field in time-based aggregations

c637bb2

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu self-assigned this Aug 27, 2025

yuancu added the enhancement New feature or request label Aug 27, 2025

Update documentation of stats to reflect that span can be used withou…

e3f904b

…t specifying a field Signed-off-by: Yuanchun Shen <[email protected]>

yuancu force-pushed the issues/4136 branch from 5725192 to e3f904b Compare August 28, 2025 03:00

yuancu marked this pull request as ready for review August 29, 2025 00:56

yuancu requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, YANG-DB, Yury-Fridlyand, acarbonetto, anirudha, dai-chen, derek-ho, forestmvey, joshuali925, kavithacm, mengweieric, noCharger, penghuo, ps48, qianheng-aws, seankao-az, vamsimanohar and ykmr1224 as code owners August 29, 2025 00:56

LantaoJin previously approved these changes Aug 29, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into issues/4136

4c20af1

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu dismissed LantaoJin’s stale review via 4c20af1 August 29, 2025 01:53

qianheng-aws reviewed Sep 17, 2025

View reviewed changes

Move @timestamp reference to AST layer

111b095

- Additionally refactored visitTimechartCommand to reuse spanLiteral definition Signed-off-by: Yuanchun Shen <[email protected]>

yuancu requested a review from RyanL1997 as a code owner September 18, 2025 07:11

yuancu force-pushed the issues/4136 branch from 66f0e42 to 43b29d0 Compare September 18, 2025 07:12

Unit test visitSpanLiteral, vistSpanClause, and visitTimechartParamter

7d7108e

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu force-pushed the issues/4136 branch from 43b29d0 to 7d7108e Compare September 18, 2025 07:14

Merge remote-tracking branch 'origin/main' into issues/4136

1d4de5a

Signed-off-by: Yuanchun Shen <[email protected]>

LantaoJin reviewed Sep 22, 2025

View reviewed changes

qianheng-aws previously approved these changes Sep 24, 2025

View reviewed changes

Revert changes to Span will always have a field with the current impl…

4a9193f

…ementation Signed-off-by: Yuanchun Shen <[email protected]>

yuancu dismissed qianheng-aws’s stale review via 4a9193f September 24, 2025 08:48

yuancu added 2 commits September 24, 2025 17:35

Merge remote-tracking branch 'origin/main' into issues/4136

1907947

Signed-off-by: Yuanchun Shen <[email protected]>

Merge remote-tracking branch 'origin/main' into issues/4136

3bf6a69

Signed-off-by: Yuanchun Shen <[email protected]>

opensearch-trigger-bot bot added the stalled label Oct 14, 2025

Merge remote-tracking branch 'origin/main' into issues/4136

9d47720

Signed-off-by: Yuanchun Shen <[email protected]>

LantaoJin previously approved these changes Oct 15, 2025

View reviewed changes

LantaoJin added backport 2.19-dev and removed stalled labels Oct 15, 2025

Throw exception for zero span

edcb72f

Signed-off-by: Yuanchun Shen <[email protected]>

yuancu dismissed LantaoJin’s stale review via edcb72f October 17, 2025 08:25

yuancu added the bug Something isn't working label Oct 17, 2025

LantaoJin approved these changes Oct 17, 2025

View reviewed changes

qianheng-aws approved these changes Oct 21, 2025

View reviewed changes

qianheng-aws merged commit c30d5d0 into opensearch-project:main Oct 21, 2025
36 checks passed

opensearch-trigger-bot bot mentioned this pull request Oct 21, 2025

[Backport 2.19-dev] Support refering to implicit @timestamp field in span #4610

Merged

yuancu deleted the issues/4136 branch October 21, 2025 06:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support refering to implicit `@timestamp` field in span #4138

Support refering to implicit `@timestamp` field in span #4138

Uh oh!

yuancu commented Aug 27, 2025 •

edited

Loading

Uh oh!

LantaoJin left a comment

Uh oh!

qianheng-aws Sep 17, 2025

Uh oh!

yuancu Sep 17, 2025

Uh oh!

LantaoJin Sep 22, 2025

Uh oh!

yuancu Sep 24, 2025

Uh oh!

opensearch-trigger-bot bot commented Oct 14, 2025

Uh oh!

RyanL1997 commented Oct 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Support refering to implicit @timestamp field in span #4138

Support refering to implicit @timestamp field in span #4138

Uh oh!

Conversation

yuancu commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

LantaoJin left a comment

Choose a reason for hiding this comment

Uh oh!

qianheng-aws Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

LantaoJin Sep 22, 2025

Choose a reason for hiding this comment

Uh oh!

yuancu Sep 24, 2025

Choose a reason for hiding this comment

Uh oh!

opensearch-trigger-bot bot commented Oct 14, 2025

Uh oh!

RyanL1997 commented Oct 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Support refering to implicit `@timestamp` field in span #4138

Support refering to implicit `@timestamp` field in span #4138

yuancu commented Aug 27, 2025 •

edited

Loading