Feature addtotals and addcoltotals

[asifabashar]

Description

Is your feature request related to a problem?
addtotals command to show total of all columns of each row as a new column , and also have option to show total of all rows of each column values to show at the end of rows.
Fixes issue #4607
From roadmap #4287

addcoltotals command to show total of each column's all rows values to show at the end of rows.
From roadmap #4287

What solution would you like?
command: addtotals ,addcoltotals
addtotals: Add totals across rows by default and also calculate total across columns when col=true
The addtotals command adds together the numeric fields in each search result.

You may specify which fields to include rather than summing all numeric fields.
The final total is stored in a new field.

The addtotals command's behavior is as follows:

When col=true, it computes the sum for every column and adds a summary row at the end containing those totals.

To label this final summary row, specify a labelfield and assign it a value using the label option.

Alternatively, instead of using the addtotals col=true command, you can use the addcoltotals command to calculate a summary event.

labelfield, if specified, is a field that will be added at the last row of the column specified by labalfield with the value set by the 'label' option.

Command Syntax:
addtotals [row=<bool>] [col=<bool>] [labelfield=<field>] [label=<string>] [fieldname=<field>] [<field-list>]
arguments description:
row: Syntax: row=<bool> . Indicates whether to compute the sum of the for each event. This works like generating a total for each row in a table. The result is stored in a new field, which is named Total by default. To use a different field name, provide the fieldname argument. Default value is true.

col : Syntax: col=<bool> . Indicates whether to append a new event—called a summary event—to the end of the event list. This summary event shows the total for each field across all events, similar to calculating column totals in a table. Default is false.

fieldname : Syntax: fieldname=<field> . Specifies the name of the field that stores the calculated sum of the field-list for each event. This argument is only applicable when row=true. Default is Total

field-list : Syntax: <field> ... . One or more numeric fields separated by spaces. Only the fields listed in the are included in the sum. If no is provided, all numeric fields are summed by default.

labelfield : Syntax: labelfield=<field> . Specifies a field to use as the label for the summary event. This argument is only applicable when col=true."

To use an existing field from your result set, provide its name as the value for the labelfield argument. For example, if the field is named salary, specify labelfield=salary. If no existing field matches the labelfield value, a new field is created using that value.

label: Syntax: label=<string>. Specifies a row label for the summary event.

If the labelfield argument refers to an existing field in the result set, the label value appears in that field for the summary row.

If the labelfield argument creates a new field, the label is placed in that new field in the summary event row. Default label is Total.

command addcoltotals: Add totals across columns of each row to show total in a new field.

addcoltotals: options
Optional Arguments
<field-list>
Syntax: <field> ... . A space-delimited list of valid field names. addcoltotals calculates sums only for the fields you include in this list. By default, the command calculates the sum for all fields.

labelfield: Syntax: labelfield=<fieldname>. Field name to add to the result set.

label : Syntax: label=<string> . Used together with the labelfield argument to add a label to the summary event. If labelfield is not specified, the label argument has no effect. Default label is Total.

Related Issues

Resolves #4607 [#4607 ]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[asifabashar]

@penghuo Please review

[coderabbitai]

Actionable comments posted: 0

♻️ Duplicate comments (7)

docs/user/ppl/cmd/addtotals.md (1)

59-69: Fix contradictory description for Example 2.

The description states "row=true value was used by default when not specified," but line 68 explicitly sets row=false. The description also mentions "after a stats command," but no stats command is present in the example.

Suggested revision:

-The example shows adding totals after a stats command where final
-summary event label is 'Sum' and row=true value was used by default
-when not specified. It also added new field specified by labelfield as
-it did not match existing field.
+The example shows adding column totals with row=false (disabling row
+totals). The summary event label is 'Sum' and a new labelfield 'Total'
+is created to display the label in the summary row.

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java (1)

104-143: Remove duplicate test method.

This test testAddColTotalsAllFields is identical to testAddColTotals (lines 19-59) - both use the same PPL query "source=EMP | fields DEPTNO, SAL, JOB | addcoltotals " and verify the same logical plan, results, and Spark SQL. Remove this duplicate or differentiate the test cases.

-  @Test
-  public void testAddColTotalsAllFields() throws IOException {
-    String ppl = "source=EMP  | fields DEPTNO, SAL, JOB | addcoltotals  ";
-    RelNode root = getRelNode(ppl);
-    ...
-  }

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java (1)

337-381: Re-enable or remove commented verifyLogical in complex options test

testAddTotalsWithAllOptionsIncludingFieldname builds expectedLogical but keeps // verifyLogical(root, expectedLogical); commented. This leaves the logical-plan path for the most complex addtotals combo (row+col, custom fieldname, labelfield) unvalidated.

Either:

• Uncomment verifyLogical(root, expectedLogical);, or
• Remove expectedLogical and add a brief TODO referencing a known mismatch if the plan is intentionally not asserted.

This keeps the test aligned with the “verify SQL generation/optimization paths” goal.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java (4)

19-27: Add class-level JavaDoc for public IT class

This public integration test class has no JavaDoc. Per the project guidelines, public classes should include JavaDoc describing their purpose and scope.

Consider adding a brief comment such as:

+/**
+ * Integration tests for PPL addtotals command with Calcite engine enabled.
+ * Covers row and column totals, custom labels/fieldnames, and interaction
+ * with upstream commands like stats.
+ */
 public class CalciteAddTotalsCommandIT extends PPLIntegTestCase {

---

104-106: Limit isNumeric visibility and avoid fully qualified self-call

isNumeric is only used inside this test class, yet it is declared public static and one call still uses the fully qualified class name.

You can tighten and simplify:

-  public static boolean isNumeric(String str) {
+  private static boolean isNumeric(String str) {
     return str != null && str.matches("-?\\d+(\\.\\d+)?");
   }

and in testAddTotalsRowFieldsNonNumeric:

-        } else if (value instanceof String) {
-          if (org.opensearch.sql.calcite.remote.CalciteAddTotalsCommandIT.isNumeric(
-              (String) value)) {
+        } else if (value instanceof String) {
+          if (isNumeric((String) value)) {
             cRowTotal = cRowTotal.add(new BigDecimal((String) (value)));
           }
         }

This keeps the helper scoped to the class and removes redundant qualification.

Also applies to: 200-211

---

248-259: Fix inconsistency between comment and assertion in testAddTotalsWithNoData

The comment states:

// Should still have totals row even with no input data

but the assertion expects zero rows:

assertEquals(0, dataRows.length()); // Only totals row

These are contradictory. Decide which behavior is intended and adjust:

• If addtotals should not emit any rows when the input is empty (row=true, col=false), update the comment to reflect that (and probably drop “Only totals row” from the inline comment).
• If it should emit a totals row, change the assertion to assertEquals(1, dataRows.length()) and add checks on that row’s contents.

Right now the test documents one behavior and asserts another.

---

278-299: Strengthen row=false tests to assert schema instead of scanning for label strings

Both testAddTotalsWithRowFalse and testAddTotalsWithFieldnameNoRow try to ensure no totals row is added when row=false by scanning all cell values for the literal "Total" / "CustomSum". A totals row, however, would contain numeric totals, not those label strings, so these tests could still pass even if a numeric totals column is incorrectly appended.

A more robust check is to assert the schema does not contain the totals column when row=false:

  @Test
  public void testAddTotalsWithRowFalse() throws IOException {
    var result =
        executeQuery(
            String.format(
                "source=%s | where age > 25 | fields age, balance | addtotals row=false",
                TEST_INDEX_ACCOUNT));

-    // With row=false, should not append totals row
-    var dataRows = result.getJSONArray("datarows");
-
-    // Verify that no totals row was added - all rows should have actual data
-    for (int i = 0; i < dataRows.length(); i++) {
-      var row = dataRows.getJSONArray(i);
-      // None of these rows should have "Total" label
-      for (int j = 0; j < row.length(); j++) {
-        if (!row.isNull(j) && row.get(j).equals("Total")) {
-          fail("Found totals row when row=false was specified");
-        }
-      }
-    }
+    // With row=false, schema should not include the Total column
+    verifySchema(result, schema("age", "bigint"), schema("balance", "bigint"));
  }

and similarly for testAddTotalsWithFieldnameNoRow:

-    // With row=false, should not append totals row regardless of fieldname
-    var dataRows = result.getJSONArray("datarows");
-
-    // Verify that no totals row was added
-    for (int i = 0; i < dataRows.length(); i++) {
-      var row = dataRows.getJSONArray(i);
-      // None of these rows should have "CustomSum" label
-      for (int j = 0; j < row.length(); j++) {
-        if (!row.isNull(j) && row.get(j).equals("CustomSum")) {
-          fail("Found totals row when row=false was specified");
-        }
-      }
-    }
+    // With row=false, schema should not include CustomSum column regardless of fieldname
+    verifySchema(result, schema("age", "bigint"), schema("balance", "bigint"));

This directly verifies the absence of any row-level totals column when row=false.

Also applies to: 347-369

🧹 Nitpick comments (8)

docs/user/ppl/cmd/addtotals.md (1)

10-12: Remove unnecessary escape characters.

The escaped apostrophes (\') are not needed in Markdown and appear as literal backslashes in some renderers.

-the case of no field-list specified.
+the case of no field-list specified.

Line 11: Change it\'s to it's.
Line 28: Change \"Total\" to "Total".

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java (1)

1416-1453: LGTM with optional refactor suggestion.

The visitor implementations correctly parse the command context, extract field lists and options, and construct the appropriate AST nodes. The pattern follows existing visitor methods in the file.

Consider extracting the common logic into a shared helper method to reduce duplication, as both methods have nearly identical structure:

private Pair<List<Field>, Map<String, Literal>> parseFieldsAndOptions(
    FieldListContext fieldListCtx,
    List<? extends ParserRuleContext> optionContexts) {
  List<Field> fieldList = fieldListCtx != null ? getFieldList(fieldListCtx) : new ArrayList<>();
  ImmutableMap.Builder<String, Literal> cmdOptionsBuilder = ImmutableMap.builder();
  optionContexts.forEach(option -> {
    String argName = option.children.get(0).toString();
    Literal value = (Literal) internalVisitExpression(option.children.get(2));
    cmdOptionsBuilder.put(argName, value);
  });
  return Pair.of(fieldList, cmdOptionsBuilder.build());
}

ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java (1)

786-801: Consider consistency with other command anonymization patterns.

The label option could contain user-provided text that other commands would mask. For example, visitChart masks options with MASK_LITERAL, and visitRex masks pattern values.

However, I acknowledge your previous response that these are column-level metadata rather than actual data values. If the decision is intentional, consider adding a brief comment explaining why these options are not anonymized, to help future maintainers understand the design decision.

 public void appendAddTotalsOptionParameters(
     List<Field> fieldList, java.util.Map<String, Literal> options, StringBuilder builder) {
+  // Note: Options (row, col, label, labelfield, fieldname) are not anonymized as they
+  // represent column-level metadata/configuration, not user data values.
   if (!fieldList.isEmpty()) {

ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java (1)

918-948: Consider adding negative test cases and addcoltotals syntax tests.

Per the retrieved learnings, grammar rules should be tested with both positive and negative cases. The current tests only cover positive cases for addtotals. Consider adding:

1. Negative test cases for invalid addtotals syntax (e.g., invalid option values, malformed syntax)
2. Syntax parser tests for addcoltotals command to match the coverage for addtotals

Example negative test:

@Test
public void testAddTotalsCommandWithInvalidOptionShouldFail() {
    exceptionRule.expect(RuntimeException.class);
    new PPLSyntaxParser().parse("source=t | addtotals row=invalid");
}

Based on learnings, test new grammar rules with positive and negative cases for PPL parser changes.

core/src/main/java/org/opensearch/sql/ast/tree/AddTotals.java (1)

20-45: Add class-level JavaDoc for the public AST node.

Per coding guidelines, all public classes should have proper JavaDoc. This AST node would benefit from documentation explaining its purpose, the meaning of its fields, and usage within the addtotals command processing.

+/**
+ * AST node representing the PPL addtotals command.
+ *
+ * <p>The addtotals command computes per-event (row) totals and optionally column totals
+ * across numeric fields.</p>
+ *
+ * @see AddColTotals for the related addcoltotals command
+ */
 @Getter
 @Setter
 @ToString
 @EqualsAndHashCode(callSuper = false)
 @RequiredArgsConstructor
 public class AddTotals extends UnresolvedPlan {

As per coding guidelines, all public classes and methods must have proper JavaDoc.

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java (1)

13-17: Add class-level JavaDoc.

Per coding guidelines, public test classes should have documentation explaining what is being tested.

+/**
+ * Tests for the PPL addcoltotals command using Calcite backend.
+ * Verifies logical plans, execution results, and Spark SQL translation.
+ */
 public class CalcitePPLAddColTotalsTest extends CalcitePPLAbstractTest {

As per coding guidelines, all public classes and methods must have proper JavaDoc.

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java (1)

73-118: Tighten helper usage and make label verification less positional

Two small cleanups:

• The call org.opensearch.sql.calcite.remote.CalciteAddColTotalsCommandIT.isNumeric(...) can be simplified to isNumeric(...) since the helper is in the same class. This improves readability and avoids over‑qualification.
• verifyColTotals assumes the summary label is always in the last column (total_row.getString(total_row.length() - 1)). That matches current tests (where labelfield is last), but is brittle if future tests use a non‑terminal label field. Passing the label field index (or name) into verifyColTotals and looking it up explicitly would make the assertion more robust.

These are non‑blocking test refactors.

Also applies to: 99-104

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java (1)

2509-2777: Minor cleanups: fix comment wording and drop unused context parameters in helpers

A few small improvements to keep this area tidy:

• In visitAddColTotals, the inline comment says // Parse options from the AddTotals node but this method handles AddColTotals. Updating the wording avoids confusion:

-    // Parse options from the AddTotals node
+    // Parse options from the AddColTotals node

• getAllNumericFields and isNumericField both accept CalcitePlanContext context but don’t use it. You can simplify the signatures and call sites:

-  private List<Field> getAllNumericFields(RelNode relNode, CalcitePlanContext context) {
+  private List<Field> getAllNumericFields(RelNode relNode) {

-    if (fieldsToAggregate.isEmpty()) {
-      fieldsToAggregate = getAllNumericFields(originalData, context);
+    if (fieldsToAggregate.isEmpty()) {
+      fieldsToAggregate = getAllNumericFields(originalData);
     }

and:

-  private boolean isNumericField(RexNode rexNode, CalcitePlanContext context) {
+  private boolean isNumericField(RexNode rexNode) {
     return rexNode.getType().getSqlTypeName().getFamily() == SqlTypeFamily.NUMERIC;
   }

with call sites updated accordingly.

• Given buildAddRowTotalAggregate is public and has a non-trivial contract (row vs col, fieldname, label/labelfield behavior), adding a short JavaDoc summarizing its parameters and the row/column totals behavior would help future maintainers.

These are non-breaking refactors and comment fixes.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d579e7b and 2391fd8.

📒 Files selected for processing (25)

• core/src/main/java/org/opensearch/sql/analysis/Analyzer.java (2 hunks)
• core/src/main/java/org/opensearch/sql/ast/AbstractNodeVisitor.java (2 hunks)
• core/src/main/java/org/opensearch/sql/ast/tree/AddColTotals.java (1 hunks)
• core/src/main/java/org/opensearch/sql/ast/tree/AddTotals.java (1 hunks)
• core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java (2 hunks)
• docs/category.json (1 hunks)
• docs/user/ppl/cmd/addcoltotals.md (1 hunks)
• docs/user/ppl/cmd/addtotals.md (1 hunks)
• docs/user/ppl/index.md (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/ppl/NewAddedCommandsIT.java (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java (2 hunks)
• integ-test/src/test/resources/expectedOutput/calcite/explain_add_col_totals.yaml (1 hunks)
• integ-test/src/test/resources/expectedOutput/calcite/explain_add_totals.yaml (1 hunks)
• ppl/src/main/antlr/OpenSearchPPLLexer.g4 (2 hunks)
• ppl/src/main/antlr/OpenSearchPPLParser.g4 (4 hunks)
• ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java (2 hunks)
• ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java (2 hunks)
• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java (1 hunks)
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java (1 hunks)
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java (1 hunks)
• ppl/src/test/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizerTest.java (1 hunks)

✅ Files skipped from review due to trivial changes (1)

• docs/user/ppl/cmd/addcoltotals.md

🚧 Files skipped from review as they are similar to previous changes (10)

• ppl/src/test/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizerTest.java
• ppl/src/main/antlr/OpenSearchPPLLexer.g4
• core/src/main/java/org/opensearch/sql/ast/AbstractNodeVisitor.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteExplainIT.java
• integ-test/src/test/java/org/opensearch/sql/ppl/NewAddedCommandsIT.java
• integ-test/src/test/resources/expectedOutput/calcite/explain_add_totals.yaml
• core/src/main/java/org/opensearch/sql/ast/tree/AddColTotals.java
• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java
• docs/category.json
• core/src/main/java/org/opensearch/sql/analysis/Analyzer.java

🧰 Additional context used

📓 Path-based instructions (7)

**/*.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

**/*.java: Use PascalCase for class names (e.g., QueryExecutor)
Use camelCase for method and variable names (e.g., executeQuery)
Use UPPER_SNAKE_CASE for constants (e.g., MAX_RETRY_COUNT)
Keep methods under 20 lines with single responsibility
All public classes and methods must have proper JavaDoc
Use specific exception types with meaningful messages for error handling
Prefer Optional<T> for nullable returns in Java
Avoid unnecessary object creation in loops
Use StringBuilder for string concatenation in loops
Validate all user inputs, especially queries
Sanitize data before logging to prevent injection attacks
Use try-with-resources for proper resource cleanup in Java
Maintain Java 11 compatibility when possible for OpenSearch 2.x
Document Calcite-specific workarounds in code

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java
• core/src/main/java/org/opensearch/sql/ast/tree/AddTotals.java
• core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java
• ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java
• ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

---

⚙️ CodeRabbit configuration file

**/*.java: - Verify Java naming conventions (PascalCase for classes, camelCase for methods/variables)

• Check for proper JavaDoc on public classes and methods
• Flag redundant comments that restate obvious code
• Ensure methods are under 20 lines with single responsibility
• Verify proper error handling with specific exception types
• Check for Optional usage instead of null returns
• Validate proper use of try-with-resources for resource management

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java
• core/src/main/java/org/opensearch/sql/ast/tree/AddTotals.java
• core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java
• ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java
• ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

integ-test/**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

End-to-end scenarios need integration tests in integ-test/ module

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java

---

⚙️ CodeRabbit configuration file

integ-test/**/*IT.java: - Verify integration tests are in correct module (integ-test/)

• Check tests can be run with ./gradlew :integ-test:integTest
• Ensure proper test data setup and teardown
• Validate end-to-end scenario coverage

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java

**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

Name integration tests with *IT.java suffix in OpenSearch SQL

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java

**/test/**/*.java

⚙️ CodeRabbit configuration file

**/test/**/*.java: - Verify test coverage for new business logic

• Check test naming follows conventions (*Test.java for unit, *IT.java for integration)
• Ensure tests are independent and don't rely on execution order
• Validate meaningful test data that reflects real-world scenarios
• Check for proper cleanup of test resources

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java

**/calcite/**/*.java

⚙️ CodeRabbit configuration file

**/calcite/**/*.java: - Follow existing patterns in CalciteRelNodeVisitor and CalciteRexNodeVisitor

• Verify SQL generation and optimization paths
• Document any Calcite-specific workarounds
• Test compatibility with Calcite version constraints

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java
• core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java

**/*Test.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

**/*Test.java: All new business logic requires unit tests
Name unit tests with *Test.java suffix in OpenSearch SQL

Files:

• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java

**/ppl/**/*.java

⚙️ CodeRabbit configuration file

**/ppl/**/*.java: - For PPL parser changes, verify grammar tests with positive/negative cases

• Check AST generation for new syntax
• Ensure corresponding AST builder classes are updated
• Validate edge cases and boundary conditions

Files:

• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java
• ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java
• ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

🧠 Learnings (12)

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Test SQL generation and optimization paths for Calcite integration changes

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java
• integ-test/src/test/resources/expectedOutput/calcite/explain_add_col_totals.yaml
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java
• core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*IT.java : Name integration tests with `*IT.java` suffix in OpenSearch SQL

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*.java : Document Calcite-specific workarounds in code

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddTotalsTest.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java
• core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: For PPL command PRs, refer docs/dev/ppl-commands.md and verify the PR satisfies the checklist

Applied to files:

• docs/user/ppl/cmd/addtotals.md
• docs/user/ppl/index.md

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to integ-test/**/*IT.java : End-to-end scenarios need integration tests in `integ-test/` module

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*.java : All public classes and methods must have proper JavaDoc

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Test new grammar rules with positive and negative cases for PPL parser changes

Applied to files:

• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*Test.java : Name unit tests with `*Test.java` suffix in OpenSearch SQL

Applied to files:

• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*Test.java : All new business logic requires unit tests

Applied to files:

• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Update corresponding AST builder classes when making PPL grammar changes

Applied to files:

• ppl/src/main/antlr/OpenSearchPPLParser.g4
• ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

📚 Learning: 2025-12-11T05:27:39.831Z

Learnt from: LantaoJin
Repo: opensearch-project/sql PR: 0
File: :0-0
Timestamp: 2025-12-11T05:27:39.831Z
Learning: In opensearch-project/sql, for SEMI and ANTI join types in CalciteRelNodeVisitor.java, the `max` option has no effect because these join types only use the left side to filter records based on the existence of matches in the right side. The join results are identical regardless of max value (max=1, max=2, or max=∞). The early return for SEMI/ANTI joins before processing the `max` option is intentional and correct behavior.

Applied to files:

• integ-test/src/test/resources/expectedOutput/calcite/explain_add_col_totals.yaml

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Follow existing patterns in `CalciteRelNodeVisitor` and `CalciteRexNodeVisitor` for Calcite integration

Applied to files:

• core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java
• ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java
• ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

🧬 Code graph analysis (2)

core/src/main/java/org/opensearch/sql/ast/tree/AddTotals.java (2)

core/src/main/java/org/opensearch/sql/ast/tree/AddColTotals.java (1)

• Getter (22-47)

core/src/main/java/org/opensearch/sql/ast/AbstractNodeVisitor.java (1)

• AbstractNodeVisitor (94-464)

ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java (1)

legacy/src/main/java/org/opensearch/sql/legacy/utils/StringUtils.java (1)

• StringUtils (17-117)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (27)

• GitHub Check: security-it-linux (25)
• GitHub Check: security-it-linux (21)
• GitHub Check: build-linux (21, doc)
• GitHub Check: bwc-tests-full-restart (21)
• GitHub Check: build-linux (25, doc)
• GitHub Check: build-linux (25, integration)
• GitHub Check: build-linux (21, integration)
• GitHub Check: build-linux (25, unit)
• GitHub Check: bwc-tests-rolling-upgrade (21)
• GitHub Check: build-linux (21, unit)
• GitHub Check: bwc-tests-full-restart (25)
• GitHub Check: bwc-tests-rolling-upgrade (25)
• GitHub Check: security-it-windows-macos (windows-latest, 21)
• GitHub Check: security-it-windows-macos (windows-latest, 25)
• GitHub Check: security-it-windows-macos (macos-14, 21)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, unit)
• GitHub Check: build-windows-macos (macos-14, 21, unit)
• GitHub Check: build-windows-macos (macos-14, 21, integration)
• GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, unit)
• GitHub Check: build-windows-macos (macos-14, 25, integration)
• GitHub Check: build-windows-macos (macos-14, 25, unit)
• GitHub Check: build-windows-macos (macos-14, 25, doc)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, integration)
• GitHub Check: build-windows-macos (macos-14, 21, doc)
• GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, integration)
• GitHub Check: test-sql-cli-integration (21)
• GitHub Check: CodeQL-Scan (java)

🔇 Additional comments (10)

docs/user/ppl/index.md (1)

81-82: Verify "stable" status for newly introduced commands.

Both addtotals and addcoltotals are marked as "stable (since 3.4)" on their introduction. Most other commands introduced in version 3.x are initially marked as "experimental" (e.g., streamstats, chart, multisearch). Please verify this is intentional or consider marking them as "experimental (since 3.4)" to allow for potential parameter adjustments based on user feedback.

integ-test/src/test/java/org/opensearch/sql/calcite/CalciteNoPushdownIT.java (1)

24-25: LGTM!

The new integration test classes are correctly added to the no-pushdown test suite, following the existing naming convention and alphabetical ordering.

ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java (1)

803-819: LGTM!

The visitor implementations correctly build the anonymized query strings for both addtotals and addcoltotals commands, following the established patterns in the class.

core/src/main/java/org/opensearch/sql/ast/tree/AddTotals.java (1)

30-34: LGTM on the attach pattern.

The attach method correctly follows the established pattern used in other UnresolvedPlan implementations like AddColTotals, returning this to allow method chaining.

integ-test/src/test/resources/expectedOutput/calcite/explain_add_col_totals.yaml (1)

1-18: LGTM - Expected explain plan structure is correct.

The YAML correctly captures the expected Calcite logical and physical plans for addcoltotals. The union-based approach (original scan + aggregated totals) aligns with the command's semantics, and the pushdown context in the physical plan shows proper optimization.

ppl/src/main/antlr/OpenSearchPPLParser.g4 (2)

587-608: Grammar rules look good but note potential ambiguity.

The grammar correctly handles flexible ordering of fieldList and options through two alternatives. However, this creates potential ambiguity that ANTLR will resolve via alternative ordering. Consider if a single rule with mixed ordering using (fieldList | addtotalsOption)* would be cleaner, though the current approach is functional.

The options correctly use stringLiteral for text values and booleanLiteral for boolean flags.

---

1644-1650: LGTM - Keywords correctly added to searchableKeyWord.

Adding ADDTOTALS, ADDCOLTOTALS, LABEL, LABELFIELD, FIELDNAME, ROW, and COL to searchableKeyWord allows these tokens to be used as field names in queries, preventing parse errors when fields have these names.

ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLAddColTotalsTest.java (1)

19-59: LGTM - Comprehensive test coverage for addcoltotals.

The test suite effectively covers:

• Basic addcoltotals behavior (all numeric fields summed)
• Single field specification
• Multiple field specification
• Label and labelfield options (creating new column or using existing)
• Changed ordering of options and field list

Each test verifies logical plan structure, execution results, and Spark SQL translation, providing good confidence in the implementation.

Also applies to: 145-184, 186-230, 232-273, 275-316, 318-376

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java (1)

34-71: addcoltotals IT coverage looks solid

The init setup and the first two tests (testAddColTotalsTotalWithTotalField, testAddColTotalsRowWithSpecificFields) correctly exercise default and field-list behavior, schema, and totals-row correctness via verifyColTotals. No functional issues spotted here.

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java (1)

2504-2715: AddTotals/AddColTotals Calcite translation looks correct and well-covered

The new visitors (visitAddColTotals, visitAddTotals) and buildAddRowTotalAggregate correctly implement:

• Default row=true, col=false for addtotals with configurable fieldname.
• col=true behavior that appends a single summary row with per-column totals and optional label/labelfield.
• addcoltotals as a pure column-totals command (no per-row totals), matching the dedicated ITs.
• Numeric-field selection via explicit field list or “all numeric fields”, with safe widening in getAggregateDataTypeFieldRef to avoid aggregation type issues.
• Label/labelField semantics exercised by the various Calcite and remote ITs.

No functional or type-safety issues stand out in this block.

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (2)

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java (2)

77-118: Refactor to honor single-responsibility and 20-line guideline.

The verifyColTotals() method is 41 lines and performs multiple responsibilities: accumulating totals from data rows, verifying computed totals match expected values, and optionally checking summary event labels.

Per coding guidelines, methods should stay under 20 lines with single responsibility.

Consider extracting separate methods:

+  private BigDecimal[] computeColumnTotals(
+      org.json.JSONArray dataRows, List<Integer> fieldIndexes) {
+    BigDecimal[] columnTotals = new BigDecimal[fieldIndexes.size()];
+    for (int i = 0; i < dataRows.length() - 1; i++) {
+      var row = dataRows.getJSONArray(i);
+      for (int j = 0; j < fieldIndexes.size(); j++) {
+        int colIndex = fieldIndexes.get(j);
+        if (columnTotals[j] == null) {
+          columnTotals[j] = new BigDecimal(0);
+        }
+        columnTotals[j] = columnTotals[j].add(extractNumericValue(row, colIndex));
+      }
+    }
+    return columnTotals;
+  }
+
+  private BigDecimal extractNumericValue(org.json.JSONArray row, int colIndex) {
+    Object value = row.isNull(colIndex) ? 0 : row.get(colIndex);
+    if (value instanceof Integer) {
+      return new BigDecimal((Integer) value);
+    } else if (value instanceof Double) {
+      return new BigDecimal((Double) value);
+    } else if (value instanceof BigDecimal) {
+      return (BigDecimal) value;
+    } else if (value instanceof String && isNumeric((String) value)) {
+      return new BigDecimal((String) value);
+    }
+    return BigDecimal.ZERO;
+  }
+
   private void verifyColTotals(
-      org.json.JSONArray dataRows, List<Integer> field_indexes, String finalSummaryEventLevel) {
-
-    BigDecimal[] cColTotals = new BigDecimal[field_indexes.size()];
-    for (int i = 0; i < dataRows.length() - 1; i++) {
-      var row = dataRows.getJSONArray(i);
-
-      // Iterate through each field in the row
-      for (int j = 0; j < field_indexes.size(); j++) {
-
-        int colIndex = field_indexes.get(j);
-        if (cColTotals[j] == null) {
-          cColTotals[j] = new BigDecimal(0);
-        }
-        Object value = row.isNull(colIndex) ? 0 : row.get(colIndex);
-        if (value instanceof Integer) {
-          cColTotals[j] = cColTotals[j].add(new BigDecimal((Integer) (value)));
-        } else if (value instanceof Double) {
-          cColTotals[j] = cColTotals[j].add(new BigDecimal((Double) (value)));
-        } else if (value instanceof BigDecimal) {
-          cColTotals[j] = cColTotals[j].add((BigDecimal) value);
-
-        } else if (value instanceof String) {
-          if (org.opensearch.sql.calcite.remote.CalciteAddColTotalsCommandIT.isNumeric(
-              (String) value)) {
-            cColTotals[j] = cColTotals[j].add(new BigDecimal((String) (value)));
-          }
-        }
-      }
-    }
+      org.json.JSONArray dataRows, List<Integer> fieldIndexes, String finalSummaryEventLevel) {
+    BigDecimal[] columnTotals = computeColumnTotals(dataRows, fieldIndexes);
     var total_row = dataRows.getJSONArray((dataRows.length() - 1));
-    for (int j = 0; j < field_indexes.size(); j++) {
-      int colIndex = field_indexes.get(j);
+    for (int j = 0; j < fieldIndexes.size(); j++) {
+      int colIndex = fieldIndexes.get(j);
       BigDecimal foundTotal = total_row.getBigDecimal(colIndex);
-      assertEquals(foundTotal.doubleValue(), cColTotals[j].doubleValue(), 0.000001);
+      assertEquals(foundTotal.doubleValue(), columnTotals[j].doubleValue(), 0.000001);
     }
     if (finalSummaryEventLevel != null) {
       String foundSummaryEventLabel = total_row.getString(total_row.length() - 1);
-
       assertEquals(foundSummaryEventLabel, finalSummaryEventLevel);
     }
   }

As per coding guidelines, keep methods under 20 lines with single responsibility.

---

48-49: Fix variable naming to follow camelCase convention.

The variable field_indexes uses snake_case instead of camelCase throughout the file. Additionally, line 100 uses an unnecessary fully qualified class name.

Per coding guidelines, use camelCase for variable names.

Apply these changes:

1. Rename field_indexes to fieldIndexes throughout
2. Rename cColTotals to columnTotals for clarity
3. Remove fully qualified class name on line 100

-    ArrayList<Integer> field_indexes = new ArrayList<>();
-    field_indexes.add(0);
+    ArrayList<Integer> fieldIndexes = new ArrayList<>();
+    fieldIndexes.add(0);

   private void verifyColTotals(
-      org.json.JSONArray dataRows, List<Integer> field_indexes, String finalSummaryEventLevel) {
+      org.json.JSONArray dataRows, List<Integer> fieldIndexes, String finalSummaryEventLevel) {
 
-    BigDecimal[] cColTotals = new BigDecimal[field_indexes.size()];
+    BigDecimal[] columnTotals = new BigDecimal[fieldIndexes.size()];

-        } else if (value instanceof String) {
-          if (org.opensearch.sql.calcite.remote.CalciteAddColTotalsCommandIT.isNumeric(
-              (String) value)) {
-            cColTotals[j] = cColTotals[j].add(new BigDecimal((String) (value)));
-          }
-        }
+        } else if (value instanceof String && isNumeric((String) value)) {
+          columnTotals[j] = columnTotals[j].add(new BigDecimal((String) value));
+        }

As per coding guidelines, use camelCase for method and variable names.

Also applies to: 67-68, 78-80, 100-102, 133-134, 156-157, 193-194

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c874251 and 2d18f7e.

📒 Files selected for processing (3)

• core/src/main/java/org/opensearch/sql/ast/tree/AddColTotals.java (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java (1 hunks)
• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (2)

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java
• core/src/main/java/org/opensearch/sql/ast/tree/AddColTotals.java

🧰 Additional context used

📓 Path-based instructions (5)

**/*.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

**/*.java: Use PascalCase for class names (e.g., QueryExecutor)
Use camelCase for method and variable names (e.g., executeQuery)
Use UPPER_SNAKE_CASE for constants (e.g., MAX_RETRY_COUNT)
Keep methods under 20 lines with single responsibility
All public classes and methods must have proper JavaDoc
Use specific exception types with meaningful messages for error handling
Prefer Optional<T> for nullable returns in Java
Avoid unnecessary object creation in loops
Use StringBuilder for string concatenation in loops
Validate all user inputs, especially queries
Sanitize data before logging to prevent injection attacks
Use try-with-resources for proper resource cleanup in Java
Maintain Java 11 compatibility when possible for OpenSearch 2.x
Document Calcite-specific workarounds in code

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

---

⚙️ CodeRabbit configuration file

**/*.java: - Verify Java naming conventions (PascalCase for classes, camelCase for methods/variables)

• Check for proper JavaDoc on public classes and methods
• Flag redundant comments that restate obvious code
• Ensure methods are under 20 lines with single responsibility
• Verify proper error handling with specific exception types
• Check for Optional usage instead of null returns
• Validate proper use of try-with-resources for resource management

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

integ-test/**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

End-to-end scenarios need integration tests in integ-test/ module

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

---

⚙️ CodeRabbit configuration file

integ-test/**/*IT.java: - Verify integration tests are in correct module (integ-test/)

• Check tests can be run with ./gradlew :integ-test:integTest
• Ensure proper test data setup and teardown
• Validate end-to-end scenario coverage

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

Name integration tests with *IT.java suffix in OpenSearch SQL

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

**/test/**/*.java

⚙️ CodeRabbit configuration file

**/test/**/*.java: - Verify test coverage for new business logic

• Check test naming follows conventions (*Test.java for unit, *IT.java for integration)
• Ensure tests are independent and don't rely on execution order
• Validate meaningful test data that reflects real-world scenarios
• Check for proper cleanup of test resources

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

**/calcite/**/*.java

⚙️ CodeRabbit configuration file

**/calcite/**/*.java: - Follow existing patterns in CalciteRelNodeVisitor and CalciteRexNodeVisitor

• Verify SQL generation and optimization paths
• Document any Calcite-specific workarounds
• Test compatibility with Calcite version constraints

Files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

🧠 Learnings (5)

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*IT.java : Name integration tests with `*IT.java` suffix in OpenSearch SQL

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Test SQL generation and optimization paths for Calcite integration changes

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to integ-test/**/*IT.java : End-to-end scenarios need integration tests in `integ-test/` module

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*.java : Document Calcite-specific workarounds in code

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*.java : All public classes and methods must have proper JavaDoc

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteAddColTotalsCommandIT.java

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (16)

• GitHub Check: security-it-linux (25)
• GitHub Check: security-it-linux (21)
• GitHub Check: security-it-windows-macos (macos-14, 21)
• GitHub Check: security-it-windows-macos (windows-latest, 21)
• GitHub Check: security-it-windows-macos (windows-latest, 25)
• GitHub Check: security-it-windows-macos (macos-14, 25)
• GitHub Check: build-windows-macos (macos-14, 25, doc)
• GitHub Check: build-windows-macos (macos-14, 25, integration)
• GitHub Check: build-windows-macos (macos-14, 25, unit)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, integration)
• GitHub Check: build-windows-macos (macos-14, 21, unit)
• GitHub Check: build-windows-macos (macos-14, 21, integration)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, unit)
• GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, integration)
• GitHub Check: build-windows-macos (macos-14, 21, doc)
• GitHub Check: test-sql-cli-integration (21)

[coderabbitai]

Actionable comments posted: 0

♻️ Duplicate comments (1)

integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java (1)

252-265: Fix stale query_string copy‑paste comments for the new tests

Both new tests exercise addtotals/addcoltotals, but the inline comments still say “Test query_string without fields parameter on remote cluster”, which is misleading and was already flagged in an earlier review. Updating them (and slightly tightening the Javadoc) will make the intent clear:

   @Test
   public void testCrossClusterAddTotals() throws IOException {
-    // Test query_string without fields parameter on remote cluster
+    // Verify addtotals works on the remote bank index with row totals over age.
@@
-  /** CrossClusterSearchIT Test for addcoltotals. */
+  /** CrossClusterSearchIT test for addcoltotals on the remote bank index. */
   @Test
   public void testCrossClusterAddColTotals() throws IOException {
-    // Test query_string without fields parameter on remote cluster
+    // Verify addcoltotals works on the remote bank index with column totals for age and balance.

🧹 Nitpick comments (1)

integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java (1)

250-259: Cross‑cluster addtotals scenario looks solid; optionally assert schema

The query exercises addtotals on the remote bank index and verifyDataRows(result, rows("Nanette", 28, 28)) confirms the extra totals column is present and populated as expected for the first row. If you want to harden this further against regressions, consider also asserting the column names (including the totals field) with verifyColumn(...), similar to neighbouring tests.

📜 Review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2d18f7e and ad94472.

📒 Files selected for processing (1)

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java (1 hunks)

🧰 Additional context used

📓 Path-based instructions (4)

**/*.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

**/*.java: Use PascalCase for class names (e.g., QueryExecutor)
Use camelCase for method and variable names (e.g., executeQuery)
Use UPPER_SNAKE_CASE for constants (e.g., MAX_RETRY_COUNT)
Keep methods under 20 lines with single responsibility
All public classes and methods must have proper JavaDoc
Use specific exception types with meaningful messages for error handling
Prefer Optional<T> for nullable returns in Java
Avoid unnecessary object creation in loops
Use StringBuilder for string concatenation in loops
Validate all user inputs, especially queries
Sanitize data before logging to prevent injection attacks
Use try-with-resources for proper resource cleanup in Java
Maintain Java 11 compatibility when possible for OpenSearch 2.x
Document Calcite-specific workarounds in code

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

---

⚙️ CodeRabbit configuration file

**/*.java: - Verify Java naming conventions (PascalCase for classes, camelCase for methods/variables)

• Check for proper JavaDoc on public classes and methods
• Flag redundant comments that restate obvious code
• Ensure methods are under 20 lines with single responsibility
• Verify proper error handling with specific exception types
• Check for Optional usage instead of null returns
• Validate proper use of try-with-resources for resource management

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

integ-test/**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

End-to-end scenarios need integration tests in integ-test/ module

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

---

⚙️ CodeRabbit configuration file

integ-test/**/*IT.java: - Verify integration tests are in correct module (integ-test/)

• Check tests can be run with ./gradlew :integ-test:integTest
• Ensure proper test data setup and teardown
• Validate end-to-end scenario coverage

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

**/*IT.java

📄 CodeRabbit inference engine (.rules/REVIEW_GUIDELINES.md)

Name integration tests with *IT.java suffix in OpenSearch SQL

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

**/test/**/*.java

⚙️ CodeRabbit configuration file

**/test/**/*.java: - Verify test coverage for new business logic

• Check test naming follows conventions (*Test.java for unit, *IT.java for integration)
• Ensure tests are independent and don't rely on execution order
• Validate meaningful test data that reflects real-world scenarios
• Check for proper cleanup of test resources

Files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

🧠 Learnings (2)

📓 Common learnings

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Verify AST generation for new PPL parser syntax

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Test SQL generation and optimization paths for Calcite integration changes

📚 Learning: 2025-12-02T17:27:55.938Z

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*IT.java : Name integration tests with `*IT.java` suffix in OpenSearch SQL

Applied to files:

• integ-test/src/test/java/org/opensearch/sql/security/CrossClusterSearchIT.java

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (28)

• GitHub Check: bwc-tests-full-restart (25)
• GitHub Check: build-linux (21, integration)
• GitHub Check: build-linux (25, unit)
• GitHub Check: build-linux (21, unit)
• GitHub Check: build-linux (21, doc)
• GitHub Check: build-linux (25, integration)
• GitHub Check: bwc-tests-rolling-upgrade (25)
• GitHub Check: build-linux (25, doc)
• GitHub Check: bwc-tests-rolling-upgrade (21)
• GitHub Check: bwc-tests-full-restart (21)
• GitHub Check: security-it-linux (21)
• GitHub Check: security-it-linux (25)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, unit)
• GitHub Check: security-it-windows-macos (macos-14, 25)
• GitHub Check: security-it-windows-macos (macos-14, 21)
• GitHub Check: security-it-windows-macos (windows-latest, 25)
• GitHub Check: security-it-windows-macos (windows-latest, 21)
• GitHub Check: build-windows-macos (macos-14, 21, doc)
• GitHub Check: build-windows-macos (macos-14, 25, unit)
• GitHub Check: build-windows-macos (macos-14, 25, integration)
• GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, integration)
• GitHub Check: build-windows-macos (macos-14, 25, doc)
• GitHub Check: build-windows-macos (macos-14, 21, unit)
• GitHub Check: build-windows-macos (windows-latest, 25, -PbuildPlatform=windows, unit)
• GitHub Check: build-windows-macos (windows-latest, 21, -PbuildPlatform=windows, integration)
• GitHub Check: build-windows-macos (macos-14, 21, integration)
• GitHub Check: CodeQL-Scan (java)
• GitHub Check: test-sql-cli-integration (21)

[dai-chen]

As I understand, we're translating to SQL Union query, could you explain bit where the most complexity comes from?

[asifabashar]

Calcite auto converts a int to long and float to double for aggregation result type. So union to a existing data of int or float type causes a type cast exception as same column with two different types are being computed to union. So this method get the type we need to cast to existing dataset so a union of two dataset a) original b) aggregated total row works without a cast exception. The cast exception can be reproduced if we remove method getAggregateDataTypeFieldRef and run CrossClusterSearchIT test.

[dai-chen]

Could you give me an example? Based on my test with Calcite SQL CLI below, I wonder is this problem only in RelNode?

0: jdbc:calcite:model=src/test/resources/mode> WITH q AS (SELECT gender, COUNT(*) AS cnt FROM account GROUP BY gender)
. . . . . . . . . . . . . . . . . . semicolon> SELECT gender, cnt FROM q
. . . . . . . . . . . . . . . . . . semicolon> UNION ALL
. . . . . . . . . . . . . . . . . . semicolon> SELECT 'Total', SUM(cnt) FROM q;
+--------+-----+
| GENDER | CNT |
+--------+-----+
| F      | 1   |
| M      | 3   |
| Total  | 4   |
+--------+-----+
3 rows selected (0.053 seconds)

[asifabashar]

@dai-chen Please check a my forked branch Pull request to test no casting use case , which I just created to test the scenerio where I commented casting for int, float type. The error is coming form CrossClusterSearchIT test, so basically it only appears when multi node results are combined where it goes through a different code path. The server side exception for the run is here .

git workflow for CrosClusterSearchIT

Exception stacktrace is below
=== Standard error of node node{:integ-test:integTestWithSecurity-0} ===
» ERROR][o.o.s.p.r.RestPPLQueryAction] [integTestWithSecurity-0] Error happened during query handling
» java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long (java.lang.Integer and java.lang.Long are in module java.base of loader 'bootstrap')
» at org.apache.calcite.avatica.util.AbstractCursor$LongAccessor.getLong(AbstractCursor.java:633) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.apache.calcite.avatica.AvaticaSite.get(AvaticaSite.java:310) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.apache.calcite.avatica.AvaticaResultSet.getObject(AvaticaResultSet.java:395) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.util.JdbcOpenSearchDataTypeConvertor.getExprValueFromSqlType(JdbcOpenSearchDataTypeConvertor.java:69) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.buildResultSet(OpenSearchExecutionEngine.java:236) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.lambda$execute$1(OpenSearchExecutionEngine.java:208) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.client.OpenSearchNodeClient.schedule(OpenSearchNodeClient.java:223) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchExecutionEngine.execute(OpenSearchExecutionEngine.java:204) ~[opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.executor.QueryService.lambda$executeWithCalcite$0(QueryService.java:108) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.calcite.CalcitePlanContext.run(CalcitePlanContext.java:121) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.executor.QueryService.executeWithCalcite(QueryService.java:98) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.executor.QueryService.execute(QueryService.java:75) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:66) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$schedule$0(OpenSearchQueryManager.java:64) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$0(OpenSearchQueryManager.java:89) [opensearch-sql-3.4.0.0-SNAPSHOT.jar:3.4.0.0-SNAPSHOT]
» at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) [opensearch-3.4.0-SNAPSHOT.jar:3.4.0-SNAPSHOT]
» at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090) [?:?]
» at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614) [?:?]
» at java.base/java.lang.Thread.run(Thread.java:1474) [?:?]

[LantaoJin]

I am okey for current workaround. Let me create a followup issue to optimize it. @coderabbitai can you help to create a github issue for tracking?

[coderabbitai]

🧩 Analysis chain

@LantaoJin, I'll create a GitHub issue to track this optimization. Let me gather the necessary information and create it for you.

---

🏁 Script executed:

#!/bin/bash

# Create a GitHub issue for tracking the type casting workaround
gh issue create \
  --repo opensearch-project/sql \
  --title "Optimize type casting workaround in addtotals/addcoltotals aggregation logic" \
  --body "## Description

A workaround was implemented in PR #4754 to handle type casting issues when performing UNION operations between original data and aggregated totals in the addtotals/addcoltotals commands.

## Current Implementation

The method \`getAggregateDataTypeFieldRef\` (line 2529 in \`CalciteRelNodeVisitor.java\`) explicitly casts:
- INTEGER → BIGINT
- FLOAT/REAL → DOUBLE

This workaround prevents \`ClassCastException\` that occurs during UNION operations when Calcite automatically converts aggregation result types.

## Problem

When combining original data (with int/float types) and aggregated totals (Calcite converts to long/double), a \`ClassCastException\` occurs in multi-node scenarios:

\`\`\`
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long
  at org.apache.calcite.avatica.util.AbstractCursor\$LongAccessor.getLong
  at org.opensearch.sql.opensearch.util.JdbcOpenSearchDataTypeConvertor.getExprValueFromSqlType
\`\`\`

This issue only manifests in CrossClusterSearchIT tests when multi-node results are combined.

## Expected Optimization

Investigate whether:
1. This is a Calcite bug that should be reported upstream
2. There's a better way to handle type compatibility in UNION operations
3. The type conversion can be handled more elegantly at a different layer

## Context

- Introduced in: PR #4754
- Discussion: https://github.com/opensearch-project/sql/pull/4754#discussion_r2612169694
- Test case demonstrating the issue: https://github.com/asifabashar/sql/pull/8/files

## Related Code

\`core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java\`:
- \`getAggregateDataTypeFieldRef\` method (line 2529)
- \`buildAddRowTotalAggregate\` method usage" \
  --assignee LantaoJin

Length of output: 53

---

@LantaoJin, I've created GitHub issue #4952 to track the optimization of the type casting workaround in the addtotals/addcoltotals aggregation logic.

The issue includes:

• Description of the current workaround in getAggregateDataTypeFieldRef
• Details about the ClassCastException that occurs in multi-node scenarios
• Context from this PR and discussion
• Suggestions for potential optimization approaches
• Links back to this PR and the test case demonstrating the issue

The issue has been assigned to you for follow-up investigation.

---

🧬 Code Graph Analysis Results

• File: core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

  • Snippet/summary: forceProjectExcept (private static)

    • Purpose: Implements a forced-exclusion projection by removing specific expressions from the project, ensuring the remaining fields align with the expected output after modifications. Used as a workaround to ensure correct field ordering when nested or deduplicated fields appear in the Calcite plan.
    • Parameters: (RelBuilder relBuilder, Iterable expressions)
      • Behavior: Builds a full projection list, removes the provided expressions from that list, and applies project(...) to enforce the desired field set.
    • Returns/Effects: Modifies the RelBuilder state to project only the desired fields, aiding correctness when Calcite’s default projection could misalign fields after transformations.
    • Key details: Used in scenarios where dedup/nesting changes require explicit control over which fields are emitted in the final projection.
    • Location hint: Related to a workaround for a Calcite projection ordering issue within the class (non-exported helper method).

  • Snippet/summary: visitDedupe (override)

    • Purpose: Handles the deduplication operation (Dedupe node) by delegating to internal buildDedup* helpers and validating options.
    • Behavior:
      • Extracts options: allowedDuplication, keepEmpty, consecutive.
      • Validates constraints (e.g., allowedDuplication > 0, non-consecutive is required).
      • Delegates to either buildDedupOrNull or buildDedupNotNull depending on keepEmpty, producing a windowed/filtered plan that deduplicates rows.
    • Exceptions/Errors: Throws IllegalArgumentException for invalid dup count or unsupported options (e.g., consecutive dedup not allowed).
    • Key details: Integrates with the rest of the plan-building to ensure dedup results are produced with correct null handling and ordering.
    • Location hint: Central logic for the Calcite-based deduplication path.

  • Snippet/summary: buildDedupOrNull (private static)

    • Purpose: Builds a deduplication plan that keeps nulls (keepempty behavior) by inserting a ROW_NUMBER window, filtering on nulls or row_number, and dropping the helper column.
    • Parameters: (CalcitePlanContext context, List dedupeFields, Integer allowedDuplication)
    • Behavior:
      • Adds a ROW_NUMBER() OVER (PARTITION BY dedupeFields ORDER BY dedupeFields) as a dedup marker.
      • Filters: isNull(a) OR isNull(b) OR row_number_dedup <= n to implement dedup with possible null retention.
      • Drops the helper column (row_number_dedup) after filtering.
    • Return/Effects: Emits a projected plan that preserves nulls per configuration and enforces dedup limits.
    • Location hint: Part of the deduplication workaround logic in CalciteRelNodeVisitor.

  • Snippet/summary: buildDedupNotNull (private static)

    • Purpose: Builds a deduplication plan that enforces non-null constraints on dedup fields and uses a ROW_NUMBER window for dedup.
    • Parameters: (CalcitePlanContext context, List dedupeFields, Integer allowedDuplication, boolean fromJoinMaxOption)
    • Behavior:
      • Applies isNotNull filters on the dedupe fields (and on any aggregated/related fields when applicable).
      • Creates a ROW_NUMBER() OVER (PARTITION BY dedupeFields ORDER BY dedupeFields) aliasing to either ROW_NUMBER_COLUMN_FOR_JOIN_MAX_DEDUP or ROW_NUMBER_COLUMN_FOR_DEDUP depending on fromJoinMaxOption.
      • Filters by row_number_dedup <= allowedDuplication, then drops the dedup helper column.
    • Return/Effects: Produces a plan that deduplicates while enforcing non-null constraints and maintains correct field output.
    • Location hint: Part of the same dedup workaround path as buildDedupOrNull.

  • Snippet/summary: containsSubqueryExpression (private)

    • Purpose: Utility to determine if any part of a given Node tree contains a SubqueryExpression.
    • Parameters: Node expr
    • Return: boolean
    • Behavior: Recursively traverses the node and its children to detect SubqueryExpression nodes.
    • Location hint: Used by dedup/regex/eval-related paths to decide if subquery correlation handling is needed.

  • Snippet/summary: visitRegex (override)

    • Purpose: Builds a regex-based filter by translating a field and a pattern into a RexNode condition.
    • Notable interplay: Validates field type, constructs REGEXP_CONTAINS call, supports negation, and applies a filter.
    • Location hint: Part of the visitor methods that translate high-level patterns into Calcite Rex operations, related to the user’s note about a possible bug/workaround around conversion behavior.

  • Snippet/summary: visitAppendPipe (override)

    • Purpose: Merges a subquery with a main plan, by resolving the subquery plan, visiting, building a sub-pipeline, and then merging.
    • Behavior: Attaches child sub-pipeline to main pipeline, then merges with mergeTableAndResolveColumnConflict (which unifies schemas and resolves conflicts).
    • Location hint: Demonstrates a path where a workaround in plan alignment might be needed when combining sub-queries and main queries, relevant to discussion about conversion/PR impact.

  • Snippet/summary: mergeTableAndResolveColumnConflict (private)

    • Purpose: Reuses a shared schema unifier (SchemaUnifier) to align two RelNode schemas and then unions them.
    • Behavior: Builds unified schema with conflict resolution, pushes projections, and unions.
    • Location hint: Central to how plan components with potential schema conflicts are merged, potentially impacted by conversion behavior or bug-workaround discussions.

Note: The above entries refer to conceptual snippets within CalciteRelNodeVisitor.java that are relevant to the user’s comment about a workaround/bug and a potential follow-up issue. If you’d like, I can extract or reformat exact code blocks from the file with precise line numbers.

---

🧠 Learnings used

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Follow existing patterns in `CalciteRelNodeVisitor` and `CalciteRexNodeVisitor` for Calcite integration

---

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Test SQL generation and optimization paths for Calcite integration changes

---

Learnt from: CR
Repo: opensearch-project/sql PR: 0
File: .rules/REVIEW_GUIDELINES.md:0-0
Timestamp: 2025-12-02T17:27:55.938Z
Learning: Applies to **/*.java : Document Calcite-specific workarounds in code

[dai-chen]

Hi @LantaoJin Just to clarify, I'm not concerned about this specific cast issue. I’m trying to understand the complexity in this RelNode translation and would like to collect some data points. As you can see from the example above, the PPL → SQL translation is very straightforward, but the current RelNode visitor logic is hard to maintain. Just wonder do we already have an issue tracking the discussion around PPL → SqlNode vs. the current PPL → RelNode discussion? Probably we can do some overall evaluation later.

[asifabashar]

@dai-chen I have applied the recommended changes, Please review. Please check the use case #4754 (comment)