Skip to content

Conversation

@ykmr1224
Copy link
Collaborator

@ykmr1224 ykmr1224 commented Oct 4, 2025

Description

  • Added mvappend(value1, value2, ...) that combines all arguments into a single array
    • Each argument could be an value, an array, or null.
  • It is needed to implement spath command without path parameter so it can merge extracted fields with existing fields.

Related Issues

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ykmr1224 ykmr1224 changed the title (Test) Add mvappend function for Calcite PPL Add mvappend function for Calcite PPL Oct 4, 2025
Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: Tomoyuki Morita <[email protected]>
@ahkcs
Copy link
Contributor

ahkcs commented Oct 6, 2025

  • Each argument could be an value, an array, or null.

How does the current logic handle primitive arrays?
I tried this test case in MVAppendFunctionImplTest.java and it returns ClassCastException error

  @Test
  public void testMvappendWithPrimitiveIntArray() {
    int[] primitiveArray = {1, 2, 3};
    Object result = MVAppendFunctionImpl.mvappend(primitiveArray, 4);
    assertEquals(Arrays.asList(1, 2, 3, 4), result);
  }

@ahkcs
Copy link
Contributor

ahkcs commented Oct 6, 2025

How does the current implementation handle type coercion?
I tested this in CalciteMVAppendFunctionIT.java and it passes:

@Test
  public void testMvappendWithIntegerAndDouble() throws IOException {
    JSONObject actual =
        executeQuery(
            source(
                TEST_INDEX_BANK,
                "eval result = mvappend(1, 2.5, 3, 4.7) | head 1 | fields result"));

    verifySchema(actual, schema("result", "array"));
    verifyDataRows(actual, rows(List.of(0, 2.5, 0, 4.7)));
  }

Is it expected the return 0 for integer value when the array has both integer and double? Or do we need some additional handling logic here

Signed-off-by: Tomoyuki Morita <[email protected]>
Comment on lines 75 to 77
if (hasStringType && hasNumericType) {
return typeFactory.createSqlType(SqlTypeName.VARCHAR);
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 questions

  1. Could explain mostGeneralType rule?
  2. if (hasStringType && hasNumericType) it should return ANY?
  3. can we leverage typeFactory.leastRestrictive(List.of(...))?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed the logic to simply return ANY when any incompatible type is detected. typeFactory.leastRestrictive does not do much job for incompatible types (It only works between structs)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Does AbstractTypeCoercion.getTightestCommonType() works?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The basic idea of AbstractTypeCoercion.getTightestCommonType() is returning widened type with no precision loss, and works only for nullability difference and number precision difference. Is it mainly used for binary operator (or function) to align the input types to execute operation, in my understanding.

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: Tomoyuki Morita <[email protected]>
@ykmr1224
Copy link
Collaborator Author

ykmr1224 commented Oct 8, 2025

Fixed logic to simply return ANY in case of mixed types. This spec might be debatable and welcome your opinion, but the idea is to leave users to cast to their preferred type. (Anyway, spath would extract values as ANY from schema perspective.)

+--------------+

PPL> source=accounts | eval names_array = array(firstname, lastname) | eval result = mvjoin(names_array, ', ') | fields result | head 1
os> source=people | eval result = mvappend(1, 'text', 2.5) | fields result | head 1
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the main reason to add a UDF rather than reusing array_append?

Copy link
Collaborator Author

@ykmr1224 ykmr1224 Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main motivation is to handle null/value/array seemlessly, so that value merge operation won't become too complicated.

@ykmr1224 ykmr1224 merged commit 5c784fe into opensearch-project:main Oct 9, 2025
33 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Oct 9, 2025
* Add mvappend function for Calcite PPL

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix annonymizer test

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix IT

Signed-off-by: Tomoyuki Morita <[email protected]>

* Minor fix

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix type coercion issue

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix test

Signed-off-by: Tomoyuki Morita <[email protected]>

---------

Signed-off-by: Tomoyuki Morita <[email protected]>
(cherry picked from commit 5c784fe)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
ykmr1224 pushed a commit that referenced this pull request Oct 10, 2025
* Add mvappend function for Calcite PPL

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix annonymizer test

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix IT

Signed-off-by: Tomoyuki Morita <[email protected]>

* Minor fix

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix type coercion issue

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix test

Signed-off-by: Tomoyuki Morita <[email protected]>

---------

Signed-off-by: Tomoyuki Morita <[email protected]>
(cherry picked from commit 5c784fe)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
penghuo pushed a commit that referenced this pull request Oct 10, 2025
* Add mvappend function for Calcite PPL



* Fix annonymizer test



* Fix IT



* Minor fix



* Fix type coercion issue



* Fix test



---------


(cherry picked from commit 5c784fe)

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
asifabashar pushed a commit to asifabashar/sql that referenced this pull request Oct 16, 2025
* Add mvappend function for Calcite PPL

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix annonymizer test

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix IT

Signed-off-by: Tomoyuki Morita <[email protected]>

* Minor fix

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix type coercion issue

Signed-off-by: Tomoyuki Morita <[email protected]>

* Fix test

Signed-off-by: Tomoyuki Morita <[email protected]>

---------

Signed-off-by: Tomoyuki Morita <[email protected]>
Signed-off-by: Asif Bashar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.19-dev calcite calcite migration releated feature PPL Piped processing language

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants