Skip to content

[BugFix]Fix multisearch UDT type loss through UNION (#5145, #5146, #5147) #5154

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ahkcs merged 5 commits into from
Mar 23, 2026
Merged

[BugFix]Fix multisearch UDT type loss through UNION (#5145, #5146, #5147) #5154

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
29fa3a0
multisearch fix
ahkcs Feb 17, 2026
81c0d3f
fix IT
ahkcs Feb 17, 2026
bb026a5
add UT
ahkcs Feb 17, 2026
6a30e67
Address PR review: simplify leastRestrictive and shorten test assertions
ahkcs Mar 12, 2026
757ed9c
Address review: use unconditional assertions in leastRestrictive tests
ahkcs Mar 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions core/src/main/java/org/opensearch/sql/calcite/utils/OpenSearchTypeFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
import org.apache.calcite.sql.SqlCollation;
import org.apache.calcite.sql.type.SqlTypeName;
import org.apache.calcite.sql.type.SqlTypeUtil;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.opensearch.sql.calcite.type.AbstractExprRelDataType;
import org.opensearch.sql.calcite.type.ExprBinaryType;
import org.opensearch.sql.calcite.type.ExprDateType;
Expand Down Expand Up @@ -377,6 +378,36 @@ public static boolean isNumericType(RelDataType fieldType) {
return false;
}

/**
* Preserves OpenSearch UDT types through set operations (UNION, INTERSECT, EXCEPT). When all
* input types share the same UDT (e.g., all are EXPR_TIMESTAMP), the result retains the UDT
* wrapper instead of being downgraded to the underlying SQL type (e.g., VARCHAR). This is
* critical for operations like multisearch that use UNION ALL, where downstream operators (bin,
* span) rely on the UDT type to determine how to process the field.
*/
@Override
public @Nullable RelDataType leastRestrictive(List<RelDataType> types) {
if (types.size() > 1) {
RelDataType first = types.get(0);
if (first instanceof AbstractExprRelDataType<?> firstUdt) {
boolean allSameUdt =
types.stream()
.allMatch(
t ->
t instanceof AbstractExprRelDataType<?> udt
&& udt.getUdt() == firstUdt.getUdt());
if (allSameUdt) {
boolean anyNullable = types.stream().anyMatch(RelDataType::isNullable);
if (anyNullable && !first.isNullable()) {
return firstUdt.createWithNullability(this, true);

@Swiddis Swiddis Mar 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Could do this in a single pass with a plain for loop instead of splitting the match checks and null checks.

@Override
public @Nullable RelDataType leastRestrictive(List<RelDataType> types) {
  if (types.size() > 1) {
    RelDataType first = types.get(0);
    if (first instanceof AbstractExprRelDataType<?> firstUdt) {
      boolean anyNullable = false;
      for (RelDataType t : types) {
        if (t instanceof AbstractExprRelDataType<?> udt
            && udt.getUdt() == firstUdt.getUdt()) {
          anyNullable |= t.isNullable();
        } else {
          return super.leastRestrictive(types); // If we hit a non-match, fallback
        }
      }
      if (anyNullable && !first.isNullable()) {
        return firstUdt.createWithNullability(this, true);
      }
      return first;
    }
  }
  return super.leastRestrictive(types);
}

@ahkcs ahkcs Mar 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

}
return first;
}
}
}
return super.leastRestrictive(types);
}
Comment thread
ahkcs marked this conversation as resolved.

/**
* Checks if the RelDataType represents a time-based field (timestamp, date, or time). Supports
* both standard SQL time types (including TIMESTAMP, TIMESTAMP_WITH_LOCAL_TIME_ZONE, DATE, TIME,
Expand Down
201 changes: 199 additions & 2 deletions integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalciteMultisearchCommandIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,10 +152,10 @@ public void testMultisearchWithTimestampInterleaving() throws IOException {

verifySchema(
result,
schema("@timestamp", null, "string"),
schema("@timestamp", null, "timestamp"),
schema("category", null, "string"),
schema("value", null, "int"),
schema("timestamp", null, "string"));
schema("timestamp", null, "timestamp"));

verifyDataRows(
result,
Expand Down Expand Up @@ -344,6 +344,203 @@ public void testMultisearchCrossIndexFieldSelection() throws IOException {
rows(null, null, "Times Square", 1002));
}

// ========================================================================
// Reproduction tests for GitHub issues #5145, #5146, #5147
// ========================================================================

/** Reproduce #5145: multisearch without further processing should return all rows. */
@Test
public void testMultisearchWithoutFurtherProcessing() throws IOException {
JSONObject result =
executeQuery(
"| multisearch [search source=opensearch-sql_test_index_time_data | where category ="
+ " \\\"A\\\"] [search source=opensearch-sql_test_index_time_data | where category"
+ " = \\\"B\\\"]");

verifySchema(
result,
schema("@timestamp", null, "timestamp"),
schema("category", null, "string"),
schema("value", null, "int"),
schema("timestamp", null, "timestamp"));

// category A has 26 rows, category B has 25 rows = 51 total
assertEquals(51, result.getInt("total"));
}

/** Reproduce #5146: span expression used after multisearch should work. */
@Test
public void testMultisearchWithSpanExpression() throws IOException {
JSONObject result =
executeQuery(
"| multisearch [search source=opensearch-sql_test_index_time_data | where category ="
+ " \\\"A\\\"] [search source=opensearch-sql_test_index_time_data2 | where category"
+ " = \\\"E\\\"] | stats avg(value) by span(@timestamp, 5m)");

verifySchema(
result,
schema("avg(value)", null, "double"),
schema("span(@timestamp,5m)", null, "timestamp"));

// Each data point falls in its own 5-min bucket (all >5min apart), so avg = single value
// Category A: 26 rows from time_test_data, Category E: 10 rows from time_test_data2
verifyDataRows(
result,
// Category A (26 rows)
rows(8945.0, "2025-07-28 00:15:00"),
rows(6834.0, "2025-07-28 03:55:00"),
rows(6589.0, "2025-07-28 07:50:00"),
rows(9367.0, "2025-07-28 11:05:00"),
rows(9245.0, "2025-07-28 15:15:00"),
rows(8917.0, "2025-07-28 19:20:00"),
rows(8384.0, "2025-07-28 23:30:00"),
rows(8798.0, "2025-07-29 03:35:00"),
rows(9306.0, "2025-07-29 07:45:00"),
rows(8873.0, "2025-07-29 11:50:00"),
rows(8542.0, "2025-07-29 15:00:00"),
rows(9321.0, "2025-07-29 19:05:00"),
rows(8917.0, "2025-07-29 23:10:00"),
rows(8756.0, "2025-07-30 03:20:00"),
rows(9234.0, "2025-07-30 07:25:00"),
rows(8679.0, "2025-07-30 11:35:00"),
rows(8765.0, "2025-07-30 15:40:00"),
rows(9187.0, "2025-07-30 19:50:00"),
rows(8862.0, "2025-07-30 23:55:00"),
rows(8537.0, "2025-07-31 03:00:00"),
rows(9318.0, "2025-07-31 07:10:00"),
rows(8914.0, "2025-07-31 11:15:00"),
rows(8753.0, "2025-07-31 15:25:00"),
rows(9231.0, "2025-07-31 19:30:00"),
rows(8676.0, "2025-07-31 23:40:00"),
rows(8762.0, "2025-08-01 03:45:00"),
// Category E (10 rows)
rows(2001.0, "2025-08-01 04:00:00"),
rows(2003.0, "2025-08-01 01:00:00"),
rows(2005.0, "2025-07-31 20:45:00"),
rows(2007.0, "2025-07-31 16:00:00"),
rows(2009.0, "2025-07-31 12:30:00"),
rows(2011.0, "2025-07-31 08:00:00"),
rows(2013.0, "2025-07-31 04:30:00"),
rows(2015.0, "2025-07-31 01:00:00"),
rows(2017.0, "2025-07-30 21:30:00"),
rows(2019.0, "2025-07-30 18:00:00"));
}

/** Reproduce #5147: bin command after multisearch should produce non-null @timestamp. */
@Test
public void testMultisearchBinTimestamp() throws IOException {
JSONObject result =
executeQuery(
"| multisearch [search source=opensearch-sql_test_index_time_data | where category ="
+ " \\\"A\\\"] [search source=opensearch-sql_test_index_time_data2 | where category"
+ " = \\\"E\\\"] | fields @timestamp, category, value | bin @timestamp span=5m");

verifySchema(
result,
schema("category", null, "string"),
schema("value", null, "int"),
schema("@timestamp", null, "timestamp"));

// bin floors @timestamp to 5-min boundaries; projectPlusOverriding moves @timestamp to end
// Category A: 26 rows from time_test_data, Category E: 10 rows from time_test_data2
verifyDataRows(
result,
// Category A (26 rows)
rows("A", 8945, "2025-07-28 00:15:00"),
rows("A", 6834, "2025-07-28 03:55:00"),
rows("A", 6589, "2025-07-28 07:50:00"),
rows("A", 9367, "2025-07-28 11:05:00"),
rows("A", 9245, "2025-07-28 15:15:00"),
rows("A", 8917, "2025-07-28 19:20:00"),
rows("A", 8384, "2025-07-28 23:30:00"),
rows("A", 8798, "2025-07-29 03:35:00"),
rows("A", 9306, "2025-07-29 07:45:00"),
rows("A", 8873, "2025-07-29 11:50:00"),
rows("A", 8542, "2025-07-29 15:00:00"),
rows("A", 9321, "2025-07-29 19:05:00"),
rows("A", 8917, "2025-07-29 23:10:00"),
rows("A", 8756, "2025-07-30 03:20:00"),
rows("A", 9234, "2025-07-30 07:25:00"),
rows("A", 8679, "2025-07-30 11:35:00"),
rows("A", 8765, "2025-07-30 15:40:00"),
rows("A", 9187, "2025-07-30 19:50:00"),
rows("A", 8862, "2025-07-30 23:55:00"),
rows("A", 8537, "2025-07-31 03:00:00"),
rows("A", 9318, "2025-07-31 07:10:00"),
rows("A", 8914, "2025-07-31 11:15:00"),
rows("A", 8753, "2025-07-31 15:25:00"),
rows("A", 9231, "2025-07-31 19:30:00"),
rows("A", 8676, "2025-07-31 23:40:00"),
rows("A", 8762, "2025-08-01 03:45:00"),
// Category E (10 rows)
rows("E", 2001, "2025-08-01 04:00:00"),
rows("E", 2003, "2025-08-01 01:00:00"),
rows("E", 2005, "2025-07-31 20:45:00"),
rows("E", 2007, "2025-07-31 16:00:00"),
rows("E", 2009, "2025-07-31 12:30:00"),
rows("E", 2011, "2025-07-31 08:00:00"),
rows("E", 2013, "2025-07-31 04:30:00"),
rows("E", 2015, "2025-07-31 01:00:00"),
rows("E", 2017, "2025-07-30 21:30:00"),
rows("E", 2019, "2025-07-30 18:00:00"));

@Swiddis Swiddis Mar 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these test assertions are pretty long, is that the minimal reproducing example? would 1-day intervals be clearer?

@ahkcs ahkcs Mar 12, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

}

/** Reproduce #5147 full pattern: bin + stats after multisearch. */
@Test
public void testMultisearchBinAndStats() throws IOException {
JSONObject result =
executeQuery(
"| multisearch [search source=opensearch-sql_test_index_time_data | where category ="
+ " \\\"A\\\"] [search source=opensearch-sql_test_index_time_data2 | where category"
+ " = \\\"E\\\"] | bin @timestamp span=5m | stats avg(value) by @timestamp");

verifySchema(
result, schema("avg(value)", null, "double"), schema("@timestamp", null, "timestamp"));

// Each data point falls in its own 5-min bucket (all >5min apart), so avg = single value
// Category A: 26 rows from time_test_data, Category E: 10 rows from time_test_data2
verifyDataRows(
result,
// Category A (26 rows)
rows(8945.0, "2025-07-28 00:15:00"),
rows(6834.0, "2025-07-28 03:55:00"),
rows(6589.0, "2025-07-28 07:50:00"),
rows(9367.0, "2025-07-28 11:05:00"),
rows(9245.0, "2025-07-28 15:15:00"),
rows(8917.0, "2025-07-28 19:20:00"),
rows(8384.0, "2025-07-28 23:30:00"),
rows(8798.0, "2025-07-29 03:35:00"),
rows(9306.0, "2025-07-29 07:45:00"),
rows(8873.0, "2025-07-29 11:50:00"),
rows(8542.0, "2025-07-29 15:00:00"),
rows(9321.0, "2025-07-29 19:05:00"),
rows(8917.0, "2025-07-29 23:10:00"),
rows(8756.0, "2025-07-30 03:20:00"),
rows(9234.0, "2025-07-30 07:25:00"),
rows(8679.0, "2025-07-30 11:35:00"),
rows(8765.0, "2025-07-30 15:40:00"),
rows(9187.0, "2025-07-30 19:50:00"),
rows(8862.0, "2025-07-30 23:55:00"),
rows(8537.0, "2025-07-31 03:00:00"),
rows(9318.0, "2025-07-31 07:10:00"),
rows(8914.0, "2025-07-31 11:15:00"),
rows(8753.0, "2025-07-31 15:25:00"),
rows(9231.0, "2025-07-31 19:30:00"),
rows(8676.0, "2025-07-31 23:40:00"),
rows(8762.0, "2025-08-01 03:45:00"),
// Category E (10 rows)
rows(2001.0, "2025-08-01 04:00:00"),
rows(2003.0, "2025-08-01 01:00:00"),
rows(2005.0, "2025-07-31 20:45:00"),
rows(2007.0, "2025-07-31 16:00:00"),
rows(2009.0, "2025-07-31 12:30:00"),
rows(2011.0, "2025-07-31 08:00:00"),
rows(2013.0, "2025-07-31 04:30:00"),
rows(2015.0, "2025-07-31 01:00:00"),
rows(2017.0, "2025-07-30 21:30:00"),
rows(2019.0, "2025-07-30 18:00:00"));
}
Comment thread
ahkcs marked this conversation as resolved.

@Test
public void testMultisearchTypeConflictWithStats() {
Exception exception =
Expand Down
2 changes: 1 addition & 1 deletion integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLAppendCommandIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ public void testAppendSchemaMergeWithTimestampUDT() throws IOException {
schema("account_number", "bigint"),
schema("firstname", "string"),
schema("age", "int"),
schema("birthdate", "string"));
schema("birthdate", "timestamp"));
verifyDataRows(actual, rows(32, null, 34, "2018-08-11 00:00:00"));
}

Expand Down
Loading