Skip to content

Fix the flaky CalcitePPLTcphIT #4846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yuancu merged 1 commit into from
Nov 24, 2025
Merged

Fix the flaky CalcitePPLTcphIT #4846

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 5 additions & 8 deletions integ-test/src/test/java/org/opensearch/sql/calcite/tpch/CalcitePPLTpchIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
import java.util.Locale;
import org.json.JSONObject;
import org.junit.Assume;
import org.junit.Ignore;
import org.junit.Test;
import org.opensearch.sql.ppl.PPLIntegTestCase;
import org.opensearch.sql.util.Retry;
Expand Down Expand Up @@ -143,21 +142,19 @@ public void testQ3() throws IOException {
rows(4423, 3055.9365, "1995-02-17 00:00:00", 0));
}

// TODO: Aggregation push down has a hard-coded limit of 1000 buckets for output, so this query
// will not return the correct results with aggregation push down and it's unstable
@Ignore
@Test
public void testQ4() throws IOException {
String ppl = sanitize(loadFromFile("tpch/queries/q4.ppl"));
JSONObject actual = executeQuery(ppl);
verifySchemaInOrder(
actual, schema("o_orderpriority", "string"), schema("order_count", "bigint"));
verifyDataRows(
actual,
rows("1-URGENT", 7),
rows("1-URGENT", 9),
rows("2-HIGH", 7),
rows("3-MEDIUM", 4),
rows("4-NOT SPECIFIED", 7),
rows("5-LOW", 10));
rows("3-MEDIUM", 9),
rows("4-NOT SPECIFIED", 8),
rows("5-LOW", 12));
Comment on lines +155 to +157
Copy link
Collaborator

@yuancu yuancu Nov 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain how blocking the index load relates to this integration test fix? The connection isn't clear to me from the current changes.

Copy link
Member Author

@LantaoJin LantaoJin Nov 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not related to the fix. I found this test was ignored due to the bucket size 1000 limitation. Just correct the results and enable it again.

}

@Test
Expand Down
13 changes: 13 additions & 0 deletions integ-test/src/test/java/org/opensearch/sql/legacy/SQLIntegTestCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,19 @@ protected synchronized void loadIndex(Index index, RestClient client) throws IOE
createIndexByRestClient(client, indexName, mapping);
loadDataByRestClient(client, indexName, dataSet);
}
// loadIndex() could directly return when isIndexExist()=true,
// e.g. the index is created in the cluster but data hasn't been flushed.
// We block loadIndex() until data loaded to resolve
// https://github.com/opensearch-project/sql/issues/4261
int countDown = 3; // 1500ms timeout
while (countDown != 0 && getDocCount(client, indexName) == 0) {
try {
Thread.sleep(500);
countDown--;
} catch (InterruptedException e) {
throw new IOException(e);
}
}
}

protected synchronized void loadIndex(Index index) throws IOException {
Expand Down
15 changes: 15 additions & 0 deletions integ-test/src/test/java/org/opensearch/sql/legacy/TestUtils.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,21 @@ public static void loadDataByRestClient(
performRequest(client, request);
}

/**
* Return how many docs in the index
*
* @param client client connection
* @param indexName index name
* @return doc count of the index
* @throws IOException
*/
public static int getDocCount(RestClient client, String indexName) throws IOException {
Request request = new Request("GET", "/" + indexName + "/_count");
Response response = performRequest(client, request);
JSONObject jsonObject = new JSONObject(getResponseBody(response));
return jsonObject.getInt("count");
}

/**
* Perform a request by REST client.
*
Expand Down
Loading