Add direct-query-core module for prometheus integration #3440

joshuali925 · 2025-03-17T20:54:07Z

Description

This PR adds direct-query-core module to SQL. It implements the following changes

move some shared classes from prometheus module to direct-query-core
- prometheus client
- client exception
- metric metadata model
- test fixtures
add direct-query-core module to support ExecuteDirectQuery and GetDirectQueryResources actions
implement query handler for executing the actions. currently the only supported data source type is prometheus

This PR does not expose the actions in SQL plugin at runtime. The follow up #3441 for direct-query module will add the rest and transport actions to SQL

Query API owner: @goyamegh
Resources API owner: @joshuali925

It is currently in draft since some tests are not fixed yet

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

...t-query-core/src/main/java/org/opensearch/sql/datasource/client/DataSourceClientFactory.java

direct-query-core/src/main/java/org/opensearch/sql/directquery/rest/model/QueryEntityType.java

joshuali925 · 2025-03-18T18:59:53Z

...-query-core/src/main/java/org/opensearch/sql/directquery/DirectQueryExecutorServiceImpl.java

+  @Override
+  public ExecuteDirectQueryResponse executeDirectQuery(ExecuteDirectQueryRequest request) {
+    // TODO: Replace with the data source query id.
+    String queryId = UUID.randomUUID().toString();


is this needed?

joshuali925 · 2025-03-18T19:01:12Z

...-query-core/src/main/java/org/opensearch/sql/directquery/DirectQueryExecutorServiceImpl.java

+                    try {
+                      return handler.executeQuery(client, request);
+                    } catch (IOException e) {
+                      return "{\"error\": \"Error executing query: " + e.getMessage() + "\"}";


if prometheus returns an error, should the SQL request fail with non-200 response?

direct-query-core/src/main/java/org/opensearch/sql/prometheus/client/PrometheusClientImpl.java

joshuali925 · 2025-03-18T19:04:36Z

direct-query-core/src/main/java/org/opensearch/sql/prometheus/client/PrometheusClientImpl.java

+    Response response = this.okHttpClient.newCall(request).execute();
+
+    logger.info("Received Prometheus response for instant query: code={}", response);
+    // Return the full response object, not just the data field


what does the comment mean?

direct-query-core/src/main/java/org/opensearch/sql/prometheus/client/PrometheusClientImpl.java

joshuali925 · 2025-03-18T19:06:38Z

...ry-core/src/main/java/org/opensearch/sql/prometheus/exception/PrometheusClientException.java

-package org.opensearch.sql.prometheus.exceptions;
+package org.opensearch.sql.prometheus.exception;

 import org.opensearch.sql.datasources.exceptions.DataSourceClientException;


this should be package org.opensearch.sql.datasource.client.exceptions.DataSourceClientException;? i think we should remove one of the two DataSourceClientException

any reason we need the datasource package under direct-query-core? or can we move it to datasource module?

Co-authored-by: Megha Goyal <[email protected]> Signed-off-by: Joshua Li <[email protected]>

opensearch-trigger-bot · 2025-05-01T15:21:29Z

This PR is stalled because it has been open for 30 days with no activity.

RyanL1997 · 2025-05-06T17:48:01Z

Hi @joshuali925 , do we still need this?

joshuali925 · 2025-05-06T17:49:30Z

we do, but currently no bandwidth

goyamegh · 2025-05-06T18:13:45Z

we do, but currently no bandwidth

@lezzago and I will take over this and take it further. Thanks.

opensearch-trigger-bot · 2025-06-07T15:21:23Z

This PR is stalled because it has been open for 30 days with no activity.

lezzago · 2025-08-25T21:44:01Z

...-core/src/main/java/org/opensearch/sql/directquery/rest/model/ExecuteDirectQueryRequest.java

+  // Optional fields
+  private Integer maxResults; // Optional: limit for Prometheus, maxDataPoints for CW
+  private Integer timeout; // Optional: number of seconds
+  private DataSourceOptions options; // Optional: Source specific arguments


Would we want a separate queryOptions or is the idea to have this contain query options as those are in a way specific to a datasource.

can't remember the details, will leave it to query API owner @goyamegh

currently this is the only usage

sql/direct-query-core/src/main/java/org/opensearch/sql/prometheus/model/PrometheusOptions.java

Lines 15 to 21 in ac6059f

public class PrometheusOptions implements DataSourceOptions {

private PrometheusQueryType queryType;

private String step; // Duration string in seconds

private String time; // ISO timestamp for instant queries

private String start; // ISO timestamp for range queries

private String end; // ISO timestamp for range queries

}

does look like query options

lezzago · 2025-08-25T21:49:09Z

...core/src/main/java/org/opensearch/sql/directquery/rest/model/ExecuteDirectQueryResponse.java

+public class ExecuteDirectQueryResponse {
+  private String queryId;
+  private String result;
+  private String sessionId;


Is this needed if we have the queryId? What benefit is there to add another tracker if we can just use queryId everywhere instead.

i remember session id is for EMR job control, and different queries might reuse the same EMR job. I might be wrong but i agree direct (sync) query shouldn't have session concept.

will leave it to query API owner @goyamegh

…-query-core Signed-off-by: Joshua Li <[email protected]>

joshuali925 mentioned this pull request Mar 17, 2025

Add direct-query module for prometheus integration #3441

Merged

7 tasks

joshuali925 commented Mar 18, 2025

View reviewed changes

Add direct-query-core module for prometheus integration

a64eefe

Co-authored-by: Megha Goyal <[email protected]> Signed-off-by: Joshua Li <[email protected]>

joshuali925 force-pushed the prometheus/direct-query-core branch from ac6059f to a64eefe Compare March 31, 2025 23:54

ps48 mentioned this pull request Apr 24, 2025

[RFC] Support for Exemplars in OpenSearch Dashboards opensearch-project/OpenSearch-Dashboards#9721

Open

opensearch-trigger-bot bot added the stalled label May 1, 2025

opensearch-trigger-bot bot removed the stalled label May 7, 2025

opensearch-trigger-bot bot added the stalled label Jun 7, 2025

ps48 mentioned this pull request Jun 21, 2025

[META] OpenSearch Observability Roadmap opensearch-project/dashboards-observability#2469

Open

ps48 added this to OpenSearch Roadmap Jun 23, 2025

github-project-automation bot moved this to New in OpenSearch Roadmap Jun 23, 2025

lezzago reviewed Aug 25, 2025

View reviewed changes

opensearch-trigger-bot bot removed the stalled label Aug 26, 2025

lezzago mentioned this pull request Sep 4, 2025

[RFC] Create OpenSearch Direct Query Plugin #4229

Open

joshuali925 changed the base branch from main to feature/direct-query-prometheus September 4, 2025 19:04

Merge branch 'feature/direct-query-prometheus' into prometheus/direct…

5d537df

…-query-core Signed-off-by: Joshua Li <[email protected]>

joshuali925 marked this pull request as ready for review September 4, 2025 19:09

joshuali925 requested review from YANG-DB, dai-chen, derek-ho, kavithacm, mengweieric, ps48 and vamsimanohar as code owners September 4, 2025 19:09

joshuali925 requested review from GumpacG, LantaoJin, MaxKsyunz, Swiddis, Yury-Fridlyand, acarbonetto, anirudha, forestmvey, noCharger, penghuo, qianheng-aws, seankao-az, ykmr1224 and yuancu as code owners September 4, 2025 19:09

joshuali925 merged commit 0ee85a3 into opensearch-project:feature/direct-query-prometheus Sep 4, 2025
20 of 21 checks passed

	public class PrometheusOptions implements DataSourceOptions {
	private PrometheusQueryType queryType;
	private String step; // Duration string in seconds
	private String time; // ISO timestamp for instant queries
	private String start; // ISO timestamp for range queries
	private String end; // ISO timestamp for range queries
	}

Add direct-query-core module for prometheus integration #3440

Add direct-query-core module for prometheus integration #3440

Uh oh!

Conversation

joshuali925 commented Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshuali925 Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

joshuali925 Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshuali925 Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshuali925 Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opensearch-trigger-bot bot commented May 1, 2025

Uh oh!

RyanL1997 commented May 6, 2025

Uh oh!

joshuali925 commented May 6, 2025

Uh oh!

goyamegh commented May 6, 2025

Uh oh!

opensearch-trigger-bot bot commented Jun 7, 2025

Uh oh!

lezzago Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

joshuali925 Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

lezzago Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

joshuali925 Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joshuali925 commented Mar 17, 2025 •

edited

Loading

joshuali925 Mar 18, 2025 •

edited

Loading

joshuali925 Aug 25, 2025 •

edited

Loading