Skip to content

Showcase support for reading wildcard tables in Elasticsearch & OpenSearch#25098

Merged
ebyhr merged 2 commits intotrinodb:masterfrom
findinpath:findinpath/elasticsearch-wildcard-table-support
Feb 25, 2025
Merged

Showcase support for reading wildcard tables in Elasticsearch & OpenSearch#25098
ebyhr merged 2 commits intotrinodb:masterfrom
findinpath:findinpath/elasticsearch-wildcard-table-support

Conversation

@findinpath
Copy link
Copy Markdown
Contributor

@findinpath findinpath commented Feb 20, 2025

Description

This contribution documents in the code that the trino-elasticsearch connector supports
querying wildcard tables through the trino-elasticsearch connector.

SELECT * from elasticsearch."default"."heartbeat-*"

Additional context and related issues

Test was inspired from #12428 trino-bigquery effort.

Querying wildcard tables is a feature supported by ES to allow searching multiple indexes at the same time (a common pattern where people shard their data sets across indexes for retention, etc): https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multiple-indices.html

Release notes

(x) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Feb 20, 2025
@github-actions github-actions bot added the elasticsearch Elasticsearch connector label Feb 20, 2025
@mosabua mosabua added the needs-docs This pull request requires changes to the documentation label Feb 20, 2025
@mosabua
Copy link
Copy Markdown
Member

mosabua commented Feb 20, 2025

If this is supported and syntax is approved by @martint we should definitely document in this connector docs. Probably also in BigQuery.. and maybe OpenSearch connector as well?

Although arguably its kinda weird that we support that syntax ..

Comment on lines 2557 to 2566
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add another test with different index definition? e.g. long vs text, 2 columns vs 1 column

Copy link
Copy Markdown
Contributor Author

@findinpath findinpath Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried both scenarios locally and the results are unpredictable - it basically depends which index is used to retrieve the table schema.
This syntax is thought specifically for tables with the same schema.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its worth adding a comment that querying wild card tables work only for the same schema tables.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. I see this test more of a "demo" and nothing else.
Bending the functionality as Yuya suggested or speaking about the limitations, is just asking for eventual changes in the future if the functionality slightly changes on the underlying data source.

@findinpath findinpath changed the title Showcase support for reading wildcard tables in Elasticsearch Showcase support for reading wildcard tables in Elasticsearch & OpenSearch Feb 21, 2025
@findinpath findinpath force-pushed the findinpath/elasticsearch-wildcard-table-support branch from 37c4837 to 42b367d Compare February 21, 2025 15:36
@github-actions github-actions bot added the opensearch OpenSearch connector label Feb 21, 2025
@findinpath
Copy link
Copy Markdown
Contributor Author

we should definitely document in this connector docs.

@mosabua see #25108

@ebyhr ebyhr removed the needs-docs This pull request requires changes to the documentation label Feb 25, 2025
@findinpath findinpath force-pushed the findinpath/elasticsearch-wildcard-table-support branch from 42b367d to a47d9aa Compare February 25, 2025 21:11
@ebyhr ebyhr force-pushed the findinpath/elasticsearch-wildcard-table-support branch from a47d9aa to e28427f Compare February 25, 2025 22:32
@ebyhr ebyhr merged commit f903f23 into trinodb:master Feb 25, 2025
16 checks passed
@github-actions github-actions bot added this to the 472 milestone Feb 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed elasticsearch Elasticsearch connector opensearch OpenSearch connector

Development

Successfully merging this pull request may close these issues.

4 participants