Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][Connector-V2][ES] Support dsl filter #4130

Merged
merged 9 commits into from
Mar 16, 2023
Merged

[Feature][Connector-V2][ES] Support dsl filter #4130

merged 9 commits into from
Mar 16, 2023

Conversation

kpretty
Copy link
Contributor

@kpretty kpretty commented Feb 15, 2023 •
edited
Loading

Purpose of this pull request

Support DSL
More convenient data ingest, improve data real-time performance, reduce unnecessary io

Elasticsearch {
    hosts = ["localhost:9200"]
    index = "seatunnel-*"
    source = ["_id","name","age"]
    query = {"range":{"firstPacket":{"gte":1669225429990,"lte":1669225429990}}}
}

Check list

@kpretty kpretty changed the title Support DSL [Feature][Connector-V2][ES]Support DSL Feb 15, 2023
@kpretty kpretty mentioned this pull request Feb 15, 2023
Closed
3 tasks
hailin0
hailin0 reviewed Feb 15, 2023
View reviewed changes
...arch-e2e/src/test/java/org/apache/seatunnel/e2e/connector/elasticsearch/ElasticsearchIT.java
@@ -197,7 +197,10 @@ private List<String> readSinkData() throws InterruptedException {
"c_bytes",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add e2e testcase（test job config）

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed @hailin0

hailin0
hailin0 reviewed Feb 15, 2023
View reviewed changes
docs/en/connector-v2/source/Elasticsearch.md Outdated
| password | string | no | - |
| index | string | yes | - |
| source | array | no | - |
| query | map | no | {"match_all": {}} |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| query | map | no | {"match_all": {}} |
| query | json | no | {"match_all": {}} |

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hailin0 right

...arch-e2e/src/test/java/org/apache/seatunnel/e2e/connector/elasticsearch/ElasticsearchIT.java Outdated
@@ -126,7 +127,8 @@ public void testElasticsearch(TestContainer container)
container.executeJob("/elasticsearch/elasticsearch_source_and_sink.conf");
Assertions.assertEquals(0, execResult.getExitCode());
List<String> sinData = readSinkData();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sinData -> sinkData

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hailin0 fixed

hailin0
hailin0 reviewed Feb 15, 2023
View reviewed changes
docs/en/connector-v2/source/Elasticsearch.md Outdated
@@ -59,6 +60,11 @@ The fields of index.
You can get the document id by specifying the field `_id`.If sink _id to other index,you need specify an alias for _id due to the Elasticsearch limit.
If you don't config source, you must config `schema`.

### query [map]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### query [map]
### query [json]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hailin0 you are right

@kpretty
Copy link
Contributor Author

kpretty commented Feb 20, 2023

TBR @hailin0

TaoZex
TaoZex reviewed Feb 23, 2023
View reviewed changes
Copy link
Contributor

@TaoZex TaoZex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

docs/en/connector-v2/source/Elasticsearch.md Outdated Show resolved Hide resolved
@TyrantLucifer TyrantLucifer added this to the 2.3.1 milestone Feb 27, 2023
@hailin0 hailin0 changed the title [Feature][Connector-V2][ES]Support DSL [Feature][Connector-V2][ES] Support dsl query Mar 14, 2023
@hailin0 hailin0 changed the title [Feature][Connector-V2][ES] Support dsl query [Feature][Connector-V2][ES] Support dsl filter Mar 14, 2023
@EricJoy2048 EricJoy2048 merged commit 79ca878 into apache:dev Mar 16, 2023
@kpretty kpretty deleted the feature-es-source-add-query branch March 26, 2023 09:51
ic4y pushed a commit to ic4y/incubator-seatunnel that referenced this pull request May 22, 2023
@kpretty @hailin0
[Feature][Connector-V2][ES] Support dsl filter (apache#4130)
06e6243 
* [Feature][Connector-V2][ES]Support DSL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TaoZex TaoZex TaoZex left review comments

@EricJoy2048 EricJoy2048 EricJoy2048 approved these changes

@hailin0 hailin0 hailin0 approved these changes

Assignees
No one assigned
Labels
approved reviewed
Projects
None yet
Milestone
2.3.1
Development

Successfully merging this pull request may close these issues.
5 participants
@kpretty @hailin0 @EricJoy2048 @TaoZex @TyrantLucifer