Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature][DataSource]Add flink datasource plugin #15640

Closed
wants to merge 1 commit into from
Closed

[Feature][DataSource]Add flink datasource plugin #15640

wants to merge 1 commit into from

Conversation

pegasas
Copy link
Contributor

@pegasas pegasas commented Feb 27, 2024 •
edited
Loading

Purpose of the pull request

fix: #15630

[Feature][Datasource] add flink datasource

Brief change log

Currently spark use thrift server for replacement of hiveserver2 to provide Spark SQL capability.
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql-gateway/overview/
Flink SQL Gateway is also a replacement solution of hiveserver2.

Verify this pull request

image
image
image
image

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

@github-actions github-actions bot added UI ui and front end related backend document labels Feb 27, 2024
@pegasas pegasas force-pushed the issues/15630 branch 2 times, most recently from a213494 to 36503c2 Compare February 27, 2024 11:45
@caishunfeng caishunfeng added the feature new feature label Feb 28, 2024
songjianet
songjianet requested changes Feb 28, 2024
View reviewed changes
Copy link
Member

@songjianet songjianet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

github-advanced-security[bot]
github-advanced-security bot found potential problems Feb 28, 2024
View reviewed changes
...java/org/apache/dolphinscheduler/plugin/datasource/flink/param/FlinkDataSourceProcessor.java
}
hosts.deleteCharAt(hosts.length() - 1);
flinkDataSourceParamDTO.setHost(hosts.toString());
flinkDataSourceParamDTO.setPort(Integer.parseInt(hostPortArray[0].split(Constants.COLON)[1]));

Check notice

Code scanning / CodeQL

Missing catch of NumberFormatException

Potential uncaught 'java.lang.NumberFormatException'.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack.
Thanks for calling out.

@pegasas pegasas force-pushed the issues/15630 branch 3 times, most recently from bd7929e to 4ea5b38 Compare March 3, 2024 11:06
@pegasas
Copy link
Contributor Author

pegasas commented Mar 25, 2024

Caused by: java.sql.SQLFeatureNotSupportedException: FlinkConnection#prepareStatement is not supported yet.
at org.apache.flink.table.jdbc.BaseConnection.prepareStatement(BaseConnection.java:42)
at org.apache.flink.table.jdbc.FlinkConnection.prepareStatement(FlinkConnection.java:49)
at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.prepareStatementAndBind(SqlTask.java:380)
... 8 common frames omitted

will do some compacity work on sql task

@pegasas pegasas force-pushed the issues/15630 branch 5 times, most recently from 5314afb to 14d308d Compare March 25, 2024 15:16
@pegasas pegasas marked this pull request as ready for review March 25, 2024 16:32
@github-actions GitHub Actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.

@github-actions github-actions bot added the Stale label Jul 24, 2024
@pegasas
Copy link
Contributor Author

pegasas commented Jul 24, 2024

This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.

remove stale

@github-actions github-actions bot removed the Stale label Jul 25, 2024
@github-actions GitHub Actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 7 days if no further activity occurs.

@github-actions github-actions bot added the Stale label Nov 22, 2024
@github-actions GitHub Actions
Copy link

This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.

@github-actions github-actions bot closed this Nov 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@songjianet songjianet songjianet requested changes

@EricGao888 EricGao888 Awaiting requested review from EricGao888 EricGao888 is a code owner

@Amy0104 Amy0104 Awaiting requested review from Amy0104

@caishunfeng caishunfeng Awaiting requested review from caishunfeng caishunfeng is a code owner

@SbloodyS SbloodyS Awaiting requested review from SbloodyS SbloodyS is a code owner

@zhuangchong zhuangchong Awaiting requested review from zhuangchong

Assignees

@pegasas pegasas

Labels
backend document feature new feature Stale UI ui and front end related
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature][Datasource] Add Flink Datasource
3 participants
@pegasas @songjianet @caishunfeng