[FLINK-35647][route] Allow symbol replacement to simplify routing rules #3428
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
docs
yuxiqian
force-pushed
the
FLINK-35647
branch
2 times, most recently
from
c5d3248
to
323fc05
Compare
flink-cdc-cli/src/test/resources/definitions/pipeline-definition-full-with-repsym.yaml
Show resolved
Hide resolved
flink-cdc-cli/src/test/resources/definitions/pipeline-definition-full-with-repsym.yaml
Show resolved
Hide resolved
...-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
Show resolved
Hide resolved
lvyanquan
reviewed
Jul 4, 2024
| private TableId resolveReplacement( | ||
| TableId originalTable, Tuple3<Selectors, String, String> route) { | ||
| if (route.f2 != null) { | ||
| return TableId.parse(route.f1.replace(route.f2, originalTable.getTableName())); |
There was a problem hiding this comment.
Can we use Map<TableId, TableId> to reduce this replace and parse operation?
There was a problem hiding this comment.
Done, used LoadingCache to store table mapping relations. As for previously discussed hashCode caching optimization, I'll open another PR to address it.
ruanhang1993
reviewed
Jul 5, 2024
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/route/RouteRule.java
Show resolved
Hide resolved
lvyanquan
approved these changes
Jul 6, 2024
...-cdc-runtime/src/main/java/org/apache/flink/cdc/runtime/operators/schema/SchemaOperator.java
Show resolved
Hide resolved
leonardBang
approved these changes
Jul 12, 2024
There was a problem hiding this comment.
Thanks @yuxiqian, @icchux and @WholeWorld-Timothy for the contribution, LGTM
This was referenced Jul 12, 2024
|
@icchux and @WholeWorld-Timothy I've added you as co-authors in this PR, and thus I closed #2908 and #2947. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This closes FLINK-35647, inspired by #2908 and #2947. Credit goes to @icchux and @WholeWorld-Timothy.
Currently, we must provide explicit sink-table in pipeline route rules, which means if we'd like to route all tables in specific database and change names in pattern, there's no choice but write all rules separately, which is a chore.
This PR allows specifying a special "replacement symbol" to resemble original table name in route rules like this:
Now,
<>will be replaced by upstream table name, so any tables likedb.tableXwill be routed tonew_db.tableXautomatically.