Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Paimon] Support specify paimon table write properties, partition keys and primary keys #6535

Merged
merged 2 commits into from
Mar 27, 2024

Conversation

dailai
Copy link
Contributor

@dailai dailai commented Mar 19, 2024

Purpose of this pull request

[Feature][Paimon] Support specify paimon table write properties, partition keys and primary keys
[Feature][Paimon] Support timestamp(n) for the paimon sink
[Feature][API] ColumnParser support define columnScale which is useful for fakesource

For example:

  1. file.format of paimon properties support orc, parquet, avro
  2. bucket of paimon properties support Normal Dynamic Bucket Mode
  3. paimon.table.partition-keys can support partition table of paimon
  4. Solve the timestamp(n) problem for the paimon sink
  5. Schema.columns can define columnScale for fakesource

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@dailai dailai force-pushed the dev branch 2 times, most recently from 619dadc to 8f8a713 Compare March 19, 2024 07:44
@Hisoka-X Hisoka-X added feature New feature paimon labels Mar 20, 2024
@dailai dailai force-pushed the dev branch 6 times, most recently from c4e8a92 to a7626ba Compare March 20, 2024 11:20
if (primaryKeys.isEmpty() && Objects.nonNull(tableSchema.getPrimaryKey())) {
primaryKeys = tableSchema.getPrimaryKey().getColumnNames();
}
if (Objects.nonNull(primaryKeys) && primaryKeys.size() > 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems impossible to be null

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I've removed it.

Copy link
Member

@Carl-Zhou-CN Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Hisoka-X
Copy link
Member

cc @TyrantLucifer

Copy link
Member

@hailin0 hailin0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hailin0 hailin0 merged commit 2b1234c into apache:dev Mar 27, 2024
7 checks passed
chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants