Skip to content

Conversation

@singhpk234
Copy link
Contributor

@singhpk234 singhpk234 commented May 16, 2022

At present partition info is not being populated to the Glue.

From checking past discussions, it looks like we decided to re-visit how to model these.

In this PR, the partition column's name are derived from {tranform}({colName}), I am open to other ways to representing this as well.

TODO : check if this works well with LF


cc @jackye1995 @rajarshisarkar @amogh-jahagirdar @arminnajafi @xiaoxuandev @yyanyy @flyrain @szehon-ho @natsukawa-kanou

@github-actions github-actions bot added the AWS label May 16, 2022
@singhpk234 singhpk234 marked this pull request as draft May 16, 2022 04:06
@singhpk234 singhpk234 force-pushed the feature/glue-partition-sort-order-storage branch from b97c292 to cd91cd9 Compare May 16, 2022 04:23
@singhpk234 singhpk234 changed the title AWS Add partition and sort info to Glue AWS: Add partition and sort info to Glue May 16, 2022
@singhpk234 singhpk234 changed the title AWS: Add partition and sort info to Glue [WIP] AWS: Add partition and sort info to Glue May 16, 2022
@singhpk234 singhpk234 force-pushed the feature/glue-partition-sort-order-storage branch from 519c0f9 to c619da9 Compare May 16, 2022 08:06
@singhpk234 singhpk234 force-pushed the feature/glue-partition-sort-order-storage branch from c619da9 to e861349 Compare May 16, 2022 09:00
@singhpk234 singhpk234 changed the title [WIP] AWS: Add partition and sort info to Glue [WIP] AWS: Add partition info to Glue May 16, 2022
@jackye1995
Copy link
Contributor

Does partition key allow properties? The info was there in the conversion, and was then removed because partition spec can also be versioned and requires some sort of properties to track which ones are the latest.

@github-actions
Copy link

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Aug 10, 2024
@github-actions
Copy link

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions bot closed this Aug 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants