Skip to content

Spark 3.4: Fix system function pushdown in CoW row-level commands#10119

Merged
aokolnychyi merged 1 commit intoapache:mainfrom
aokolnychyi:system-pushdown-3.4
Apr 11, 2024
Merged

Spark 3.4: Fix system function pushdown in CoW row-level commands#10119
aokolnychyi merged 1 commit intoapache:mainfrom
aokolnychyi:system-pushdown-3.4

Conversation

@aokolnychyi
Copy link
Contributor

@aokolnychyi aokolnychyi commented Apr 10, 2024

This PR cherry-picks PR #9873 to Spark 3.4.

@github-actions github-actions bot added the spark label Apr 10, 2024
@aokolnychyi aokolnychyi mentioned this pull request Apr 10, 2024
Closed
@aokolnychyi
Copy link
Contributor Author

aokolnychyi commented Apr 10, 2024

I forgot 3.4 is a bit different, I may need to make some changes. Let me see.

@aokolnychyi aokolnychyi force-pushed the system-pushdown-3.4 branch from cef4cdd to f18d7f8 Compare April 10, 2024 23:21
@aokolnychyi
Copy link
Contributor Author

aokolnychyi commented Apr 10, 2024

I excluded the unnecessary branch in ReplaceStaticInvoke and adapted the tests.

szehon-ho
szehon-ho approved these changes Apr 11, 2024
View reviewed changes
Copy link
Member

@szehon-ho szehon-ho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, ReplaceData is not available in Spark 3.4?

@tmnd1991
Copy link

tmnd1991 commented Apr 11, 2024

lgtm, ReplaceData is not available in Spark 3.4?

ReplaceData is there in 3.4 but has 1 less argument (groupFilterCondition: Option[Expression]) but I see it's not used at all in ReplaceStaticInvoke logic, so I would say that we could add that branch too in 3.4 but definitely @aokolnychyi had a good reason to remove it 😄

@aokolnychyi
Copy link
Contributor Author

aokolnychyi commented Apr 11, 2024

In 3.4, we use custom ReplaceIcebergData and rewrite the operations differently. I think that logic is only needed in 3.5.

@aokolnychyi aokolnychyi merged commit 290a6a0 into apache:main Apr 11, 2024
@aokolnychyi
Copy link
Contributor Author

aokolnychyi commented Apr 11, 2024

Thanks for reviewing, @szehon-ho @tmnd1991 @nastra!

@aokolnychyi aokolnychyi added this to the Iceberg 1.5.1 milestone Apr 17, 2024
amogh-jahagirdar pushed a commit to amogh-jahagirdar/iceberg that referenced this pull request Apr 18, 2024
@aokolnychyi @amogh-jahagirdar
Spark 3.4: Fix system function pushdown in CoW row-level commands (ap…
a2376a7 
…ache#10119)
@amogh-jahagirdar amogh-jahagirdar mentioned this pull request Apr 18, 2024
Merged
nastra pushed a commit that referenced this pull request Apr 18, 2024
@amogh-jahagirdar @aokolnychyi
Spark 3.4: Fix system function pushdown in CoW row-level commands (#1…
cbb8530 
…0119) (#10171)

Co-authored-by: Anton Okolnychyi <aokolnychyi@apple.com>
@lurnagao-dahua
Copy link
Contributor

lurnagao-dahua commented Aug 26, 2024

Hi,Is there a plan to port to Spark 3.3?

szehon-ho pushed a commit to szehon-ho/iceberg that referenced this pull request Sep 16, 2024
@amogh-jahagirdar @aokolnychyi
Spark 3.4: Fix system function pushdown in CoW row-level commands (ap…
63551f7 
…ache#10119) (apache#10171)

Co-authored-by: Anton Okolnychyi <aokolnychyi@apple.com>
sasankpagolu pushed a commit to sasankpagolu/iceberg that referenced this pull request Oct 27, 2024
@aokolnychyi
Spark 3.4: Fix system function pushdown in CoW row-level commands (ap…
99ee0a9 
…ache#10119)
zachdisc pushed a commit to zachdisc/iceberg that referenced this pull request Dec 23, 2024
@aokolnychyi @zachdisc
Spark 3.4: Fix system function pushdown in CoW row-level commands (ap…
3f69fa0 
…ache#10119)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nastra nastra nastra approved these changes

@szehon-ho szehon-ho szehon-ho approved these changes

Assignees

No one assigned

Labels

spark

Projects

None yet

Milestone

Iceberg 1.5.1

Development

Successfully merging this pull request may close these issues.

5 participants

@aokolnychyi @tmnd1991 @lurnagao-dahua @nastra @szehon-ho