Spark-3.0: Fix UnresolvedException for some filters in rewrite_data_files #3794
Conversation
| // Add a dummy prefix linking to the table to collect the resolved spark expression from optimized plan. | ||
| val prefix = String.format("SELECT 42 from %s where ", tableName) | ||
| val logicalPlan = session.sessionState.sqlParser.parsePlan(prefix + where) | ||
| val optimizedLogicalPlan = session.sessionState.executePlan(logicalPlan).optimizedPlan |
There was a problem hiding this comment.
The only change compared to spark-3.2 PR is this. executePlan accepts only one argument in spark-3.0 and spark-3.1
|
|
||
| List<Object[]> output = sql( | ||
| "CALL %s.system.rewrite_data_files(table => '%s')", catalogName, tableIdent); | ||
| "CALL %s.system.rewrite_data_files(table => '%s')", catalogName, tableIdent); |
There was a problem hiding this comment.
Nit: Usually we try to avoid formatting only changes (even if they are the correct formatting), as the larger diff makes it harder for people that maintain forks to cherry-pick commits. If you aren’t changing the line, I’d suggest leaving it as is.
Same comment applies to all formatting only changes within a PR. 😊
There was a problem hiding this comment.
Agree. But I got a specific comment to fix indentation for this file in the spark-3.2 PR. Hence adopting it here also.
But I think checkstyle should catch over indented code in PR, so it can be caught and fixed in the respective PR itself. Let me explore why it is not catching
|
Thanks, @ajantha-bhat! |
3 participants
This PR is a backport of #3757 to spark-3.0