Skip to content

[SPARK-38939][SQL][FOLLOWUP] Replace named parameter with comment in ReplaceColumns #36487

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dcoliversun wants to merge 1 commit into from
Closed

[SPARK-38939][SQL][FOLLOWUP] Replace named parameter with comment in ReplaceColumns #36487

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...yst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2AlterTableCommands.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ case class ReplaceColumns(
require(table.resolved)
val deleteChanges = table.schema.fieldNames.map { name =>
// REPLACE COLUMN should require column to exist
TableChange.deleteColumn(Array(name), ifExists = false)
Copy link
Contributor

@cloud-fan cloud-fan May 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I'm a bit confused. named parameter has been widely used in the Spark codebase, what's wrong with it?

Copy link
Contributor Author

@dcoliversun dcoliversun May 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, named parameter is right. But there is a compilation error when k8s integration test. Please see PR description for more information. And you can exec this command to check it again

build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=r "kubernetes-integration-tests/test"

Copy link
Contributor

@cloud-fan cloud-fan May 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know why the issue only happens in this place?

Copy link
Contributor Author

@dcoliversun dcoliversun May 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because deleteColumn#TableChange is Java file. I replace named parameter with comment, because it's used in Apache Spark.

spark/core/src/main/scala/org/apache/spark/deploy/ExternalShuffleService.scala

Line 119 in ad81ba8

blockHandler.applicationRemoved(appId, true /* cleanupLocalDirs */)

TableChange.deleteColumn(Array(name), false /* ifExists */)
}
val addChanges = columnsToAdd.map { col =>
assert(col.path.isEmpty)
Expand Down