Skip to content

[SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 #5811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed

[SPARK-7277][SQL] Throw exception if the property mapred.reduce.tasks is set to -1 #5811

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
68a1c70
Show warning message if mapred.reduce.tasks is set to -1.
viirya Apr 30, 2015
4ede705
Throw exception instead of warning message.
viirya May 1, 2015
fd9c817
Merge remote-tracking branch 'upstream/master' into no_neg_reduce_tasks
viirya May 4, 2015
e518f96
Consider other wrong setting values.
viirya May 4, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/commands.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,15 @@ case class SetCommand(
logWarning(
s"Property ${SQLConf.Deprecated.MAPRED_REDUCE_TASKS} is deprecated, " +
s"automatically converted to ${SQLConf.SHUFFLE_PARTITIONS} instead.")
sqlContext.setConf(SQLConf.SHUFFLE_PARTITIONS, value)
Seq(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=$value"))
if (value == "-1") {
logWarning(
s"Set this property to -1 for automatically determining the number of reducers " +
s"is not supported, showing current ${SQLConf.SHUFFLE_PARTITIONS} instead.")
Copy link
Contributor

@squito squito Apr 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: "Setting this property ..."

Copy link
Contributor

@squito squito Apr 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually can I suggest a small rewording a think is a little clearer?

Setting this property to -1 for automatically determing the number of reducers is not supported. Ignoring request, and showing ${SQLConf.SHUFFLE_PARTITIONS} instead

Seq(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=${sqlContext.conf.numShufflePartitions}"))
} else {
sqlContext.setConf(SQLConf.SHUFFLE_PARTITIONS, value)
Seq(Row(s"${SQLConf.SHUFFLE_PARTITIONS}=$value"))
}

// Configures a single property.
case Some((key, Some(value))) =>
Expand Down