Skip to content

[SPARK-25636][CORE] spark-submit cuts off the failure reason when there is an error connecting to master #22623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
devaraj-kavali wants to merge 9 commits into from
Closed

[SPARK-25636][CORE] spark-submit cuts off the failure reason when there is an error connecting to master #22623

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
dfba592
[SPARK-25636][CORE] spark-submit swallows the failure reason when there
Oct 3, 2018
81e1730
Removed handling of SparkException in overridden doSubmit
Oct 5, 2018
a82e75f
Updated the tests according to the change
Oct 5, 2018
20c4e01
Refactored the test change
Oct 8, 2018
96104bd
Refactored the test change
Oct 8, 2018
9e60602
Refactored the test change
Oct 8, 2018
71e3e30
Refactored the test change
Oct 8, 2018
131b3af
MesosClusterDispatcherSuite tests have the case that exitedCleanly=true
Oct 9, 2018
1002e44
Removed assert redundant message
Oct 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -927,8 +927,6 @@ object SparkSubmit extends CommandLineUtils with Logging {
} catch {
case e: SparkUserAppException =>
exitFn(e.exitCode)
case e: SparkException =>
printErrorAndExit(e.getMessage())
}
}

Expand Down
17 changes: 11 additions & 6 deletions core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,20 +74,25 @@ trait TestPrematureExit {
@volatile var exitedCleanly = false
mainObject.exitFn = (_) => exitedCleanly = true

@volatile var exception: Exception = null
val thread = new Thread {
override def run() = try {
mainObject.main(input)
} catch {
// If exceptions occur after the "exit" has happened, fine to ignore them.
// These represent code paths not reachable during normal execution.
case e: Exception => if (!exitedCleanly) throw e
// Capture the exception to check whether the exception contains searchString or not
case e: Exception => exception = e
}
}
thread.start()
thread.join()
val joined = printStream.lineBuffer.mkString("\n")
if (!joined.contains(searchString)) {
fail(s"Search string '$searchString' not found in $joined")
if (exitedCleanly) {
val joined = printStream.lineBuffer.mkString("\n")
assert(joined.contains(searchString), s"Search string '$searchString' not found in $joined")

@vanzin vanzin Oct 9, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message is redundant, assert already gives you a nice message, e.g.

[info]   "foo" did not contain "bar" (blah.scala)

} else {
assert(exception != null)
if (!exception.getMessage.contains(searchString)) {
throw exception
}
}
}
}
Expand Down