Skip to content

[SPARK-36070][CORE] Log time cost info for writing rows out and committing the task #33279

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 2 commits into from
Closed

[SPARK-36070][CORE] Log time cost info for writing rows out and committing the task #33279

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7950524
[SPARK-36070][CORE] Add time cost info for writing rows out and commi…
yaooqinn Jul 9, 2021
fb3af2a
debug info
yaooqinn Jul 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions core/src/main/scala/org/apache/spark/mapred/SparkHadoopMapRedUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ import org.apache.hadoop.mapreduce.{OutputCommitter => MapReduceOutputCommitter}
import org.apache.spark.{SparkEnv, TaskContext}
import org.apache.spark.executor.CommitDeniedException
import org.apache.spark.internal.Logging
import org.apache.spark.util.Utils

object SparkHadoopMapRedUtil extends Logging {
/**
Expand All @@ -47,8 +48,8 @@ object SparkHadoopMapRedUtil extends Logging {
// Called after we have decided to commit
def performCommit(): Unit = {
try {
committer.commitTask(mrTaskContext)
logInfo(s"$mrTaskAttemptID: Committed")
val (_, timeCost) = Utils.timeTakenMs(committer.commitTask(mrTaskContext))
logInfo(s"$mrTaskAttemptID: Committed. Elapsed time: $timeCost ms.")
} catch {
case cause: IOException =>
logError(s"Error committing the output of task: $mrTaskAttemptID", cause)
Expand Down
10 changes: 8 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,8 +301,14 @@ object FileFormatWriter extends Logging {
try {
Utils.tryWithSafeFinallyAndFailureCallbacks(block = {
// Execute the task to write rows out and commit the task.
dataWriter.writeWithIterator(iterator)
dataWriter.commit()
val taskAttemptID = taskAttemptContext.getTaskAttemptID
val (res, timeCost) = Utils.timeTakenMs {
logDebug("$taskAttemptID starts to write and commit.")
dataWriter.writeWithIterator(iterator)
dataWriter.commit()
}
logInfo(s"$taskAttemptID finished to write and commit. Elapsed time: $timeCost ms.")
Copy link
Contributor

@cloud-fan cloud-fan Jul 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some more thought, I think it's better to use SQL metrics for it. It's very hard to know max/min/avg by reading the logs.

@AngersZhuuuu I think you tried it before. Can you restore the work?

Copy link
Contributor

@AngersZhuuuu AngersZhuuuu Jul 14, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some more thought, I think it's better to use SQL metrics for it. It's very hard to know max/min/avg by reading the logs.

@AngersZhuuuu I think you tried it before. Can you restore the work?

Yea, working on this

res
})(catchBlock = {
// If there is an error, abort the task
dataWriter.abort()
Expand Down