[SPARK-51444][CORE] Remove the unreachable if branch from TaskSchedulerImpl#statusUpdate

[LuciferYang]

What changes were proposed in this pull request?

After SPARK-51184 (#49913), the variables failedExecutor and reason defined in the function TaskSchedulerImpl#statusUpdate are not reassigned after being initialized to None.

spark/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala

Lines 783 to 811 in f40bf4d

	var failedExecutor: Option[String] = None
	var reason: Option[ExecutorLossReason] = None
	synchronized {
	try {
	Option(taskIdToTaskSetManager.get(tid)) match {
	case Some(taskSet) =>
	assert(state != TaskState.LOST)
	if (TaskState.isFinished(state)) {
	cleanupTaskState(tid)
	taskSet.removeRunningTask(tid)
	if (state == TaskState.FINISHED) {
	taskResultGetter.enqueueSuccessfulTask(taskSet, tid, serializedData)
	} else if (Set(TaskState.FAILED, TaskState.KILLED).contains(state)) {
	taskResultGetter.enqueueFailedTask(taskSet, tid, state, serializedData)
	}
	}
	if (state == TaskState.RUNNING) {
	taskSet.taskInfos(tid).launchSucceeded()
	}
	case None =>
	logError(log"Ignoring update with state ${MDC(LogKeys.TASK_STATE, state)} for " +
	log"TID ${MDC(LogKeys.TASK_ID, tid)} because its task set is gone (this is " +
	log"likely the result of receiving duplicate task finished status updates) or its " +
	log"executor has been marked as failed.")
	}
	} catch {
	case e: Exception => logError("Exception in statusUpdate", e)
	}
	}

Therefore, failedExecutor.isDefined will always be false, and the following if branch will never be entered:

spark/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala

Lines 812 to 817 in f40bf4d

	// Update the DAGScheduler without holding a lock on this, since that can deadlock
	if (failedExecutor.isDefined) {
	assert(reason.isDefined)
	dagScheduler.executorLost(failedExecutor.get, reason.get)
	backend.reviveOffers()
	}

.

So, the this pr cleans it up.

Why are the changes needed?

Code cleanup.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

Was this patch authored or co-authored using generative AI tooling?

No

[dongjoon-hyun]

+1, LGTM. Thank you, @LuciferYang and @amoghantarkar .
Merged to master for Apache Spark 4.1.0.

[LuciferYang]

Thanks @dongjoon-hyun and @amoghantarkar