Skip to content

[SPARK-28302] [CORE] Make sure to generate unique output file for SparkLauncher on Windows #25076

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Ngone51 wants to merge 2 commits into from
Closed

[SPARK-28302] [CORE] Make sure to generate unique output file for SparkLauncher on Windows #25076

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b55dfc6
make sure to generate unique output file
Ngone51 Jul 8, 2019
9273f33
address comment
Ngone51 Jul 9, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions bin/spark-class2.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,12 @@ if not "x%JAVA_HOME%"=="x" (

rem The launcher library prints the command to be executed in a single line suitable for being
rem executed by the batch interpreter. So read all the output of the launcher into a variable.
:gen
set LAUNCHER_OUTPUT=%temp%\spark-class-launcher-output-%RANDOM%.txt
rem SPARK-28302: %RANDOM% would return the same number if we call it instantly after last call,
rem so we should make sure to generate unique file to avoid process collision of writing into a
Copy link
Member

@dongjoon-hyun dongjoon-hyun Jul 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit.

  • make sure -> make it sure.
  • into a same file -> into the same file.

rem same file concurrently.
if exist %LAUNCHER_OUTPUT% goto :gen
"%RUNNER%" -Xmx128m -cp "%LAUNCH_CLASSPATH%" org.apache.spark.launcher.Main %* > %LAUNCHER_OUTPUT%
for /f "tokens=*" %%i in (%LAUNCHER_OUTPUT%) do (
set SPARK_CMD=%%i
Expand Down