Skip to content

[SPARK-31759][Deploy] Support configurable max number of rotate logs for spark daemons #28580

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
yaooqinn wants to merge 3 commits into from
Closed

[SPARK-31759][Deploy] Support configurable max number of rotate logs for spark daemons #28580

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
311d0f3
[SPARK-31759][Deploy] Support configurable max number of rotate logs …
yaooqinn May 19, 2020
c56f152
address comments
yaooqinn May 20, 2020
a54c6bd
address comments
yaooqinn May 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions conf/spark-env.sh.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@
# Generic options for the daemons used in the standalone deploy mode
# - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs)
# - SPARK_LOG_MAX_FILES Max log files of Spark daemons can rotate to, at least 5.
# - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp)
# - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER)
# - SPARK_NICENESS The scheduling priority for daemons. (Default: 0)
Expand Down
10 changes: 7 additions & 3 deletions sbin/spark-daemon.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
#
# SPARK_CONF_DIR Alternate conf dir. Default is ${SPARK_HOME}/conf.
# SPARK_LOG_DIR Where log files are stored. ${SPARK_HOME}/logs by default.
# SPARK_LOG_MAX_FILES Max log files of Spark daemons can rotate to, at least 5.
Comment thread
HyukjinKwon marked this conversation as resolved.
Outdated
# SPARK_MASTER host:path where spark code should be rsync'd from
# SPARK_PID_DIR The pid files are stored. /tmp by default.
# SPARK_IDENT_STRING A string representing this instance of spark. $USER by default
Expand Down Expand Up @@ -74,10 +75,13 @@ shift
spark_rotate_log ()
{
log=$1;
num=5;
if [ -n "$2" ]; then
num=$2

if [[ ${SPARK_LOG_MAX_FILES} -gt 5 ]] 2>/dev/null; then
Comment thread
HyukjinKwon marked this conversation as resolved.
Outdated
num=${SPARK_LOG_MAX_FILES}
else
num=5
Comment thread
HyukjinKwon marked this conversation as resolved.
Outdated
fi

if [ -f "$log" ]; then # rotate logs
while [ $num -gt 1 ]; do
prev=`expr $num - 1`
Expand Down