[SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 #18599

liu-zhaokun · 2017-07-11T08:00:25Z

https://issues.apache.org/jira/browse/SPARK-21372
Spark writes one log file even I set the number of spark_rotate_log to 0.Meanwhile,the number of spark_rotate_log also can set to a string type and other type.So I think we should make some changes.

…k_rotate_log to 0

AmplabJenkins · 2017-07-11T08:02:10Z

Can one of the admins verify this patch?

jiangxb1987 · 2017-07-11T09:12:26Z

Did you see any failure in production environment caused by this issue?

srowen · 2017-07-11T09:18:54Z

sbin/spark-daemon.sh

    fi
+
+    expr $num + 0 &>/dev/null
+    if [ $? -ne 0 ] || [ $num -eq 0 ]; then


Are you trying to verify if num is a number? I get it but you would already get a bash error in $num -eq 0 if so and we don't otherwise validate script input that way. Isn't this forcing num to be nonzero, when I thought the issue was handling the case of 0 correctly?

@srowen
Yes, we should avoid this scenario that num isn't number.If num isn't a number,num will be set to 5.

liu-zhaokun · 2017-07-11T09:35:10Z

@jiangxb1987
When I want to change the number of log file,I test the scenario that num=0,and spark writes one log file.When I test the scenario that set num to Invaild symbol，the logic is wrong.So I think we should modify it to guarantee this feature.

jiangxb1987 · 2017-07-11T13:34:34Z

AFAIK the function spark_rotate_log is never used with num param set, unless you set log param manually, in that case I think you are responsible for the data validation, it would be interesting to see how this issue is triggered, could you share your script please? Thank you!

liu-zhaokun · 2017-07-11T15:06:57Z

@jiangxb1987
I want to change the number of log file ,so I passed num param manually by modify code in this file.In fact,this bug is there，even I don't trigger it,right?Shouldn't we modify it?

srowen · 2017-07-11T15:26:35Z

@liu-zhaokun I don't understand what bug you are referring to. We'd need to understand how an unmodified script displays some incorrect behavior.

liu-zhaokun · 2017-07-11T15:41:07Z

@srowen
If I set num to 0,there shouldn't create a logfile or we should make some deal for 0，and If I set num some invaild symbol,the function doesn't work normally,right？I am sorry for my poor English.

srowen · 2017-07-11T15:47:28Z

How would you set this with normal operation?
If you change the script to set a value that wasn't expected at that point given how the script works, of course it may fail. That's not a bug.

liu-zhaokun · 2017-07-12T01:39:20Z

sbin/spark-daemon.sh

    log=$1;
    num=5;
    if [ -n "$2" ]; then
 	num=$2


@srowen
There provide a param,num, to set the number of logfile,but it wasn't used in line 179.Since it doesn't work,can I remove the param "num" and use it just as a variable？

liu-zhaokun · 2017-07-12T01:41:09Z

@srowen
I think spark provides the param,so I can pass it in the script.

srowen · 2017-07-12T07:59:28Z

Let's close this. You aren't providing a scenario where an input causes a problem, and I think are still saying the error happens when you modify the script.

liu-zhaokun · 2017-07-12T08:07:44Z

@srowen
The param "num" which defined in function spark_rotate_log() isn't used,if users don't modify the script,it will never be invoked,so could I create a new PR to delete the param and replace it with a fixed number 5?

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017 Closes apache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closes apache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closes apache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closes apache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closes apache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closes apache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closes apache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closes apache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closes apache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closes apache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closes apache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closes apache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closes apache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closes apache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closes apache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closes apache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closes apache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closes apache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closes apache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closes apache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closes apache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closes apache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closes apache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closes apache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closes apache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closes apache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closes apache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closes apache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closes apache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closes apache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closes apache#18432 - resolve com.esotericsoftware.kryo.KryoException Closes apache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closes apache#18585 - SPARK-21359 Closes apache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closes apache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closes apache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closes apache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closes apache#18667 - Fix the simpleString used in error messages Closes apache#18782 - Branch 2.1 Added: Closes apache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closes apache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closes apache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closes apache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closes apache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <[email protected]> Closes apache#18780 from HyukjinKwon/close-prs.

[SPARK-21372] spark writes one log file even I set the number of spar…

7c71bc7

…k_rotate_log to 0

srowen reviewed Jul 11, 2017

View reviewed changes

liu-zhaokun commented Jul 12, 2017

View reviewed changes

srowen mentioned this pull request Jul 31, 2017

[INFRA] Close stale PRs #18780

Closed

asfgit closed this in 3a45c7f Aug 5, 2017

[SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 #18599

[SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 #18599

Uh oh!

Conversation

liu-zhaokun commented Jul 11, 2017

Uh oh!

AmplabJenkins commented Jul 11, 2017

Uh oh!

jiangxb1987 commented Jul 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen Jul 11, 2017

Choose a reason for hiding this comment

Uh oh!

liu-zhaokun Jul 11, 2017

Choose a reason for hiding this comment

Uh oh!

liu-zhaokun commented Jul 11, 2017

Uh oh!

jiangxb1987 commented Jul 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liu-zhaokun commented Jul 11, 2017

Uh oh!

srowen commented Jul 11, 2017

Uh oh!

liu-zhaokun commented Jul 11, 2017

Uh oh!

srowen commented Jul 11, 2017

Uh oh!

liu-zhaokun Jul 12, 2017

Choose a reason for hiding this comment

Uh oh!

liu-zhaokun commented Jul 12, 2017

Uh oh!

srowen commented Jul 12, 2017

Uh oh!

liu-zhaokun commented Jul 12, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jiangxb1987 commented Jul 11, 2017 •

edited

Loading

jiangxb1987 commented Jul 11, 2017 •

edited

Loading