[SPARK-14937][ML][Document]spark.ml LogisticRegression sqlCtx in scala is inconsistent with java and python #12717
Conversation
|
Test build #57047 has finished for PR 12717 at commit
|
|
retest this please. |
|
Test build #57051 has finished for PR 12717 at commit
|
|
Test build #57053 has finished for PR 12717 at commit
|
|
@yanboliang Can you take a look ? It is a simple fix. Thanks! |
| val sqlCtx = new SQLContext(sc) | ||
| import sqlCtx.implicits._ | ||
| val sqlContext = new SQLContext(sc) | ||
| import sqlContext.implicits._ |
There was a problem hiding this comment.
Is this import still needed? (I forget.) It's weird to use a variable name in the import isn't it? (I know it was already like that.) Otherwise this does look like all instances of sqlCtx in examples, yes.
There was a problem hiding this comment.
It's necessary, otherwise compiler will complains when use DataFrame methods.
There was a problem hiding this comment.
I don't think this import is needed anymore. It used to be there to use the toDF() method back when there was no libsvm data source.
There was a problem hiding this comment.
Let me try to fix it in this PR. Thanks!
There was a problem hiding this comment.
@jkbradley
69: val bestThreshold = fMeasure.where($"F-Measure" === maxFMeasure)
70: .select("threshold").head().getDouble(0)
Still needs the import. Otherwise, it won't be compiled.
There was a problem hiding this comment.
Oh, OK, I guess it's the dollar sign notation
|
This looks good to me. Thanks! |
|
LGTM too, thanks all! |
5 participants
What changes were proposed in this pull request?
In spark.ml document, the LogisticRegression scala example uses sqlCtx. It is inconsistent with java and python examples which use sqlContext. In addition, a user can't copy & paste to run the example in spark-shell as sqlCtx doesn't exist in spark-shell while sqlContext exists.
Change the scala example referred by the spark.ml example.
How was this patch tested?
Compile the example scala file and it passes compilation.