Skip to content

[ML][docs][minor] Define LabeledDocument/Document classes in CV example #5135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
petro-rudenko wants to merge 2 commits into from
Closed

[ML][docs][minor] Define LabeledDocument/Document classes in CV example #5135

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1d35383
[SQL][docs][minor] Define LabeledDocument/Document classes in CV example
petro-rudenko Mar 23, 2015
5190c75
Fix primitive types for java examples.
petro-rudenko Mar 24, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions docs/ml-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -565,6 +565,11 @@ import org.apache.spark.ml.tuning.{ParamGridBuilder, CrossValidator}
import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.sql.{Row, SQLContext}

// Labeled and unlabeled instance types.
// Spark SQL can infer schema from case classes.
case class LabeledDocument(id: Long, text: String, label: Double)
case class Document(id: Long, text: String)

val conf = new SparkConf().setAppName("CrossValidatorExample")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
Expand Down Expand Up @@ -655,6 +660,36 @@ import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SQLContext;

// Labeled and unlabeled instance types.

@srowen srowen Mar 24, 2015

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK so this is intentionally duplicated from the example above? I guess that's reasonable since the point is to be self-contained, and I don't imagine there's a lot of maintenance overhead in trying to evolve both copies together.

@petro-rudenko petro-rudenko Mar 24, 2015

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it's annoying when copy/pasting a bunch of code into spark shell and it fails because these classes are not declared.

// Spark SQL can infer schema from Java Beans.
public class Document implements Serializable {
private Long id;

@mengxr mengxr Mar 24, 2015

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long -> long

private String text;

public Document(Long id, String text) {
this.id = id;
this.text = text;
}

public Long getId() { return this.id; }
public void setId(Long id) { this.id = id; }

public String getText() { return this.text; }
public void setText(String text) { this.text = text; }
}

public class LabeledDocument extends Document implements Serializable {
private Double label;

public LabeledDocument(Long id, String text, Double label) {
super(id, text);
this.label = label;
}

public Double getLabel() { return this.label; }
public void setLabel(Double label) { this.label = label; }
}

SparkConf conf = new SparkConf().setAppName("JavaCrossValidatorExample");
JavaSparkContext jsc = new JavaSparkContext(conf);
SQLContext jsql = new SQLContext(jsc);
Expand Down