Skip to content

[SPARK-17104][SQL] LogicalRelation.newInstance should follow the semantics of MultiInstanceRelation #14682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
viirya wants to merge 3 commits into from
Closed

[SPARK-17104][SQL] LogicalRelation.newInstance should follow the semantics of MultiInstanceRelation #14682

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f9a8c76
Fix LogicalRelation.newInstance.
viirya Aug 17, 2016
e243323
Address comment.
viirya Aug 19, 2016
e7fe68b
Update the doc.
viirya Aug 20, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/LogicalRelation.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,11 +79,18 @@ case class LogicalRelation(
/** Used to lookup original attribute capitalization */
val attributeMap: AttributeMap[AttributeReference] = AttributeMap(output.map(o => (o, o)))

def newInstance(): this.type =
/**
* Returns a new instance of this LogicalRelation. According to the semantics of
* MultiInstanceRelation, this method should returns a copy of this object with
* unique expression ids. Thus we don't respect the `expectedOutputAttributes` and
Copy link
Contributor

@cloud-fan cloud-fan Aug 19, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't expectedOutputAttributes.map(_.map(_.newInstance())) work?

Copy link
Member Author

@viirya viirya Aug 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once expectedOutputAttributes is None, we still need to have new instances of relation's output. output already covers both relation's output and expectedOutputAttributes.

Copy link
Contributor

@cloud-fan cloud-fan Aug 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if expectedOutputAttributes is None, output is defined as relation.schema.toAttributes, we will get different expr ids.

Copy link
Member Author

@viirya viirya Aug 19, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, yeah, you are right. It should work. Both are ok for me. I will update it.

Copy link
Contributor

@cloud-fan cloud-fan Aug 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update the doc?

Copy link
Member Author

@viirya viirya Aug 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

* create new instances of original output attributes.
*/
override def newInstance(): this.type = {
LogicalRelation(
relation,
expectedOutputAttributes,
Some(output.map(_.newInstance())),
metastoreTableIdentifier).asInstanceOf[this.type]
}

override def refresh(): Unit = relation match {
case fs: HadoopFsRelation => fs.refresh()
Expand Down
7 changes: 7 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/parquetSuites.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -589,6 +589,13 @@ class ParquetMetastoreSuite extends ParquetPartitioningTest {
}
}
}

test("self-join") {
val table = spark.table("normal_parquet")
val selfJoin = table.as("t1").join(table.as("t2"))
checkAnswer(selfJoin,
sql("SELECT * FROM normal_parquet x JOIN normal_parquet y"))
}
}

/**
Expand Down