Skip to content

[SPARK-13390][SQL]Fix the issue due to Iterator.map().toSeq is not Serializable #11313

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zsxwing wants to merge 1 commit into from
Closed

[SPARK-13390][SQL]Fix the issue due to Iterator.map().toSeq is not Serializable #11313

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -579,7 +579,7 @@ class SQLContext private[sql](
val className = beanClass.getName
val beanInfo = Introspector.getBeanInfo(beanClass)
val rows = SQLContext.beansToRows(data.asScala.iterator, beanInfo, attrSeq)
DataFrame(self, LocalRelation(attrSeq, rows.toSeq))
DataFrame(self, LocalRelation(attrSeq, rows.toArray))
Copy link
Contributor

@holdenk holdenk Feb 23, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

toArray could be kind of expensive for large local iterators - maybe we should only do this if necessary?

Copy link
Contributor

@JoshRosen JoshRosen Feb 23, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you have to materialize it anyways in order to support multiple scans.

Copy link
Contributor

@holdenk holdenk Feb 23, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yah I know we need to materialize anyways (toSeq forces that too on Iterators) - but I meant more the cost of copying all the objects to an array (see https://github.com/scala/scala/blob/v2.11.1/src/library/scala/collection/TraversableOnce.scala#L269 ) so maybe only calling toArray if necessary would be worth while (but perhaps not).

Copy link
Member

@srowen srowen Feb 23, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it was the same before -- it was materialized into a Seq. I think Josh is suggesting this is inherently necessary before making a LocalRelation so there's no meaningful opportunity for lazy eval (?) If so then this seems fine.

}

/**
Expand Down