Skip to content

[SPARK-2010] [PySpark] [SQL] support nested structure in SchemaRDD #1598

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 27 commits into from
Closed

[SPARK-2010] [PySpark] [SQL] support nested structure in SchemaRDD #1598

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
644665a
use tuple and namedtuple for schemardd
davies Jul 25, 2014
2c8debc
Merge branch 'master' into nested
davies Jul 30, 2014
a435b5a
add docs and code refactor
davies Jul 30, 2014
547bf3e
Merge branch 'master' into nested
davies Jul 30, 2014
bc6e9e1
switch to new Schema API
davies Jul 30, 2014
182fb46
refactor
davies Jul 30, 2014
2cc2d45
refactor
davies Jul 30, 2014
d69d397
refactor
davies Jul 30, 2014
7f6f251
address all comments
davies Jul 31, 2014
c4ddc30
fix conflict between name of fields and variables
davies Jul 31, 2014
b3559b4
use generated Row instead of namedtuple
davies Jul 31, 2014
0eaaf56
fix doc tests
davies Jul 31, 2014
84679b3
Merge branch 'master' of github.com:apache/spark into nested
davies Jul 31, 2014
9d9af55
use arrry to applySchema and infer schema in Python
davies Jul 31, 2014
f5df97f
refactor, address comments
davies Jul 31, 2014
9d8447c
apply schema provided by string of names
davies Jul 31, 2014
6b258b5
fix pep8
davies Aug 1, 2014
c79ca67
fix serialization of nested data
davies Aug 1, 2014
63de8f8
fix typo
davies Aug 1, 2014
353a3f2
fix code style
davies Aug 1, 2014
e9c0d5c
remove string typed schema
davies Aug 1, 2014
51aa135
use Row to infer schema
davies Aug 1, 2014
1e5b801
improve cache of classes
davies Aug 1, 2014
61b2292
add @deprecated to pythonToJavaMap
davies Aug 1, 2014
abe9e6e
address comments
davies Aug 1, 2014
8852aaf
check type of schema
davies Aug 1, 2014
f1d15b6
verify schema with the first few rows
davies Aug 1, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions python/pyspark/rdd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,9 +311,9 @@ def map(self, f, preservesPartitioning=False):
>>> sorted(rdd.map(lambda x: (x, 1)).collect())
[('a', 1), ('b', 1), ('c', 1)]
"""
def func(split, iterator):
def func(_, iterator):
return imap(f, iterator)
return PipelinedRDD(self, func, preservesPartitioning)
return self.mapPartitionsWithIndex(func, preservesPartitioning)

def flatMap(self, f, preservesPartitioning=False):
"""
Expand Down Expand Up @@ -1070,7 +1070,7 @@ def func(split, iterator):
if not isinstance(x, basestring):
x = unicode(x)
yield x.encode("utf-8")
keyed = PipelinedRDD(self, func)
keyed = self.mapPartitionsWithIndex(func)
keyed._bypass_serializer = True
keyed._jrdd.map(self.ctx._jvm.BytesToString()).saveAsTextFile(path)

Expand Down Expand Up @@ -1268,7 +1268,7 @@ def add_shuffle_key(split, iterator):
yield pack_long(split)
yield outputSerializer.dumps(items)

keyed = PipelinedRDD(self, add_shuffle_key)
keyed = self.mapPartitionsWithIndex(add_shuffle_key)
keyed._bypass_serializer = True
with _JavaStackTrace(self.context) as st:
pairRDD = self.ctx._jvm.PairwiseRDD(
Expand Down
Loading