Skip to content

[SPARK-20090][PYTHON] Add StructType.fieldNames in PySpark #18618

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HyukjinKwon wants to merge 3 commits into from
Closed

[SPARK-20090][PYTHON] Add StructType.fieldNames in PySpark #18618

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
efe113f
Add StructType.fieldNames in PySpark
HyukjinKwon Jul 13, 2017
eaa910d
Make the return type to a list instead of a tuple
HyukjinKwon Jul 15, 2017
86493be
Add a note for deprecation for names attribute in StructType
HyukjinKwon Jul 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions python/pyspark/sql/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1216,21 +1216,25 @@ def test_struct_type(self):
struct1 = StructType().add("f1", StringType(), True).add("f2", StringType(), True, None)
struct2 = StructType([StructField("f1", StringType(), True),
StructField("f2", StringType(), True, None)])
self.assertEqual(struct1.fieldNames(), tuple(struct2.names))
self.assertEqual(struct1, struct2)

struct1 = StructType().add("f1", StringType(), True).add("f2", StringType(), True, None)
struct2 = StructType([StructField("f1", StringType(), True)])
self.assertNotEqual(struct1.fieldNames(), tuple(struct2.names))
self.assertNotEqual(struct1, struct2)

struct1 = (StructType().add(StructField("f1", StringType(), True))
.add(StructField("f2", StringType(), True, None)))
struct2 = StructType([StructField("f1", StringType(), True),
StructField("f2", StringType(), True, None)])
self.assertEqual(struct1.fieldNames(), tuple(struct2.names))
self.assertEqual(struct1, struct2)

struct1 = (StructType().add(StructField("f1", StringType(), True))
.add(StructField("f2", StringType(), True, None)))
struct2 = StructType([StructField("f1", StringType(), True)])
self.assertNotEqual(struct1.fieldNames(), tuple(struct2.names))
self.assertNotEqual(struct1, struct2)

# Catch exception raised during improper construction
Expand All @@ -1249,11 +1253,11 @@ def test_struct_type(self):
self.assertIs(struct1[0], struct1.fields[0])
self.assertEqual(struct1[0:1], StructType(struct1.fields[0:1]))
with self.assertRaises(KeyError):
not_a_field = struct1["f9"]
_ = struct1["f9"]
with self.assertRaises(IndexError):
not_a_field = struct1[9]
_ = struct1[9]
with self.assertRaises(TypeError):
not_a_field = struct1[9.9]
_ = struct1[9.9]

def test_parse_datatype_string(self):
from pyspark.sql.types import _all_atomic_types, _parse_datatype_string
Expand Down
12 changes: 11 additions & 1 deletion python/pyspark/sql/types.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,7 +445,7 @@ class StructType(DataType):

This is the data type representing a :class:`Row`.

Iterating a :class:`StructType` will iterate its :class:`StructField`s.
Iterating a :class:`StructType` will iterate its :class:`StructField`\\s.

@HyukjinKwon HyukjinKwon Jul 13, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before

2017-07-13 2 28 03

After

2017-07-13 2 48 36

@holdenk holdenk Jul 20, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank's for fixing the documentation issue while you were here :) +1

A contained :class:`StructField` can be accessed by name or position.

>>> struct1 = StructType([StructField("f1", StringType(), True)])
Expand Down Expand Up @@ -562,6 +562,16 @@ def jsonValue(self):
def fromJson(cls, json):
return StructType([StructField.fromJson(f) for f in json["fields"]])

def fieldNames(self):
"""
Returns all field names in a tuple.

@HyukjinKwon HyukjinKwon Jul 13, 2017

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2017-07-13 1 50 58

>>> struct = StructType([StructField("f1", StringType(), True)])
>>> struct.fieldNames()
('f1',)
"""
return tuple(self.names)

def needConversion(self):
# We need convert Row()/namedtuple into tuple()
return True
Expand Down