Skip to content

[SPARK-12457] [SQL] Add ExpressionDescription to collection functions. #10418

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gatorsmile wants to merge 6 commits into from
Closed

[SPARK-12457] [SQL] Add ExpressionDescription to collection functions. #10418

Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e49b036
Add ExpressionDescription to collection functions.
gatorsmile Dec 21, 2015
8aa414e
Merge remote-tracking branch 'upstream/master' into ExpDesc4C
gatorsmile Dec 21, 2015
c03371c
address comments.
gatorsmile Dec 21, 2015
afdcb1c
added three test cases.
gatorsmile Dec 21, 2015
99cb7d8
use triple quotes.
gatorsmile Dec 22, 2015
64f32a4
address the comments.
gatorsmile Dec 29, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions ...alyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ import org.apache.spark.sql.types._
/**
* Given an array or map, returns its size.
*/
@ExpressionDescription(
usage = "_FUNC_(column) - Returns the length of the array or map stored in the column",
extended = "> SELECT _FUNC_(array(\"b\", \"d\", \"c\", \"a\"));\n 4")
Copy link
Contributor

@marmbrus marmbrus Dec 21, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i would use triple quotes for strings that contain quotes

Copy link
Member Author

@gatorsmile gatorsmile Dec 22, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried to do it, but realized annotation argument needs to be a constant. A newline marker \n is an issue here. We are unable to call stripMargin after triple quotes. Thus, I called stripMargin when building ExpressionInfo. Do you like it?

case class Size(child: Expression) extends UnaryExpression with ExpectsInputTypes {
override def dataType: DataType = IntegerType
override def inputTypes: Seq[AbstractDataType] = Seq(TypeCollection(ArrayType, MapType))
Expand All @@ -44,6 +47,11 @@ case class Size(child: Expression) extends UnaryExpression with ExpectsInputType
* Sorts the input array in ascending / descending order according to the natural ordering of
* the array elements and returns it.
*/
@ExpressionDescription(
usage = "_FUNC_(array, ascendingOrder) - Sorts the input array for the given column in " +
Copy link
Contributor

@hvanhovell hvanhovell Dec 28, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will fail to compile in scala 2.11. Use a multiline string here, see: #10488

Copy link
Member Author

@gatorsmile gatorsmile Dec 29, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, I will change all of them.

"ascending / descending order, according to the natural ordering of the array elements",
extended = "> SELECT _FUNC_(array(\"b\", \"d\", \"c\", \"a\"), false);\n " +
"[\"d\",\"c\",\"b\",\"a\"]")
case class SortArray(base: Expression, ascendingOrder: Expression)
extends BinaryExpression with ExpectsInputTypes with CodegenFallback {

Expand Down Expand Up @@ -125,6 +133,9 @@ case class SortArray(base: Expression, ascendingOrder: Expression)
/**
* Checks if the array (left) has the element (right)
*/
@ExpressionDescription(
usage = "_FUNC_(array, value) - Returns true if the array contains the value",
extended = "> SELECT _FUNC_(array(2, 9, 8), 9);\n true")
case class ArrayContains(left: Expression, right: Expression)
extends BinaryExpression with ImplicitCastInputTypes {

Expand Down