[SPARK-36683][SQL] Add new built-in SQL functions: SEC and CSC #33988

yutoacts · 2021-09-14T04:16:58Z

What changes were proposed in this pull request?

Add new built-in SQL functions: secant and cosecant, and add them as Scala and Python functions.

Why are the changes needed?

Cotangent has been supported in Spark SQL but Secant and Cosecant are missing though I believe they can be used as much as cot.
Related Links: SPARK-20751 SPARK-36660

Does this PR introduce any user-facing change?

Yes, users can now use these functions.

How was this patch tested?

Unit tests

sarutak

@yutoacts Than you for adding these new built-in functions.
Could you add tests to MathFunctionsSuite and test_functions.py?

sql/core/src/test/resources/sql-tests/inputs/operators.sql

sql/core/src/test/resources/sql-tests/results/operators.sql.out

sarutak · 2021-09-14T05:50:13Z

Jenkins, add to whitelist.

SparkQA · 2021-09-14T06:40:51Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47745/

SparkQA · 2021-09-14T06:49:58Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47745/

sarutak · 2021-09-14T10:04:47Z

For the Python code, it's better to follow @HyukjinKwon 's comments in #33992

SparkQA · 2021-09-14T10:15:36Z

Test build #143242 has finished for PR 33988 at commit 9ce80cb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yutoacts · 2021-09-15T02:22:55Z

For the Python code, it's better to follow @HyukjinKwon 's comments in #33992

OK, and seems like #33992 could conflict with this PR. I'll wait for it to be merged before pushing test_functions.py.

…SPARK-36683

SparkQA · 2021-09-15T03:19:26Z

Test build #143292 has finished for PR 33988 at commit 105631e.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-15T03:24:09Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47795/

SparkQA · 2021-09-15T05:32:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47798/

SparkQA · 2021-09-15T05:40:55Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47798/

SparkQA · 2021-09-15T07:33:08Z

Test build #143295 has finished for PR 33988 at commit abc64b6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

…6683

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala

…SPARK-36683

SparkQA · 2021-09-17T06:00:04Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47899/

SparkQA · 2021-09-17T06:09:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47899/

SparkQA · 2021-09-17T06:23:38Z

Test build #143400 has finished for PR 33988 at commit f0b0887.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2021-09-17T06:37:15Z

@yutoacts you can use dev/lint-python to check style for Python code.

SparkQA · 2021-09-17T06:48:32Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47906/

SparkQA · 2021-09-17T06:57:19Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47906/

SparkQA · 2021-09-17T07:07:12Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47907/

SparkQA · 2021-09-17T07:18:26Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47907/

yutoacts · 2021-09-17T07:37:00Z

@yutoacts you can use dev/lint-python to check style for Python code.

Thank you. It should be fixed now.

SparkQA · 2021-09-17T08:51:15Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47910/

SparkQA · 2021-09-17T09:01:19Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47910/

SparkQA · 2021-09-17T10:00:24Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47912/

SparkQA · 2021-09-17T10:00:48Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47915/

SparkQA · 2021-09-17T10:09:08Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47915/

SparkQA · 2021-09-17T10:09:26Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47912/

SparkQA · 2021-09-17T12:26:06Z

Test build #143402 has finished for PR 33988 at commit bf2179a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-17T13:48:45Z

Test build #143404 has finished for PR 33988 at commit a67b93f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-17T14:07:14Z

Test build #143407 has finished for PR 33988 at commit 0578140.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2021-09-20T13:39:34Z

Merged to master. Thanks @yutoacts .

### What changes were proposed in this pull request? Add secant and cosecant as R functions. ### Why are the changes needed? [SEC and CSC have been added](#33988) thus these functions need R support. ### Does this PR introduce _any_ user-facing change? Yes, users can now call those functions as R functions. ### How was this patch tested? unit tests added. Closes #34067 from yutoacts/SPARK-36824. Authored-by: Yuto Akutsu <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>

Yuto Akutsu added 2 commits September 14, 2021 11:44

Add new built-in SQL functions: SEC and CSC

e60f6a1

Add tests to MathExpressionSuite.scala

48b3247

github-actions bot added CORE PYTHON SQL labels Sep 14, 2021

add test results in operators.sql.out

9ce80cb

sarutak reviewed Sep 14, 2021

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/operators.sql Outdated Show resolved Hide resolved

sql/core/src/test/resources/sql-tests/results/operators.sql.out Outdated Show resolved Hide resolved

Yuto Akutsu added 2 commits September 15, 2021 11:32

added test to MathFunctionsSuite and fixed operators.sql

068432b

Merge branch 'SPARK-36683' of https://github.com/yutoacts/spark into …

105631e

…SPARK-36683

re-generate operators.sql.out and fix MathFunctionSuite

abc64b6

Merge branch 'master' of https://github.com/apache/spark into SPARK-3…

937a3c8

…6683

sarutak reviewed Sep 16, 2021

View reviewed changes

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala Show resolved Hide resolved

Yuto Akutsu added 2 commits September 16, 2021 13:43

Merge branch 'SPARK-36683' of https://github.com/yutoacts/spark into …

0b808e2

…SPARK-36683

add unit test in test_functions.py and generate golden result file

e0e90a1

github-actions bot added BUILD R STRUCTURED STREAMING labels Sep 16, 2021

yutoacts closed this Sep 16, 2021

yutoacts force-pushed the SPARK-36683 branch from e0e90a1 to 0666f5c Compare September 16, 2021 04:55

add short descriptions in functions.py

f0b0887

fixed lint error and wrong method calling

2df5825

Yuto Akutsu added 2 commits September 17, 2021 16:47

removed unnecessary spaces

bf2179a

edit assert_close in sqlutils.py: support negative infinity

a67b93f

"not math.isinf" to "math.isfinite" in assert_close

0578140

sarutak approved these changes Sep 17, 2021

View reviewed changes

HyukjinKwon approved these changes Sep 20, 2021

View reviewed changes

sarutak closed this in 30d17b6 Sep 20, 2021

yutoacts mentioned this pull request Sep 22, 2021

[SPARK-36824][R] Add sec and csc as R functions #34067

Closed

yutoacts deleted the SPARK-36683 branch September 22, 2021 06:12

[SPARK-36683][SQL] Add new built-in SQL functions: SEC and CSC #33988

[SPARK-36683][SQL] Add new built-in SQL functions: SEC and CSC #33988

Uh oh!

Conversation

yutoacts commented Sep 14, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sarutak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sarutak commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

sarutak commented Sep 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

yutoacts commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

sarutak commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

yutoacts commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

SparkQA commented Sep 17, 2021

Uh oh!

sarutak commented Sep 20, 2021

Uh oh!

Reviewers

Assignees

Labels

sarutak commented Sep 14, 2021 •

edited

Loading