[SPARK-33769][SQL] Improve the next-day function of the sql component to deal with Column type

[chongguang]

What changes were proposed in this pull request?

The proposition of this pull request is described in this JIRA ticket: https://issues.apache.org/jira/browse/SPARK-33769

It proposes to improve the next-day function of the sql component to deal with Column type for the parameter dayOfWeek.

Why are the changes needed?

It makes this functionality easier to use.
Actually the signature of this function is:

def next_day(date: Column, dayOfWeek: String): Column.

It accepts the dayOfWeek parameter as a String. However in some cases, the dayOfWeek is in a Column, so a different value for each row of the dataframe.
A current workaround is to use the NextDay function like this:

NextDay(dateCol.expr, dayOfWeekCol.expr).

The proposition is to add another signature for this function:

def next_day(date: Column, dayOfWeek: Column): Column

In fact it is already the case for some other functions in this scala object, exemple:

def date_sub(start: Column, days: Int): Column = date_sub(start, lit(days))
def date_sub(start: Column, days: Column): Column = withExpr { DateSub(start.expr, days.expr) }

or

def add_months(startDate: Column, numMonths: Int): Column = add_months(startDate, lit(numMonths))
def add_months(startDate: Column, numMonths: Column): Column = withExpr {
AddMonths(startDate.expr, numMonths.expr)
}

This pull request is the same idea for the function next_day.

Does this PR introduce any user-facing change?

Yes
With this pull request, users of spark will have a new signature of the function:

def next_day(date: Column, dayOfWeek: Column): Column

But the existing function signature should still work:

def next_day(date: Column, dayOfWeek: String): Column

So this change should be retrocompatible.

How was this patch tested?

The unit tests of the next_day function has been enhanced.
It tests the dayOfWeek parameter both as String and Column.
I also added a test case for the existing signature where the dayOfWeek is a non valid String. This should return null.

[chongguang]

cc @HyukjinKwon

[HyukjinKwon]

Can you add @since 3.2.0?

[chongguang]

done

[HyukjinKwon]

ok to test

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37387/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37387/

[SparkQA]

Test build #132785 has finished for PR 30761 at commit 52f4213.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37416/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37416/

[HyukjinKwon]

Merged to master.

[SparkQA]

Test build #132815 has finished for PR 30761 at commit 7c32905.

• This patch fails SparkR unit tests.
• This patch merges cleanly.
• This patch adds no public classes.