[SPARK-30539][PYTHON][SQL] Add DataFrame.tail in PySpark

[HyukjinKwon]

What changes were proposed in this pull request?

#26809 added Dataset.tail API. It should be good to have it in PySpark API as well.

Why are the changes needed?

To support consistent APIs.

Does this PR introduce any user-facing change?

No. It adds a new API.

How was this patch tested?

Manually tested and doctest was added.

[dongjoon-hyun]

+1, LGTM (only a minor comment on doc-string)

[SparkQA]

Test build #116900 has finished for PR 27251 at commit 565e3be.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #116912 has finished for PR 27251 at commit 438c8c9.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #116920 has finished for PR 27251 at commit 438c8c9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Also, cc @BryanCutler .

[dongjoon-hyun]

Merged to master. Thank you, @HyukjinKwon .

[HyukjinKwon]

Thanks @dongjoon-hyun!

[zero323]

Just thinking out loud ‒ should there be tailToPandas variant? For head we can easily do df.limit(n).toPandas().

[HyukjinKwon]

We could have things like df.limit(n, reverse=True).toPandas(); however, this implementation was not added yet because we should think how it works with other SQL APIs. It looks pretty clear to have it in Scala/Python APIs; however, I wasn't sure about SQL.