[SPARK-42169] [SQL] Implement code generation for to_csv function (StructsToCsv)

[NarekDW]

What changes were proposed in this pull request?

This PR enhances StructsToCsv class with doGenCode function instead of extending it from CodegenFallback trait (performance improvement).

Why are the changes needed?

It will improve performance.

Does this PR introduce any user-facing change?

No

How was this patch tested?

an additional test case were added to org.apache.spark.sql.CsvFunctionsSuite class.

[MaxGekk]

Does CSVBenchmark show any performance improvements?

[NarekDW]

@MaxGekk
This is from master branch:

This is from current branch:

On my local machine - to_csv related operations in average are about ~20 % faster with this change in CSVBenchmark.

[MaxGekk]

@NarekDW Could you regenerate benchmark results using GitHub actions, see
https://spark.apache.org/developer-tools.html (Running benchmarks in your forked repository)
and update:

• CSVBenchmark-results.txt
• CSVBenchmark-jdk11-results.txt
• CSVBenchmark-jdk17-results.txt

in your PR.

[NarekDW]

@NarekDW Could you regenerate benchmark results using GitHub actions, see https://spark.apache.org/developer-tools.html (Running benchmarks in your forked repository) and update:

• CSVBenchmark-results.txt
• CSVBenchmark-jdk11-results.txt
• CSVBenchmark-jdk17-results.txt

in your PR.

@MaxGekk sorry for the late response. I've added benchmark results from GitHub actions.
I've added 2 commits: in pre-last commit I've updated benchmarks results from master branch and in the last commit I've added benchmarks results from current branch.

Execution links for master branch:
Java 8
Java 11
Java 17

Execution links for current branch:
Java 8
Java 11
Java 17

P.S. Scala 2.12 was used for all executions.

[NarekDW]

@jaceklaskowski thank you for the review. @MaxGekk just a reminder.

[MaxGekk]

Could you clarify this test title, please. I don't see anything related to 64KB in the test.

Doesn't the test just duplicates existing one: to_csv - struct?

[NarekDW]

Sure, checkEvaluation will execute this test in both modes without codegen and with codegen modes. In case of codegen mode, if the generated java code (method) will be larger than 64 kb in size it won't be able to compile it, and the test will fail, as Java has a 64kb limit on the size of methods.
It doesn't duplicate to_csv - struct test case, because the purpose of this test - is to generate big StructType(with 5000 literals in current case) and test it on StructsToCsv expression to be sure that generated java code doesn't exceed the limit of 64 kb in size.

[MaxGekk]

@NarekDW Could you rebase this PR on the recent master, please.

[NarekDW]

@MaxGekk sure, I've rebased, but there were conflicts in benchmarks. It will take some time to regenerate them.

[MaxGekk]

LGTM, waiting for benchmark results.

[NarekDW]

@MaxGekk sorry for delay, benchmark results are updated.

[MaxGekk]

+1, LGTM. Merging to master.
Thank you, @NarekDW and @HyukjinKwon @jaceklaskowski for review.