[SPARK-43273][SQL] Support lz4raw compression codec for Parquet

[wangyum]

What changes were proposed in this pull request?

Parquet 1.13.0 supports LZ4_RAW codec. Please see https://issues.apache.org/jira/browse/PARQUET-2196.

This PR adds lz4raw to the supported list of spark.sql.parquet.compression.codec.

Why are the changes needed?

Support writing Parquet files with lz4raw compression codec.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Unit test and manual testing:

spark.sql("set spark.sql.parquet.compression.codec=lz4raw")
spark.range(10).write.parquet("/tmp/spark/lz4raw")
spark.read.parquet("/tmp/spark/lz4raw").show(false)

yumwang@LM-SHC-16508156 lz4raw % ll /tmp/spark/lz4raw
total 16
-rw-r--r--@ 1 yumwang  wheel    0 Jun  8 12:10 _SUCCESS
-rw-r--r--@ 1 yumwang  wheel  487 Jun  8 12:10 part-00000-c6786f4d-b5a6-406d-96a1-37bf0ceeeac7-c000.lz4raw.parquet
-rw-r--r--@ 1 yumwang  wheel  489 Jun  8 12:10 part-00001-c6786f4d-b5a6-406d-96a1-37bf0ceeeac7-c000.lz4raw.parquet

[dongjoon-hyun]

Thank you, @wangyum . Could you add this new codec here?

spark/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceCodecSuite.scala

Lines 54 to 64 in 2adb8e1

	class ParquetCodecSuite extends FileSourceCodecSuite {
	override def format: String = "parquet"
	override val codecConfigName: String = SQLConf.PARQUET_COMPRESSION.key
	// Exclude "lzo" because it is GPL-licenced so not included in Hadoop.
	// Exclude "brotli" because the com.github.rdblue:brotli-codec dependency is not available
	// on Maven Central.
	override protected def availableCodecs: Seq[String] = {
	Seq("none", "uncompressed", "snappy", "gzip", "zstd", "lz4")
	}
	}

[dongjoon-hyun]

May I ask why we need to move this line?

[wangyum]

In order to keep the order consistent:

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

Lines 986 to 987 in 4e78ff2

	"`spark.sql.parquet.compression.codec`. Acceptable values include: none, uncompressed, " +
	"snappy, gzip, lzo, brotli, lz4, lz4raw, zstd.")

[dongjoon-hyun]

Got it. It makes sense.

[dongjoon-hyun]

+1, LGTM (Pending CIs)

[LuciferYang]

+1, LGTM

[dongjoon-hyun]

The first commit passed all tests already (except pyspark-pandas-slow-connect). And, I verified the second commit manually. Merged to master.

[info] ParquetCodecSuite:
01:42:55.154 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[info] - write and read - file source parquet - codec: none (10 seconds, 419 milliseconds)
[info] - write and read - file source parquet - codec: uncompressed (1 second, 513 milliseconds)
[info] - write and read - file source parquet - codec: snappy (1 second, 457 milliseconds)
[info] - write and read - file source parquet - codec: gzip (1 second, 201 milliseconds)
[info] - write and read - file source parquet - codec: zstd (1 second, 435 milliseconds)
[info] - write and read - file source parquet - codec: lz4 (1 second, 163 milliseconds)
[info] - write and read - file source parquet - codec: lz4raw (1 second, 282 milliseconds)
01:43:14.972 WARN org.apache.spark.sql.execution.datasources.ParquetCodecSuite:

===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.datasources.ParquetCodecSuite, threads: ForkJoinPool.commonPool-worker-6 (daemon=true), QueryStageCreator-0 (daemon=true), ForkJoinPool.commonPool-worker-4 (daemon=true), QueryStageCreator-5 (daemon=true), QueryStageCreator-4 (daemon=true), ForkJoinPool.commonPool-worker-7 (daemon=true), QueryStageCreator-1 (daemon=true), rpc-boss-3-1 (daemon=true), ForkJoinPool.commonPool-worker-5 (daemon=true), QueryStageCreator-6 (daemon=true), shuffle-boss-6-1 (daemon=true), ForkJoinPool.commonPool-worker-1 (daemon=true), ForkJoinPool.commonPool-worker-3 (daemon=true), QueryStageCreator-2 (daemon=true), QueryStageCreator-3 (daemon=true), ForkJoinPool.commonPool-worker-2 (daemon=true) =====
[info] Run completed in 22 seconds, 883 milliseconds.
[info] Total number of tests run: 7
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 7, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 487 s (08:07), completed Jun 8, 2023 1:43:15 AM