Skip to content

[SPARK-34863][SQL] Support complex types for Parquet vectorized reader #34659

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sunchao wants to merge 15 commits into from
Closed

[SPARK-34863][SQL] Support complex types for Parquet vectorized reader #34659

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
1ac4793
wip
sunchao Nov 9, 2021
1452d53
more tests
sunchao Nov 9, 2021
22fdbe4
update golden files
sunchao Nov 9, 2021
033d129
more fixes
sunchao Nov 18, 2021
ee50e4c
address comments
sunchao Dec 2, 2021
999df58
fix logic to enable/disable batch reads
sunchao Dec 20, 2021
4782dfb
fix lint
sunchao Dec 22, 2021
8a5b5b7
Address perf issue in ParquetSchemaConverter
sunchao Jan 4, 2022
aaad383
space
sunchao Jan 25, 2022
7e96a18
address comments
sunchao Feb 10, 2022
8c6c2ec
remove more unnecessary changes
sunchao Mar 3, 2022
71192f4
address more comments
sunchao Mar 30, 2022
4f6efa9
address more comments
sunchao Mar 31, 2022
cd6ec99
disable by default
sunchao Mar 31, 2022
91da32d
address comments and turn on config
sunchao Apr 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1009,6 +1009,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)

val PARQUET_VECTORIZED_READER_NESTED_COLUMN_ENABLED =
buildConf("spark.sql.parquet.enableNestedColumnVectorizedReader")
.doc("Enables vectorized Parquet decoding for nested columns (e.g., struct, list, map). " +
s"Requires ${PARQUET_VECTORIZED_READER_ENABLED.key} to be enabled.")
.version("3.3.0")
.booleanConf
.createWithDefault(true)
Copy link

@agrawaldevesh agrawaldevesh Nov 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am liking the confidence :-)

Copy link
Member Author

@sunchao sunchao Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to make sure we can pass all Parquet related tests :) I plan to turn it off later separately.

Copy link
Member

@viirya viirya Mar 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure we remember to turn it off as default before merging. 😄


val PARQUET_RECORD_FILTER_ENABLED = buildConf("spark.sql.parquet.recordLevelFilter.enabled")
.doc("If true, enables Parquet's native record-level filtering using the pushed down " +
"filters. " +
Expand Down Expand Up @@ -3926,6 +3934,9 @@ class SQLConf extends Serializable with Logging {

def parquetVectorizedReaderEnabled: Boolean = getConf(PARQUET_VECTORIZED_READER_ENABLED)

def parquetVectorizedReaderNestedColumnEnabled: Boolean =
getConf(PARQUET_VECTORIZED_READER_NESTED_COLUMN_ENABLED)

def parquetVectorizedReaderBatchSize: Int = getConf(PARQUET_VECTORIZED_READER_BATCH_SIZE)

def columnBatchSize: Int = getConf(COLUMN_BATCH_SIZE)
Expand Down
Loading