Add `advanced_parquet_index.rs` example of index in into parquet files #10701

alamb · 2024-05-28T19:06:00Z

Which issue does this PR close?

Rationale for this change

This shows how to use the APIs in #9929

Building and using external indexes in DataFusion is an important feature. Adding an example of how to do so will help drive the design and APIs and help other people discover it more easiy

Specifically, this example illustrates how to configure the ParquetExec to read only some row groups / pages within a file and shows how to avoid reading the metadata each time. This is required for low latency parquet query access

What changes are included in this PR?

Add a new example: parquet_index_advanced

Are these changes tested?

Are there any user-facing changes?

This is an example

alamb · 2024-06-03T19:54:10Z

datafusion/core/src/datasource/physical_plan/parquet/mod.rs

@@ -143,6 +145,12 @@ pub use writer::plan_to_parquet;
 /// custom reader is used, it supplies the metadata directly and this parameter
 /// is ignored. [`ParquetExecBuilder::with_metadata_size_hint`] for more details.
 ///
+/// * External indexes: you can use custom external indexes to control exactly


This is the new feature of ParquetExec: provide a custom external index

alamb · 2024-06-13T11:04:58Z

datafusion/common/src/column.rs

@@ -127,6 +127,13 @@ impl Column {
            })
    }

+    /// return the column's name.


I added some small access APIs to make the example easier to read. I can revert these or put them in a different PR if reviewers prefer

adriangb · 2024-06-13T20:31:00Z

datafusion-examples/examples/advanced_parquet_index.rs

+    /// Create a new IndexTableProvider
+    /// * `dir` - the directory containing the parquet files
+    /// * `object_store` - the object store implementation to use for reading the files


Seems like this is only taking an object store now and creating the directory internally

good call -- fixed in 79e1476

…t_index

Weijun-H

Nice example! Thanks @alamb 👍

datafusion-examples/README.md

datafusion-examples/examples/advanced_parquet_index.rs

@Weijun-H

Thank you @Weijun-H Co-authored-by: Alex Huang <[email protected]>

Co-authored-by: Alex Huang <[email protected]>

alamb · 2024-06-14T11:45:22Z

datafusion-examples/examples/advanced_parquet_index.rs

+/// [`SessionContext::read_parquet`] or [`ListingTable`], which also do file
+/// pruning based on parquet statistics (using the same underlying APIs)
+///
+/// # Diagram


I added a diagram here to try and explain what is going on visually for those so inclined

Just curious, do you generate these by hand? I was making them by hand with auto complete assistance, but wondering if there's a better tool.

I use this https://monodraw.helftone.com/

I think there are other online ascii art diagrammers, but that monodraw works well for me

…t_index

alamb · 2024-06-18T12:44:52Z

This PR is just waiting for a committer to approve it so I can merge it in. @crepererum or @tustvold might you have a moment to do so?

advancedxy

Thanks for the example, the ascii doc is pure art.

advancedxy · 2024-06-19T13:43:14Z

datafusion-examples/examples/advanced_parquet_index.rs

+    // in this case, the access plan specifies skipping 8 row groups
+    // and scanning 2 of them. The skipped row groups are not read at all
+    //
+    // [Skip, Skip, Scan, Skip, Skip, Skip, Skip, Scan, Skip, Skip]
+    //
+    // Note that the parquet reader only does 2 IOs - one for the data from each
+    // row group.


hmm, I think it might be more clear to move these comments in L184 and L185?

It seems a bit unclear and weird considering the following comments in L197.

yeah, I agree having the comments about what happened after the statement that does it is a big strange. I updated it in 58b2ba6

WHile doing that I realized the comments are old -- this final query makes only a single IO. I updated that as well

advancedxy · 2024-06-19T13:43:27Z

datafusion-examples/examples/advanced_parquet_index.rs

+    // In this case, the access plan specifies skipping all but the last row group
+    // and within the last row group, reading only the row with id 950
+    //
+    // [Skip, Skip, Skip, Skip, Skip, Skip, Skip, Skip, Skip, Selection(skip 49, select 1, skip 50)]
+    //
+    // In order to prune pages, the Page Index must be loaded. This PageIndex is
+    // loaded in a separate IO request, so the parquet reader makes 2 IO
+    // requests for this query.


…t_index

alamb · 2024-06-20T10:14:48Z

@crepererum do you by any chance have a few minutes to review and approve this PR?

alamb · 2024-06-21T16:48:16Z

I can't merge this PR until a committer approves it:

@Dandandan or @thinkharderdev any chance you have some time to review an example (mostly comments)?

thinkharderdev

very nice!

thinkharderdev · 2024-06-21T19:50:38Z

datafusion-examples/examples/advanced_parquet_index.rs

+/// which will skip unneeded data pages:
+///
+/// ```text
+///         ┌───────────────────────┐   If the RowSelection does not include any


Maybe add a note here that this can only happen if the parquet file has a PageIndex?

Good call -- done in d2f6477

…t_index

alamb · 2024-06-22T12:17:15Z

Thank you everyone for the great comments and feedback

@Weijun-H

apache#10701) * Add `advanced_parquet_index.rs` example of indexing into parquet files * pre-load page index * fix comment * Apply suggestions from code review Thank you @Weijun-H Co-authored-by: Alex Huang <[email protected]> * Add ASCII ART * Update datafusion-examples/README.md Co-authored-by: Alex Huang <[email protected]> * Update datafusion-examples/examples/advanced_parquet_index.rs Co-authored-by: Alex Huang <[email protected]> * Improve / clarify comments based on review * Add page index caveat --------- Co-authored-by: Alex Huang <[email protected]>

@Weijun-H

apache#10701) * Add `advanced_parquet_index.rs` example of indexing into parquet files * pre-load page index * fix comment * Apply suggestions from code review Thank you @Weijun-H Co-authored-by: Alex Huang <[email protected]> * Add ASCII ART * Update datafusion-examples/README.md Co-authored-by: Alex Huang <[email protected]> * Update datafusion-examples/examples/advanced_parquet_index.rs Co-authored-by: Alex Huang <[email protected]> * Improve / clarify comments based on review * Add page index caveat --------- Co-authored-by: Alex Huang <[email protected]>

@Weijun-H

apache#10701) * Add `advanced_parquet_index.rs` example of indexing into parquet files * pre-load page index * fix comment * Apply suggestions from code review Thank you @Weijun-H Co-authored-by: Alex Huang <[email protected]> * Add ASCII ART * Update datafusion-examples/README.md Co-authored-by: Alex Huang <[email protected]> * Update datafusion-examples/examples/advanced_parquet_index.rs Co-authored-by: Alex Huang <[email protected]> * Improve / clarify comments based on review * Add page index caveat --------- Co-authored-by: Alex Huang <[email protected]>

alamb changed the title ~~Alamb/advanced parquet index~~ Add parquet_index_advanced.rs example of index in into parquet files May 28, 2024

This was referenced May 28, 2024

Minor: Clarify docs on EnabledStatistics apache/arrow-rs#5812

Merged

Minor: Add examples for ColumnPath::from apache/arrow-rs#5813

Merged

alamb force-pushed the alamb/advanced_parquet_index branch 2 times, most recently from 2f1b23f to 3221378 Compare May 29, 2024 21:35

github-actions bot added the core Core DataFusion crate label May 29, 2024

alamb mentioned this pull request May 31, 2024

Add ParquetAccessPlan, unify RowGroup selection and PagePruning selection #10738

Merged

alamb force-pushed the alamb/advanced_parquet_index branch from b7af342 to 2e860ad Compare June 3, 2024 19:53

alamb commented Jun 3, 2024

View reviewed changes

alamb mentioned this pull request Jun 6, 2024

Support user defined ParquetAccessPlan in ParquetExec, validation to ParquetAccessPlan::select #10813

Merged

alamb force-pushed the alamb/advanced_parquet_index branch 3 times, most recently from 8969b55 to 7ff0bbd Compare June 12, 2024 21:14

Add advanced_parquet_index.rs example of indexing into parquet files

96ee03c

alamb force-pushed the alamb/advanced_parquet_index branch from 7ff0bbd to 96ee03c Compare June 13, 2024 11:02

alamb marked this pull request as ready for review June 13, 2024 11:21

alamb commented Jun 13, 2024

View reviewed changes

alamb changed the title ~~Add parquet_index_advanced.rs example of index in into parquet files~~ Add advanced_parquet_index.rs example of index in into parquet files Jun 13, 2024

hengfeiyang approved these changes Jun 13, 2024

View reviewed changes

adriangb reviewed Jun 13, 2024

View reviewed changes

alamb added 3 commits June 13, 2024 20:14

pre-load page index

031878f

Merge remote-tracking branch 'apache/main' into alamb/advanced_parque…

9d00ed9

…t_index

fix comment

79e1476

This was referenced Jun 14, 2024

Document when the ParquetRecordBatchReader will re-read metadata apache/arrow-rs#5887

Merged

Docs: clarify when the parquet reader will read from object store when using cached metadata #10909

Merged

Weijun-H approved these changes Jun 14, 2024

View reviewed changes

alamb and others added 4 commits June 14, 2024 06:29

Apply suggestions from code review

9f3a5f9

Thank you @Weijun-H Co-authored-by: Alex Huang <[email protected]>

Add ASCII ART

0a2bcfa

Update datafusion-examples/README.md

05febf8

Co-authored-by: Alex Huang <[email protected]>

Update datafusion-examples/examples/advanced_parquet_index.rs

9030e7b

Co-authored-by: Alex Huang <[email protected]>

alamb commented Jun 14, 2024

View reviewed changes

alamb mentioned this pull request Jun 17, 2024

Advanced example for building an external index for Row Groups *within* parquet files #10580

Closed

Merge remote-tracking branch 'apache/main' into alamb/advanced_parque…

9be984d

…t_index

alamb added the documentation Improvements or additions to documentation label Jun 18, 2024

advancedxy approved these changes Jun 19, 2024

View reviewed changes

alamb added 2 commits June 19, 2024 17:19

Merge remote-tracking branch 'apache/main' into alamb/advanced_parque…

e9caa0d

…t_index

Improve / clarify comments based on review

58b2ba6

github-actions bot removed the documentation Improvements or additions to documentation label Jun 19, 2024

advancedxy approved these changes Jun 20, 2024

View reviewed changes

thinkharderdev approved these changes Jun 21, 2024

View reviewed changes

alamb added 2 commits June 21, 2024 17:02

Merge remote-tracking branch 'apache/main' into alamb/advanced_parque…

28dceb7

…t_index

Add page index caveat

d2f6477

alamb merged commit ea46e82 into apache:main Jun 22, 2024
23 checks passed

alamb deleted the alamb/advanced_parquet_index branch June 22, 2024 12:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `advanced_parquet_index.rs` example of index in into parquet files #10701

Add `advanced_parquet_index.rs` example of index in into parquet files #10701

alamb commented May 28, 2024 •

edited

Loading

alamb Jun 3, 2024

alamb Jun 13, 2024

adriangb Jun 13, 2024

alamb Jun 14, 2024

Weijun-H left a comment

alamb Jun 14, 2024

adriangb Jun 14, 2024

alamb Jun 15, 2024

alamb commented Jun 18, 2024

advancedxy left a comment

advancedxy Jun 19, 2024

alamb Jun 19, 2024

advancedxy Jun 19, 2024

alamb commented Jun 20, 2024

alamb commented Jun 21, 2024

thinkharderdev left a comment

thinkharderdev Jun 21, 2024

alamb Jun 21, 2024

alamb commented Jun 22, 2024

Add advanced_parquet_index.rs example of index in into parquet files #10701

Add advanced_parquet_index.rs example of index in into parquet files #10701

Conversation

alamb commented May 28, 2024 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Weijun-H left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Jun 18, 2024

advancedxy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Jun 20, 2024

alamb commented Jun 21, 2024

thinkharderdev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alamb commented Jun 22, 2024

Add `advanced_parquet_index.rs` example of index in into parquet files #10701

Add `advanced_parquet_index.rs` example of index in into parquet files #10701

alamb commented May 28, 2024 •

edited

Loading