Blog post about efficient filter representation in Parquet filter pushdown by hhhizzz · Pull Request #10 · hhhizzz/arrow-rs

hhhizzz · 2025-11-30T11:36:36Z

apache#8843

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

alamb

First of all, this is amazing @hhhizzz -- I really like how you frame the post / deocder as a mini query engine and then proceed to teach the reader what late materialization is and how it works.

The figures are also quite amazing.

I suggest you incorporate any suggestions you would like, and then we move this content to the arrow-site repo for final polish and iteration.

Also, if you are willing, I am happy to help work on / propose changes directly to the blog, but I don't want to make a bunch of conflicts if you still have changes you plan to make.

Mechanics of posting to Arrow Blog

To publish this content on the arrow blog (https://arrow.apache.org/blog), we need to make a PR to https://github.com/apache/arrow-site. Once the PR to arrow-site is merged, it will automatically be published

Here is an example of such a PR apache/arrow-site#720 (I am also happy to help translate this content into a PR as well)

alamb · 2025-12-02T12:27:18Z

I found this figure slightly confusing due to the fact that the upper right was different than the other two images

Added the Query in image.

alamb · 2025-12-02T12:27:59Z

I didn't see this image referenced anywhere in the file

Added in the doc

alamb · 2025-12-02T12:29:41Z

I really like this image as a high level overview of the late materialization process -- but it didn't seem to be referenced anywhere in the text

alamb · 2025-12-02T12:31:40Z

I found this image a little confusing for two reasons:

There is no Step 2

The "Decoded B Values" didn't make sense -- there were 3 decoded values, but Row Mask 1 seems to have 5 that pass (so I think Decoded B Values should have 5 green boxes)

Fixed in the new image

alamb · 2025-12-02T12:34:06Z

This is (also) an amazing image. One minor suggestion is to use a term other than "Track"

Perhaps instead of "Track A (Selection)" you could use a term like "A > 10 Filter Results (Selection)"

And instead of "Track B" use a term like "B < 5 Filter Results"

And then instead of "Result Track" use a term "Final Filter Result"

Updated in the new image.

alamb · 2025-12-02T13:40:07Z

+Starting from [v57.1.0](https://github.com/apache/arrow-rs/tree/57.1.0), `RowSelection` can switch between RLE (selectors) and bitmask. Bitmasks are faster when gaps are tiny and sparsity is high; RLE is friendlier to page-level skips. Details show up again in 3.3.
+
+**Execution sketch** (`SELECT * FROM table WHERE A > 10 AND B < 5`):
+1. **Initial**: `selection = None` (equivalent to "select all").


Eventually I suggest we add links to the code / docs for ArrayReader, ReadPlanBuilder, etc -- I suggest we do this after a few more passes on the content

alamb · 2025-12-02T13:41:25Z

+
+---
+
+## 2. Key Mechanics


I suggest calling this section "Late Materialization in the Rust Parquet Reader" and stop after section 2.2 where you go through the actual code implementation that matches the high level overview you gave in section 1

(in other words, I suggest starting a new major section starting at what is currently "### 2.3 Smart caching"

alamb · 2025-12-02T13:49:42Z

+
+![](fig3.png)
+
+### 2.3 Smart caching


I think this would be a great place to start the "## 3. Engineering challenges" section

It would help I think if the introduction to that section said something like

While relatively straightforward in theory, actually implementing Late Materialization in a production grade system such as arrow-rs requires significant engineering. Previously the effort required meant that such technology is typically only available in proprietary engines and has been a struggle to implement in the open source community (see the DataFusion ticket about enabling filter pushdown). Now, after several years of effort, we are finally close to having Late Materialization as fast or faster in all cases. Getting to this point required several major implementation details, which are described below

alamb · 2025-12-02T13:52:20Z

+
+![](fig4.png)
+
+### 3.3 Adaptive RowSelection policy (bitmask vs. RLE)


I recommend moving this section to the top of the engineering challenges section -- and title it "Row Selection Representation"

The idea being that different RowSelection representations have different tradeoffs for different filter patterns, and that in order to improve performance, we needed to provide different implementations

alamb · 2025-12-02T13:54:10Z

+```
+
+#### 3.3.2 The bitmask trap: missing pages
+The bitmask introduces a new failure mode. Consider a simple example:


If we want to include this section, i think we need to set it up a bit more carefully (give it some more context).

Maybe we can talk about it as "an example of why high performance engineering is so complicated -- we can also link to the ticket for the follow on work. I'll dig that up later

hhhizzz · 2025-12-02T16:41:21Z

@alamb Thanks for the review! I’ve updated the images using the new drawing method. It’s getting late here (UTC+8), so I’ll update the rest tomorrow.
Feel free to make any changes directly! And it would be really helpful if you can help to transfer it to the arrow-site repo.

alamb · 2025-12-02T19:48:32Z

@alamb Thanks for the review! I’ve updated the images using the new drawing method. It’s getting late here (UTC+8), so I’ll update the rest tomorrow. Feel free to make any changes directly! And it would be really helpful if you can help to transfer it to the arrow-site repo.

Thank you!

I will try and make a PR in the arrow-site repo tomorrow. Note that if I create the PR, then in order for you to edit it, you will need to make PRs to my fork (which is fine)

If you would like to retain the ability to edit the post directly (which is probably good) you will need to open the PR.

I can create a branch and then you could potentially open a PR 🤔

hhhizzz · 2025-12-03T04:04:02Z

@alamb Thanks for the review! I’ve updated the images using the new drawing method. It’s getting late here (UTC+8), so I’ll update the rest tomorrow. Feel free to make any changes directly! And it would be really helpful if you can help to transfer it to the arrow-site repo.

Thank you!

I will try and make a PR in the arrow-site repo tomorrow. Note that if I create the PR, then in order for you to edit it, you will need to make PRs to my fork (which is fine)

If you would like to retain the ability to edit the post directly (which is probably good) you will need to open the PR.

I can create a branch and then you could potentially open a PR 🤔

I just realized I also want to publish a Chinese translation at the same time. I think it’s better if I create the PR and update both the English and Chinese versions based on your comments or anyone else’s feedback.

hhhizzz · 2025-12-03T16:01:48Z

@alamb I submitted the PR, let me know if anything I can improve. apache/arrow-site#740

@alamb

…ads (#740) - closes apache/arrow-rs#8843 See preview URL: https://hhhizzz.github.io/arrow-site/blog/ cc @alamb Original blog and Chinese translation. See earlier draft here: hhhizzz/arrow-rs#10 --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Add blog

952f687

chatgpt-codex-connector Bot reviewed Nov 30, 2025

View reviewed changes

Comment thread parquet/tests/tpcds_inv_quantity.rs Outdated

github-actions Bot added the parquet label Nov 30, 2025

hhhizzz added 2 commits November 30, 2025 20:10

Some miner fix

2a1724f

Some miner fix

4da9036

hhhizzz mentioned this pull request Nov 30, 2025

Blog post about efficient filter representation in Parquet filter pushdown apache/arrow-rs#8843

Closed

alamb reviewed Dec 2, 2025

View reviewed changes

hhhizzz added 2 commits December 2, 2025 22:10

remove the unrelated data

cd51206

Update the image

fd633d6

Update the doc

2538d84

hhhizzz closed this Dec 3, 2025

alamb mentioned this pull request Dec 5, 2025

Blog: Practical Dive Into Late Materialization in arrow-rs Parquet Reads apache/arrow-site#740

Merged


		![](fig4.png)

		### 3.3 Adaptive RowSelection policy (bitmask vs. RLE)

Conversation

hhhizzz commented Nov 30, 2025

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Mechanics of posting to Arrow Blog

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hhhizzz commented Dec 2, 2025

Uh oh!

alamb commented Dec 2, 2025

Uh oh!

hhhizzz commented Dec 3, 2025

Uh oh!

hhhizzz commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants