TEST: use new async parquet reader #18385

alamb · 2025-10-30T14:45:23Z

This is a test PR to confirm the changes in

Rewrite ParquetRecordBatchStream in terms of the PushDecoder arrow-rs#8159

Do not cause any performance regressions

alamb · 2025-10-30T16:41:26Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_push_decoder (21071e2) to 607325a diff using: clickbench_pushdown
Results will be posted here when complete

alamb · 2025-10-30T17:11:41Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_test_push_decoder
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_test_push_decoder ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.49 ms │                 2.25 ms │ +1.11x faster │
│ QQuery 1     │    55.14 ms │                50.23 ms │ +1.10x faster │
│ QQuery 2     │   140.95 ms │               140.23 ms │     no change │
│ QQuery 3     │   168.33 ms │               172.90 ms │     no change │
│ QQuery 4     │  1167.87 ms │              1129.83 ms │     no change │
│ QQuery 5     │  1578.39 ms │              1553.46 ms │     no change │
│ QQuery 6     │     2.21 ms │                 2.20 ms │     no change │
│ QQuery 7     │    74.81 ms │                69.28 ms │ +1.08x faster │
│ QQuery 8     │  1496.01 ms │              1510.03 ms │     no change │
│ QQuery 9     │  1895.52 ms │              1880.73 ms │     no change │
│ QQuery 10    │   507.94 ms │               490.43 ms │     no change │
│ QQuery 11    │   577.62 ms │               554.06 ms │     no change │
│ QQuery 12    │  1623.83 ms │              1615.84 ms │     no change │
│ QQuery 13    │  2509.92 ms │              2563.63 ms │     no change │
│ QQuery 14    │  1703.16 ms │              1685.61 ms │     no change │
│ QQuery 15    │  1279.70 ms │              1260.35 ms │     no change │
│ QQuery 16    │  2680.64 ms │              2736.33 ms │     no change │
│ QQuery 17    │  2719.88 ms │              2714.60 ms │     no change │
│ QQuery 18    │  5432.67 ms │              5231.24 ms │     no change │
│ QQuery 19    │   149.75 ms │               147.08 ms │     no change │
│ QQuery 20    │  2041.01 ms │              2030.34 ms │     no change │
│ QQuery 21    │  2502.95 ms │              2460.44 ms │     no change │
│ QQuery 22    │  4288.61 ms │              4272.66 ms │     no change │
│ QQuery 23    │  1157.07 ms │              1210.83 ms │     no change │
│ QQuery 24    │   274.18 ms │               251.35 ms │ +1.09x faster │
│ QQuery 25    │   705.13 ms │               676.99 ms │     no change │
│ QQuery 26    │   360.51 ms │               365.26 ms │     no change │
│ QQuery 27    │  3260.37 ms │              3126.34 ms │     no change │
│ QQuery 28    │ 24287.11 ms │             23704.26 ms │     no change │
│ QQuery 29    │  1017.57 ms │               965.79 ms │ +1.05x faster │
│ QQuery 30    │  2245.61 ms │              1977.17 ms │ +1.14x faster │
│ QQuery 31    │  2123.12 ms │              1967.44 ms │ +1.08x faster │
│ QQuery 32    │  4756.22 ms │              4647.30 ms │     no change │
│ QQuery 33    │  6052.26 ms │              5886.23 ms │     no change │
│ QQuery 34    │  6242.68 ms │              6185.08 ms │     no change │
│ QQuery 35    │  2142.31 ms │              2029.97 ms │ +1.06x faster │
│ QQuery 36    │    25.55 ms │                27.83 ms │  1.09x slower │
│ QQuery 37    │    25.82 ms │                27.09 ms │     no change │
│ QQuery 38    │    25.63 ms │                25.17 ms │     no change │
│ QQuery 39    │    26.31 ms │                25.34 ms │     no change │
│ QQuery 40    │    27.92 ms │                27.23 ms │     no change │
│ QQuery 41    │    26.98 ms │                26.11 ms │     no change │
│ QQuery 42    │    26.33 ms │                26.21 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 89408.08ms │
│ Total Time (alamb_test_push_decoder)   │ 87452.72ms │
│ Average Time (HEAD)                    │  2079.26ms │
│ Average Time (alamb_test_push_decoder) │  2033.78ms │
│ Queries Faster                         │          8 │
│ Queries Slower                         │          1 │
│ Queries with No Change                 │         34 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

alamb · 2025-10-30T17:11:44Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_push_decoder (21071e2) to 607325a diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

alamb · 2025-10-30T17:56:57Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_test_push_decoder
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_test_push_decoder ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2776.33 ms │              2747.99 ms │     no change │
│ QQuery 1     │  1328.17 ms │              1287.18 ms │     no change │
│ QQuery 2     │  2546.02 ms │              2411.56 ms │ +1.06x faster │
│ QQuery 3     │  1187.94 ms │              1209.33 ms │     no change │
│ QQuery 4     │  2280.97 ms │              2251.89 ms │     no change │
│ QQuery 5     │ 28524.18 ms │             27998.24 ms │     no change │
│ QQuery 6     │  4255.90 ms │              4350.39 ms │     no change │
│ QQuery 7     │  3546.16 ms │              3539.33 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 46445.67ms │
│ Total Time (alamb_test_push_decoder)   │ 45795.91ms │
│ Average Time (HEAD)                    │  5805.71ms │
│ Average Time (alamb_test_push_decoder) │  5724.49ms │
│ Queries Faster                         │          1 │
│ Queries Slower                         │          0 │
│ Queries with No Change                 │          7 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_test_push_decoder ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.64 ms │                 2.57 ms │     no change │
│ QQuery 1     │    51.64 ms │                51.81 ms │     no change │
│ QQuery 2     │   140.01 ms │               143.97 ms │     no change │
│ QQuery 3     │   162.55 ms │               168.17 ms │     no change │
│ QQuery 4     │  1148.26 ms │              1127.32 ms │     no change │
│ QQuery 5     │  1515.87 ms │              1544.56 ms │     no change │
│ QQuery 6     │     2.19 ms │                 2.31 ms │  1.06x slower │
│ QQuery 7     │    55.28 ms │                59.58 ms │  1.08x slower │
│ QQuery 8     │  1473.14 ms │              1480.76 ms │     no change │
│ QQuery 9     │  1863.04 ms │              1920.86 ms │     no change │
│ QQuery 10    │   433.76 ms │               394.75 ms │ +1.10x faster │
│ QQuery 11    │   465.75 ms │               449.21 ms │     no change │
│ QQuery 12    │  1473.71 ms │              1460.31 ms │     no change │
│ QQuery 13    │  2232.15 ms │              2241.46 ms │     no change │
│ QQuery 14    │  1317.09 ms │              1305.85 ms │     no change │
│ QQuery 15    │  1249.75 ms │              1223.81 ms │     no change │
│ QQuery 16    │  2737.27 ms │              2713.18 ms │     no change │
│ QQuery 17    │  2694.34 ms │              2695.66 ms │     no change │
│ QQuery 18    │  5046.61 ms │              5029.62 ms │     no change │
│ QQuery 19    │   128.33 ms │               130.61 ms │     no change │
│ QQuery 20    │  2061.23 ms │              2083.27 ms │     no change │
│ QQuery 21    │  2383.01 ms │              2412.93 ms │     no change │
│ QQuery 22    │  4106.84 ms │              4081.94 ms │     no change │
│ QQuery 23    │ 13124.63 ms │             13092.55 ms │     no change │
│ QQuery 24    │   230.51 ms │               221.51 ms │     no change │
│ QQuery 25    │   523.45 ms │               523.84 ms │     no change │
│ QQuery 26    │   215.79 ms │               225.98 ms │     no change │
│ QQuery 27    │  3007.59 ms │              2876.25 ms │     no change │
│ QQuery 28    │ 22972.32 ms │             22942.02 ms │     no change │
│ QQuery 29    │   991.47 ms │               975.66 ms │     no change │
│ QQuery 30    │  1363.44 ms │              1334.95 ms │     no change │
│ QQuery 31    │  1375.42 ms │              1369.78 ms │     no change │
│ QQuery 32    │  4885.64 ms │              4691.12 ms │     no change │
│ QQuery 33    │  5857.62 ms │              5794.20 ms │     no change │
│ QQuery 34    │  6192.46 ms │              6044.52 ms │     no change │
│ QQuery 35    │  2097.17 ms │              2037.86 ms │     no change │
│ QQuery 36    │   119.73 ms │               125.58 ms │     no change │
│ QQuery 37    │    52.74 ms │                56.11 ms │  1.06x slower │
│ QQuery 38    │   123.45 ms │               123.62 ms │     no change │
│ QQuery 39    │   199.36 ms │               214.08 ms │  1.07x slower │
│ QQuery 40    │    44.76 ms │                41.78 ms │ +1.07x faster │
│ QQuery 41    │    40.89 ms │                39.44 ms │     no change │
│ QQuery 42    │    32.83 ms │                32.86 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 96195.73ms │
│ Total Time (alamb_test_push_decoder)   │ 95488.20ms │
│ Average Time (HEAD)                    │  2237.11ms │
│ Average Time (alamb_test_push_decoder) │  2220.66ms │
│ Queries Faster                         │          2 │
│ Queries Slower                         │          4 │
│ Queries with No Change                 │         37 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ alamb_test_push_decoder ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 174.40 ms │               179.86 ms │     no change │
│ QQuery 2     │  28.81 ms │                27.91 ms │     no change │
│ QQuery 3     │  35.52 ms │                38.61 ms │  1.09x slower │
│ QQuery 4     │  29.19 ms │                28.58 ms │     no change │
│ QQuery 5     │  77.52 ms │                79.17 ms │     no change │
│ QQuery 6     │  19.95 ms │                19.51 ms │     no change │
│ QQuery 7     │ 228.11 ms │               228.03 ms │     no change │
│ QQuery 8     │  35.82 ms │                34.29 ms │     no change │
│ QQuery 9     │ 110.58 ms │               113.49 ms │     no change │
│ QQuery 10    │  60.91 ms │                60.63 ms │     no change │
│ QQuery 11    │  17.80 ms │                17.29 ms │     no change │
│ QQuery 12    │  53.01 ms │                51.18 ms │     no change │
│ QQuery 13    │  49.08 ms │                47.42 ms │     no change │
│ QQuery 14    │  13.85 ms │                13.95 ms │     no change │
│ QQuery 15    │  26.08 ms │                25.21 ms │     no change │
│ QQuery 16    │  28.13 ms │                24.76 ms │ +1.14x faster │
│ QQuery 17    │ 159.89 ms │               151.62 ms │ +1.05x faster │
│ QQuery 18    │ 340.29 ms │               333.54 ms │     no change │
│ QQuery 19    │  37.22 ms │                38.63 ms │     no change │
│ QQuery 20    │  50.85 ms │                50.97 ms │     no change │
│ QQuery 21    │ 363.03 ms │               345.64 ms │     no change │
│ QQuery 22    │  20.97 ms │                20.53 ms │     no change │
└──────────────┴───────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 1961.00ms │
│ Total Time (alamb_test_push_decoder)   │ 1930.84ms │
│ Average Time (HEAD)                    │   89.14ms │
│ Average Time (alamb_test_push_decoder) │   87.77ms │
│ Queries Faster                         │         2 │
│ Queries Slower                         │         1 │
│ Queries with No Change                 │        19 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

alamb · 2025-10-30T18:17:51Z

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1017-gcp #18~24.04.1-Ubuntu SMP Tue Sep 23 17:51:44 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/test_push_decoder (21071e2) to 607325a diff using: clickbench_pushdown
Results will be posted here when complete

alamb · 2025-10-30T18:33:11Z

🤖: Benchmark completed

Details

Comparing HEAD and alamb_test_push_decoder
--------------------
Benchmark clickbench_pushdown.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ alamb_test_push_decoder ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.80 ms │                 2.32 ms │ +1.21x faster │
│ QQuery 1     │    53.48 ms │                55.58 ms │     no change │
│ QQuery 2     │   143.82 ms │               143.74 ms │     no change │
│ QQuery 3     │   163.94 ms │               165.43 ms │     no change │
│ QQuery 4     │  1184.19 ms │              1256.83 ms │  1.06x slower │
│ QQuery 5     │  1585.21 ms │              1525.51 ms │     no change │
│ QQuery 6     │     2.14 ms │                 2.56 ms │  1.20x slower │
│ QQuery 7     │    73.87 ms │                72.05 ms │     no change │
│ QQuery 8     │  1544.75 ms │              1495.37 ms │     no change │
│ QQuery 9     │  1935.81 ms │              1936.69 ms │     no change │
│ QQuery 10    │   505.38 ms │               490.36 ms │     no change │
│ QQuery 11    │   593.99 ms │               547.13 ms │ +1.09x faster │
│ QQuery 12    │  1691.75 ms │              1627.43 ms │     no change │
│ QQuery 13    │  2639.94 ms │              2562.09 ms │     no change │
│ QQuery 14    │  1773.46 ms │              1665.89 ms │ +1.06x faster │
│ QQuery 15    │  1285.22 ms │              1264.29 ms │     no change │
│ QQuery 16    │  2784.57 ms │              2678.13 ms │     no change │
│ QQuery 17    │  2706.22 ms │              2668.85 ms │     no change │
│ QQuery 18    │  5329.87 ms │              5087.38 ms │     no change │
│ QQuery 19    │   148.36 ms │               146.31 ms │     no change │
│ QQuery 20    │  2073.01 ms │              2026.16 ms │     no change │
│ QQuery 21    │  2529.03 ms │              2461.37 ms │     no change │
│ QQuery 22    │  4420.95 ms │              4268.38 ms │     no change │
│ QQuery 23    │  1225.25 ms │              1150.61 ms │ +1.06x faster │
│ QQuery 24    │   268.15 ms │               271.87 ms │     no change │
│ QQuery 25    │   694.65 ms │               681.87 ms │     no change │
│ QQuery 26    │   364.57 ms │               364.55 ms │     no change │
│ QQuery 27    │  3348.96 ms │              3135.36 ms │ +1.07x faster │
│ QQuery 28    │ 23840.45 ms │             23769.52 ms │     no change │
│ QQuery 29    │   995.63 ms │               996.39 ms │     no change │
│ QQuery 30    │  2270.46 ms │              1997.47 ms │ +1.14x faster │
│ QQuery 31    │  2170.19 ms │              1938.40 ms │ +1.12x faster │
│ QQuery 32    │  4491.13 ms │              4437.91 ms │     no change │
│ QQuery 33    │  5960.55 ms │              5777.31 ms │     no change │
│ QQuery 34    │  6172.09 ms │              6240.81 ms │     no change │
│ QQuery 35    │  2177.59 ms │              2016.76 ms │ +1.08x faster │
│ QQuery 36    │    26.12 ms │                26.42 ms │     no change │
│ QQuery 37    │    26.67 ms │                25.57 ms │     no change │
│ QQuery 38    │    25.44 ms │                25.53 ms │     no change │
│ QQuery 39    │    26.10 ms │                25.30 ms │     no change │
│ QQuery 40    │    27.11 ms │                27.27 ms │     no change │
│ QQuery 41    │    25.75 ms │                26.77 ms │     no change │
│ QQuery 42    │    25.78 ms │                26.21 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 89334.37ms │
│ Total Time (alamb_test_push_decoder)   │ 87111.78ms │
│ Average Time (HEAD)                    │  2077.54ms │
│ Average Time (alamb_test_push_decoder) │  2025.86ms │
│ Queries Faster                         │          8 │
│ Queries Slower                         │          2 │
│ Queries with No Change                 │         33 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘

alamb · 2025-10-30T18:41:53Z

TLDR is I don't see any substantial change here (means test has passed!)

TEST: use new async parquet reader

21071e2

alamb mentioned this pull request Oct 30, 2025

Rewrite ParquetRecordBatchStream in terms of the PushDecoder apache/arrow-rs#8159

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

TEST: use new async parquet reader #18385

TEST: use new async parquet reader #18385

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

TEST: use new async parquet reader #18385

Are you sure you want to change the base?

TEST: use new async parquet reader #18385

Uh oh!

Conversation

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025

Uh oh!

alamb commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

alamb commented Oct 30, 2025 •

edited

Loading