Merge up from main and fix logical conflict in zip kernel#1
Merged
maxburke merged 34 commits intourbanlogiq:arrow-6408from Jan 10, 2026
Merged
Merge up from main and fix logical conflict in zip kernel#1maxburke merged 34 commits intourbanlogiq:arrow-6408from
maxburke merged 34 commits intourbanlogiq:arrow-6408from
Conversation
# Which issue does this PR close? N/A # Rationale for this change because `Rows` own the data and offsets vector, it should use the capacity for tracking the size that it uses # What changes are included in this PR? replace `len` with `capacity` and added test # Are these changes tested? yes (and of course they are failing on main and passing with this fix) # Are there any user-facing changes? more accurate size
…f space (apache#9043) # Which issue does this PR close? - Closes apache#9024. # Rationale for this change the ci container starts with 63gb / 72gb used, the 9GB remaining disk space is barely enough for a cross build in 7 languages that leads to ci being stuck. this is what a debug step after initialize container shows === CONTAINER DISK USAGE === Filesystem Size Used Avail Use% Mounted on overlay 72G 63G 9.5G 87% / # What changes are included in this PR? - add resource monitoring to build process - add a clean up step to remove unnecessary software (cuts 6GB of space) === Cleaning up host disk space === Disk space before cleanup: Filesystem Size Used Avail Use% Mounted on overlay 72G 63G 9.5G 87% / Disk space after cleanup: Filesystem Size Used Avail Use% Mounted on overlay 72G 57G 16G 79% / - add a small optimization to shallow clone (only clone most recent commit not full history) for github repos optimization results we have 6.1 GB left after build === After Build === Filesystem Size Used Avail Use% Mounted on overlay 72G 66G 6.1G 92% / # Are these changes tested? tested by github ci # Are there any user-facing changes? no --------- Signed-off-by: lyang24 <lanqingy93@gmail.com>
# Which issue does this PR close? - Closes apache#8946 # What changes are included in this PR? The PR adds support for parsing array index (eg. `foo.bar[3]`) with the help of parse_path fn. Currently the parser silently parses invalid segments as Field (eg., `foo[0`, `[0]`(parsed as index), `foo0]`, `foo[0][`) #### Feedback requested Whether to add stricter validation (throw an error) and reject the segment ? Or to keep the current behavior ? # Are these changes tested? yes, only for valid inputs # Are there any user-facing changes? no
# Which issue does this PR close? - RElated to apache/datafusion#19477 # Rationale for this change While profiling apache/datafusion#19477 I noticed some additional clones we could avoid <img width="1504" height="927" alt="Screenshot 2025-12-26 at 12 03 00 PM" src="https://github.com/user-attachments/assets/958f76e2-53d0-4008-b224-d9275984cd1a" /> I doubt this will be a huge deal but it does remove some allocations int he parquet read path # What changes are included in this PR? Use `into_data` rather than `to_data` # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
# Which issue does this PR close? N/A # Rationale for this change I have a PR to improve zip perf for Utf8View/BinaryView scalars and I need benchmarks for that. - apache#8963 # What changes are included in this PR? This extends the zip benchmarks by one new Input Generator for StringViews and two more functions to test scalar combinations of different StringViews combinations. # Are these changes tested? N/A # Are there any user-facing changes? No
# Which issue does this PR close? - Closes apache#9055. # What changes are included in this PR? Changes to docstrings for `min` and `max` functions. # Are these changes tested? Yes # Are there any user-facing changes? Yes. These doc pages will be updated: https://docs.rs/arrow/latest/arrow/compute/fn.min.html https://docs.rs/arrow/latest/arrow/compute/fn.max.html
…che#8984) # Which issue does this PR close? - Closes apache#8873 . # What changes are included in this PR? Unify the `CastOptions` usage in `parquet-variant-compute` Currently, there is only `arrow::compute::CastOptions` in `parquet-variant-compute` now, the existing `parquet-variant-compute/CastOptions` has replaced by the `arrow::compute::CastOptions` with the equal behavior <img width="1263" height="424" alt="image" src="https://github.com/user-attachments/assets/2e3e17aa-1e69-42fb-91af-43a7cde4bb95" /> # Are these changes tested? The existing tests covered the logic # Are there any user-facing changes? This will break some public API in `parquet-variant-compute`, but this crate is `experiment` now, so maybe we don't need to wait for a major release.
…d is non-nullable and have non-zero `null_density` (apache#9046) # Which issue does this PR close? N/A # Rationale for this change if decimal field is non nullable and have null density we should not have nulls in the genrated array # What changes are included in this PR? Override `null_density` for non nested and non dictionary to avoid future problems like this added assertion and test # Are these changes tested? yes # Are there any user-facing changes? working generate for `Decimal128` and `Decimal256` Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…he#9052) # Which issue does this PR close? its a nitpick to replace "allocation + memcpy" with "allocation only". # Rationale for this change remove the value clone in decode path `decoded_values.push(decoded_data.clone())` and taking from decoded_data directly # What changes are included in this PR? # Are these changes tested? i think the current testing suite will do # Are there any user-facing changes? no Signed-off-by: lyang24 <lanqingy93@gmail.com>
…e#9054) # Which issue does this PR close? N/A # Rationale for this change Make row conversion faster # What changes are included in this PR? created "manual" iterator over the byte array and offsets with optimizations for no nulls # Are these changes tested? Existing tests # Are there any user-facing changes? No
# Which issue does this PR close? - Closes apache#9063. # What changes are included in this PR? Fixed exaxmple headers and removes empty first lines # Are these changes tested? Yes # Are there any user-facing changes? Yes, the following doc pages will be updated: https://docs.rs/arrow/latest/arrow/compute/fn.min_boolean.html https://docs.rs/arrow/latest/arrow/compute/fn.max_boolean.html https://docs.rs/arrow/latest/arrow/compute/fn.lexsort.html https://docs.rs/arrow/latest/arrow/compute/fn.take_record_batch.html https://docs.rs/arrow/latest/arrow/compute/fn.shift.html
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#7236 # Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> The previous documentation for `BooleanBufferBuilder::reserve` mentioned that reading new bytes is undefined behavior. However, as noted in the issue, the safe APIs only expose initialized slices, making this warning misleading. # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Remove misleading UB warning from `BooleanBufferBuilder::reserve`. - Unify documentation for the `reserve` methods to follow a consistent format. # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> No, because it's doc change # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> Yes
…est (apache#9073) # Which issue does this PR close? N/A # Rationale for this change More coverage, I might not be compiling for tests and still want this validation # What changes are included in this PR? replace test with debug assertion # Are these changes tested? No # Are there any user-facing changes? not API change
…pache#8963) # Which issue does this PR close? - Closes apache#8724 # Rationale for this change It's explained in the issue. # What changes are included in this PR? This adds a special implementation for Utf8View/BinaryView scalars for zip based on the design from apache#8653. It also includes tests. Benchmarks are available here: - apache#8988 # Are these changes tested? Yes. # Are there any user-facing changes? There is a new struct `ByteViewScalarImpl`. <details close> <summary>Benchmarks</summary> System: Apple M1 Max with 10 cores on macOS 26.1 ``` group branch main ----- ------ ---- zip_8192_from_string_views size 10 and string_views size 10/non_null_scalar_vs_null_scalar/10pct_true 1.00 3.5±0.04µs ? ?/sec 37.06 128.9±1.36µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_null_scalar_vs_null_scalar/1pct_true 1.00 3.5±0.07µs ? ?/sec 35.76 125.1±1.76µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_null_scalar_vs_null_scalar/50pct_nulls 1.00 3.7±0.12µs ? ?/sec 36.91 136.8±2.17µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_null_scalar_vs_null_scalar/50pct_true 1.00 3.5±0.06µs ? ?/sec 40.30 139.9±2.11µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_null_scalar_vs_null_scalar/90pct_true 1.00 3.6±0.10µs ? ?/sec 30.57 108.5±2.62µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_null_scalar_vs_null_scalar/99pct_true 1.00 3.5±0.05µs ? ?/sec 28.40 99.8±2.12µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_null_scalar_vs_null_scalar/all_false 1.00 3.5±0.02µs ? ?/sec 36.04 127.4±3.14µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_null_scalar_vs_null_scalar/all_true 1.00 3.5±0.08µs ? ?/sec 27.39 97.1±1.11µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_nulls_scalars/10pct_true 1.00 28.2±0.37µs ? ?/sec 2.70 75.9±0.61µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_nulls_scalars/1pct_true 1.00 7.2±0.24µs ? ?/sec 9.89 71.4±12.56µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_nulls_scalars/50pct_nulls 1.00 51.0±2.97µs ? ?/sec 1.75 89.4±2.50µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_nulls_scalars/50pct_true 1.00 62.1±1.00µs ? ?/sec 1.61 99.7±4.68µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_nulls_scalars/90pct_true 1.00 28.8±0.64µs ? ?/sec 2.63 75.7±1.22µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_nulls_scalars/99pct_true 1.00 7.7±0.11µs ? ?/sec 8.98 69.0±0.74µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_nulls_scalars/all_false 1.00 3.7±0.13µs ? ?/sec 19.06 69.8±1.55µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/non_nulls_scalars/all_true 1.00 3.6±0.10µs ? ?/sec 18.90 68.0±1.12µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/null_vs_non_null_scalar/10pct_true 1.00 3.8±0.07µs ? ?/sec 28.85 108.4±3.09µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/null_vs_non_null_scalar/1pct_true 1.00 3.8±0.09µs ? ?/sec 25.83 98.7±2.71µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/null_vs_non_null_scalar/50pct_nulls 1.00 3.9±0.06µs ? ?/sec 32.25 127.3±7.41µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/null_vs_non_null_scalar/50pct_true 1.00 3.7±0.06µs ? ?/sec 37.66 139.5±3.00µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/null_vs_non_null_scalar/90pct_true 1.00 3.8±0.16µs ? ?/sec 34.52 129.5±1.53µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/null_vs_non_null_scalar/99pct_true 1.00 3.7±0.05µs ? ?/sec 33.83 124.8±1.28µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/null_vs_non_null_scalar/all_false 1.00 3.8±0.09µs ? ?/sec 26.08 98.8±2.02µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 10/null_vs_non_null_scalar/all_true 1.00 3.8±0.08µs ? ?/sec 32.56 123.9±1.48µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_null_scalar_vs_null_scalar/10pct_true 1.00 3.6±0.06µs ? ?/sec 36.09 129.8±6.06µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_null_scalar_vs_null_scalar/1pct_true 1.00 3.6±0.35µs ? ?/sec 34.05 122.9±5.06µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_null_scalar_vs_null_scalar/50pct_nulls 1.00 3.7±0.12µs ? ?/sec 36.77 137.9±5.49µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_null_scalar_vs_null_scalar/50pct_true 1.00 3.6±0.09µs ? ?/sec 38.23 137.4±3.35µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_null_scalar_vs_null_scalar/90pct_true 1.00 3.6±0.06µs ? ?/sec 29.20 104.8±1.64µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_null_scalar_vs_null_scalar/99pct_true 1.00 3.6±0.15µs ? ?/sec 26.94 96.9±2.73µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_null_scalar_vs_null_scalar/all_false 1.00 3.6±0.05µs ? ?/sec 34.97 127.5±5.81µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_null_scalar_vs_null_scalar/all_true 1.00 3.8±1.05µs ? ?/sec 24.98 95.0±2.14µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_nulls_scalars/10pct_true 1.00 28.9±0.46µs ? ?/sec 2.69 77.7±1.57µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_nulls_scalars/1pct_true 1.00 7.3±0.09µs ? ?/sec 9.81 71.6±1.96µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_nulls_scalars/50pct_nulls 1.00 50.3±1.16µs ? ?/sec 1.74 87.7±1.14µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_nulls_scalars/50pct_true 1.00 63.5±1.44µs ? ?/sec 1.59 100.7±1.97µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_nulls_scalars/90pct_true 1.00 29.8±0.48µs ? ?/sec 2.64 78.6±2.85µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_nulls_scalars/99pct_true 1.00 8.2±0.12µs ? ?/sec 8.54 69.7±0.91µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_nulls_scalars/all_false 1.00 3.8±0.07µs ? ?/sec 18.77 71.6±1.51µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/non_nulls_scalars/all_true 1.00 3.8±0.11µs ? ?/sec 18.31 68.8±1.10µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/null_vs_non_null_scalar/10pct_true 1.00 3.8±0.07µs ? ?/sec 27.36 104.3±1.35µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/null_vs_non_null_scalar/1pct_true 1.00 3.8±0.07µs ? ?/sec 24.86 94.8±1.12µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/null_vs_non_null_scalar/50pct_nulls 1.00 4.0±0.04µs ? ?/sec 29.84 117.9±1.34µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/null_vs_non_null_scalar/50pct_true 1.00 3.9±0.21µs ? ?/sec 35.19 137.1±3.87µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/null_vs_non_null_scalar/90pct_true 1.00 3.8±0.06µs ? ?/sec 32.78 125.8±1.73µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/null_vs_non_null_scalar/99pct_true 1.00 3.8±0.11µs ? ?/sec 31.87 121.5±1.47µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/null_vs_non_null_scalar/all_false 1.00 3.8±0.07µs ? ?/sec 25.36 95.5±1.89µs ? ?/sec zip_8192_from_string_views size 10 and string_views size 100/null_vs_non_null_scalar/all_true 1.00 3.9±0.20µs ? ?/sec 30.83 121.7±3.36µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_null_scalar_vs_null_scalar/10pct_true 1.00 3.7±0.73µs ? ?/sec 35.72 132.2±6.77µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_null_scalar_vs_null_scalar/1pct_true 1.00 3.6±0.04µs ? ?/sec 35.35 125.8±2.79µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_null_scalar_vs_null_scalar/50pct_nulls 1.00 3.8±0.11µs ? ?/sec 36.05 136.0±2.59µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_null_scalar_vs_null_scalar/50pct_true 1.00 3.6±0.13µs ? ?/sec 39.36 142.5±6.32µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_null_scalar_vs_null_scalar/90pct_true 1.00 3.6±0.11µs ? ?/sec 29.63 107.5±2.03µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_null_scalar_vs_null_scalar/99pct_true 1.00 3.6±0.08µs ? ?/sec 28.40 102.2±6.74µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_null_scalar_vs_null_scalar/all_false 1.00 3.6±0.05µs ? ?/sec 34.83 126.0±2.12µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_null_scalar_vs_null_scalar/all_true 1.00 3.6±0.05µs ? ?/sec 27.38 98.6±1.62µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_nulls_scalars/10pct_true 1.00 29.9±2.79µs ? ?/sec 2.51 75.1±0.98µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_nulls_scalars/1pct_true 1.00 7.2±0.16µs ? ?/sec 9.48 68.3±1.01µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_nulls_scalars/50pct_nulls 1.00 50.5±1.90µs ? ?/sec 1.68 84.6±1.27µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_nulls_scalars/50pct_true 1.00 64.4±0.60µs ? ?/sec 1.53 98.6±1.71µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_nulls_scalars/90pct_true 1.00 29.7±0.61µs ? ?/sec 2.57 76.1±1.15µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_nulls_scalars/99pct_true 1.00 7.9±0.09µs ? ?/sec 8.89 70.5±2.13µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_nulls_scalars/all_false 1.00 3.7±0.06µs ? ?/sec 18.31 67.8±0.86µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/non_nulls_scalars/all_true 1.00 3.7±0.06µs ? ?/sec 18.35 67.9±1.16µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/null_vs_non_null_scalar/10pct_true 1.00 3.8±0.12µs ? ?/sec 28.20 107.5±2.55µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/null_vs_non_null_scalar/1pct_true 1.00 3.9±0.16µs ? ?/sec 25.73 99.5±2.19µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/null_vs_non_null_scalar/50pct_nulls 1.00 4.1±0.14µs ? ?/sec 29.98 122.2±2.27µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/null_vs_non_null_scalar/50pct_true 1.00 3.8±0.08µs ? ?/sec 37.05 140.1±2.01µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/null_vs_non_null_scalar/90pct_true 1.00 3.9±0.20µs ? ?/sec 33.52 131.8±3.10µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/null_vs_non_null_scalar/99pct_true 1.00 3.8±0.09µs ? ?/sec 33.55 127.6±3.56µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/null_vs_non_null_scalar/all_false 1.00 3.8±0.08µs ? ?/sec 26.47 100.8±5.55µs ? ?/sec zip_8192_from_string_views size 100 and string_views size 100/null_vs_non_null_scalar/all_true 1.00 3.9±0.06µs ? ?/sec 32.05 124.6±2.16µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_null_scalar_vs_null_scalar/10pct_true 1.00 3.6±0.40µs ? ?/sec 35.16 126.4±1.92µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_null_scalar_vs_null_scalar/1pct_true 1.00 3.5±0.07µs ? ?/sec 35.43 123.6±4.98µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_null_scalar_vs_null_scalar/50pct_nulls 1.00 3.7±0.06µs ? ?/sec 36.06 132.4±1.80µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_null_scalar_vs_null_scalar/50pct_true 1.00 3.6±0.06µs ? ?/sec 38.44 136.9±2.82µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_null_scalar_vs_null_scalar/90pct_true 1.00 3.5±0.04µs ? ?/sec 29.82 105.2±2.25µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_null_scalar_vs_null_scalar/99pct_true 1.00 3.5±0.08µs ? ?/sec 27.48 96.9±1.69µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_null_scalar_vs_null_scalar/all_false 1.00 3.6±0.12µs ? ?/sec 33.80 123.0±2.52µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_null_scalar_vs_null_scalar/all_true 1.00 3.6±0.14µs ? ?/sec 26.74 95.0±1.74µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_nulls_scalars/10pct_true 1.00 27.9±0.32µs ? ?/sec 2.65 73.9±1.31µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_nulls_scalars/1pct_true 1.00 6.9±0.09µs ? ?/sec 9.64 67.0±0.92µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_nulls_scalars/50pct_nulls 1.00 49.0±0.60µs ? ?/sec 1.73 84.7±2.45µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_nulls_scalars/50pct_true 1.00 62.4±2.22µs ? ?/sec 1.56 97.1±2.37µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_nulls_scalars/90pct_true 1.00 28.7±0.37µs ? ?/sec 2.59 74.1±1.17µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_nulls_scalars/99pct_true 1.00 7.8±0.20µs ? ?/sec 8.69 67.7±1.34µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_nulls_scalars/all_false 1.00 3.6±0.09µs ? ?/sec 18.78 68.2±2.16µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/non_nulls_scalars/all_true 1.00 3.6±0.05µs ? ?/sec 19.10 68.4±11.77µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/null_vs_non_null_scalar/10pct_true 1.00 3.8±0.21µs ? ?/sec 27.30 104.1±1.34µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/null_vs_non_null_scalar/1pct_true 1.00 3.7±0.04µs ? ?/sec 25.76 95.8±2.00µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/null_vs_non_null_scalar/50pct_nulls 1.00 4.2±0.96µs ? ?/sec 28.05 118.0±1.17µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/null_vs_non_null_scalar/50pct_true 1.00 3.9±0.13µs ? ?/sec 35.42 136.6±3.78µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/null_vs_non_null_scalar/90pct_true 1.00 3.8±0.10µs ? ?/sec 33.31 125.5±1.89µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/null_vs_non_null_scalar/99pct_true 1.00 3.8±0.04µs ? ?/sec 32.36 121.6±1.80µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/null_vs_non_null_scalar/all_false 1.00 3.7±0.04µs ? ?/sec 25.64 95.1±0.98µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 10/null_vs_non_null_scalar/all_true 1.00 3.9±0.07µs ? ?/sec 31.19 121.2±2.69µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_null_scalar_vs_null_scalar/10pct_true 1.00 3.5±0.04µs ? ?/sec 35.69 126.5±2.89µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_null_scalar_vs_null_scalar/1pct_true 1.00 3.6±0.05µs ? ?/sec 33.84 120.9±1.68µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_null_scalar_vs_null_scalar/50pct_nulls 1.00 3.7±0.10µs ? ?/sec 35.72 133.2±3.49µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_null_scalar_vs_null_scalar/50pct_true 1.00 3.6±0.12µs ? ?/sec 38.28 136.0±2.11µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_null_scalar_vs_null_scalar/90pct_true 1.00 3.5±0.06µs ? ?/sec 29.81 104.4±1.56µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_null_scalar_vs_null_scalar/99pct_true 1.00 3.5±0.08µs ? ?/sec 27.69 98.1±2.86µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_null_scalar_vs_null_scalar/all_false 1.00 3.6±0.10µs ? ?/sec 33.58 122.3±1.77µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_null_scalar_vs_null_scalar/all_true 1.00 3.5±0.08µs ? ?/sec 26.79 94.7±1.02µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_nulls_scalars/10pct_true 1.00 29.0±0.51µs ? ?/sec 2.59 75.1±1.08µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_nulls_scalars/1pct_true 1.00 7.4±0.10µs ? ?/sec 9.41 69.2±1.76µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_nulls_scalars/50pct_nulls 1.00 50.2±0.54µs ? ?/sec 1.70 85.2±1.17µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_nulls_scalars/50pct_true 1.00 64.1±1.59µs ? ?/sec 1.51 96.9±1.22µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_nulls_scalars/90pct_true 1.00 29.8±0.36µs ? ?/sec 2.55 75.9±2.47µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_nulls_scalars/99pct_true 1.00 8.2±0.17µs ? ?/sec 8.24 67.8±1.11µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_nulls_scalars/all_false 1.00 3.8±0.07µs ? ?/sec 17.96 68.8±1.15µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/non_nulls_scalars/all_true 1.00 3.8±0.12µs ? ?/sec 17.37 66.1±0.97µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/null_vs_non_null_scalar/10pct_true 1.00 3.8±0.27µs ? ?/sec 27.57 105.2±3.06µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/null_vs_non_null_scalar/1pct_true 1.00 3.7±0.08µs ? ?/sec 25.44 94.8±0.94µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/null_vs_non_null_scalar/50pct_nulls 1.00 3.9±0.07µs ? ?/sec 30.10 118.6±2.83µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/null_vs_non_null_scalar/50pct_true 1.00 3.9±0.30µs ? ?/sec 35.20 135.6±1.67µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/null_vs_non_null_scalar/90pct_true 1.00 3.9±0.55µs ? ?/sec 32.58 125.9±2.14µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/null_vs_non_null_scalar/99pct_true 1.00 3.8±0.36µs ? ?/sec 32.47 122.9±4.15µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/null_vs_non_null_scalar/all_false 1.00 3.8±0.10µs ? ?/sec 25.24 94.9±0.97µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 100/null_vs_non_null_scalar/all_true 1.00 3.8±0.09µs ? ?/sec 31.58 120.3±1.65µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_null_scalar_vs_null_scalar/10pct_true 1.00 3.5±0.04µs ? ?/sec 37.39 131.4±4.74µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_null_scalar_vs_null_scalar/1pct_true 1.00 3.5±0.09µs ? ?/sec 35.84 126.8±3.56µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_null_scalar_vs_null_scalar/50pct_nulls 1.00 3.7±0.06µs ? ?/sec 37.15 137.8±3.16µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_null_scalar_vs_null_scalar/50pct_true 1.00 3.5±0.06µs ? ?/sec 39.19 138.9±4.82µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_null_scalar_vs_null_scalar/90pct_true 1.00 3.6±0.04µs ? ?/sec 30.30 107.9±5.71µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_null_scalar_vs_null_scalar/99pct_true 1.00 3.6±0.05µs ? ?/sec 27.33 97.7±2.10µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_null_scalar_vs_null_scalar/all_false 1.00 3.6±0.06µs ? ?/sec 34.64 124.7±2.24µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_null_scalar_vs_null_scalar/all_true 1.00 3.7±0.19µs ? ?/sec 26.17 96.9±1.75µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_nulls_scalars/10pct_true 1.00 28.7±0.55µs ? ?/sec 2.66 76.2±1.45µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_nulls_scalars/1pct_true 1.00 7.2±0.12µs ? ?/sec 9.58 69.0±0.80µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_nulls_scalars/50pct_nulls 1.00 49.5±1.15µs ? ?/sec 1.75 86.8±2.09µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_nulls_scalars/50pct_true 1.00 62.6±0.88µs ? ?/sec 1.65 103.4±16.82µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_nulls_scalars/90pct_true 1.00 29.1±0.49µs ? ?/sec 2.69 78.3±2.51µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_nulls_scalars/99pct_true 1.00 7.8±0.09µs ? ?/sec 9.01 70.2±1.72µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_nulls_scalars/all_false 1.00 3.7±0.06µs ? ?/sec 18.77 68.7±0.73µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/non_nulls_scalars/all_true 1.00 3.6±0.10µs ? ?/sec 18.73 68.2±1.44µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/null_vs_non_null_scalar/10pct_true 1.00 3.9±0.11µs ? ?/sec 27.68 106.9±2.29µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/null_vs_non_null_scalar/1pct_true 1.00 3.9±0.19µs ? ?/sec 26.12 101.9±8.79µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/null_vs_non_null_scalar/50pct_nulls 1.00 4.1±0.07µs ? ?/sec 29.91 122.7±3.28µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/null_vs_non_null_scalar/50pct_true 1.00 3.8±0.14µs ? ?/sec 36.82 141.4±3.69µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/null_vs_non_null_scalar/90pct_true 1.00 3.8±0.10µs ? ?/sec 34.15 131.4±2.99µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/null_vs_non_null_scalar/99pct_true 1.00 3.8±0.06µs ? ?/sec 32.89 125.2±3.21µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/null_vs_non_null_scalar/all_false 1.00 3.8±0.06µs ? ?/sec 26.05 99.2±2.30µs ? ?/sec zip_8192_from_string_views size 3 and string_views size 3/null_vs_non_null_scalar/all_true 1.00 4.0±0.33µs ? ?/sec 32.00 126.7±25.05µs ? ?/sec ``` </details>
# Which issue does this PR close? - Closes apache#9085 # Rationale for this change Fix a regression introduced in apache#8996 # What changes are included in this PR? 1. Add test coverage for nullif kernel 1. Undeprecate `bitwise_unary_op_helper` 2. Document subtle differences 3. Restore nullif kernel from apache#8996 # Are these changes tested Yes # Are there any user-facing changes? Fix (not yet released) bug
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - related to apache/datafusion#19477 # Rationale for this change While working on apache/datafusion#19477, and profiling ClickBench q7, I noticed that the RowSelectors was being cloned to resolve the strategy -- for a large number of selections this is expensive and shows up in the traces <img width="1724" height="1074" alt="Screenshot 2025-12-28 at 4 49 49 PM" src="https://github.com/user-attachments/assets/72c6fd22-9377-48ef-ba80-6bc03b177cf7" /> ```shell samply record -- ./datafusion-cli-alamb_enable_pushdown -f q.sql > /dev/null 2>& ``` We should change the code to avoid cloning the RowSelectors when resolving the strategy. # Changes Don't clone / allocate while resolving the strategy. I don't expect this to have a massive impact, but it did show up in the profile FYI @hhhizzz -- perhaps you could review this PR # Are these changes tested? Yes by CI # Are there any user-facing changes? small performance improvement
…pache#9094) # Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Part of apache#8082. # Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> To support `Lists/Array` in `variant_get`, it's better to move `ArrayVariantToArrowRowBuilder` from `shred_variant` to `variant_to_arrow` and be shared with `variant_get`. In the meantime, some code movement in `variant_to_arrow` would help to get a better overview of the overall implementation # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> This PR can be reviewed commit by commit: - Move `VariantToArrowRowBuilder` and related impl to the top of `variant_to_arrow` - Push the `FixedSizeList` check from `make_variant_to_shredded_variant_arrow_row_builder` down to `ArrayVariantToArrowRowBuilder` - Move `ArrayVariantToArrowRowBuilder` to `variant_to_arrow` # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Covered by existing tests # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> No
…9000) # Which issue does this PR close? - Closes apache#8999 # Rationale for this change This PR fixes a bug in the row-to-column conversion for Union types when multiple union columns are present in the same row converter Previously, the row slice was being consumed from reading their data correctly. The fix tracks bytes consumed per row across all union fields, this way it properly advances row slices
# Which issue does this PR close? - Closes apache#5163 # Rationale for this change I've implemented this function at least twice in other codebases, and `arrow-rs` now has 4 variants. # What changes are included in this PR? New public function to test if a `DataType` is any decimal variant. # Are these changes tested? Yes # Are there any user-facing changes? New function including docs.
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #NNN. # Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Add convenient method to insert all endpoints. # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Add `FlightInfo::with_endpoints` method # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> CI. # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> Yes.
…apache#9088) # Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #NNN. # Rationale for this change Previous benchmark is too fast to deterministically measure the performance improvement because they run only in 2-7 microsecond. <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
…pache#9099) # Which issue does this PR close? - Closes apache#9098. # Rationale for this change Don't require strict equality for nested fields (including inner field name/metadata), just require that nested data types are logically equivalent. # What changes are included in this PR? Use `a.equals_datatype(b)` instead of `a == b` at the start of `LevelInfoBuilder::types_compatible`. # Are these changes tested? Yes. # Are there any user-facing changes?
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#9106 # Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> This trait is not meant to be overridden, and doing so will break many kernels in sometimes subtle ways. # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> Seals the Array trait to prevent implementation outside of arrow-array. # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…#9057) # Which issue does this PR close? - related to apache/datafusion#19477 # Rationale for this change I am tracking down allocations in various parts of the parquet reader (to remove them) and I ran across this in the cached reader. # What changes are included in this PR? Use `Arc::clone` to make it clear there is no deep cloning going on . I don't expect this will have any impact on actual performance # Are these changes tested? By CI # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
# Which issue does this PR close? - Closes apache#9084. # What changes are included in this PR? Documentation on union types encoding in https://arrow.apache.org/rust/arrow_row/struct.RowConverter.html. # Are these changes tested? Yes. # Are there any user-facing changes? Yes. https://arrow.apache.org/rust/arrow_row/struct.RowConverter.html will get updated. --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
# Which issue does this PR close? - Part of apache#8465 # Rationale for this change - See apache#8465 # What changes are included in this PR? 1. Update version to 57.2.0 2. Add CHANGELOG. See rendered version https://github.com/alamb/arrow-rs/blob/alamb/prepare_57.2.0/CHANGELOG.md # Are these changes tested? By CI and I am testing them manually in DataFusion here: - apache/datafusion#19355 # Are there any user-facing changes? Version and changelog
…ask (apache#9051) # Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#8859 # Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> > Currently the default behavior is to parse the full vector of encoding stats, but given the limited use of this information we should instead default to the more concise and performant bitmask. # What changes are included in this PR? <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> - Implement `Default` for `ParquetMetaDataOptions` with `encoding_stats_as_mask: true` - Update Thrift decoding logic to default to bitmask even if options are missing - Update documentation to reflect the new default behavior - Update existing tests to maintain coverage for full statistics - Add new tests to verify default behavior and full stats option # Are these changes tested? <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. --> Yes
# Which issue does this PR close? - releated to apache#7392 # Rationale for this change Keep website updated with our planned release schedule # What changes are included in this PR? Update README with planned release schedule # Are these changes tested? N/A # Are there any user-facing changes? Update readme
# Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes #NNN. # Rationale for this change <!-- Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed. Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes. --> Add targeted JSON reader benchmarks to track performance for wide objects, hex-encoded binary inputs, and projection workloads. # What changes are included in this PR? - Add `arrow-json/benches/wide_object.rs` for wide-object decode/serialize benchmarks. - Add `arrow-json/benches/binary_hex.rs` for hex string decoding into Binary/FixedSizeBinary/BinaryView. - Add `arrow-json/benches/wide_projection.rs` for full vs projected schema decoding. <!-- There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR. --> # Are these changes tested? No <!-- We typically require tests for all PRs in order to: 1. Prevent the code from being accidentally broken by subsequent changes 2. Serve as another way to document the expected behavior of the code If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> # Are there any user-facing changes? No <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
…TSEC-2025-0141 ) (apache#9104) # Which issue does this PR close? <!-- We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. --> - Closes apache#9009 - Also addresses https://rustsec.org/advisories/RUSTSEC-2025-0141 -- this is only used for testing but still # Rationale for this change https://crates.io/crates/bincode is unmaintained <img width="833" height="657" alt="Screenshot 2026-01-06 at 5 57 13 PM" src="https://github.com/user-attachments/assets/9e18a6e5-eb45-4470-94b7-e17b89ad558e" /> There also appears to be some sort of drama related to the maintainer While we only use this code in tests, it would be nice to avoid issues sooner rather than later # What changes are included in this PR? Change to use postcard # Are these changes tested? by ci # Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please call them out. -->
…er::from_bitwise_binary_op` (apache#9090) # Which issue does this PR close? - Part of apache#8806 - Closes apache#8854 - Closes apache#8807 This is the next step after - apache#8996 # Rationale for this change - we can help rust / LLVM generate more optimal code by processing u64 words at a time when the buffer is already u64 aligned (see apache#8807) Also, it is hard to find the code to create new Buffers by applying bitwise unary operations. # What changes are included in this PR? - Introduce optimized `BooleanBuffer::from_bitwise_binary` - Migrate several kernels that use `bitwise_bin_op_helper` to use the new BooleanBuffer # Are these changes tested? Yes new tests are added Performance results show 30% performance improvement for the `and` and `or` kernels for aligned buffers (common case) # Are there any user-facing changes? A new API
…r::finish (apache#8031) # Which issue does this PR close? This pr wants to optimize the logic of `ObjectBuilder::finish` - Closes apache#7978 . # Rationale for this change This pr wants to optimize the logic of `ObjectBuilder::finish` # What changes are included in this PR? This PR wants to optimize `ObjectBuilder::finish` with packedu3 iterator # Are these changes tested? This pr was covered by existing test # Are there any user-facing changes? No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This targets the PR in arrow-rs from @maxburke
I merged up from main to test and found a logical conflict
I proposed a fix in
(if you merge this PR it will update the arrow-rs PR: apache#9016)