Change to use `resolver v2`, test more feature flag combinations in CI, fix errors (#1630) #1822

tustvold · 2022-06-08T21:29:47Z

Which issue does this PR close?

Rationale for this change

We continue to have issues with various feature flag combinations resulting in compile errors. A particularly pernicious variant of this occurs you need to enable a feature of an optional dependency for tests, for example, arrow prettyprint within parquet. To do this you add the dependency as a dev-dependency, with the feature enabled.

However, the way feature resolution traditionally works, is that even when building the library crate on its own, it would take into account features enabled by the dev-dependencies. This would mask issues with feature flags. The previous solution to this has been to have test/dependency crates that simply depend on the library, without the dev-dependencies.

What changes are included in this PR?

rust-lang/cargo#7916 added an option to skip using dev-dependencies when resolving features, and this was stabilised in feature resolver v2 which was released in Rust 1.51. In particular with the new feature resolver

Dev-dependencies do not activate features unless building a target that needs them (like tests or examples).

This PR therefore:

Switches to using the new feature resolver
Uses this to test different feature flag combinations in CI
Removes the old test/dependency workaround
Fixes the bugs this turned up

Thanks to @carols10cents for the pointer ❤️

Are there any user-facing changes?

No

tustvold · 2022-06-08T21:30:20Z

.github/workflows/rust.yml

-          cargo test --no-default-features
+
+          # re-run tests on arrow crate with all supported features
+          cargo test -p arrow --features=force_validate,prettyprint


I merged these two test runs together as I couldn't see an obvious reason to separate them, and this will make CI slightly faster

I think the original rationale was to try and test without the default features (mostly I think to try and catch build errors, which this PR does in another way)

tustvold · 2022-06-08T21:31:08Z

.github/workflows/rust.yml

          cargo run --example builders
          cargo run --example dynamic_types
          cargo run --example read_csv
          cargo run --example read_csv_infer_schema
-          cargo check --no-default-features
+


I moved the cargo check logic into here as opposed to a separate job so it can benefit from the compilation already performed above.

Maybe it would be worth making a separate named run step (as is done https://github.com/apache/arrow-rs/pull/1822/files#diff-73e17259d77e5fbef83b2bdbbe4dc40a912f807472287f7f45b77e0cbf78792dR188)

- name: Check compilation with simd features

So when/if this check fails it will be easier to figure out what is wrong

Working on this, running into nonsense with environment variables and caching

Cargo.toml

tustvold · 2022-06-08T21:32:09Z

parquet/src/errors.rs

@@ -135,6 +135,7 @@ macro_rules! eof_err {
    ($fmt:expr, $($args:expr),*) => (ParquetError::EOF(format!($fmt, $($args),*)));
 }

+#[cfg(any(feature = "arrow", test))]


This removes a warning

tustvold · 2022-06-08T21:32:21Z

parquet/src/record/api.rs

@@ -27,7 +27,7 @@ use crate::data_type::{ByteArray, Decimal, Int96};
 use crate::errors::{ParquetError, Result};
 use crate::schema::types::ColumnDescPtr;

-#[cfg(feature = "cli")]
+#[cfg(any(feature = "cli", test))]


Drive by refactor to make consistent with rest of crate which tests everything

tustvold · 2022-06-08T21:36:29Z

.github/workflows/rust.yml

+          cargo check -p parquet --no-default-features --features arrow --all-targets
+
+          # Test compilation of parquet_derive macro with different feature combinations
+          cargo check -p parquet_derive


To be completely honest I'm not sure why we were testing this, parquet_derive doesn't have any dev-dependencies or feature flags, but I guess it can't hurt to be thorough

tustvold · 2022-06-08T21:39:05Z

.github/workflows/rust.yml

+          cargo check -p arrow
+          cargo check -p arrow --no-default-features
+
+          # Test compilation of arrow targets with different feature combinations


It would theoretically be more correct to test each target individually, so that feature resolution is not impacted by other targets, in practice this is probably good enough

tustvold · 2022-06-08T21:40:30Z

parquet/Cargo.toml

@@ -39,38 +39,44 @@ brotli = { version = "3.3", optional = true }
 flate2 = { version = "1.0", optional = true }
 lz4 = { version = "1.23", optional = true }
 zstd = { version = "0.11.1", optional = true, default-features = false }
-chrono = { version = "0.4", default-features = false }
+chrono = { version = "0.4", default-features = false, features = ["alloc"] }


This is the actual fix for #1630

Unfortunately this is required by the record API which is tightly coupled with the file APIs, and so there isn't an easy way to hide this behind a feature flag. Perhaps something for another day, I'm a bit feature flagged out 😆

codecov-commenter · 2022-06-09T07:46:41Z

Codecov Report

Merging #1822 (86dfef5) into master (ba38ebe) will increase coverage by 0.10%.
The diff coverage is 95.89%.

❗ Current head 86dfef5 differs from pull request most recent head 89c0429. Consider uploading reports for the commit 89c0429 to get more accurate results

@@            Coverage Diff             @@
##           master    #1822      +/-   ##
==========================================
+ Coverage   83.44%   83.54%   +0.10%     
==========================================
  Files         199      200       +1     
  Lines       56651    56798     +147     
==========================================
+ Hits        47272    47452     +180     
+ Misses       9379     9346      -33

Impacted Files	Coverage Δ
arrow/src/buffer/mod.rs	`73.33% <ø> (ø)`
parquet/src/errors.rs	`29.62% <ø> (ø)`
arrow/src/buffer/scalar.rs	`93.47% <93.47%> (ø)`
arrow/src/array/equal/list.rs	`96.47% <100.00%> (+0.08%)`	⬆️
arrow/src/array/equal/mod.rs	`96.20% <100.00%> (+0.10%)`	⬆️
parquet/src/record/api.rs	`96.82% <100.00%> (+4.79%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ba38ebe...89c0429. Read the comment docs.

alamb

The idea looks great - thanks @tustvold

I want to review the actual test output before I approve this PR and it still appears to be running

alamb · 2022-06-09T13:48:28Z

.github/workflows/rust.yml

-          cargo test --no-default-features
+
+          # re-run tests on arrow crate with all supported features
+          cargo test -p arrow --features=force_validate,prettyprint


I think the original rationale was to try and test without the default features (mostly I think to try and catch build errors, which this PR does in another way)

Cargo.toml

alamb · 2022-06-09T13:52:11Z

.github/workflows/rust.yml

          cargo run --example builders
          cargo run --example dynamic_types
          cargo run --example read_csv
          cargo run --example read_csv_infer_schema
-          cargo check --no-default-features
+


Maybe it would be worth making a separate named run step (as is done https://github.com/apache/arrow-rs/pull/1822/files#diff-73e17259d77e5fbef83b2bdbbe4dc40a912f807472287f7f45b77e0cbf78792dR188)

- name: Check compilation with simd features

So when/if this check fails it will be easier to figure out what is wrong

tustvold · 2022-06-09T14:51:55Z

Setting as draft whilst I try to get CI happy

tustvold · 2022-06-09T16:46:56Z

I had to rework the caching to allow splitting up the stages into smaller steps, without having to duplicate "export CARGO_HOME". into every step. Setting these variable at the top-level caused rustup to fail to install correctly, and so rather than using CARGO_HOME to move cargo's location, instead we now just cache the relevant cargo paths. I moved this logic into the "Prepare Rust Build Environment" action to reduce duplication.

.github/actions/setup-builder/action.yaml

Don't install unused components

tustvold · 2022-06-09T17:16:02Z

Here is an example of it restoring from a cache key (without lockfile suffix), and then publishing the lockfile specific cache key 🎉

This then gets picked up by https://github.com/apache/arrow-rs/runs/6817593817?check_suite_focus=true

tustvold · 2022-06-09T17:44:22Z

MIRI appears to be unhappy... 👀

alamb

Beautiful

tustvold · 2022-06-09T17:48:59Z

Appears MIRI version is too old to supposed namespaced deps - rust-lang/cargo#5565

Attempting to update it in - #1828

…istency

alamb · 2022-06-10T10:26:18Z

Thank you @tustvold

github-actions bot added arrow Changes to the arrow crate parquet Changes to the parquet crate parquet-derive labels Jun 8, 2022

Test more feature flag combinations in CI (apache#1630)

75f9b49

tustvold force-pushed the feature-flag-consistency branch from 376f38e to 75f9b49 Compare June 8, 2022 21:35

tustvold commented Jun 8, 2022

View reviewed changes

tustvold added 2 commits June 8, 2022 22:56

Clippy lints

ac2f0e2

Fix clippy fix

6d953cc

tustvold added 3 commits June 9, 2022 13:11

Fix running examples from workspace root

23897c1

Format

457a749

Fix arrow benchmark features

39c4eea

alamb changed the title ~~Test more feature flag combinations in CI (#1630)~~ Test more feature flag combinations in CI, fix errors (#1630) Jun 9, 2022

alamb changed the title ~~Test more feature flag combinations in CI, fix errors (#1630)~~ Change to use resolver v2, test more feature flag combinations in CI, fix errors (#1630) Jun 9, 2022

alamb reviewed Jun 9, 2022

View reviewed changes

tustvold added 2 commits June 9, 2022 15:01

Split up CI yaml

fba5907

Add docs

b020e6e

tustvold marked this pull request as draft June 9, 2022 14:51

tustvold force-pushed the feature-flag-consistency branch 8 times, most recently from 9b0324a to b701d2b Compare June 9, 2022 16:11

Rework caching

ed413af

tustvold force-pushed the feature-flag-consistency branch from b701d2b to ed413af Compare June 9, 2022 16:33

tustvold commented Jun 9, 2022

View reviewed changes

.github/actions/setup-builder/action.yaml Outdated Show resolved Hide resolved

tustvold force-pushed the feature-flag-consistency branch 2 times, most recently from 68d1441 to 7142dff Compare June 9, 2022 17:02

Use lockfile for cache key

89c0429

Don't install unused components

tustvold force-pushed the feature-flag-consistency branch from 7142dff to 89c0429 Compare June 9, 2022 17:04

tustvold marked this pull request as ready for review June 9, 2022 17:39

alamb approved these changes Jun 9, 2022

View reviewed changes

tustvold mentioned this pull request Jun 9, 2022

Update MIRI pin #1828

Merged

Merge remote-tracking branch 'upstream/master' into feature-flag-cons…

c6cf1c5

…istency

jhorstmann mentioned this pull request Jun 9, 2022

AVX512 + simd binary and/or kernels slower than autovectorized version #1829

Closed

tustvold merged commit db41b33 into apache:master Jun 9, 2022

tustvold mentioned this pull request Jun 10, 2022

Larger CI Runners to Prevent MIRI OOMing and Improve CI Times #1833

Closed

tustvold mentioned this pull request Jul 28, 2022

remove redundant CI benchmark check, cleanups #2212

Merged

This was referenced Oct 28, 2022

parquet allows use of arrow without base64 #811

Closed

Unresolved import arrow::ipc #872

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change to use `resolver v2`, test more feature flag combinations in CI, fix errors (#1630) #1822

Change to use `resolver v2`, test more feature flag combinations in CI, fix errors (#1630) #1822

tustvold commented Jun 8, 2022

tustvold Jun 8, 2022

alamb Jun 9, 2022

tustvold Jun 8, 2022

alamb Jun 9, 2022

tustvold Jun 9, 2022

tustvold Jun 8, 2022

tustvold Jun 8, 2022

tustvold Jun 8, 2022

tustvold Jun 8, 2022

tustvold Jun 8, 2022

codecov-commenter commented Jun 9, 2022 •

edited

Loading

alamb left a comment

alamb Jun 9, 2022

alamb Jun 9, 2022

tustvold commented Jun 9, 2022

tustvold commented Jun 9, 2022

tustvold commented Jun 9, 2022

tustvold commented Jun 9, 2022

alamb left a comment

tustvold commented Jun 9, 2022 •

edited

Loading

alamb commented Jun 10, 2022

Change to use resolver v2, test more feature flag combinations in CI, fix errors (#1630) #1822

Change to use resolver v2, test more feature flag combinations in CI, fix errors (#1630) #1822

Conversation

tustvold commented Jun 8, 2022

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jun 9, 2022 • edited Loading

Codecov Report

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tustvold commented Jun 9, 2022

tustvold commented Jun 9, 2022

tustvold commented Jun 9, 2022

tustvold commented Jun 9, 2022

alamb left a comment

Choose a reason for hiding this comment

tustvold commented Jun 9, 2022 • edited Loading

alamb commented Jun 10, 2022

Change to use `resolver v2`, test more feature flag combinations in CI, fix errors (#1630) #1822

Change to use `resolver v2`, test more feature flag combinations in CI, fix errors (#1630) #1822

codecov-commenter commented Jun 9, 2022 •

edited

Loading

tustvold commented Jun 9, 2022 •

edited

Loading