Skip to content

Add Cargo.lock #14483

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
alamb merged 27 commits into from
Feb 8, 2025
Merged

Add Cargo.lock #14483

Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
65ce278
Add `Cargo.lock`
mbrobbel Feb 4, 2025
4b6eb70
Style fix
mbrobbel Feb 4, 2025
c41b1bc
Update `Cargo.lock`
mbrobbel Feb 4, 2025
b76ad41
Remove unused path
mbrobbel Feb 4, 2025
4b2e5e3
Remove cli-specific ci job and dependabot config
mbrobbel Feb 4, 2025
f847184
Remove home pin
mbrobbel Feb 4, 2025
2e16216
Make cli test work with backtrace feature
mbrobbel Feb 4, 2025
586334c
More changes resulting from moving the cli crate in the workspace
mbrobbel Feb 4, 2025
25a05f2
Merge branch 'main' into cargo-lock
mbrobbel Feb 4, 2025
6643145
Merge branch 'main' into cargo-lock
mbrobbel Feb 4, 2025
37078eb
Exclude `depcheck` `Cargo.lock`
mbrobbel Feb 4, 2025
3ee82a5
Remove `--locked` from `depcheck` run
mbrobbel Feb 4, 2025
0c9d9cb
Merge branch 'main' into cargo-lock
mbrobbel Feb 5, 2025
32b2178
Refer to guidance instead of updated guidance
mbrobbel Feb 5, 2025
684c7df
Remove `--locked` from benchmark script
mbrobbel Feb 5, 2025
9d0d5cb
Only run with `--locked` in `linux-build-lib` job of `Rust` workflow
mbrobbel Feb 5, 2025
29d8521
Remove unrelated formatting changes
mbrobbel Feb 5, 2025
51d6bf9
Merge branch 'main' into cargo-lock
mbrobbel Feb 5, 2025
152f1cf
Add a note about using `--locked`
mbrobbel Feb 5, 2025
a911120
Merge branch 'main' into cargo-lock
mbrobbel Feb 5, 2025
9291eae
Merge branch 'main' into cargo-lock
mbrobbel Feb 5, 2025
f1cdd72
Merge remote-tracking branch 'apache/main' into cargo-lock
alamb Feb 8, 2025
ce4a4ef
Update cargo.lock
alamb Feb 8, 2025
5d30308
Update .github/workflows/rust.yml
alamb Feb 8, 2025
eb3046f
Update Cargo.lock
alamb Feb 8, 2025
8fe3cb9
Merge branch 'cargo-lock' of github.com:mbrobbel/datafusion into carg…
alamb Feb 8, 2025
347cc0a
fix yaml
alamb Feb 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 0 additions & 15 deletions .github/dependabot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,24 +28,9 @@ updates:
# arrow is bumped manually
- dependency-name: "arrow*"
update-types: ["version-update:semver-major"]
- package-ecosystem: cargo
directory: "datafusion-cli/"
schedule:
interval: daily
open-pull-requests-limit: 10
target-branch: main
labels: [auto-dependencies]
ignore:
# arrow is bumped manually
- dependency-name: "arrow*"
update-types: ["version-update:semver-major"]
# datafusion is bumped manually
- dependency-name: "datafusion*"
update-types: ["version-update:semver-major"]
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
open-pull-requests-limit: 10
labels: [auto-dependencies]

4 changes: 3 additions & 1 deletion .github/workflows/dependencies.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,11 @@ on:
push:
paths:
- "**/Cargo.toml"
- "**/Cargo.lock"
pull_request:
paths:
- "**/Cargo.toml"
- "**/Cargo.lock"
# manual trigger
# https://docs.github.com/en/actions/managing-workflow-runs/manually-running-a-workflow
workflow_dispatch:
Expand All @@ -50,4 +52,4 @@ jobs:
- name: Check dependencies
run: |
cd dev/depcheck
cargo run
cargo run
8 changes: 4 additions & 4 deletions .github/workflows/extended.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
with:
rust-version: stable
- name: Prepare cargo build
run: cargo check --profile ci --all-targets
run: cargo check --profile ci --all-targets --locked

# Run extended tests (with feature 'extended_tests')
linux-test-extended:
Expand All @@ -65,7 +65,7 @@ jobs:
with:
rust-version: stable
- name: Run tests (excluding doctests)
run: cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests
run: cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests --locked
- name: Verify Working Directory Clean
run: git diff --exit-code

Expand All @@ -87,7 +87,7 @@ jobs:
- name: Run tests
run: |
cd datafusion
cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-sqllogictest --workspace --lib --tests --features=force_hash_collisions,avro,extended_tests
cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-sqllogictest --workspace --lib --tests --features=force_hash_collisions,avro,extended_tests --locked

@findepi findepi Feb 5, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cargo test shouldn't need the --locked flag (here & other places)

it would suffice to verify cargo lock updatodateness once in single job

@mbrobbel mbrobbel Feb 5, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I added it everywhere to let all these jobs fail early when the lock file needs to be updated (to save CI resources), because in that case all those jobs are going to have to run again when the lock file is updated. But I'm also fine with only keeping it for one check.

@findepi findepi Feb 5, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fail-fast is not unreasonable, i didn't think about this.

otoh one could say same thing about formatting issues. there is definitely tradeoff between conserving CI resources and maxing our single PR feedback signal.
let's put cargo.lock consistency in the "formatting bucket", ie reuse previous design decision for now. i am open to revisiting it.

@mbrobbel mbrobbel Feb 5, 2025 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll just keep it for

datafusion/.github/workflows/extended.yml

Line 37 in 684c7df

linux-build-lib:
which is required to pass before a bunch of additional jobs are started.

Edit: I meant to link:

datafusion/.github/workflows/rust.yml

Line 51 in 684c7df

linux-build-lib:

@findepi findepi Feb 5, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!
can you please add a code comment there why --locked is useful?

@mbrobbel mbrobbel Feb 5, 2025

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

152f1cf


sqllogictest-sqlite:
name: "Run sqllogictests with the sqlite test suite"
Expand All @@ -104,4 +104,4 @@ jobs:
with:
rust-version: stable
- name: Run sqllogictest
run: cargo test --profile release-nonlto --test sqllogictests -- --include-sqlite
run: cargo test --profile release-nonlto --test sqllogictests --locked -- --include-sqlite
176 changes: 67 additions & 109 deletions .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ jobs:
with:
rust-version: stable

@alamb alamb Feb 7, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a nice idea to only use --locked in one test 👍

- name: Prepare cargo build
run: cargo check --profile ci --all-targets --features integration-tests
run: cargo check --profile ci --all-targets --features integration-tests --locked

# cargo check common, functions and substrait with no default features
linux-cargo-check-no-default-features:
Expand All @@ -78,28 +78,22 @@ jobs:
- name: Check datafusion without default features
# Some of the test binaries require the parquet feature still
#run: cargo check --all-targets --no-default-features -p datafusion
run: cargo check --profile ci --no-default-features -p datafusion
run: cargo check --profile ci --no-default-features -p datafusion --locked

- name: Check datafusion-common without default features
run: cargo check --profile ci --all-targets --no-default-features -p datafusion-common
run: cargo check --profile ci --all-targets --no-default-features -p datafusion-common --locked

- name: Check datafusion-functions without default features
run: cargo check --profile ci --all-targets --no-default-features -p datafusion-functions
run: cargo check --profile ci --all-targets --no-default-features -p datafusion-functions --locked

- name: Check datafusion-substrait without default features
run: cargo check --profile ci --all-targets --no-default-features -p datafusion-substrait
run: cargo check --profile ci --all-targets --no-default-features -p datafusion-substrait --locked

- name: Check workspace in debug mode
run: cargo check --profile ci --all-targets --workspace
run: cargo check --profile ci --all-targets --workspace --locked

- name: Check workspace with additional features
run: cargo check --profile ci --workspace --benches --features avro,json,integration-tests

- name: Check Cargo.lock for datafusion-cli
run: |
# If this test fails, try running `cargo update` in the `datafusion-cli` directory
# and check in the updated Cargo.lock file.
cargo check --profile ci --manifest-path datafusion-cli/Cargo.toml --locked
run: cargo check --profile ci --workspace --benches --features avro,json,integration-tests --locked

# cargo check datafusion to ensure that the datafusion crate can be built with only a
# subset of the function packages enabled.
Expand All @@ -116,25 +110,25 @@ jobs:
with:
rust-version: stable
- name: Check datafusion (nested_expressions)
run: cargo check --profile ci --no-default-features --features=nested_expressions -p datafusion
run: cargo check --profile ci --no-default-features --features=nested_expressions -p datafusion --locked

- name: Check datafusion (crypto)
run: cargo check --profile ci --no-default-features --features=crypto_expressions -p datafusion
run: cargo check --profile ci --no-default-features --features=crypto_expressions -p datafusion --locked

- name: Check datafusion (datetime_expressions)
run: cargo check --profile ci --no-default-features --features=datetime_expressions -p datafusion
run: cargo check --profile ci --no-default-features --features=datetime_expressions -p datafusion --locked

- name: Check datafusion (encoding_expressions)
run: cargo check --profile ci --no-default-features --features=encoding_expressions -p datafusion
run: cargo check --profile ci --no-default-features --features=encoding_expressions -p datafusion --locked

- name: Check datafusion (math_expressions)
run: cargo check --profile ci --no-default-features --features=math_expressions -p datafusion
run: cargo check --profile ci --no-default-features --features=math_expressions -p datafusion --locked

- name: Check datafusion (regex_expressions)
run: cargo check --profile ci --no-default-features --features=regex_expressions -p datafusion
run: cargo check --profile ci --no-default-features --features=regex_expressions -p datafusion --locked

- name: Check datafusion (string_expressions)
run: cargo check --profile ci --no-default-features --features=string_expressions -p datafusion
run: cargo check --profile ci --no-default-features --features=string_expressions -p datafusion --locked

# cargo check datafusion-functions to ensure that the datafusion-functions crate can be built with
# only a subset of the function packages enabled.
Expand All @@ -151,22 +145,22 @@ jobs:
with:
rust-version: stable
- name: Check datafusion-functions (crypto)
run: cargo check --profile ci --all-targets --no-default-features --features=crypto_expressions -p datafusion-functions
run: cargo check --profile ci --all-targets --no-default-features --features=crypto_expressions -p datafusion-functions --locked

- name: Check datafusion-functions (datetime_expressions)
run: cargo check --profile ci --all-targets --no-default-features --features=datetime_expressions -p datafusion-functions
run: cargo check --profile ci --all-targets --no-default-features --features=datetime_expressions -p datafusion-functions --locked

- name: Check datafusion-functions (encoding_expressions)
run: cargo check --profile ci --all-targets --no-default-features --features=encoding_expressions -p datafusion-functions
run: cargo check --profile ci --all-targets --no-default-features --features=encoding_expressions -p datafusion-functions --locked

- name: Check datafusion-functions (math_expressions)
run: cargo check --profile ci --all-targets --no-default-features --features=math_expressions -p datafusion-functions
run: cargo check --profile ci --all-targets --no-default-features --features=math_expressions -p datafusion-functions --locked

- name: Check datafusion-functions (regex_expressions)
run: cargo check --profile ci --all-targets --no-default-features --features=regex_expressions -p datafusion-functions
run: cargo check --profile ci --all-targets --no-default-features --features=regex_expressions -p datafusion-functions --locked

- name: Check datafusion-functions (string_expressions)
run: cargo check --profile ci --all-targets --no-default-features --features=string_expressions -p datafusion-functions
run: cargo check --profile ci --all-targets --no-default-features --features=string_expressions -p datafusion-functions --locked

# Run tests
linux-test:
Expand All @@ -185,29 +179,7 @@ jobs:
with:
rust-version: stable
- name: Run tests (excluding doctests)
run: cargo test --profile ci --exclude datafusion-examples --exclude ffi_example_table_provider --exclude datafusion-benchmarks --workspace --lib --tests --bins --features avro,json,backtrace,integration-tests
- name: Verify Working Directory Clean
run: git diff --exit-code

linux-test-datafusion-cli:
name: cargo test datafusion-cli (amd64)
needs: linux-build-lib
runs-on: ubuntu-latest
container:
image: amd64/rust
steps:
- uses: actions/checkout@v4
with:
submodules: true
fetch-depth: 1
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: stable
- name: Run tests (excluding doctests)
run: |
cd datafusion-cli
cargo test --profile ci --lib --tests --bins --all-features
run: cargo test --profile ci --exclude datafusion-examples --exclude ffi_example_table_provider --exclude datafusion-benchmarks --workspace --lib --tests --bins --features avro,json,backtrace,integration-tests --locked
- name: Verify Working Directory Clean
run: git diff --exit-code

Expand All @@ -229,7 +201,7 @@ jobs:
- name: Run examples
run: |
# test datafusion-sql examples
cargo run --profile ci --example sql
cargo run --profile ci --example sql --locked
# test datafusion-examples
ci/scripts/rust_example.sh
- name: Verify Working Directory Clean
Expand All @@ -252,10 +224,7 @@ jobs:
with:
rust-version: stable
- name: Run doctests
run: |
cargo test --profile ci --doc --features avro,json
cd datafusion-cli
cargo test --profile ci --doc --all-features
run: cargo test --profile ci --doc --features avro,json --locked
- name: Verify Working Directory Clean
run: git diff --exit-code

Expand Down Expand Up @@ -321,8 +290,8 @@ jobs:
# increase stack size to fix stack overflow
export RUST_MIN_STACK=20971520
export TPCH_DATA=`realpath datafusion/sqllogictest/test_files/tpch/data`
cargo test plan_q --package datafusion-benchmarks --profile ci --features=ci -- --test-threads=1
INCLUDE_TPCH=true cargo test --profile ci --package datafusion-sqllogictest --test sqllogictests
cargo test plan_q --package datafusion-benchmarks --profile ci --features=ci --locked -- --test-threads=1
INCLUDE_TPCH=true cargo test --profile ci --package datafusion-sqllogictest --test sqllogictests --locked
- name: Verify Working Directory Clean
run: git diff --exit-code

Expand Down Expand Up @@ -358,51 +327,46 @@ jobs:
- name: Run sqllogictest
run: |
cd datafusion/sqllogictest
PG_COMPAT=true PG_URI="postgresql://postgres:postgres@$POSTGRES_HOST:$POSTGRES_PORT/db_test" cargo test --profile ci --features=postgres --test sqllogictests
PG_COMPAT=true PG_URI="postgresql://postgres:postgres@$POSTGRES_HOST:$POSTGRES_PORT/db_test" cargo test --profile ci --features=postgres --test sqllogictests --locked
env:
# use postgres for the host here because we have specified a container for the job
POSTGRES_HOST: postgres
POSTGRES_PORT: ${{ job.services.postgres.ports[5432] }}

# Temporarily commenting out the Windows flow, the reason is enormously slow running build
# Waiting for new Windows 2025 github runner
# Details: https://github.com/apache/datafusion/issues/13726
#
# windows:
# name: cargo test (win64)
# runs-on: windows-latest
# steps:
# - uses: actions/checkout@v4
# with:
# submodules: true
# - name: Setup Rust toolchain
# uses: ./.github/actions/setup-windows-builder
# - name: Run tests (excluding doctests)
# shell: bash
# run: |
# export PATH=$PATH:$HOME/d/protoc/bin
# cargo test --lib --tests --bins --features avro,json,backtrace
# cd datafusion-cli
# cargo test --lib --tests --bins --all-features

# Commenting out intel mac build as so few users would ever use it
# Details: https://github.com/apache/datafusion/issues/13846
# macos:
# name: cargo test (macos)
# runs-on: macos-latest
# steps:
# - uses: actions/checkout@v4
# with:
# submodules: true
# fetch-depth: 1
# - name: Setup Rust toolchain
# uses: ./.github/actions/setup-macos-builder
# - name: Run tests (excluding doctests)
# shell: bash
# run: |
# cargo test run --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --workspace --lib --tests --bins --features avro,json,backtrace
# cd datafusion-cli
# cargo test run --profile ci --lib --tests --bins --all-features
# Temporarily commenting out the Windows flow, the reason is enormously slow running build
# Waiting for new Windows 2025 github runner
# Details: https://github.com/apache/datafusion/issues/13726
#
# windows:
# name: cargo test (win64)
# runs-on: windows-latest
# steps:
# - uses: actions/checkout@v4
# with:
# submodules: true
# - name: Setup Rust toolchain
# uses: ./.github/actions/setup-windows-builder
# - name: Run tests (excluding doctests)
# shell: bash
# run: |
# export PATH=$PATH:$HOME/d/protoc/bin
# cargo test --lib --tests --bins --features avro,json,backtrace

# Commenting out intel mac build as so few users would ever use it
# Details: https://github.com/apache/datafusion/issues/13846
# macos:
# name: cargo test (macos)
# runs-on: macos-latest
# steps:
# - uses: actions/checkout@v4
# with:
# submodules: true
# fetch-depth: 1
# - name: Setup Rust toolchain
# uses: ./.github/actions/setup-macos-builder
# - name: Run tests (excluding doctests)
# shell: bash
# run: cargo test run --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --workspace --lib --tests --bins --features avro,json,backtrace

macos-aarch64:
name: cargo test (macos-aarch64)
Expand All @@ -416,10 +380,7 @@ jobs:
uses: ./.github/actions/setup-macos-aarch64-builder
- name: Run tests (excluding doctests)
shell: bash
run: |
cargo test --profile ci --lib --tests --bins --features avro,json,backtrace,integration-tests
cd datafusion-cli
cargo test --profile ci --lib --tests --bins --all-features
run: cargo test --profile ci --lib --tests --bins --features avro,json,backtrace,integration-tests --locked

test-datafusion-pyarrow:
name: cargo test pyarrow (amd64)
Expand All @@ -444,7 +405,7 @@ jobs:
with:
rust-version: stable
- name: Run datafusion-common tests
run: cargo test --profile ci -p datafusion-common --features=pyarrow
run: cargo test --profile ci -p datafusion-common --features=pyarrow --locked

vendor:
name: Verify Vendored Code
Expand Down Expand Up @@ -615,19 +576,19 @@ jobs:
# (Min Supported Rust Version) than the one specified in the
# `rust-version` key of `Cargo.toml`.
#
# To reproduce:
# 1. Install the version of Rust that is failing. Example:
# To reproduce:
# 1. Install the version of Rust that is failing. Example:
# rustup install 1.80.1
# 2. Run the command that failed with that version. Example:
# cargo +1.80.1 check -p datafusion
#
#
# To resolve, either:
# 1. Change your code to use older Rust features,
# 1. Change your code to use older Rust features,
# 2. Revert dependency update
# 3. Update the MSRV version in `Cargo.toml`
#
# Please see the DataFusion Rust Version Compatibility Policy before
# updating Cargo.toml. You may have to update the code instead.
# updating Cargo.toml. You may have to update the code instead.
# https://github.com/apache/datafusion/blob/main/README.md#rust-version-compatibility-policy
cargo msrv --output-format json --log-target stdout verify
- name: Check datafusion-substrait
Expand All @@ -636,6 +597,3 @@ jobs:
- name: Check datafusion-proto
working-directory: datafusion/proto
run: cargo msrv --output-format json --log-target stdout verify
- name: Check datafusion-cli
working-directory: datafusion-cli
run: cargo msrv --output-format json --log-target stdout verify
2 changes: 0 additions & 2 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,6 @@ venv/*

# Rust
target
Cargo.lock
!datafusion-cli/Cargo.lock

rusty-tags.vi
.history
Expand Down
Loading