chore: remove sql feature and sqlparser dependency from DataFusion

[Copilot]

Removes the sql feature (and its transitive datafusion-sql/sqlparser dependencies) from our DataFusion integration, updating to DataFusion 53.0.0 which fixes the recursive_protection feature so it no longer activates sql.

Cargo.toml

• datafusion: default-features = false, explicitly list all needed features including avro and recursive_protection (both now safe in v53 — avro no longer pulls datafusion-sql, and recursive_protection no longer activates sql) — omitting sql
• datafusion-expr, datafusion-functions-nested: default-features = false to prevent their sql default from re-enabling datafusion-expr/sql via feature unification
• Removed separate recursive_protection entries for individual crates (datafusion-common, datafusion-expr, datafusion-physical-expr, datafusion-optimizer, datafusion-physical-optimizer) — datafusion/recursive_protection now handles all sub-crates correctly in v53

wildcard.rs

Replace sqlparser::ast types and expr_to_sql with the non-sql equivalents from datafusion_expr::expr:

// Before
use datafusion::sql::unparser::expr_to_sql;
use datafusion::sql::sqlparser::ast;
let ilike = wildcard_options.ilike_pattern.map(|x| ast::IlikeSelectItem { pattern: x });
let item = ast::ReplaceSelectElement { expr: expr_to_sql(&expression)?, ... };

// After
let ilike = wildcard_options.ilike_pattern.map(|x| expr::IlikeSelectItem { pattern: x });
let item = expr::ReplaceSelectElement { expr: expression.to_string(), ... };

A make_ident helper reduces repetition when constructing expr::Ident with default quote_style/span values.

error.rs

Remove the DataFusionError::SQL match arm — that variant is gated behind datafusion-common's sql feature, which is now disabled.

avro/mod.rs and codec.rs

Use the original import paths via datafusion re-exports (datafusion::datasource::file_format::avro::AvroFormat and datafusion::datasource::physical_plan::AvroSource). In DataFusion 53, the avro feature is clean and no longer pulls in datafusion-sql, so no direct datafusion-datasource-avro dependency is needed in individual crates.

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[linhr]

@copilot Please investigate why datafusion-sql and sqlparser still show up in Cargo.lock.

[codecov]

Codecov Report

❌ Patch coverage is 3.12500% with 31 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ates/sail-plan/src/resolver/expression/wildcard.rs	3.12%	31 Missing ⚠️

@@            Coverage Diff             @@
##             main    #1437      +/-   ##
==========================================
- Coverage   75.12%   74.79%   -0.34%     
==========================================
  Files         870      877       +7     
  Lines      114384   122097    +7713     
==========================================
+ Hits        85936    91321    +5385     
- Misses      28448    30776    +2328     

Flag	Coverage Δ		*Carryforward flag
ibis-tests	19.24% <ø> (+<0.01%)	⬆️	Carriedforward from 7e06f79
python-unit-tests	57.02% <3.44%> (+0.66%)	⬆️
rust-slow-tests	43.05% <ø> (+<0.01%)	⬆️	Carriedforward from 7e06f79
rust-unit-tests	38.55% <3.12%> (-0.16%)	⬇️
spark-tests	33.15% <ø> (+<0.01%)	⬆️	Carriedforward from 7e06f79

*This pull request uses carry forward flags. Click here to find out more.

Files with missing lines	Coverage Δ
crates/sail-common-datafusion/src/error.rs	44.56% <ø> (-4.25%)	⬇️
...ates/sail-plan/src/resolver/expression/wildcard.rs	54.68% <3.12%> (-9.24%)	⬇️

... and 97 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

@copilot Please investigate why datafusion-sql and sqlparser still show up in Cargo.lock.

They still appeared because datafusion's avro feature directly enables datafusion-sql:

avro = [
    "datafusion-common/avro",
    "datafusion-datasource-avro",
    "unicode_expressions",
    "datafusion-sql/recursive_protection",
    "datafusion-sql",   # <-- this pulled in datafusion-sql
]

Fixed in facc882 by removing avro from datafusion's features and instead depending directly on datafusion-datasource-avro. The imports in sail-data-source and sail-execution are updated to use datafusion_datasource_avro directly.

Note: datafusion-sql and sqlparser still appear in Cargo.lock as resolved optional dependencies of datafusion, datafusion-common, and datafusion-expr (all of which have sqlparser as an optional dep for their sql feature). This is standard Cargo behavior — the resolver records all optional deps even when unactivated. They are confirmed NOT compiled: cargo build --message-format=json shows neither sqlparser nor datafusion-sql in the set of compiled artifacts.

[lonless9]

@copilot Merge from origin/main branch and resolve the conflicts. Then follow apache/datafusion#20625 that mentioned

When enabling the recursive_protection feature for the datafusion crate, the sql feature is enabled. This is undesirable if the downstream project would like the sql feature to be off.

This issue has been handled in the DataFusion 53 version so now again try to remove the sql feature and sqlparser dependency.

[Copilot]

@copilot Merge from origin/main branch and resolve the conflicts. Then follow apache/datafusion#20625 that mentioned

When enabling the recursive_protect...

Done in d5ab5c5. Merged from origin/main (DataFusion 53.0.0) and resolved the conflict.

With DataFusion 53, recursive_protection no longer activates sql (apache/datafusion#20625), so the Cargo.toml is simplified:

• datafusion: default-features = false, features include recursive_protection but NOT sql
• datafusion-expr and datafusion-functions-nested: still need default-features = false since their own defaults include sql
• Removed the separate recursive_protection entries for datafusion-common, datafusion-optimizer, etc. (now handled via datafusion/recursive_protection)

sqlparser and datafusion-sql are confirmed NOT compiled (verified via --message-format=json).

[linhr]

@copilot Add back the avro feature for datafusion and keep the original way of importing the Avro data source.

[Copilot]

@copilot Add back the avro feature for datafusion and keep the original way of importing the Avro data source.

Done in d9c1921. Added avro back to datafusion's features and reverted to the original import paths (datafusion::datasource::file_format::avro::AvroFormat and datafusion::datasource::physical_plan::AvroSource). The datafusion-datasource-avro direct dependencies are also removed from individual crates. In DataFusion 53, the avro feature no longer pulls in datafusion-sql, so sqlparser and datafusion-sql are still confirmed NOT compiled.