Coerce MemTable schema to fix Binary/BinaryView mismatch on reduce sink

[vinaykpud]

Description

Fixes a Substrait Binary vs MemTable BinaryView type-mismatch on the reduce-sink path. Cross-index queries (JOIN, IN-subquery) over parquet-backed indices with ip or binary fields fail at plan-binding time with:

Substrait error: Field '<x>' in Substrait schema has a different type (Binary)
than the corresponding field in the table schema (BinaryView).

Root cause

derive_schema_from_partial_plan (in analytics-backend-datafusion/rust/src/api.rs) builds a throwaway MemTable to bind named-table refs in the substrait wire plan. It applies transform_schema_to_view (flips Binary → BinaryView) but never applies schema_coerce::coerce_inferred_schema to the MemTable schema before from_substrait_plan. The substrait consumer then sees wire Binary against MemTable BinaryView and rejects the binding.

PR #21631 introduced the coercer for the table-scan path

Fix

One-line addition: apply coerce_inferred_schema to the MemTable schema after transform_schema_to_view and coerce_unsupported_timestamp_precision. The MemTable now lands in the same Binary/Int64/Float32 shape as the wire.

Verification

Verified locally against mch2's 2-shard QA branch (cherry-picked qa: default analytics-engine datasets to 2 shards + un-mute commits). Across 5 suites named in mch2's failure report bucket #1 (Binary/IP, 170 units):

• Before fix: every Binary/BinaryView query 500s with the message above.
• After fix: 399 PPL queries executed across the suites, 0 Binary/BinaryView errors.

Residual failures all attributable to other report buckets (unimplemented scalar functions, RexCall casts, decimal divide, etc.) — none introduced by this change.

Related Issues

• Builds on Added scan support for indexed fieldTypes #21631 (scan support for `ip`/`binary` field types — introduced `schema_coerce`)

Check List

• Functionality includes testing — Rust unit tests in `schema_coerce.rs` cover the `BinaryView → Binary` rewrite; e2e verified via QA suite re-run
• API changes companion pull request — N/A
• Public documentation issue/PR — N/A

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 No relevant tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Schema Coercion Ordering The schema is coerced twice in sequence: first by coerce_unsupported_timestamp_precision (line 1157), then by coerce_inferred_schema (line 1158). If both functions modify overlapping field types, the second coercion could override the first, potentially undoing timestamp precision fixes. This matters if coerce_inferred_schema transforms timestamp fields without preserving the precision adjustments made by coerce_unsupported_timestamp_precision. let arrow_schema = coerce_unsupported_timestamp_precision(&arrow_schema); let arrow_schema = crate::schema_coerce::coerce_inferred_schema(Arc::new(arrow_schema)); let table = MemTable::try_new(arrow_schema, vec![vec![]])?;

[github-actions]

✅ Gradle check result for 8f7b3a2: SUCCESS

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.50%. Comparing base (dad63c0) to head (8f7b3a2).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #21910      +/-   ##
============================================
- Coverage     73.51%   73.50%   -0.01%     
+ Complexity    75582    75573       -9     
============================================
  Files          6034     6034              
  Lines        342661   342661              
  Branches      49294    49294              
============================================
- Hits         251918   251885      -33     
- Misses        70712    70753      +41     
+ Partials      20031    20023       -8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.