feat!: Add support for sparse transform expressions (#1199)

## What changes are proposed in this pull request?

Log replay needs to define per-file transforms for row of metadata that
survives data skipping. Unfortunately, `Expression::Struct` is "dense"
(mentions _every_ output field) and this produces excessive overhead
when injecting the (usually very few) partition columns for tables with
wide schemas (hundreds or thousands of columns). Column mapping tables
are even worse, because they don't _change_ the columns at all -- they
just need to apply the output schema to the input data.

Solution: Define a new `Expression::Transform` that is a "sparse"
representation of the _changes_ to be made to a given top-level schema
or nested struct. Input columns can be dropped or replaced, and new
output columns can be injected after an input column of choice (or
prepended to the output schema). The engine's expression evaluator does
the actual work to transfer unchanged input columns across while
building the output `EngineData`.

Update log replay to use the new transform capability, so that the cost
is `O(partition_columns)` instead of `O(schema_width)`. For
non-partitioned tables with column mapping mode enabled, this translates
to an empty (identity) transform which the default engine expression
evaluator has been updated to optimize as a special case (just
`apply_schema` directly to the input and return).

Result: Scan times are cut by nearly half in the `metadata_bench`
benchmark:
```
Benchmarking scan_metadata/scan_metadata: Warming up for 3.0000 s
Warning: Unable to complete 20 samples in 5.0s. You may wish to increase target time to 5.1s, or reduce sample count to 10.
scan_metadata/scan_metadata
                        time:   [250.51 ms 253.72 ms 257.45 ms]
                        change: [-45.173% -44.306% -43.415%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 20 measurements (5.00%)
  1 (5.00%) high mild
```

### This PR affects the following public APIs

Added a new `Expression::Transform` enum variant.

## How was this change tested?

New and existing unit tests, existing benchmarks.

Author: scovich

ffi/src/expressions/engine_visitor.rs

@@ -482,6 +482,11 @@ fn visit_expression_impl(
         Expression::Opaque(OpaqueExpression { op, exprs }) => {
             visit_expression_opaque(visitor, op, exprs, sibling_list_id)
         }
+        Expression::Transform(_) => {
+            // Minimal FFI support: Transform expressions are treated as unknown
+            // TODO: Implement full Transform FFI support in future version
+            visit_unknown(visitor, sibling_list_id, "Transform")
+        }
         Expression::Unknown(name) => visit_unknown(visitor, sibling_list_id, name),
     }
 }

kernel/src/engine/arrow_expression/apply_schema.rs

@@ -183,3 +183,59 @@ pub(crate) fn apply_schema_to(array: &ArrayRef, schema: &DataType) -> DeltaResul
     };
     Ok(array)
 }
+
+#[cfg(test)]
+mod apply_schema_validation_tests {
+    use super::*;
+
+    use std::sync::Arc;
+
+    use crate::arrow::array::{Int32Array, StructArray};
+    use crate::arrow::datatypes::{
+        DataType as ArrowDataType, Field as ArrowField, Schema as ArrowSchema,
+    };
+    use crate::schema::{DataType, StructField, StructType};
+
+    #[test]
+    fn test_apply_schema_basic_functionality() {
+        // Test that apply_schema works for basic field transformation
+        let input_array = create_test_struct_array_2_fields();
+        let target_schema = create_target_schema_2_fields();
+
+        // This should succeed - basic schema application
+        let result = apply_schema_to_struct(&input_array, &target_schema);
+        assert!(result.is_ok(), "Basic schema application should succeed");
+
+        let result_array = result.unwrap();
+        assert_eq!(
+            result_array.len(),
+            input_array.len(),
+            "Row count should be preserved"
+        );
+        assert_eq!(result_array.num_columns(), 2, "Should have 2 columns");
+    }
+
+    // Helper functions to create test data
+    fn create_test_struct_array_2_fields() -> StructArray {
+        let field1 = ArrowField::new("a", ArrowDataType::Int32, false);
+        let field2 = ArrowField::new("b", ArrowDataType::Int32, false);
+        let schema = ArrowSchema::new(vec![field1, field2]);
+
+        let a_data = Int32Array::from(vec![1, 2, 3]);
+        let b_data = Int32Array::from(vec![4, 5, 6]);
+
+        StructArray::try_new(
+            schema.fields.clone(),
+            vec![Arc::new(a_data), Arc::new(b_data)],
+            None,
+        )
+        .unwrap()
+    }
+
+    fn create_target_schema_2_fields() -> StructType {
+        StructType::new([
+            StructField::new("a", DataType::INTEGER, false),
+            StructField::new("b", DataType::INTEGER, false),
+        ])
+    }
+}