Added substrait converter for the fragments

[vinaykpud]

Description

Add Substrait-based fragment converter to the DataFusion backend plugin, enabling end-to-end query execution from DSL query → Calcite RelNode → Substrait plan → native DataFusion runtime → Arrow result batches → materialized rows.

• Fix classloader dependency issues: flatbuffers-java, arrow-format, jackson-databind, jackson-annotations moved to bundled in analytics-engine (parent plugin). Arrow's ArrowType subclasses are loaded by the parent classloader and transitively reference these jars — child plugins can't provide them. Removed duplicates from analytics-backend-datafusion to avoid jar hell.

• Fix Substrait TCCL issue: Substrait uses Jackson polymorphic deserialization which resolves class names via the thread context classloader. On search threads the TCCL is the system CL and can't see plugin classes. Moved extension catalog loading to plugin bootstrap with a one-time TCCL swap so it's cached before any query runs.

• Fix Arrow memory leak: batchesToRows() iterated VectorSchemaRoot batches without closing them. Added batch.close() after materializing rows so the per-query allocator closes cleanly.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for c5398f2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[mch2]

overall looks like a good start here for substrait conversion, thanks @vinaykpud .

[sandeshkr419]

Execution failed for task ':sandbox:plugins:analytics-backend-datafusion:thirdPartyAudit'.
> Audit of third party dependencies failed

@vinaykpud lets fix this as well. Thanks!

[github-actions]

❌ Gradle check result for 302985b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[vinaykpud]

@sandeshkr419
Sure.
The missing class SqlDdlParserImpl is Calcite's DDL parser,it handles SQL statements like CREATE TABLE, DROP TABLE, ALTER TABLE. It lives in a separate calcite-server module and we don't need DDL parsing. So fixing by adding it to ignored list

[github-actions]

✅ Gradle check result for c39e324: SUCCESS

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.33%. Comparing base (436e4a6) to head (9a81b59).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #21348      +/-   ##
============================================
- Coverage     73.34%   73.33%   -0.02%     
+ Complexity    74223    74205      -18     
============================================
  Files          5958     5958              
  Lines        337309   337349      +40     
  Branches      48664    48687      +23     
============================================
- Hits         247408   247380      -28     
- Misses        70188    70246      +58     
- Partials      19713    19723      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Gradle check result for 9a81b59: SUCCESS