build: Add spark-4.1 profile and shims

[manuzhang]

Which issue does this PR close?

First step of #2792.

Rationale for this change

What changes are included in this PR?

Add spark-4.1 profile and shims.

How are these changes tested?

Added UTs.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.16%. Comparing base (f09f8af) to head (31cf1ca).
⚠️ Report is 728 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2829      +/-   ##
============================================
+ Coverage     56.12%   59.16%   +3.03%     
- Complexity      976     1477     +501     
============================================
  Files           119      167      +48     
  Lines         11743    15188    +3445     
  Branches       2251     2523     +272     
============================================
+ Hits           6591     8986    +2395     
- Misses         4012     4917     +905     
- Partials       1140     1285     +145     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.