build: Stop running Comet's Spark 4 tests on Linux for PR builds

[andygrove]

Which issue does this PR close?

Part of #1786

Rationale for this change

Until we resolve #1786, let's stop running the problematic test so that we can unblock other PRs.

We are still testing Spark 4 on macos.

What changes are included in this PR?

How are these changes tested?

[parthchandra]

Can we make the Spark 4 pipeline available as a manually triggered pipeline? We do need to make sure we are more or less stable wrt Spark 4.

[andygrove]

Can we make the Spark 4 pipeline available as a manually triggered pipeline? We do need to make sure we are more or less stable wrt Spark 4.

Sure. I moved it to a separate workflow for now. We are still running Spark 4 tests on macos.

[parthchandra]

lgtm

[comphead]

lgtm thanks @andygrove

[andygrove]

It is also worth noting that we are still running the Spark SQL tests for Spark 4

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 59.45%. Comparing base (f09f8af) to head (390a90f).
Report is 216 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1802      +/-   ##
============================================
+ Coverage     56.12%   59.45%   +3.32%     
- Complexity      976     1147     +171     
============================================
  Files           119      128       +9     
  Lines         11743    12528     +785     
  Branches       2251     2355     +104     
============================================
+ Hits           6591     7448     +857     
+ Misses         4012     3889     -123     
- Partials       1140     1191      +51     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.