[GLUTEN-7336][CORE] Bump Spark version to v3.5.3

[Yohahaha]

What changes were proposed in this pull request?

Summary:

1. bump spark version to 3.5.3
2. disable hudi tests in GHA, [CORE] Reopen hudi GHA tests with Spark353 #7571
3. bump delta version to 3.2.1
4. iceberg module may breaks [CORE] Iceberg and hudi module may breaks with Spark35 #7637, but GHA was passed currently.

Fixes: #7336

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

#7336

[github-actions]

Run Gluten Clickhouse CI

[zhouyuan]

@Yohahaha can you please do a rebase? there's a small conflict on GHA script

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[zhouyuan]

CC @baibaichen @zzcclp

[baibaichen]

@zhouyuan

I need some time to do performance testing, maybe I'll have a chance to merge next week.

[Yohahaha]

@zhouyuan

I need some time to do performance testing, maybe I'll have a chance to merge next week.

why CH backend's performance may drop when upgrade spark version?

[baibaichen]

why CH backend's performance may drop when upgrade spark version?

It's not said that performance will decrease, it's just a standard process for upgrading versions. In fact, when upgrading to 3.5.2, we found that two TPCDS queries were a bit faster.

[baibaichen]

LGTM

[github-actions]

Run Gluten Clickhouse CI

[zhztheplayer]

Assuming it's unintentional?

[Yohahaha]

maybe it's Github diff issue.

[jackylee-ch]

Have you checked the iceberg with Spark 3.5.3? It seems that Spark 3.5.3 has some problems with iceberg and the iceberg community would not support the Spark 3.5.3 release in iceberg-11160.

[Yohahaha]

Have you checked the iceberg with Spark 3.5.3? It seems that Spark 3.5.3 has some problems with iceberg and the iceberg community would not support the Spark 3.5.3 release in iceberg-11160.

no, I only check the GHA passed.

IMO, Gluten only depends on Spark, we have no guarantee that gluten-iceberg will works well after this PR if iceberg has decided to skip Spark 3.5.3.

[zzcclp]

can wo merge this pr first if ci passed, if there are some problems with the iceberg, will raise another pr to fix ?

[Yohahaha]

#7637
Open above issue for gluten-iceberg if we need fix it.