feat(glue-alpha): include extra jars parameter in pyspark jobs

[gontzalm]

Issue # (if applicable)

Closes #33225.

Reason for this change

PySpark jobs with extra JAR dependencies cannot be defined with the new L2 constructs introduced in v2.177.0.

Description of changes

Add the extraJars parameter in the PySpark job L2 constructs.

Checklist

• My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

[gontzalm]

Exemption Request: no changes in README or integration tests needed.

[aws-cdk-automation]

The pull request linter fails with the following errors:

❌ Features must contain a change to a README file.
❌ Features must contain a change to an integration test file and the resulting snapshot.

If you believe this pull request should receive an exemption, please comment and provide a justification. A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed, add Clarification Request to a comment.

✅ A exemption request has been requested. Please wait for a maintainer's review.

[aws-cdk-automation]

AWS CodeBuild CI Report

• CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
• Commit ID: 399e8df
• Result: SUCCEEDED
• Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

[natalie-white-aws]

One of the authors of the new L2 here - We talked about this during RFC and implementation phases as a potential anti-pattern. Can you share why you need extra jars for a python job?

[gontzalm]

Hi Natalie, we need to use the spark-xml package in order to read XML files in Spark v3 (as you probably know, this package will be included in Spark v4). This package must be provided via the extraJars parameter, because AWS Glue does not accept installing packages via --conf spark.jars.packages=<maven coordinates>.

[natalie-white-aws]

Thanks for the extra clarification. Let me get with the Glue service team; it sounds like this may be more of a Glue feature request than something we should work around in the L2 construct. Stay tuned.

[humanzz]

+1 to this. Some libraries that provide additional spark capabilities require a jar, even if one is actually using spark via python (pyspark).

Here's a chatgpt-generated list of examples https://chatgpt.com/share/67a8e12d-ccd8-800e-a641-75e58db91d7b

[natalie-white-aws]

We had some internal discussions and (in addition to the data here) decided this is a valid use case. But we should add them to all 3 PySpark job types.