Skip to content

Move GCP args from DataprocSubmitter to CLI #915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Jun 30, 2025
Merged

Move GCP args from DataprocSubmitter to CLI #915

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
15f15cf
Pass gcp args from cli to DataprocSubmitter
david-zlai Jun 30, 2025
24fcb24
Merge branch 'main' into davidhan/jun30
david-zlai Jun 30, 2025
568cd8e
add gcp args in cli
david-zlai Jun 30, 2025
1769121
actually remove
david-zlai Jun 30, 2025
c582714
fix python
david-zlai Jun 30, 2025
12abd7d
last fix
david-zlai Jun 30, 2025
210c93b
stop copying to old wheel
david-zlai Jun 30, 2025
6071430
remove gcp args
david-zlai Jun 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 14 additions & 8 deletions api/python/ai/chronon/repo/gcp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,14 +298,20 @@ def generate_dataproc_submitter_args(

elif job_type == JobType.SPARK:
main_class = "ai.chronon.spark.Driver"
return final_args.format(
user_args=user_args,
jar_uri=jar_uri,
job_type=job_type.value,
main_class=main_class,
zipline_version=self._version,
job_id=self.job_id,
) + (f" --files={gcs_file_args}" if gcs_file_args else "")
return " ".join([
final_args.format(
user_args=user_args,
jar_uri=jar_uri,
job_type=job_type.value,
main_class=main_class,
zipline_version=self._version,
job_id=self.job_id,
), "--is-gcp",
f"--gcp-project-id={GcpRunner.get_gcp_project_id()}",
f"--gcp-bigtable-instance-id={GcpRunner.get_gcp_bigtable_instance_id()}--",
f" --files={gcs_file_args}" if gcs_file_args else "",
]
)
else:
raise ValueError(f"Invalid job type: {job_type}")

Expand Down
7 changes: 1 addition & 6 deletions cloud_gcp/src/main/scala/ai/chronon/integrations/cloud_gcp/DataprocSubmitter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -440,12 +440,7 @@ object DataprocSubmitter {
case TypeSparkJob => {
val bigtableInstanceId = envMap(GcpBigtableInstanceIdEnvVar).getOrElse("")
val projectId = envMap(GcpProjectIdEnvVar).getOrElse(throw new Exception(s"GcpProjectId not set"))
val gcpArgsToPass = Array.apply(
"--is-gcp",
s"--gcp-project-id=${projectId}",
s"--gcp-bigtable-instance-id=$bigtableInstanceId"
)
Array.concat(userArgs, gcpArgsToPass)
Array.concat(userArgs)
}
case TypeFlinkJob => {
val additionalArgsToFilterOut = Set(
Expand Down
1 change: 1 addition & 0 deletions spark/src/main/scala/ai/chronon/spark/submission/JobSubmitter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,7 @@ object JobSubmitterConstants {
val AdditionalJarsUriArgKeyword = "--additional-jars"
val FilesArgKeyword = "--files"
val ConfTypeArgKeyword = "--conf-type"
val ConfPathArgKeyword = "--conf-path"
val LocalConfPathArgKeyword = "--local-conf-path"
val OriginalModeArgKeyword = "--original-mode"
val ZiplineVersionArgKeyword = "--zipline-version"
Expand Down
Loading