Update to a different metadata path in GCS everytime to avoid collisions #771
Conversation
WalkthroughThe Changes
Possibly related PRs
Suggested reviewers
Poem
Note ⚡️ AI Code Reviews for VS Code, Cursor, WindsurfCodeRabbit now has a plugin for VS Code, Cursor and Windsurf. This brings AI code reviews directly in the code editor. Each commit is reviewed immediately, finding bugs before the PR is raised. Seamless context handoff to your AI code agent ensures that you can easily incorporate review feedback. Note ⚡️ Faster reviews with cachingCodeRabbit now supports caching for code and dependencies, helping speed up reviews. This means quicker feedback, reduced wait times, and a smoother review experience overall. Cached data is encrypted and stored securely. This feature will be automatically enabled for all accounts on May 16th. To opt out, configure 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
⏰ Context from checks skipped due to timeout of 90000ms (29)
🔇 Additional comments (7)
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
|
||
| val jobId = UUID.randomUUID().toString | ||
| val jobId = JobSubmitter | ||
| .getArgValue(args, JobIdArgKeyword) |
There was a problem hiding this comment.
probably just make this required then, and fail if it's not passed.
…ons (#771) ## Summary ^^^ Currently, we'll face unexpected behavior if multiple people are working and iterating on the same GroupBy/Join and changing the conf because we'll upload to the same GCS path. This change adds the job id to the destination GCS path. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Refactor** - Streamlined job submission to upload a single metadata configuration file, simplifying the process. - Enhanced job ID management by requiring and propagating a job ID argument, improving job tracking and consistency. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…ons (#771) ## Summary ^^^ Currently, we'll face unexpected behavior if multiple people are working and iterating on the same GroupBy/Join and changing the conf because we'll upload to the same GCS path. This change adds the job id to the destination GCS path. ## Checklist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Refactor** - Streamlined job submission to upload a single metadata configuration file, simplifying the process. - Enhanced job ID management by requiring and propagating a job ID argument, improving job tracking and consistency. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…ons (#771) ## Summary ^^^ Currently, we'll face unexpected behavior if multiple people are working and iterating on the same GroupBy/Join and changing the conf because we'll upload to the same GCS path. This change adds the job id to the destination GCS path. ## Cheour clientslist - [ ] Added Unit Tests - [ ] Covered by existing CI - [x] Integration tested - [ ] Documentation update <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Refactor** - Streamlined job submission to upload a single metadata configuration file, simplifying the process. - Enhanced job ID management by requiring and propagating a job ID argument, improving job traour clientsing and consistency. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Summary
^^^
Currently, we'll face unexpected behavior if multiple people are working and iterating on the same GroupBy/Join and changing the conf because we'll upload to the same GCS path.
This change adds the job id to the destination GCS path.
Checklist
Summary by CodeRabbit