Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve Vertex Scheduler Character Limit Issue for Long Pipeline Names #2262

Open
1 task
strickvl opened this issue Jan 10, 2024 · 9 comments
Open
1 task
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@strickvl
Copy link
Contributor

Open Source Contributors Welcomed!

Please comment below if you would like to work on this issue!

Contact Details [Optional]

[email protected]

What happened?

A user encountered an issue where the Vertex scheduler in ZenML does not accept pipeline names longer than 64 characters. This limitation requires users to shorten their pipeline names to successfully schedule their pipelines on Vertex AI, which can be restrictive and inconvenient.

Task Description

Investigate and address the issue with the Vertex scheduler's character limit for pipeline names in ZenML. The goal is to either increase the character limit or implement a mechanism that automatically handles longer names without user intervention.

Expected Outcome

Users should be able to schedule pipelines on Vertex AI without the constraint of a 64-character limit on pipeline names.
ZenML should provide a seamless experience, either by accommodating longer names or by intelligently managing them within the existing Vertex scheduler limitations.
This fix will enhance user convenience and flexibility in naming pipelines.

Steps to Implement

  • Reproduce the issue to understand the exact limitations and behavior of the Vertex scheduler with respect to pipeline names.
  • Explore solutions such as increasing the character limit or automatically abbreviating/transforming long pipeline names while preserving their uniqueness and recognizability.
  • Implement the chosen solution in a way that is transparent and intuitive for the user.
  • Test the implementation with various pipeline name lengths to ensure compatibility and functionality.
  • Update documentation to inform users about the handling of pipeline names in Vertex scheduler and any best practices or limitations.

Additional Context

This issue is crucial for users who require descriptive and potentially lengthy names for their pipelines, especially in complex or large-scale MLOps environments.

Code of Conduct

  • I agree to follow this project's Code of Conduct
@strickvl strickvl added bug Something isn't working good first issue Good for newcomers labels Jan 10, 2024
@strickvl
Copy link
Contributor Author

@christianversloot if I'm not completely mistaken, we had a conversation about this in Slack? Unfortunately the slack history hasn't been retained.

@christianversloot
Copy link
Contributor

Correct, but that was for SageMaker orchestrator. There, I believe we limit name length to circumvent this error. It could perhaps be inspiration for a fix here :)

@strickvl
Copy link
Contributor Author

@christianversloot ah yes! #1505 was the PR.

@ashutosh887
Copy link

@strickvl @christianversloot
I would like to work on this... Please assign!

@Merthoshan
Copy link

has the bug been solved? if not i would like to solve it

@strickvl
Copy link
Contributor Author

The bug has not been solved yet @Merthoshan. @ashutosh887 are you working on it, if not, can let @Merthoshan take a shot.

@ashutosh887
Copy link

Where can we discuss if we're facing some issues understanding the bug? @strickvl
I've worked it

@strickvl
Copy link
Contributor Author

Here is the place.

@Prabhat-Thapa45
Copy link

is there a simpler way to replicate this I could take this but It's going to be difficult for me to replicate this if this needs me to setup GCP which requires some cost. @strickvl

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

5 participants