-
Notifications
You must be signed in to change notification settings - Fork 359
Trackio when available #930
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Co-authored-by: Copilot <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You need to update
- the docs
- the pyproject.toml
- the typer kwargs to explain that it's either trackio or wandb
self.should_push_to_hub = push_to_hub | ||
self.should_save_details = save_details | ||
self.wandb = wandb | ||
self.wandb = use_wandb |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
attribute could also be called use_wandb
instead
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds support for Trackio as an alternative to WandB for experiment tracking, with automatic fallback to WandB when Trackio is not available. The change enhances the logging capabilities by providing users with multiple tracking backend options.
- Adds optional Trackio dependency and import logic with WandB fallback
- Updates CLI help text and documentation to reflect Trackio support
- Modifies parameter names from
wandb
touse_wandb
for clarity
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
File | Description |
---|---|
src/lighteval/main_vllm.py | Updates CLI help text and parameter name for tracking backend |
src/lighteval/main_sglang.py | Updates CLI help text and parameter name for tracking backend |
src/lighteval/main_endpoint.py | Updates CLI help text and parameter name for tracking backend across multiple functions |
src/lighteval/main_accelerate.py | Updates CLI help text and parameter name for tracking backend |
src/lighteval/logging/evaluation_tracker.py | Implements Trackio import logic with WandB fallback and updates parameter handling |
pyproject.toml | Adds optional Trackio dependency |
docs/source/saving-and-reading-results.mdx | Updates documentation to describe Trackio usage and configuration |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Co-authored-by: Copilot <[email protected]>
* use trackio when available * Update src/lighteval/logging/evaluation_tracker.py Co-authored-by: Copilot <[email protected]> * update doc * change attr name * fix trackio typo * Update docs/source/saving-and-reading-results.mdx Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]>
No description provided.