Skip to content

Conversation

NathanHB
Copy link
Member

No description provided.

@NathanHB NathanHB requested a review from Copilot August 20, 2025 11:50
Copilot

This comment was marked as outdated.

@HuggingFaceDocBuilderDev
Copy link
Collaborator

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Member

@clefourrier clefourrier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to update

  • the docs
  • the pyproject.toml
  • the typer kwargs to explain that it's either trackio or wandb

self.should_push_to_hub = push_to_hub
self.should_save_details = save_details
self.wandb = wandb
self.wandb = use_wandb
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

attribute could also be called use_wandb instead

@NathanHB NathanHB requested a review from Copilot August 20, 2025 12:33
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for Trackio as an alternative to WandB for experiment tracking, with automatic fallback to WandB when Trackio is not available. The change enhances the logging capabilities by providing users with multiple tracking backend options.

  • Adds optional Trackio dependency and import logic with WandB fallback
  • Updates CLI help text and documentation to reflect Trackio support
  • Modifies parameter names from wandb to use_wandb for clarity

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/lighteval/main_vllm.py Updates CLI help text and parameter name for tracking backend
src/lighteval/main_sglang.py Updates CLI help text and parameter name for tracking backend
src/lighteval/main_endpoint.py Updates CLI help text and parameter name for tracking backend across multiple functions
src/lighteval/main_accelerate.py Updates CLI help text and parameter name for tracking backend
src/lighteval/logging/evaluation_tracker.py Implements Trackio import logic with WandB fallback and updates parameter handling
pyproject.toml Adds optional Trackio dependency
docs/source/saving-and-reading-results.mdx Updates documentation to describe Trackio usage and configuration

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@NathanHB NathanHB requested a review from clefourrier August 20, 2025 14:07
@clefourrier clefourrier merged commit 0da2477 into main Aug 20, 2025
5 checks passed
NathanHB added a commit that referenced this pull request Sep 19, 2025
* use trackio when available

* Update src/lighteval/logging/evaluation_tracker.py

Co-authored-by: Copilot <[email protected]>

* update doc

* change attr name

* fix trackio typo

* Update docs/source/saving-and-reading-results.mdx

Co-authored-by: Copilot <[email protected]>

---------

Co-authored-by: Copilot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants