Skip to content

feat: add gpu mem and util logging to wandb/tensorboard#37

Merged
terrykong merged 7 commits intomainfrom
more-metric-logging
Mar 24, 2025
Merged

feat: add gpu mem and util logging to wandb/tensorboard#37
terrykong merged 7 commits intomainfrom
more-metric-logging

Conversation

@terrykong
Copy link
Collaborator

@terrykong terrykong commented Mar 23, 2025

What does this PR do ?

  • Adds a simple unified way to log gpu memory and utilization to tensorboard and wandb
  • Move the logger initialization earlier in each algorithm setup() so we can log memory/util ealier (previously the logging started after vLLM workers were spun up)
  • small fix to repr since I observed it's possible to fail if torch.distributed hadn't been setup yet
  • turn sft wandb logging default to False so it matches grpo and doesn't error if someone doesn't set WANDB_API_KEY
  • vllm tests now work on a slurm ray cluster. previously they checked locally for gpus, which won't work since our head node has no GPUs. can revisit this when we look at mocking ray

Issues

List issues that this PR closes (syntax):

Closes #35

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this 

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Mar 23, 2025
@terrykong terrykong force-pushed the more-metric-logging branch from 0952265 to 9c8cf81 Compare March 23, 2025 06:08
terrykong and others added 4 commits March 23, 2025 14:18
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
Co-authored-by: Parth Chadha <pchadha@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
Signed-off-by: Terry Kong <terryk@nvidia.com>
avialable on the head node)

Signed-off-by: Terry Kong <terryk@nvidia.com>
@terrykong
Copy link
Collaborator Author

Functional tests need a followup PR to fix #41

Running them manually, will merge once they pass

@terrykong terrykong enabled auto-merge (squash) March 24, 2025 20:51
@terrykong terrykong merged commit 4d62783 into main Mar 24, 2025
11 of 13 checks passed
@terrykong terrykong deleted the more-metric-logging branch March 24, 2025 21:05
KiddoZhu pushed a commit that referenced this pull request May 6, 2025
Signed-off-by: Terry Kong <terryk@nvidia.com>
Co-authored-by: Parth Chadha <pchadha@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add GPU memory and utilization logging for ray programs

2 participants