Skip to content

fix: Decouple exposed_generation time from weight_sync time#1052

Merged
parthchadha merged 1 commit intoNVIDIA-NeMo:faster-strictfifofrom
youngeunkwon0405:async-refit-timing-fix
Sep 3, 2025
Merged

fix: Decouple exposed_generation time from weight_sync time#1052
parthchadha merged 1 commit intoNVIDIA-NeMo:faster-strictfifofrom
youngeunkwon0405:async-refit-timing-fix

Conversation

@youngeunkwon0405
Copy link
Contributor

What does this PR do ?

Example output after this fix

📊 Training Results:
  • Loss: 0.1032
  • Avg Reward: 0.3486
  • Buffer Size: 64
  • Avg Trajectory Age: 1.00 steps

⏱️  Timing:
  • Total step time: 66.55s
  • exposed_generation: 32.52s (48.9%)
  • policy_training: 25.05s (37.6%)
  • policy_and_reference_logprobs: 4.78s (7.2%)
  • buffer_sampling: 1.38s (2.1%)
  • logprob_inference_prep: 1.34s (2.0%)
  • training_prep: 1.06s (1.6%)
  • data_processing: 0.23s (0.4%)
  • weight_sync: 0.13s (0.2%)

Issues

List issues that this PR closes (syntax):

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com>
@youngeunkwon0405 youngeunkwon0405 changed the title Decouple exposed_generation time from weight_sync time fix: Decouple exposed_generation time from weight_sync time Sep 3, 2025
@youngeunkwon0405 youngeunkwon0405 self-assigned this Sep 3, 2025
@parthchadha parthchadha merged commit 7ca538d into NVIDIA-NeMo:faster-strictfifo Sep 3, 2025
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants