Conversation
Signed-off-by: Wei Du <wedu@nvidia.com>
Collaborator
Author
Kipok
reviewed
Aug 13, 2025
Collaborator
Kipok
left a comment
Kipok
left a comment
There was a problem hiding this comment.
please also do the same for grpo script. And ideally reuse the logic, so that the nsight_cmd is defined in one function and called in both scripts
nemo_skills/pipeline/nemo_rl/sft.py
Outdated
| wandb_project: str = typer.Option("nemo-skills", help="Weights & Biases project name"), | ||
| wandb_group: str = typer.Option(None, help="Weights & Biases group name."), | ||
| disable_wandb: bool = typer.Option(False, help="Disable wandb logging"), | ||
| nsys_profile: bool = typer.Option(False, help="Profile GPU with Nsight Systems for selected Ray workers via env var matching."), |
Collaborator
There was a problem hiding this comment.
we don't need this, let's just have one nsys_step_range and if it's specified, we do profiling with that range
nemo_skills/pipeline/nemo_rl/sft.py
Outdated
|
|
||
| if nsys_profile: | ||
| if "RAY_LOG_SYNC_FREQUENCY" not in env_variables: | ||
| raise typer.BadParameter( |
Collaborator
There was a problem hiding this comment.
don't raise error, just add it directly to cluster config's env vars with a logic like this https://github.com/NVIDIA/NeMo-Skills/blob/main/nemo_skills/pipeline/utils/exp.py#L452
Signed-off-by: Wei Du <wedu@nvidia.com>
Collaborator
Author
|
@Kipok I revised according to the comments |
Kipok
reviewed
Aug 13, 2025
nemo_skills/pipeline/nemo_rl/grpo.py
Outdated
| disable_wandb: bool = typer.Option(False, help="Disable wandb logging"), | ||
| profile_step_range: str = typer.Option( | ||
| None, | ||
| help="Environment variable to control which training steps the profiler captures. " |
Collaborator
There was a problem hiding this comment.
Suggested change
| help="Environment variable to control which training steps the profiler captures. " | |
| help="Controls which training steps the nsys profiler captures. " |
nemo_skills/pipeline/nemo_rl/grpo.py
Outdated
| with_ray=True, | ||
| installation_command=installation_command, | ||
| ) | ||
| with temporary_env_update(cluster_config, {"RAY_LOG_SYNC_FREQUENCY": 20}): |
Collaborator
There was a problem hiding this comment.
we should only do this if profile is specified. You can use {} as a second argument in other cases
| wandb_project: str = typer.Option("nemo-skills", help="Weights & Biases project name"), | ||
| wandb_group: str = typer.Option(None, help="Weights & Biases group name."), | ||
| disable_wandb: bool = typer.Option(False, help="Disable wandb logging"), | ||
| profile_step_range: str = typer.Option( |
Collaborator
There was a problem hiding this comment.
same comments here as in the grpo file
SeanNaren
pushed a commit
to SeanNaren/NeMo-Skills
that referenced
this pull request
Aug 15, 2025
Signed-off-by: Wei Du <wedu@nvidia.com> Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
shtoshni
pushed a commit
that referenced
this pull request
Aug 15, 2025
Signed-off-by: Wei Du <wedu@nvidia.com> Signed-off-by: Shubham Toshniwal <stoshniwal@nvidia.com>
SeanNaren
pushed a commit
to SeanNaren/NeMo-Skills
that referenced
this pull request
Aug 18, 2025
Signed-off-by: Wei Du <wedu@nvidia.com> Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
SeanNaren
pushed a commit
to SeanNaren/NeMo-Skills
that referenced
this pull request
Aug 18, 2025
Signed-off-by: Wei Du <wedu@nvidia.com> Signed-off-by: SeanNaren <snarenthiran@nvidia.com>
wasiahmad
pushed a commit
that referenced
this pull request
Oct 1, 2025
Signed-off-by: Wei Du <wedu@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Added support for nsys_profile