Suppress noisy external-library log lines on non-rank-0 processes - #5590
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/claude review |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| logger.info(f'Setting logging level to {logging_level}') | ||
| logging.getLogger().setLevel(logging_level) | ||
|
|
||
| if not is_rank0(): |
There was a problem hiding this comment.
can we just use this instead: https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/training/utils/log_utils.py#L66-L70
was going to remove the version in this file (initialize.py) in a different PR that hasn't been merged yet. but it seems that it would do the same thing?
There was a problem hiding this comment.
Where should I call the version in log_utils?
There was a problem hiding this comment.
does it not work if called in the same place this one is called? ie log_utils.setup_logging should just replace initialize.setup_logging?
6d6f616 to
4a188c1
Compare
Add DeprecationWarning to the warning filter list on non-rank-0. Override
warnings.showwarning as a fallback for libraries like CUTLASS DSL that use
catch_warnings() with simplefilter("always") to bypass filters. Suppress
noisy Python loggers (GroupedGemm variants, absl) on non-rank-0 in
setup_logging().
Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>
4a188c1 to
7027e38
Compare
|
/ok to test 7027e38 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/31615942188 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA/Megatron-LM/actions/runs/31624303817 |
5 participants
Summary
DeprecationWarningto the warning filter list on non-rank-0 processes inpretrain_gpt.pyandpretrain_hybrid.py. Overridewarnings.showwarningas a fallback for libraries like CUTLASS DSL that usecatch_warnings()withsimplefilter("always")to bypass filters.GroupedGemmQuantSm100,GroupedGemmDsreluSm100,GroupedGemmSreluSm100,GroupedGemmWgradSm100,absl) on non-rank-0 insetup_logging().device_idtotorch.distributed.init_process_group()to eliminate the NCCL "Guessing device ID" C++ warning on every rank.Together these reduce log output from ~160K lines to ~3K lines on a 128-GPU job.
Test plan
🤖 Generated with Claude Code