Skip to content

fix: suppress non-axolotl logs unless it's warning or higher#2724

Merged
NanoCode012 merged 10 commits into
mainfrom
fix/logging-level
May 31, 2025
Merged

fix: suppress non-axolotl logs unless it's warning or higher#2724
NanoCode012 merged 10 commits into
mainfrom
fix/logging-level

Conversation

@NanoCode012

@NanoCode012 NanoCode012 commented May 27, 2025

Copy link
Copy Markdown
Collaborator

Description

By default, we set Axolotl to LOG all debugs which can be annoying to users and doesn't respect the LOG_LEVEL env.

Secondly, we had other loggers set to INFO, so moving it up to WARNING to not show their non-axolotl INFO logs

Motivation and Context

How has this been tested?

Screenshots (if appropriate)

Types of changes

Social Handles (Optional)

Summary by CodeRabbit

  • New Features
    • Improved logging behavior with more flexible control over log levels using environment variables.
    • Enhanced filtering to display relevant log messages based on user-configured settings.

@coderabbitai

coderabbitai Bot commented May 27, 2025

Copy link
Copy Markdown
Contributor

"""

Walkthrough

The logging configuration was updated by introducing a custom filter and logger class to control log record emission based on environment variables. The root logger's level is now set via an environment variable, and the axolotl logger's level is also configurable. The filter ensures only relevant log records are allowed, and all loggers use the new logger class.

Changes

File(s) Change Summary
src/axolotl/logging_config.py Added AxolotlOrWarnErrorFilter and AxolotlLogger classes; updated configure_logging to set logger class, apply filter globally, and use environment-driven log levels including setting ACCELERATE_LOG_LEVEL.

Sequence Diagram(s)

sequenceDiagram
    participant Env as Environment
    participant Config as configure_logging()
    participant LoggerClass as AxolotlLogger
    participant Filter as AxolotlOrWarnErrorFilter
    participant Logger

    Env->>Config: Provide LOG_LEVEL, AXOLOTL_LOG_LEVEL
    Config->>LoggerClass: Set as global logger class
    Config->>Logger: Create root and axolotl loggers
    Config->>Logger: Attach AxolotlOrWarnErrorFilter
    Logger->>Filter: Filter log records based on env levels
    Filter-->>Logger: Allow or drop record
Loading

Poem

In the warren, logs now flow with care,
With filters set by rabbits aware.
Axolotls and warnings, each with their say,
Only the worthy logs hop out to play.
With custom class and levels anew,
Our logging’s as sharp as morning dew!
🐇✨
"""


📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c738feb and 999990b.

📒 Files selected for processing (1)
  • src/axolotl/logging_config.py (3 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/axolotl/logging_config.py
⏰ Context from checks skipped due to timeout of 90000ms (8)
  • GitHub Check: PyTest from Source Dist (3.11, 2.6.0)
  • GitHub Check: PyTest from Source Dist (3.11, 2.7.0)
  • GitHub Check: PyTest (3.11, 2.7.0)
  • GitHub Check: PyTest from Source Dist (3.11, 2.5.1)
  • GitHub Check: PyTest (3.11, 2.5.1)
  • GitHub Check: PyTest (3.11, 2.6.0)
  • GitHub Check: pre-commit
  • GitHub Check: pre-commit
✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Explain this complex logic.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai explain this code block.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and explain its main purpose.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@codecov

codecov Bot commented May 27, 2025

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

📢 Thoughts on this report? Let us know!

@NanoCode012

Copy link
Copy Markdown
Collaborator Author

New:


[2025-05-27 09:55:45,424] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
The following values were not passed to `accelerate launch` and had defaults used instead:
        `--num_processes` was set to a value of `2`
                More than one GPU was found, enabling multi-GPU training.
                If this was unintended please pass in `--num_processes=1`.
        `--num_machines` was set to a value of `1`
        `--mixed_precision` was set to a value of `'no'`
        `--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
[2025-05-27 09:55:55,746] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-05-27 09:55:56,091] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
INFO 05-27 09:55:58 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-27 09:55:58 [__init__.py:239] Automatically detected platform cuda.
INFO 05-27 09:55:59 [importing.py:53] Triton module has been replaced with a placeholder.
INFO 05-27 09:55:59 [__init__.py:239] Automatically detected platform cuda.
[2025-05-27 09:56:00,622] [INFO] [axolotl.utils.schemas.config.check_eval_packing:731] [PID:1495] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
[2025-05-27 09:56:00,777] [INFO] [axolotl.normalize_config:238] [PID:1495] [RANK:0] cuda memory usage baseline: 0.000GB (+0.543GB misc)

Root logs are gone like from, root.spawn, root.register

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 0d87da2 and 49369db.

📒 Files selected for processing (2)
  • src/axolotl/logging_config.py (4 hunks)
  • src/axolotl/utils/config/__init__.py (1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (7)
  • GitHub Check: PyTest (3.11, 2.7.0)
  • GitHub Check: PyTest from Source Dist (3.11, 2.7.0)
  • GitHub Check: PyTest from Source Dist (3.11, 2.5.1)
  • GitHub Check: PyTest (3.11, 2.5.1)
  • GitHub Check: PyTest from Source Dist (3.11, 2.6.0)
  • GitHub Check: PyTest (3.11, 2.6.0)
  • GitHub Check: pre-commit
🔇 Additional comments (6)
src/axolotl/logging_config.py (5)

15-28: LGTM! Well-designed filter implementation.

The AxolotlOrWarnErrorFilter correctly implements the filtering logic to allow WARNING+ messages from any logger while restricting INFO messages to only axolotl-prefixed loggers. The logic is clear and aligns with the PR objectives to reduce log noise.


31-38: LGTM! Clean logger class implementation.

The AxolotlLogger class properly inherits from the standard Logger and automatically applies the custom filter. The implementation is minimal and focused, following the single responsibility principle.


86-86: LGTM! Root logger level change aligns with objectives.

Changing the root logger default from "INFO" to "WARNING" effectively reduces noise from non-axolotl libraries while still respecting the LOG_LEVEL environment variable. This addresses the core issue mentioned in the PR description.


90-90: LGTM! Axolotl logger level maintains flexibility.

The axolotl logger now defaults to "INFO" instead of "DEBUG" while still respecting the LOG_LEVEL environment variable. This provides a good balance between informativeness and noise reduction.


101-101: LGTM! Proper integration of custom logger class.

Setting the global logger class ensures that all subsequent logging.getLogger() calls will create AxolotlLogger instances with the filtering behavior. This is the correct way to apply the custom logger globally.

src/axolotl/utils/config/__init__.py (1)

24-24: LGTM! Dynamic logger naming improves modularity.

Changing from hardcoded "axolotl" to __name__ makes the logger name dynamic based on the module path (axolotl.utils.config). This aligns perfectly with the new AxolotlOrWarnErrorFilter which allows INFO+ messages for loggers starting with "axolotl", and it follows logging best practices for hierarchical logger naming.

Comment thread src/axolotl/logging_config.py Outdated
@NanoCode012

Copy link
Copy Markdown
Collaborator Author

Current logs:

root@8f1edf8c8831:/workspace/axolotl# axolotl train examples/llama-3/fft-8b-liger-fsdp.yaml
The following values were not passed to `accelerate launch` and had defaults used instead:
        `--num_processes` was set to a value of `2`
                More than one GPU was found, enabling multi-GPU training.
                If this was unintended please pass in `--num_processes=1`.
        `--num_machines` was set to a value of `1`
        `--mixed_precision` was set to a value of `'no'`
        `--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
[2025-05-27 11:15:36,375] [INFO] [axolotl.integrations.base.register:337] [PID:15257] [RANK:0] Attempting to load plugin: axolotl.integrations.liger.LigerPlugin
[2025-05-27 11:15:36,379] [INFO] [axolotl.integrations.base.register:340] [PID:15257] [RANK:0] Plugin loaded successfully: axolotl.integrations.liger.LigerPlugin
[2025-05-27 11:15:36,460] [INFO] [axolotl.integrations.base.register:337] [PID:15258] [RANK:1] Attempting to load plugin: axolotl.integrations.liger.LigerPlugin
[2025-05-27 11:15:36,464] [INFO] [axolotl.utils.schemas.config.check_eval_packing:731] [PID:15257] [RANK:0] explicitly setting `eval_sample_packing` to match `sample_packing`
[2025-05-27 11:15:36,465] [INFO] [axolotl.integrations.base.register:340] [PID:15258] [RANK:1] Plugin loaded successfully: axolotl.integrations.liger.LigerPlugin
[2025-05-27 11:15:36,567] [INFO] [axolotl.utils.schemas.config.check_eval_packing:731] [PID:15258] [RANK:1] explicitly setting `eval_sample_packing` to match `sample_packing`
[2025-05-27 11:15:36,600] [INFO] [axolotl.utils.config.normalize_config:238] [PID:15257] [RANK:0] cuda memory usage baseline: 0.000GB (+2.864GB misc)

     #@@ #@@      @@# @@#
    @@  @@          @@  @@           =@@#                               @@                 #@    =@@#.
    @@    #@@@@@@@@@    @@           #@#@=                              @@                 #@     .=@@
      #@@@@@@@@@@@@@@@@@            =@# @#     ##=     ##    =####=+    @@      =#####+  =#@@###.   @@
    @@@@@@@@@@/  +@@/  +@@          #@  =@=     #@=   @@   =@#+  +#@#   @@    =@#+  +#@#   #@.      @@
    @@@@@@@@@@  ##@@  ##@@         =@#   @#      =@# @#    @@      @@   @@    @@      #@   #@       @@
     @@@@@@@@@@@@@@@@@@@@          #@=+++#@=      =@@#     @@      @@   @@    @@      #@   #@       @@
                                  =@#=====@@     =@# @#    @@      @@   @@    @@      #@   #@       @@
    @@@@@@@@@@@@@@@@  @@@@        #@      #@=   #@=  +@@   #@#    =@#   @@.   =@#    =@#   #@.      @@
                                 =@#       @#  #@=     #@   =#@@@@#=    +#@@=  +#@@@@#=    .##@@+   @@
    @@@@  @@@@@@@@@@@@@@@@

[2025-05-27 11:15:36,645] [WARNING] [axolotl.cli.checks.check_user_token:44] [PID:15257] [RANK:0] Error verifying HuggingFace token. Remember to log in using `huggingface-cli login` and get your access token from https://huggingface.co/settings/tokens if you want to use gated models or datasets.
[2025-05-27 11:15:36,670] [INFO] [axolotl.utils.config.normalize_config:238] [PID:15258] [RANK:1] cuda memory usage baseline: 0.000GB (+2.864GB misc)
[2025-05-27 11:15:37,185] [INFO] [axolotl.utils.data.sft.load_tokenized_prepared_datasets:260] [PID:15257] [RANK:0] Loading prepared dataset from disk at last_run_prepared/87c278d65d5df45c9a901e3a4b556df0...
[2025-05-27 11:15:37,193] [INFO] [axolotl.utils.data.sft.load_tokenized_prepared_datasets:262] [PID:15257] [RANK:0] Prepared dataset loaded from disk...
[rank0]:[W527 11:15:37.549223591 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[rank1]:[W527 11:15:37.599762850 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 1]  using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.

@NanoCode012 NanoCode012 changed the title fix: increase log level for root loggers and axolotl's fix: suppress non-axolotl logs unless it's warning or higher May 27, 2025
Comment thread src/axolotl/logging_config.py
@NanoCode012 NanoCode012 added the hold don't merge this yet label May 28, 2025
@salmanmohammadi

Copy link
Copy Markdown
Contributor

@NanoCode012 I ended up landing the rank-0 logging PR separately if you're up for merging it in and updating this PR

@NanoCode012

NanoCode012 commented May 30, 2025

Copy link
Copy Markdown
Collaborator Author

After merged main in

root@0b67df1fb895:/workspace/axolotl# axolotl train examples/llama-3/instruct-lora-8b.yml                                                                                               
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/click/core.py:1193: UserWarning: The parameter --merge-lora is used more than once. Remove its duplicate as parameters should 
be unique.                                                                                                                                                                              
  parser = self.make_parser(ctx)                                                                                                                                                        
/root/miniconda3/envs/py3.11/lib/python3.11/site-packages/click/core.py:1186: UserWarning: The parameter --merge-lora is used more than once. Remove its duplicate as parameters should 
be unique.                                                                                                                                                                              
  self.parse_args(ctx, args)                                                                                                                                                            
The following values were not passed to `accelerate launch` and had defaults used instead:                                                                                              
        `--num_processes` was set to a value of `2`                                                                                                                                     
                More than one GPU was found, enabling multi-GPU training.                                                                                                               
                If this was unintended please pass in `--num_processes=1`.                                                                                                              
        `--num_machines` was set to a value of `1`                                                                                                                                      
        `--mixed_precision` was set to a value of `'no'`                                                                                                                                
        `--dynamo_backend` was set to a value of `'no'`                                                                                                                                 
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.                                                                                 
[2025-05-30 10:25:09,150] [INFO] [axolotl.utils.config.log_gpu_memory_usage:107] [PID:23348] [RANK:0] cuda memory usage baseline: 0.000GB (+0.543GB misc)                               
                                                                                                                                                                                        
     #@@ #@@      @@# @@#                                                                                                                                                               
    @@  @@          @@  @@           =@@#                               @@                 #@    =@@#.                                                                                  
    @@    #@@@@@@@@@    @@           #@#@=                              @@                 #@     .=@@                                                                                  
      #@@@@@@@@@@@@@@@@@            =@# @#     ##=     ##    =####=+    @@      =#####+  =#@@###.   @@                                                                                  
    @@@@@@@@@@/  +@@/  +@@          #@  =@=     #@=   @@   =@#+  +#@#   @@    =@#+  +#@#   #@.      @@                                                                                  
    @@@@@@@@@@  ##@@  ##@@         =@#   @#      =@# @#    @@      @@   @@    @@      #@   #@       @@                                                                                  
     @@@@@@@@@@@@@@@@@@@@          #@=+++#@=      =@@#     @@      @@   @@    @@      #@   #@       @@                                                                                  
                                  =@#=====@@     =@# @#    @@      @@   @@    @@      #@   #@       @@                                                                                  
    @@@@@@@@@@@@@@@@  @@@@        #@      #@=   #@=  +@@   #@#    =@#   @@.   =@#    =@#   #@.      @@                                                                                  
                                 =@#       @#  #@=     #@   =#@@@@#=    +#@@=  +#@@@@#=    .##@@+   @@                                                                                  
    @@@@  @@@@@@@@@@@@@@@@                                                                                                                                                              
                                                                                                                                                                                        
[2025-05-30 10:25:09,213] [WARNING] [axolotl.cli.checks.check_user_token:45] [PID:23348] [RANK:0] Error verifying HuggingFace token. Remember to log in using `huggingface-cli login` an
d get your access token from https://huggingface.co/settings/tokens if you want to use gated models or datasets.                                                                        
[rank1]:[W530 10:25:09.512872753 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 1]  using GPU 1 to perform barrier as devices used by this process are currently unknown. This can p
otentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.   
[2025-05-30 10:25:09,839] [INFO] [axolotl.utils.data.sft.load_tokenized_prepared_datasets:270] [PID:23348] [RANK:0] Unable to find prepared dataset in last_run_prepared/8339d42bbb38a3d
53825a15842d247a4
[2025-05-30 10:25:09,839] [INFO] [axolotl.utils.data.sft.load_tokenized_prepared_datasets:271] [PID:23348] [RANK:0] Loading raw datasets...
[2025-05-30 10:25:09,839] [WARNING] [axolotl.utils.data.sft.load_tokenized_prepared_datasets:273] [PID:23348] [RANK:0] Processing datasets during training can lead to VRAM instability.
 Please pre-process your dataset.
[2025-05-30 10:25:09,839] [INFO] [axolotl.utils.data.sft.load_tokenized_prepared_datasets:280] [PID:23348] [RANK:0] No seed provided, using default seed of 42
[2025-05-30 10:25:09,839] [WARNING] [axolotl.utils.data.sft.load_tokenized_prepared_datasets:273] [PID:23348] [RANK:0] Processing datasets during training can lead to VRAM instability.
 Please pre-process your dataset.
[2025-05-30 10:25:09,839] [INFO] [axolotl.utils.data.sft.load_tokenized_prepared_datasets:280] [PID:23348] [RANK:0] No seed provided, using default seed of 42
[2025-05-30 10:25:10,836] [INFO] [axolotl.utils.data.sft.get_dataset_wrapper:485] [PID:23348] [RANK:0] Loading dataset: fozziethebeat/alpaca_messages_2k_test with base_type: chat_templ
ate and prompt_style: None
[2025-05-30 10:25:10,881] [INFO] [axolotl.prompt_strategies.chat_template.__call__:777] [PID:23348] [RANK:0] Using chat template:
---
{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|s
tart_header_id|>' + message['role'] + '<|end_header_id|>

'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|st
art_header_id|>assistant<|end_header_id|>

' }}{% endif %}
---
[2025-05-30 10:25:11,222] [INFO] [axolotl.utils.data.utils.drop_long_seq_in_dataset:177] [PID:23348] [RANK:0] min_input_len: 22
[2025-05-30 10:25:11,222] [INFO] [axolotl.utils.data.utils.drop_long_seq_in_dataset:179] [PID:23348] [RANK:0] max_input_len: 534
[2025-05-30 10:25:11,997] [INFO] [axolotl.utils.data.sft.load_tokenized_prepared_datasets:350] [PID:23348] [RANK:0] Saving merged prepared dataset to disk... last_run_prepared/8339d42b
bb38a3d53825a15842d247a4

@NanoCode012

NanoCode012 commented May 30, 2025

Copy link
Copy Markdown
Collaborator Author

Fixed logger to respect the env LOG_LEVEL and AXOLOTL_LOG_LEVEL

Verified with LOG_LEVEL='ERROR' AXOLOTL_LOG_LEVEL='WARNING' axolotl train examples/llama-3/instruct-lora-8b.yml that no INFO are shown from axolotl

@salmanmohammadi salmanmohammadi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

         ,
     /\^/`\
    | \/   |
    | |    |
    \ \    /
     '\\//'
       ||
       ||
       ||
       ||  ,
   |\  ||  |\
   | | ||  | |
   | | || / /
    \ \||/ /
     `\\//`
    ^^^^^^^^

@NanoCode012 NanoCode012 merged commit d5d0dc5 into main May 31, 2025
16 checks passed
@NanoCode012 NanoCode012 deleted the fix/logging-level branch May 31, 2025 05:13
@coderabbitai coderabbitai Bot mentioned this pull request Jun 18, 2025
@coderabbitai coderabbitai Bot mentioned this pull request Sep 13, 2025
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hold don't merge this yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants