Allowed and default values for LOG_FORMAT and LOG_FORMAT_FILE

[dbeatty10]

Preview

What are you changing in this pull request and why?

resolves #5577

Allowed and default values are in the source code here.

TLDR

The LOG_FORMAT and LOG_FORMAT_FILE global config flags in dbt control the format of log outputs. There are three formats:

1. text

   • Outputs unstructured text with minimal metadata and second-precision timestamps in UTC.
   • Default format for console output.

   Example:

   23:30:16  Running with dbt=1.8.0
   23:30:17  Registered adapter: postgres=1.8.0
   

2. debug

   • Outputs unstructured text with detailed metadata including log level, thread ID, and microsecond-precision timestamps in the system's local time zone.
   • Default format for the logs/dbt.log file.

   Example:

   ============================== 16:12:08.555032 | 9089bafa-4010-4f38-9b42-564ec9106e07 ==============================
   16:12:08.555032 [info ] [MainThread]: Running with dbt=1.8.0
   16:12:08.751069 [info ] [MainThread]: Registered adapter: postgres=1.8.0
   

3. json

   • Outputs structured JSON-L format logs.

   Example:

   {"data": {"log_version": 3, "version": "=1.8.0"}, "info": {"category": "", "code": "A001", "extra": {}, "invocation_id": "82131fa0-d2b4-4a77-9436-019834e22746", "level": "info", "msg": "Running with dbt=1.8.0", "name": "MainReportVersion", "pid": 7875, "thread": "MainThread", "ts": "2024-05-29T23:32:54.993336Z"}}
   {"data": {"adapter_name": "postgres", "adapter_version": "=1.8.0"}, "info": {"category": "", "code": "E034", "extra": {}, "invocation_id": "82131fa0-d2b4-4a77-9436-019834e22746", "level": "info", "msg": "Registered adapter: postgres=1.8.0", "name": "AdapterRegistered", "pid": 7875, "thread": "MainThread", "ts": "2024-05-29T23:32:56.437986Z"}}
   

Note: The debug log format should not be confused with the debug log level. The log level determines the number of log messages, while the log format determines the detail in each log message.

More info

There are two different sub-classes of _Logger:

1. _TextLogger (default)
2. _JsonLogger

_TextLogger has two different line formats:

1. DebugText (default)
   • Same as PlainText below except it includes a header at the beginning of each invocation (which has a timestamp and the invocation ID) and each message includes the log level of the message, thread ID, and the timestamp has a microsecond precision using the Python system time zone
2. PlainText
   • Same as DebugText above except it does not include a header at the beginning of each invocation and each message does not include the log level of the message or the thread ID. The timestamp for each message has a second precision represented in UTC.

_JsonLogger has a single line format:

1. Json

Those (3) line formats correspond to the options for the LOG_FORMAT and LOG_FORMAT_FILE global config flags:

1. debug -> DebugText line format (default for the logs/dbt.log file) outputs unstructured text with more metadata at the beginning of each log line and upon each dbt invocation
2. text -> PlainText line format (default for console output) outputs unstructured text with minimal metadata at the beginning of each log line and none upon each dbt invocation
3. json -> Json line format outputs structured JSON-L

Warning

The debug log_level can easily be confused with the debug log_format since they have the same name. But the log_level and log_format are independent configurations whose options can be mixed'n'matched.

Think of log levels as affecting the total number of log messages and the log format as affecting the length of each log message: the debug log level includes the maximal number of log messages whereas the debug log format includes more detailed information per log message

e.g. The debug log level filters out the least number of potential log messages -- none! And the debug log format has longer lines because it includes the most detailed debugging metadata per log message.

To avoid confusion, we would have been better off giving the debug log format a unique name instead (like detailed). A better name for the text log format would have been basic, simple, or brief.

Note

I don't know the precision reasons why the PlainText line format is in UTC with low-resolution while the DebugText with a local time zone and high resolution

Checklist

• Review the Content style guide so my content adheres to these guidelines.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs-getdbt-com	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 30, 2024 3:03pm

[iamtodor]

@dbeatty10 thank you!

[iamtodor]

LGTM! 🔥

[matthewshaver]

Thanks @dbeatty10