More consistency for global CLI flags

[jtcohen6]

Describe the feature

Every "global" CLI flag should:

• Precede the dbt subcommand (dbt --global-flag run)
• Be truly global, i.e. usable with all subcommands
• Have a corresponding environment variable and profile config
• Relate to logistical details of development or deployment: log format + coloring, how to handle warnings, whether/where to write artifacts, etc. Anything project-related should be defined within project code; these are things that wouldn't make sense to version control alongside the rest.

If set in multiple ways, the hierarchy should always be: flag > env var > profile config.

Each flag should have a corresponding and addressable Jinja value, perhaps under flags. While e.g. {{env_var(DBT_NO_WRITE_JSON)}} should only return true if the specific environment variable is on, I think that the value of {{flags.NO_WRITE_JSON}} should be true whether set by CLI flag, env var, or profile config.

Current global flags/configs

I'm not talking about standard CLI fare, such as --version and --help.

• -r, --record-timing-info
• -d, --debug
• --log-format
• --no-write-json
• -S, --strict: I believe we should retire this one, since it no longer serves a functional purpose and users think it does something different (strict field validation for schema.yml #1570, Clarify --strict flag usage docs.getdbt.com#337)
• --warn-error
• --partial-parse, config.partial_parse
• --use-colors, --no-use-colors, config.use_colors: needs documentation update
• config.printer_width
• config.anonymous_usage_stats (see also: Support console env var DO_NOT_TRACK #3540)
• --use-experimental-parser

Proposed

I do think that some flags should be promoted to "global" status:

• --no-version-check: Right now, it works for some subcommands but not others, and I personally find it pesky to remember which ones are which. It should work for every subcommand and precede the subcommand.
• -x, --fail-fast: Supported for run and test, why not all the rest?
• --profiles-dir, DBT_PROFILES_DIR: Already global, it should also be supported when preceding the subcommand.

Documentation

We should then create a handy table of environment variables available to dbt's invocation context. That would include:

• All global flags
• Subcommand-specific flags / env vars, such as DBT_DEFER_TO_STATE + DBT_ARTIFACT_STATE_PATH

We might also mention there that:

• Custom environment variables prefixed with DBT_ENV_CUSTOM_ENV_ will appear in the metadata.env dict of dbt JSON artifacts
• Custom environment variables prefixed with DBT_ENV_SECRET_ will be scrubbed from logs
• All environment variables are addressable using {{env_var()}}

Describe alternatives you've considered

• Leaving these incomplete and inconsistent

Who will this benefit?

• More complex deployments of dbt
• Readers of more consistent documentation

[jtcohen6]

See also:

• Global CLI commands should work both for commands and for subcommands, i.e. dbt --debug run == dbt run --debug (Make flags to the dbt command also be valid flags to all subcommands #1618)
• Set log_format: text | json from profiles.yml (Allow log format to be configured from profiles.yml #1961)

[leahwicz]

• Some global flags can be set as env vars -> this can be a different effort
• "Reconsider our runtime configuration setup" issue (we can split this out into another v1.0 issue, estimation 8)