[CT-2627] [Bug] --warn-error-options doesn't support configuration of turning test warnings into failures

[joellabes]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I wrote about using the warn-error-options functionality on LI, and @epapineau asked to work with me to implement it in the internal analytics project.

After we got it working, we realised we might have to exclude tests with a warn severity as well as "nothing to do", and went poking around in types.py where we found LogTestResult (we assume that's the right one!)

Because it extends DynamicLevel instead of WarnLevel, nothing happened when we included it in the dictionary of include/excludes. However, dbt build --warn-error would create an error from a warn-level test (or freshness check etc).

dbt test --warn-error-options {"include": "all"} and dbt test --warn-error should be identical, right?

This happens to be convenient bug for us, because we didn't want to include them anyway and were trying to add them to the exclude-list, but a bug nonetheless.

Expected Behavior

dbt test --warn-error-options {"include": "all"} and dbt test --warn-error to be identical

Steps To Reproduce

Make a singular test:

{{ config(warn_if = '>0', error_if ="> 10") }}

select 1 as issue

Run dbt --warn-error-options '{"include":"all"}' test -s single_line
and dbt --warn-error test -s single_line

Relevant log output

(dbt-prod) joel@Joel-Labes joel-sandbox % dbt --warn-error-options '{"include":"all"}' test -s single_line
...
02:00:19  Finished running 1 test in 0 hours 0 minutes and 5.12 seconds (5.12s).
02:00:19  
02:00:19  Completed with 1 warning:
02:00:19  
02:00:19  Warning in test single_line (tests/single_line.sql)
02:00:19  Got 1 result, configured to warn if >0
02:00:19  
02:00:19  Done. PASS=0 WARN=1 ERROR=0 SKIP=0 TOTAL=1


(dbt-prod) joel@Joel-Labes joel-sandbox % dbt --warn-error test -s single_line 
...
02:00:40  Finished running 1 test in 0 hours 0 minutes and 5.06 seconds (5.06s).
02:00:40  
02:00:40  Completed with 1 error and 0 warnings:
02:00:40  
02:00:40  Failure in test single_line (tests/single_line.sql)
02:00:40    Got 1 result, configured to fail if >0
02:00:40  
02:00:40  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Environment

- OS: Mac
- Python: 3.8.10
- dbt: 1.5.0

Which database adapter are you using with dbt?

No response

Additional Context

No response

[dbeatty10]

Thanks for discovering and raising this @joellabes and @epapineau 🤩

I was able to reproduce what you reported ✅

Root cause

I suspect it might be the difference between the logic in the test task and warn_or_error.

Potential solution

Not the most elegant solution (and completely untested!), but updating this:

dbt-core/core/dbt/task/test.py

Line 160 in 2d23782

if get_flags().WARN_ERROR:

to this might do the trick 🤷:

        if get_flags().WARN_ERROR or get_flags().WARN_ERROR_OPTIONS.includes(type(event).__name__):

[jtcohen6]

Hmm! Failures in tests and source freshness checks are a bit different from our other warnings-to-exceptions, because dbt handles "error" results gracefully, rather than interrupting its entire process.

As a way for users to configure --warn-error-options, LogTestResult does seem like the most sensible option here. To Doug's point, we might want to update line 160 to something like:

        if get_flags().WARN_ERROR or get_flags().WARN_ERROR_OPTIONS.includes("LogTestResult"):

That event has a "dynamic" log level, which is set conditionally based on the status returned by the TestStatus, using the status_to_level method. That's another (alternative? additional?) place where we should be elevating the log level from warn to error. But I'm not sure it makes sense to actually raise an exception there.

dbt-core/core/dbt/events/types.py

Lines 1551 to 1563 in 6f51de4

	@classmethod
	def status_to_level(cls, status):
	# The statuses come from TestStatus
	level_lookup = {
	"fail": EventLevel.ERROR,
	"pass": EventLevel.INFO,
	"warn": EventLevel.WARN,
	"error": EventLevel.ERROR,
	}
	if status in level_lookup:
	return level_lookup[status]
	else:
	return EventLevel.INFO

[emmyoop]

Need to determine if we should fix this or document why we won't fix it.

[dbeatty10]

If I'm interpreting @jtcohen6's message correctly, the answer is "yes, we should fix this". And the specific way we may do so is:

        if get_flags().WARN_ERROR or get_flags().WARN_ERROR_OPTIONS.includes("LogTestResult"):

That being said, this probably isn't a very high priority (since it didn't actually affect the reporters negatively, they merely noticed and reported it).

Is that a fair summary @jtcohen6 ?

[joellabes]

Regardless of the answer ☝️, just want to also flag for the sake of completeness that source freshness tests have the same dynamic capabilities, so I expect are covered by the same issue where --warn-error doesn't have the same result as --warn-error-options {"include": "all"}

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[dbeatty10]

We are not prioritizing this ourselves right now, but labeling as a good_first_issue in case a community member wants to contribute to this.

See implementation hint here.

[jx2lee]

@dbeatty10 could i take this issue?
I've been looking for issue labeled good_first_issue, it's so much fun. 🤩

[dbeatty10]

That would be awesome @jx2lee 👍