Fix configuration of turning test warnings into failures

[jx2lee]

resolves #7761

Problem

dbt --warn-error-option '{"include":"all"}' test -s single_line
and
dbt --warn-error test -s single_line have different results.

╰─ dbt --warn-error-options '{"include":"all"}' test -s single_line
15:04:07  Running with dbt=1.8.0-a1
15:04:07  Registered adapter: postgres=1.8.0-a1
15:04:07  Found 10 models, 19 tests, 6 seeds, 6 sources, 0 exposures, 15 metrics, 627 macros, 0 groups, 5 semantic models
15:04:07  
15:04:08  Concurrency: 1 threads (target='dev')
15:04:08  
15:04:08  1 of 1 START test single_line .................................................. [RUN]
15:04:08  1 of 1 WARN 1 single_line ...................................................... [WARN 1 in 0.07s]
15:04:08  
15:04:08  Finished running 1 test in 0 hours 0 minutes and 0.87 seconds (0.87s).
15:04:08  
15:04:08  Completed with 1 warning:
15:04:08  
15:04:08  Warning in test single_line (tests/single_line.sql)
15:04:08  Got 1 result, configured to warn if >0
15:04:08  
15:04:08    compiled Code at target/compiled/jaffle_shop/tests/single_line.sql
15:04:08  
15:04:08  Done. PASS=0 WARN=1 ERROR=0 SKIP=0 TOTAL=1

╭─ ~/workspace/opensource/jaffle-sl-template  on main !2 ?5 ▓▒░····························································································································░▒▓ ✔  took 6s  3.9.16 Py  at 00:04:09 
╰─ dbt --warn-error test -s single_line                            
15:04:18  Running with dbt=1.8.0-a1
15:04:18  Registered adapter: postgres=1.8.0-a1
15:04:18  Found 10 models, 19 tests, 6 seeds, 6 sources, 0 exposures, 15 metrics, 627 macros, 0 groups, 5 semantic models
15:04:18  
15:04:19  Concurrency: 1 threads (target='dev')
15:04:19  
15:04:19  1 of 1 START test single_line .................................................. [RUN]
15:04:19  1 of 1 FAIL 1 single_line ...................................................... [FAIL 1 in 0.07s]
15:04:19  
15:04:19  Finished running 1 test in 0 hours 0 minutes and 0.87 seconds (0.87s).
15:04:19  
15:04:19  Completed with 1 error and 0 warnings:
15:04:19  
15:04:19  Failure in test single_line (tests/single_line.sql)
15:04:19    Got 1 result, configured to fail if >0
15:04:19  
15:04:19    compiled Code at target/compiled/jaffle_shop/tests/single_line.sql
15:04:19  
15:04:19  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Solution

Modfiy suggested line, and then have same results

╰─ dbt --warn-error-options '{"include":"all"}' test -s single_line
15:06:56  Running with dbt=1.8.0-a1
15:06:56  Registered adapter: postgres=1.8.0-a1
15:06:56  Found 10 models, 19 tests, 6 seeds, 6 sources, 0 exposures, 15 metrics, 627 macros, 0 groups, 5 semantic models
15:06:56  
15:06:57  Concurrency: 1 threads (target='dev')
15:06:57  
15:06:57  1 of 1 START test single_line .................................................. [RUN]
15:06:57  1 of 1 FAIL 1 single_line ...................................................... [FAIL 1 in 0.07s]
15:06:57  
15:06:57  Finished running 1 test in 0 hours 0 minutes and 0.88 seconds (0.88s).
15:06:57  
15:06:57  Completed with 1 error and 0 warnings:
15:06:57  
15:06:57  Failure in test single_line (tests/single_line.sql)
15:06:57    Got 1 result, configured to fail if >0
15:06:57  
15:06:57    compiled Code at target/compiled/jaffle_shop/tests/single_line.sql
15:06:57  
15:06:57  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

╭─ ~/workspace/opensource/jaffle-sl-template  on main !2 ?5 ▓▒░··························································································································░▒▓ 1 ✘  took 6s  3.9.16 Py  at 00:06:59 
╰─ dbt --warn-error test -s single_line                            
15:07:06  Running with dbt=1.8.0-a1
15:07:07  Registered adapter: postgres=1.8.0-a1
15:07:07  Found 10 models, 19 tests, 6 seeds, 6 sources, 0 exposures, 15 metrics, 627 macros, 0 groups, 5 semantic models
15:07:07  
15:07:08  Concurrency: 1 threads (target='dev')
15:07:08  
15:07:08  1 of 1 START test single_line .................................................. [RUN]
15:07:08  1 of 1 FAIL 1 single_line ...................................................... [FAIL 1 in 0.07s]
15:07:08  
15:07:08  Finished running 1 test in 0 hours 0 minutes and 0.85 seconds (0.85s).
15:07:08  
15:07:08  Completed with 1 error and 0 warnings:
15:07:08  
15:07:08  Failure in test single_line (tests/single_line.sql)
15:07:08    Got 1 result, configured to fail if >0
15:07:08  
15:07:08    compiled Code at target/compiled/jaffle_shop/tests/single_line.sql
15:07:08  
15:07:08  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX
• This PR includes type annotations for new and modified functions

[jx2lee]

@MichelleArk
could you please let me know when you might be able to review it? Thank you in advance for your time and consideration.

[MichelleArk]

Let's add a test to these changes!

There isn't really a great existing file for these changes because we're planning on refactoring the existing tests/functional/adapter/test_singular_test.py soon (#9513)

I'd recommend creating a functional test as follows:

1. Create a new file under tests/functional/test_singular_tests.py
2. Set up test class with a tests fixture. Something like:

single_test_sql = """
{{ config(warn_if = '>0', error_if ="> 10") }}

select 1 as issue
"""

class TestSingularTestWarnError:
   @pytest.fixture(scope="class")
   def tests(self):
       return {"single_test.sql": single_test_sql"}

3. Implementing test methods that run dbt test with --warn-error-options as in your PR description, and assert the expected error is raised (similar example)

You'll need postgres setup locally for dbt-core to run the functional tests, there's some documentation here

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.33%. Comparing base (f683e36) to head (269fda0).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9347      +/-   ##
==========================================
- Coverage   88.07%   78.33%   -9.74%     
==========================================
  Files         178      178              
  Lines       22458    22459       +1     
==========================================
- Hits        19779    17594    -2185     
- Misses       2679     4865    +2186     

Flag	Coverage Δ
integration	73.82% <100.00%> (-11.62%)	⬇️
unit	61.78% <50.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[MichelleArk]

Looks great! Thank you for adding the testing. Merging this in now, and it should be available for the 1.8rc1 on May 2: #9784

[jx2lee]

@MichelleArk thank you for review! 🥹