Allow instances of generic data tests to be documented

[clrcrl]

Describe the feature

Currently, you can document pretty much everything but a data test - if you try, it fails on a parsing error. This ticket is for generic data tests. A separate ticket addresses singular data tests: #9005

Acceptance criteria

• You can add a description to a generic test
• You can use a docs block to capture your test description
• That description shows up in the docs site

Examples

version: 2

models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - unique
            description: The order_id is unique for every row in the orders model
          - not_null
            description: '{{ doc("not_null_test") }}'

Concerns

• adding documentation to tests will slow down parsing (full parse), particularly for really common tests like unique and not_null
• We have special processing to speed up unique and not_null test parsing. Would that have to change?
• Ensure the solution is performant

[RobMcZag]

I would say that for me a tests: label would imply all kind of tests, including schema tests, but looking at the current naming it looks like the most reasonable.
Current names related to tests:

• data tests are in the tests folder, not data-tests
• in dbt_projectwe specify a folder for test-paths, not data-test-paths

[gshank]

I know we've discussed putting schema tests in the tests directory instead of the macros directory, but I don't recall what the resolution of that was. Is that on the roadmap somewhere?

[jtcohen6]

@gshank @kwigley I'd like to use our discussion of this issue as a jumping-off point to discuss other test-related work we scoped out earlier this year, and to determine whether / which of these should go on the v1.0 list as well.

The ones that come to my mind are:

• Rename "schema" + "data" tests #3234
• Configurable description in yaml config for generic tests #3249
• define generic tests in configured tests-paths directory #3250
• Improve test FQNs #3259
• Better node identification #3348

All of those would represent a breaking change to some degree. There's ways we can offer backwards compatibility; easier for some, harder for others.

[leahwicz]

Work:

• New parser for tests
• Work will need to be done in partial parsing
• Involve doc blocks (perf hit)

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[NicolasPA]

I think the ticket activity doesn't reflect the demand for this feature, there has been at least 6 users asking about in on Slack this past year: https://getdbt.slack.com/archives/CBSQTAPLG/p1631304882432300

It might be because the wording used in the ticket has changed:
schema test → generic test
data test → singular test

I think custom tests are code as much as models and macros are, so documenting them is equally important.

@jtcohen6: I think a new tests: key in .yml files is exactly how we'd want to do this. Singular tests wouldn't be too hard. Generic tests would be trickier, since the good version (IMO) allows for templated descriptions that includes the values of {{ model }}, {{ column_name }}, and potentially other arguments.

IMO, having the static description as is in written in the YAML without replacing the values would already be nice and useful, the more complex version could be kept for later.

[jeff-skoldberg-ct]

I came to +1 this post. Looking forward to support for documenting Singular Tests. Thanks!

[gnilrets]

+1 from me too.

[ChristianP89]

+1 from my side as well. It's quite important to be able to document what a test does.

[nbkrd1]

+1 from me as well.

[nbllh1]

+1 from me too.

[GeorgeHC97]

+1

[victorrgez]

+1

[victorrgez]

+1

This should be a bug, not a feature request imo. As others have mentioned, dbt docs supports descriptions for tests, but it is not yet implemented in dbt core. When I write a custom generic test, the user sees the following under "Description"

This test is not currently documented

This phrasing implies the ability to document the test exists.

@jomccr Please note that generic tests can be documented as a Macro (inside macros: block in yaml). It is singular tests that cannot be documented as far as I know!!

You can create a macros.yml file inside macros/ which refer both to macros and generic tests and it works, does not matter if your generic test is inside macros or tests/generic folder.

As an example I'm providing the file tests/generic/test_not_empty_string.sql whose content is [1] and the yaml code that documents it [2], which is placed inside macros/macros.yml. I'm attaching the screenshot of the documentation as well:

[1]:

{% test not_empty_string(model, column_name) %}

/*
    Complementary test to default `not_null` test as it checks that there is not an empty string
*/

with only_relevant_column as (
    SELECT
        {{ column_name }}
    FROM
        {{ model }}
)

SELECT
    *
FROM
    only_relevant_column
WHERE
    {{ column_name }} = ''

{% endtest %}

[2]:

  - name: test_not_empty_string
    description: Complementary test to default `not_null` test as it checks that there is not an empty string. It only accepts columns of type string. It is deprecated as this functionality has been included in the custom implementation of `not_null`
    arguments:
      - name: model
        type: string
        description: Model Name
      - name: column_name
        type: string
        description: Column name that should not be an empty string

[ybressler]

+1
This should be a bug, not a feature request imo. As others have mentioned, dbt docs supports descriptions for tests, but it is not yet implemented in dbt core. When I write a custom generic test, the user sees the following under "Description"

This test is not currently documented

This phrasing implies the ability to document the test exists.

@jomccr Please note that generic tests can be documented as a Macro (inside macros: block in yaml). It is singular tests that cannot be documented as far as I know!!

You can create a macros.yml file inside macros/ which refer both to macros and generic tests and it works, does not matter if your generic test is inside macros or tests/generic folder.

As an example I'm providing the file tests/generic/test_not_empty_string.sql whose content is [1] and the yaml code that documents it [2], which is placed inside macros/macros.yml. I'm attaching the screenshot of the documentation as well:

[1]:

{% test not_empty_string(model, column_name) %}

/*
    Complementary test to default `not_null` test as it checks that there is not an empty string
*/

with only_relevant_column as (
    SELECT
        {{ column_name }}
    FROM
        {{ model }}
)

SELECT
    *
FROM
    only_relevant_column
WHERE
    {{ column_name }} = ''

{% endtest %}

[2]:

  - name: test_not_empty_string
    description: Complementary test to default `not_null` test as it checks that there is not an empty string. It only accepts columns of type string. It is deprecated as this functionality has been included in the custom implementation of `not_null`
    arguments:
      - name: model
        type: string
        description: Model Name
      - name: column_name
        type: string
        description: Column name that should not be an empty string

This documents your test "as a macro" >> but is the same documentation generated when you view the test, linked in the model documentation?

[oliviamaquar]

+1 to be able to add descriptions to singular tests

[AlexanderStephenson]

I have finally got this to work in dbt cloud but it is not obvious.

My generic test (macro) is called check_row_count_not_zero and is stored under tests/generic.

To document this I have to update the macros.yml in the macros folder, note the name has to have the prefix test_

• name: test_check_row_count_not_zero
  description: A generic test that will throw and error is a model has zero rows.
  arguments:
  • name: model
    type: char
    description: The name of the model to test for zero rows.

With this the documentation now shows

Description
A generic test that will throw and error is a model has zero rows.

[SimonDegeorge]

+1 to be able to add descriptions to singular tests

[anai-s]

+1 to be able to add descriptions to singular tests

[babschlott]

+1 to be able to add descriptions to singular tests

[jeff-skoldberg-gmds]

@dbeatty10 , this thread makes me wonder, what is the official way dbt Labs tracks feature requests? Most platforms have an idea page with an upvote count and their product team works on the most upvoted ideas. In this case, everyone is randomly "+1"ing and there's no way for anyone to tell this is a popular idea.

Should we be doing something else?

[FishtownBuildBot]

Opened a new issue in dbt-labs/docs.getdbt.com: dbt-labs/docs.getdbt.com#6293