Parity for source freshness task/command

[jtcohen6]

Describe the feature

dbt source snapshot-freshness is quite limited and a bit verbose relative to other dbt task-commands.

1. Source freshness should support full node selection syntax, including things like:

   • --select tag:something (Ability to run snapshot-freshness with tag selection #2369)
   • --exclude and --selector (Add --exclude for source snapshot-freshness #2729)
   • --select +downstream_resource, i.e. check freshness of all sources upstream of downstream_resource. I think this would be especially compelling for exposures!

2. Should this be renamed to avoid confusion with dbt snapshot? I think yes, and it could be as simple as calling it dbt source freshness instead—although that's not a verb! Open to suggestions.

We'd obviously do the renaming in a way that's backwards-compatible. Ideally, we'd rename and add functionality in the same release.

Describe alternatives you've considered

Leaving dbt source snapshot-freshness as is: clunky to type, wonky to use

Additional context

Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here.

Who will this benefit?

All current and potential users of source freshness, which is a relatively underused + under-appreciated feature! Maybe in part because it feels a bit clunky :)

[emilieschario]

When running this command, I also think it should have a summary statement which it doesn't have right now (e.g. 10 models pass, etc.). Do you consider this part of parity or should it be a separate issue @jtcohen6?

[jtcohen6]

Good call @emilieschario! I think we could work that into this issue for now, and we'll split it out if we need to.

In general, we don't make the same commitments today about log / stdout stability as we now do for artifacts and user-supplied code. There's a lot of little stdout things I'd like to tidy up or touch up, but they're not high-priority prerequisites for 1.0.

[kwigley]

• What is the exit criteria for this issue?
  • I think all constructs exist, exit criteria can be aligning selection behavior with other commands.
• What are the high-level items of the work that need to be done (i.e. create x, split out y, etc.)
  • update cli parsing to add to sub parsers to handle selection args
  • update selection logic in the freshness task
  • create new command pointing to the same entry point
• What are the open questions on this issue that still need answers?
  • what exactly should we rename the cli command?
  • does state selection make sense for this command?

[kwigley]

@jtcohen6 2 quick questions:

1. I'm going back and forth here. What should we rename this cli command to?

• dbt source freshness
• dbt source-freshness
• dbt data-freshness
• dbt check freshness

2. Should the current dbt source snapshot-freshness command stay in tact (this is a simple alias) with a deprecation warning?

[jtcohen6]

1. I'm leaning dbt source freshness. I realize it's a noun, not a verb (which other commands are), but I don't think that's so bad. The colloquial expression/documentation can refer to this as "checking source freshness."
2. Yes!

[Kiollpt]

@jtcohen6
I am curious about --select +downstream_resource you mentioned
I would like to automatically check all upstream sources if they are fresh given the model
Does this work now?

[jtcohen6]

@Kiollpt Yes, that should work now! Have you given it a try with a recent version of dbt (v0.21+)?

[Kiollpt]

@jtcohen6 It works well after I tried.