You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Source Freshness UI (when accessed from the Data Sources link in the main dbt Cloud menu) would display the status of each source as of the most recent job that tested its freshness.
Only sources currently in the project manifest need to be included. A link at the top of each environment's main page – https://cloud.getdbt.com/#/accounts/X/projects/X/environments/X/ – would be convenient, too.
Additional context
Currently, data in the Source Freshness UI is populated only for those sources --selected in the most recently completed job containing a snapshot-freshness statement. This means that in projects with several "freshness SLAs", there's no one-stop, comprehensive view of source freshness for the whole project.
Consider a project with the following source freshness SLAs and scheduled jobs:
If a user views the Source Freshness UI between (roughly speaking) 0615 and 0630, they'll see the freshness status for source_b but UNTRACKED for source_a. At any other time, source_b will display as UNTRACKED, and only source_a's status will be available.
Describe alternatives you've considered
Browse the Job History to find the last run of each Source Freshness job, and view the results of each job separately.
Who will this benefit?
Any project with multiple source freshness SLAs, especially as node selection syntax enhancements (via #2987) make it easier to define more granular source freshness checks.
The text was updated successfully, but these errors were encountered:
@ljhopkins2 Thanks for opening! I totally agree with what you've laid out here.
Today, the Source Freshness UI in dbt Cloud is powered by the sources.json artifact, grabbed from the most recent run of the configured job. In order to grab all metadata from the selected environment, merging information from multiple jobs into a single view, we'd need some kind of Metadata API... the foundations of which are soon to be in GA. (If you want a sneak peak at some of the exciting new documentation: dbt-labs/docs.getdbt.com#713.)
As those docs (will) state:
Please note that at present, queries to the Metadata API exclusively return metadata about the most recent Run for a given dbt Cloud Job. We are continuously expanding the capabilities of the metadata API and we welcome your feedback and suggestions at [email protected].
So the limitation you've described will remain in place for the immediate future. That said, I believe we have the right foundations in place to support exactly the kind of experience you're after.
I am going to close this issue, only insofar as it's not a code change we'll be making in this repo (dbt Core). But it's definitely on our minds (cc @barryaron)
Describe the feature
The Source Freshness UI (when accessed from the Data Sources link in the main dbt Cloud menu) would display the status of each source as of the most recent job that tested its freshness.
Only sources currently in the project manifest need to be included. A link at the top of each environment's main page –
https://cloud.getdbt.com/#/accounts/X/projects/X/environments/X/
– would be convenient, too.Additional context
Currently, data in the Source Freshness UI is populated only for those sources
--selected
in the most recently completed job containing asnapshot-freshness
statement. This means that in projects with several "freshness SLAs", there's no one-stop, comprehensive view of source freshness for the whole project.Consider a project with the following source freshness SLAs and scheduled jobs:
source_a
job_a: dbt source snapshot-freshness --select source_a
0,30 * * * *
source_b
job_b: dbt source snapshot-freshness --select source_b
15 6 * * *
If a user views the Source Freshness UI between (roughly speaking) 0615 and 0630, they'll see the freshness status for
source_b
butUNTRACKED
forsource_a
. At any other time,source_b
will display asUNTRACKED
, and onlysource_a
's status will be available.Describe alternatives you've considered
Browse the Job History to find the last run of each Source Freshness job, and view the results of each job separately.
Who will this benefit?
Any project with multiple source freshness SLAs, especially as node selection syntax enhancements (via #2987) make it easier to define more granular source freshness checks.
The text was updated successfully, but these errors were encountered: