Display latest freshness check for all sources, irrespective of job, in the Data Sources UI

[ljhopkins2]

Describe the feature

The Source Freshness UI (when accessed from the Data Sources link in the main dbt Cloud menu) would display the status of each source as of the most recent job that tested its freshness.

Only sources currently in the project manifest need to be included. A link at the top of each environment's main page – https://cloud.getdbt.com/#/accounts/X/projects/X/environments/X/ – would be convenient, too.

Additional context

Currently, data in the Source Freshness UI is populated only for those sources --selected in the most recently completed job containing a snapshot-freshness statement. This means that in projects with several "freshness SLAs", there's no one-stop, comprehensive view of source freshness for the whole project.

Consider a project with the following source freshness SLAs and scheduled jobs:

Source	Freshness SLA	Job	Schedule
source_a	Hourly	job_a: dbt source snapshot-freshness --select source_a	0,30 * * * *
source_b	Daily by 0600	job_b: dbt source snapshot-freshness --select source_b	15 6 * * *

If a user views the Source Freshness UI between (roughly speaking) 0615 and 0630, they'll see the freshness status for source_b but UNTRACKED for source_a. At any other time, source_b will display as UNTRACKED, and only source_a's status will be available.

Describe alternatives you've considered

Browse the Job History to find the last run of each Source Freshness job, and view the results of each job separately.

Who will this benefit?

Any project with multiple source freshness SLAs, especially as node selection syntax enhancements (via #2987) make it easier to define more granular source freshness checks.

[jtcohen6]

@ljhopkins2 Thanks for opening! I totally agree with what you've laid out here.

Today, the Source Freshness UI in dbt Cloud is powered by the sources.json artifact, grabbed from the most recent run of the configured job. In order to grab all metadata from the selected environment, merging information from multiple jobs into a single view, we'd need some kind of Metadata API... the foundations of which are soon to be in GA. (If you want a sneak peak at some of the exciting new documentation: dbt-labs/docs.getdbt.com#713.)

As those docs (will) state:

Please note that at present, queries to the Metadata API exclusively return metadata about the most recent Run for a given dbt Cloud Job. We are continuously expanding the capabilities of the metadata API and we welcome your feedback and suggestions at [email protected].

So the limitation you've described will remain in place for the immediate future. That said, I believe we have the right foundations in place to support exactly the kind of experience you're after.

I am going to close this issue, only insofar as it's not a code change we'll be making in this repo (dbt Core). But it's definitely on our minds (cc @barryaron)