[CT-643] Support package resources with path: selector method

[jtcohen6]

Prompts:

• first internal Slack thread
• second internal Slack thread

Short description

The dbt Cloud IDE uses dbt list behind the scenes to power the DAG viz. When it provides +dbt_packages/package_name/models/.../model_name.sql+, dbt-core doesn't return any resources.

The reason: dbt-core's path: selector method doesn't support resources defined in packages.

dbt-core/core/dbt/graph/selector_methods.py

Lines 288 to 289 in 72c17c4

	if Path(real_node.root_path) != root:
	continue

Proposed resolution

Update FileSelectorMethod to accept the full relative path of resources defined in packages, including the install path and the package name, as a way to select that resource.

For example, dbt_packages/dbt_project_evaluator/models/marts/dag/fct_direct_join_to_source.sql to select fct_direct_join_to_source.

Acceptance criteria

• dbt ls -s full/relative/path/to/package/model.sql should return that model

Suggested tests

dbt-core/tests/unit/test_graph_selector_methods.py

Line 1150 in e81f7fd

def test_select_file(manifest):

https://github.com/dbt-labs/dbt-core/blob/main/tests/functional/list/test_list.py

Alternatives

Both of these alternatives (the original resolutions I suggested in ~2022) would require additional logic on the "client" side. I believe a preferable resolution allows clients to pass the full (relative) path to all resources, both those defined in the root project and those defined in packages, without any additional logic.

1. Start returning package resources based on their original_file_path (always relative), and expect services passing file names into dbt list to trim off dbt_packages/package_name/. Because models with the same name can be defined in multiple installed packages, this could return multiple models that have the same relative path. For example, if I also models/my_model.sql, even if they're defined in different packages. I think that's fair, and in keeping with expected behavior.

2. Add a new FileIDSelector. dbt-core does have a notion of an internal "file ID" that looks like package_name://relative/file/path, which it uses to uniquely identify each file during partial parsing. That still requires knowing the name of the package, though. I'm pretty sure dbt deps takes care of this. (It should be the same for Hub packages, based on how dbt deps renames the package after installation, but not necessarily the same for non-Hub packages.)

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[racheldaniel]

The XP team is watching this as a possible solution for issues with compile/preview not working on dbt packages in the IDE. Per slack thread with core this is to be reopened.

[ChenyuLInx]

We might need to insert dbt packages in the original_file_path

[ChenyuLInx]

Whoever picks this one up please pair with @gshank

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.