Fix failing dbt-fusion tests when run in parallel in CI

[pankajkoti]

Problem

The dbt Fusion integration tests were experiencing resource conflicts when multiple matrix jobs ran in parallel against the same Snowflake schema. All jobs were attempting to create and operate on tables/views with identical names (e.g., customers, orders, stg_payments), leading to:

• Race conditions between parallel jobs
• dbt test failures due to duplicate records

Initial attempt to resolve:

As described in issue #1887, I attempted to specify a separate schema in the environment variable to uniquely identify the PR, and matrix entry. However, since such schemas don't pre-exist in our database, when dbt tries to create them, we encounter failures because the Snowflake CI account does not have the necessary permissions to create schemas.

Working solution proposed in the PR:

Instead of creating separate schemas, this PR implements resource-level isolation within a shared schema:

1. Unique Resource Prefixes: Each matrix job gets a unique prefix like COSMOS_<PR_IDENTIFIER>_PY311_AF210_DBT20_
2. Custom dbt Macro: Added generate_alias_name macro that prefixes all table/view names when RESOURCE_PREFIX is set (this is set for CI jobs)
3. Safe Environment Handling: Uses GitHub Actions step outputs instead of $GITHUB_ENV (one of the initial attempts to export RESOURCE_PREFIX and make it available for subsequent steps in the CI jobs) to avoid security issues flagged by Zizmor
4. Automatic Cleanup: Added a cleanup step (calling a Python script) that removes all objects created by each job after completion, as otherwise it would just keep on populating a huge number of tables and views for each PR * matrix entry in the database.

Successful job run: https://github.com/astronomer/astronomer-cosmos/actions/runs/16622909325

And a specific dbt fusion tests job example: https://github.com/astronomer/astronomer-cosmos/actions/runs/16622909325/job/47031551362?pr=1896

closes: #1887

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	624e5f5
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/6890dd7d7867730008632535

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.06%. Comparing base (c9fb4ab) to head (4bda381).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1896   +/-   ##
=======================================
  Coverage   98.06%   98.06%           
=======================================
  Files          85       85           
  Lines        5312     5312           
=======================================
  Hits         5209     5209           
  Misses        103      103           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull Request Overview

This PR fixes failing dbt-fusion tests when run in parallel in CI by implementing resource-level isolation to prevent race conditions. Instead of creating separate schemas, it uses unique prefixes for table/view names within a shared schema.

• Implements unique resource prefixes per matrix job to avoid naming conflicts
• Adds a custom dbt macro to automatically prefix table/view names when RESOURCE_PREFIX is set
• Includes automatic cleanup of Snowflake resources after each job completion

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File	Description
.github/workflows/test.yml	Adds resource prefix generation and cleanup steps for dbt-fusion tests
dev/dags/dbt/jaffle_shop/macros/generate_alias_name.sql	Custom dbt macro to prefix table/view names with RESOURCE_PREFIX
scripts/ci_dbtf_delete_snowflake_resources.py	Python script to clean up Snowflake resources with matching prefix
tests/dbt/test_graph.py	Updates expected hash values for test assertions

[tatiana]

This looks great, @pankajkoti , I really liked the approach and implementation. This will make our dbt Fusion tests more robust and less flaky.