feat: Add support for StarRocks profile mapping#2256
Conversation
This commit introduces the StarrocksUserPasswordProfileMapping class, enabling Cosmos to automatically map Airflow connections to dbt profiles for StarRocks. Key changes: Added StarrocksUserPasswordProfileMapping to support starrocks connection type. Registered the new profile mapping in cosmos/profiles/ init .py . Added dbt-starrocks to dbt-all and dbt-starrocks optional dependencies in pyproject.toml . Added comprehensive unit tests in tests/profiles/starrocks/test_starrocks_user_pass.py .
✅ Deploy Preview for astronomer-cosmos canceled.
|
There was a problem hiding this comment.
Pull request overview
This PR adds support for mapping Airflow MySQL connections to dbt StarRocks profiles. StarRocks uses the MySQL protocol, so the integration leverages the existing MySQL connection type while mapping to dbt's StarRocks adapter.
Key changes:
- Implements
StarrocksUserPasswordProfileMappingclass for StarRocks profile generation - Adds dbt-starrocks as an optional dependency in the project configuration
- Includes comprehensive test coverage for the new profile mapping functionality
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| cosmos/profiles/starrocks/user_pass.py | Core implementation of StarRocks profile mapping using MySQL connection type |
| cosmos/profiles/starrocks/init.py | Package initialization exposing the StarRocks profile mapping class |
| cosmos/profiles/init.py | Registers StarRocks profile mapping in the automatic profile discovery system |
| pyproject.toml | Adds dbt-starrocks to optional dependencies |
| tests/profiles/starrocks/test_starrocks_user_pass.py | Test suite validating connection claiming, profile generation, and environment variables |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Good. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Good suggestion Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2256 +/- ##
=======================================
Coverage 98.03% 98.04%
=======================================
Files 98 100 +2
Lines 6377 6401 +24
=======================================
+ Hits 6252 6276 +24
Misses 125 125 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
tatiana
left a comment
There was a problem hiding this comment.
@kurkim0661, this looks great, thank you very much for the contribution! We'll be releasing it as part of Cosmos 1.13. The implementation, tests, and PR description are excellent, including the limitations bit. Excellent work!
Features * Support cross-referencing models across dbt projects using dbt-loom by @pankajkoti in #2271 * Support use of YAML selectors when using ``LoadMode.DBT_MANIFEST`` by @YourRoyalLinus in #2261 * Introduce ``ExecutionMode.WATCHER_KUBERNETES`` to use the watcher with ``KubernetesPodOperator`` by @tatiana in #2207 * Add support for StarRocks profile mapping by @kurkim0661 in #2256 * Allow pushing URIs as XComs for Cosmos tasks by @corsettigyg in #2275 * Support defining custom callbacks alongside the ``WATCHER_KUBERNETES`` callback by @johnhoran in #2307 Enhancements * Refactor: remove duplicate ``_construct_dest_file_path`` by @jx2lee in #2077 * Leverage Airflow ``::group::`` to group logs associated with DAG parsing by @tatiana in #2235 * Refactor ``DbtConsumerWatcherSensor`` for reusability by @tatiana in #2245 * Restore plain text output when using ``ExecutionMode.WATCHER`` by @tiovader in #2241 Bug Fixes * Fix running empty models or ephemeral nodes in ``ExecutionMode.WATCHER`` by @tatiana in #2279 * Improve watcher producer task priority in scheduling and the UI by @tatiana in #2237 * Fix typos and formatting issues in documentation by @pankajkoti in #2259 * Allow watcher producer retries without erroring by @tatiana in #2283 * Fix ``TestBehavior.AFTER_ALL`` is missing project_name information when loading project using manifest file by @tuantran0910 in #2242 * Fix duplicate log lines in watcher subprocess execution and format timestamps by @pankajkoti in #2301 Docs * Add Watcher Kubernetes documentation by @tatiana in #2303 * Document newly added telemetry metrics in the privacy notice by @pankajkoti in #2249 * Add compatibility policy document by @pankajastro in #2251 * Improve watcher documentation related to dbt threads by @tatiana in #2273 * Fix link in watcher execution mode documentation by @jedcunningham in #2277 * Update Apache Airflow minimum compatibility policy by @tatiana in #2285 * Clarify Cosmos runtime support until "End of Basic Support" by @jedcunningham in #2286 * Update watcher docs by @tatiana in #2298 * Update watcher kubernetes documentation by @tatiana in #2306 Others * Add Airflow 3 DAG versioning tests for Cosmos by @michal-mrazek in #2177 * Add dbt Core 1.11 to the test matrix by @tatiana in #2230 * Add integration tests using InvocationMode.SUBPROCESS and validate output by @tatiana in #2287 * Fix main branch failing tests by @tatiana in #2296 * Update pre-commit hooks to the latest versions by @jedcunningham in #2289 * Pre-commit autoupdates by @pre-commit in #2222, #2264, #2274 and #2290 * Dependabot updates by @dependabot in #2218, #2219, #2220, #2280 and #2284 * Add Scarf metrics to understand Cosmos feature usage patterns - Add telemetry tracking for dbt docs plugin usage by @pankajkoti in #2240 - Add DAG run telemetry metrics for load mode, invocation, and render_config parameters by @pankajkoti in #2223 - Collect profile metrics for DAG runs by @pankajastro in #2228 - Compress telemetry metadata to reduce serialized DAG size by @pankajkoti in #2252 - Skip storing telemetry metadata when emission is disabled by @pankajkoti in #2278 - Hide telemetry metadata parameters from the Airflow trigger UI by @pankajkoti in #2247 closes: astronomer/oss-integrations-private#317 --------- Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Description
Introduces the StarrocksUserPasswordProfileMapping class, enabling Cosmos to automatically map Airflow connections to dbt profiles for StarRocks.
Test environment
Concerns
Both issues shouldn’t affect existing users, but they could cause problems for users who want to use StarRocks.
Related Issue(s)
Closes: #2255
Breaking Change?
Checklist