Allow to set task display name as a user-defined function

[corsettigyg]

Description

re-opened #1756 after resetting my fork by mistake.

Currently cosmos allows to override the task_id value with the param normalize_task_id, but there is no process in place to override the display_name . At my current company we have some logic in place using the task_id and corresponding suffixes, but we would like to still change how tasks are displayed in the airflow UI, plus from a user's perspective it is sometimes confusing to have the suffix being displayed in the airflow UI instead of just having the name of the model itself. It would be nice to have a way to keep the task_id as it is while overriding the display_name instead so if a user is already using the task_ids for some custom logic, it is possible to change how the tasks are displayed in the UI without causing issues.

a small disclaimer: I took the liberty to rename task_display_name to task_name in the code since it was not exactly correct, and since this PR aims to separate both concepts, task_name is a more generic name in case that the display_name is not set 😄

Related Issue(s)

Breaking Change?

No

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Name	Link
🔨 Latest commit	80c9ee9
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/685425797038dd0008382268
😎 Deploy Preview	https://deploy-preview-1761--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[Copilot]

Pull Request Overview

This PR introduces the ability for users to set a custom display name for tasks using a user-defined function without affecting the underlying task_id.

• Added a new parameter, normalize_task_display_name, in RenderConfig and updated documentation.
• Updated task generation logic in cosmos/airflow/graph.py to incorporate the new display name normalization functionality.
• Adjusted documentation in configuration files to explain the new feature.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
docs/configuration/task-display-name.rst	Added explanation and example usage for normalize_task_display_name.
docs/configuration/render-config.rst	Documented the new normalize_task_display_name parameter.
cosmos/config.py	Added the normalize_task_display_name field in the RenderConfig class.
cosmos/airflow/graph.py	Updated functions to handle the new normalize_task_display_name parameter when generating task metadata.

[tatiana]

Hi @corsettigyg ! Thanks a lot for the ongoing contributions.

I'd love to understand more about this use case. Does your company currently use the same or different approaches to create display_name and task ID?

[corsettigyg]

hey @tatiana

lets say I have a dbt task called my_model_run and I use normalize_task_id to make it into my_model. The airflow user will still see my_model_run through the UI but if I want to create, lets say a sensor poking the task, I should now refer to my_model due to the fact that airflow decoupled both elements and task_id is the id used for anything that is airflow internal.

The use-case we have at my current company is that we want to rename how cosmos tasks appear in the airflow UI AND the task ID itself, so they will match each other after the renaming. the reason is because it is convenient to do some string manipulation to have the cosmos task = full table path in the warehouse. there are many users creating sensors on top of our dbt dags and for them, task_id and task_display_name are foreign concepts to begin with, so we want to align them with our custom logic.

This PR precisely enables setting the task_id & task_display_name the way the user wants, giving an extra layer of flexibility that is not available since the task_display_name is embedded into the codebase. In our case, it is because we want to rename task_id and have task_id = display_name, but for some other companies they might have distinct use-cases, which will be attainable now

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.01%. Comparing base (05ca4e8) to head (80c9ee9).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1761   +/-   ##
=======================================
  Coverage   98.01%   98.01%           
=======================================
  Files          85       85           
  Lines        5279     5287    +8     
=======================================
+ Hits         5174     5182    +8     
  Misses        105      105           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tatiana]

@giovannicorsetti I find it a bit tricky how Airflow ended up disassociating task_id from task_display_name, and I can fully understand your use case. Thank you very much for all the work and patience. We'll release this feature in Cosmos 1.11.