Add profile_config arg to DbtKubernetesBaseOperator

[david-mag]

Description

When creating DbtTaskGroups and DbtDags, a ProfileConfig needs to be provided, since it's a required positional argument of the DbtAirflowConverter class. However, when providing a profile_config in execution_mode="kubernetes", the following Exception is thrown:

AirflowException: Invalid arguments were passed to DbtRunKubernetesOperator (task_id: my_first_dbt_model_run). Invalid arguments were:
**kwargs: {'profile_config': ProfileConfig(profile_name='jaffle_shop', target_name='dev', profiles_yml_filepath='jaffle_shop/profiles.yml', profile_mapping=None)}

In my humble attempt of a solution, I make the profile_config an optional argument of the DbtKubernetesBaseOperator class (mainly to satisfy the tests ... since its a required argument in the previous part of the "pipeline", having it being a required argument would make sense imho) and parse it inside the build_kube_args method (similar, but not identically) to how it is done in the run_command method of the DbtLocalBaseOperator class.

This solution fixes the thrown exception and has the added benefit of giving the user that want to run dbt inside kubernetes the option to specify --profile and --target, which is a useful feature when wanting to schedule the same model in airflow, pointing to dev and prod targets.

I've tested this solution in my GCC environment.

Related Issue(s)

This PR resolves the issue in issue #493

Breaking Change?

This should fix something that is broken atm, as far as I can see.

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	776ec59
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/64f6f84973c8540008cac3d4

[codecov]

Codecov Report

Patch coverage: 71.42% and project coverage change: -0.18% ⚠️

Comparison is base (033a4b3) 91.48% compared to head (776ec59) 91.30%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #505      +/-   ##
==========================================
- Coverage   91.48%   91.30%   -0.18%     
==========================================
  Files          49       49              
  Lines        1914     1921       +7     
==========================================
+ Hits         1751     1754       +3     
- Misses        163      167       +4     

Files Changed	Coverage Δ
cosmos/operators/kubernetes.py	70.00% <71.42%> (-2.05%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jlaneve]

nice, this looks great! we should add the same support in the Docker base operator. I've filed #512 as a follow up!

[agneku]

Hi @david-mag, @jlaneve, I'm now trying to create a DbtDag with execution_mode='kubernetes'. Even though I don't receive the profile_config error anymore (since the 1.1.0 release), I got a new error:
airflow.exceptions.AirflowException: Invalid arguments were passed to DbtRunKubernetesOperator (task_id: final_model2_run). Invalid arguments were: **kwargs: {'emit_datasets': True}

The emit_datasets arg is not something I set on my own. Here's the DbtDag for reference:

transform_data = DbtDag(
    profile_config=profile_config,
    project_config=ProjectConfig(DBT_PROJECT_PATH),
    execution_config=ExecutionConfig(
        execution_mode=ExecutionMode.KUBERNETES,
    ),
    operator_args={
        "queue": "kubernetes",
        "image": DBT_IMAGE,
        "image_pull_policy": "Always",
        "get_logs": True,
        "is_delete_operator_pod": False,
    },
    dag_id = "dbt_test",
    start_date=datetime(2023, 8, 1),
    schedule=None,
    catchup=False,
)

is this a related issue?

[david-mag]

As far as I can see, with the release of this fix, more "unknown" arguments have been added, which broke the kubernetes DAGs and TaskGroups again right away.

[qimumu9406]

@david-mag I also encountered the same problem. If using cosmos on kubernetes, it seems this is a fatal problem.