Kubernetes Pod Operator convert container_resources

[johnhoran]

Description

Slightly odd and I'm not sure if this is the best solution, but this is the code base I have access to, so lets go with it.

I'm trying to specify resource limits in cosmos using https://astronomer.github.io/astronomer-cosmos/configuration/operator-args.html#overriding-operator-arguments-per-dbt-node-or-group-of-nodes, specifically I'm adding

    meta:
      cosmos:
        operator_kwargs:
          container_resources:
            limits:
              memory: "2000Mi"

to a DBT project and when running locally using astro dev start this works correctly. However when running in astronomer hosted, it doesn't apply. Researching further if I have an operator with

KubernetesPodOperator(
        ...
        container_resources={
            "limits": {"cpu": "100m", "memory": "100Mi", "ephemeral-storage": "512Mi"},
            "requests": {
                "cpu": "100m",
                "memory": "100Mi",
                "ephemeral-storage": "512Mi",
            },
        },
    )

then the resources will be overridden with the defaults set on the deployment in astronomer hosted, though running locally it works correctly. If I specify them as

KubernetesPodOperator(
        ...
        container_resources=k8s.V1ResourceRequirements(
            limits={"cpu": "100m", "memory": "100Mi", "ephemeral-storage": "512Mi"},
            requests={"cpu": "100m", "memory": "100Mi", "ephemeral-storage": "512Mi"},
        ),
    )

then it all works correctly, but this precludes my ability to set them in cosmos as the argument will always be passed as a dictionary there.

It seems whatever code astronomer is using on the backend to apply the default container resources doesn't work with anything except V1ResourceRequirements. Arguably this is correct behaviour since the signature in the kubernetes pod operator specifies the type to be V1ResourceRequirements | None, even though the code itself does work with dictionaries.

Anyway the proposed solution is to convert this argument to the correct type before it is passed to the kubernetes pod operator. Thoughts?

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	9c3a2e8
🔍 Latest deploy log	https://app.netlify.com/projects/sunny-pastelito-5ecb04/deploys/6859373d1d8970000925f062

[Copilot]

Pull Request Overview

This pull request addresses the issue with container resource specifications by converting a dictionary input into a V1ResourceRequirements object so that the resource limits are correctly applied in Astronomer hosted deployments.

• Added a conversion step for container_resources when provided as a dict.
• Ensured compatibility with the expected type in KubernetesPodOperator.

[tatiana]

Hi @johnhoran thank you very much for raising this and creating the PR so things work in Astro. Which type of executor are you using? Where is the k8s cluster you're instantiating these pods?

I'm following up with the Astro Airflow Infra team to gain a deeper understanding of why this override occurs and if there are any alternative solutions.

That said, I'm happy with the change you're proposing. Please, could you create unit tests for this use case, to validate the desired behaviour?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.01%. Comparing base (e45bf11) to head (9c3a2e8).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1821   +/-   ##
=======================================
  Coverage   98.01%   98.01%           
=======================================
  Files          85       85           
  Lines        5287     5290    +3     
=======================================
+ Hits         5182     5185    +3     
  Misses        105      105           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[johnhoran]

Hi @johnhoran thank you very much for raising this and creating the PR so things work in Astro. Which type of executor are you using? Where is the k8s cluster you're instantiating these pods?

Hi @tatiana, We are running in astronomer 13 in a dedicated hosted AWS cluster using the kubernetes executor.

[tatiana]

@johnhoran I discussed this matter with fellow Astronomers, and @kaxil raised this question:

Did you (or someone from your team) set some defaults in Astro UI as defined in https://www.astronomer.io/docs/astro/deployment-resources/#configure-kubernetes-pod-resources ?

This may justify the behaviour you're seeing

[johnhoran]

@johnhoran I discussed this matter with fellow Astronomers, and @kaxil raised this question:

Did you (or someone from your team) set some defaults in Astro UI as defined in https://www.astronomer.io/docs/astro/deployment-resources/#configure-kubernetes-pod-resources ?

This may justify the behaviour you're seeing

As far as I know it isn't possible to create a deployment without the defaults set. I guess the real question is if this behaviour is intentional or not, like I said, it is arguably correct.