Reduce Mccabe code complexity to 8

[joppevos]

Description

Created this small PR to reduce the maccabe score to 8 and refactored the code flagged as too complex.

I removed flake8, since we can use ruffs maccabe linter.

The following functions have been addressed

> pre-commit run flake8 --all-files
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

cosmos/airflow/graph.py:196:1: C901 'build_airflow_graph' is too complex (9)
cosmos/converter.py:101:5: C901 'DbtToAirflowConverter.__init__' is too complex (9)
cosmos/dbt/parser/project.py:277:5: C901 'LegacyDbtProject.__post_init__' is too complex (10)
cosmos/dbt/selector.py:197:1: C901 'select_nodes' is too complex (9)

I picked out the functions which stood out to me as complex. If we want to reduce it to 6, there are plenty more functions to tackle. For some functions I would consider it unnecessary to drop it to 6.

cosmos/config.py:108:9: C901 `__init__` is too complex (7 > 6)
cosmos/dbt/parser/project.py:63:9: C901 `_config_selector_ooo` is too complex (7 > 6)
cosmos/dbt/parser/project.py:98:5: C901 `extract_python_file_upstream_requirements` is too complex (7 > 6)
cosmos/dbt/parser/project.py:165:9: C901 `extract_sql_file_requirements` is too complex (7 > 6)
cosmos/dbt/selector.py:52:9: C901 `load_from_statement` is too complex (7 > 6)
cosmos/dbt/selector.py:119:9: C901 `_should_include_node` is too complex (7 > 6)
cosmos/hooks/subprocess.py:34:9: C901 `run_command` is too complex (8 > 6)
cosmos/operators/base.py:137:9: C901 `get_env` is too complex (7 > 6)
cosmos/operators/base.py:186:9: C901 `add_global_flags` is too complex (7 > 6)
cosmos/operators/local.py:136:9: C901 `store_compiled_sql` is too complex (7 > 6)
cosmos/profiles/base.py:183:9: C901 `get_dbt_value` is too complex (8 > 6)
Found 11 errors.

Related Issue(s)

This PR continues on the amazing work of @jbandoro in on #525
Related issue #641

Breaking Change?

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

👷 Deploy Preview for amazing-pothos-a3bca0 processing.

Name	Link
🔨 Latest commit	d7454e9
🔍 Latest deploy log	https://app.netlify.com/sites/amazing-pothos-a3bca0/deploys/656f1bafade1320008ee3a2f

[codecov]

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (abf7f27) 93.07% compared to head (d7454e9) 93.12%.

Files	Patch %	Lines
cosmos/dbt/selector.py	94.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #738      +/-   ##
==========================================
+ Coverage   93.07%   93.12%   +0.05%     
==========================================
  Files          55       55              
  Lines        2410     2429      +19     
==========================================
+ Hits         2243     2262      +19     
  Misses        167      167              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

This looks great, @joppevos , a minor comment on separating part of the change into a separate PR!

[tatiana]

@joppevos tiny error raised on the pre-commit check:

ruff..........................................................................Failed
- hook id: ruff
- exit code: 1

cosmos/converter.py:9:8: F811 Redefinition of unused `copy` from line 6
Found 1 error.

[joppevos]

One more function has gotten more complex than 8 after the rebase from main.

cosmos/dbt/selector.py:198:5: C901 'SelectorConfig.load_from_statement' is too complex (9)

I will solve it

[tatiana]

Thanks a lot for working on this, @joppevos ! It really helps with the overall code maintainability.

[joppevos]

Thanks a lot for working on this, @joppevos ! It really helps with the overall code maintainability.

I'm happy to do it! Refactoring teaches me how Cosmos works on the inside.