diff --git a/docs/getting_started/watcher-execution-mode.rst b/docs/getting_started/watcher-execution-mode.rst index 04f7e3082b..803c5d18ed 100644 --- a/docs/getting_started/watcher-execution-mode.rst +++ b/docs/getting_started/watcher-execution-mode.rst @@ -87,6 +87,45 @@ Depending on the dbt workflow topology, if your dbt DAG previously took 5 minute We plan to repeat these benchmarks and share the code with the community in the future. + +.. note:: + ``ExecutionMode.WATCHER`` relies on the ``threads`` value defined in your dbt profile. Start with a conservative value that matches the CPU capacity of your Airflow workers, then gradually increase it to find the sweet spot between faster runs and acceptable memory/CPU usage. + +When we ran the `astronomer/cosmos-benchmark `_ project with ``ExecutionMode.WATCHER``, that same ``threads`` setting directly affected runtime: moving from 1 to 8 threads reduced the end-to-end ``dbt build`` duration from roughly 26 seconds to about 4 seconds (see table above), while 16 threads squeezed it to around 2 seconds at the cost of higher CPU usage. Use those numbers as a reference point when evaluating how thread counts scale in your own environment. + +To increase the number of threads, edit your dbt ``profiles.yml`` (or Helm values if you manage the profile there) and update the ``threads`` key for the target you use with Cosmos: + +.. code-block:: yaml + + your_dbt_project: + target: prod + outputs: + prod: + type: postgres + host: your-host + user: your-user + password: your-password + schema: analytics + threads: 8 # increase or decrease to match available resources + + +If you prefer to manage threads through Cosmos profile mappings instead of editing ``profiles.yml`` directly, pass ``profile_args={"threads": }`` to your ``ProfileConfig``. For example, using the built-in ``PostgresUserPasswordProfileMapping``: + +.. code-block:: python + + from cosmos.config import ProfileConfig + from cosmos.profiles import PostgresUserPasswordProfileMapping + + profile_config = ProfileConfig( + profile_name="jaffle_shop", + target_name="prod", + profile_mapping=PostgresUserPasswordProfileMapping( + conn_id="postgres_connection", + profile_args={"threads": 8}, + ), + ) + + ------------------------------------------------------------------------------- Example Usage of ``ExecutionMode.WATCHER``