Refactor handling of watcher producer-only dbt flags#2717
Merged
Conversation
When a watcher consumer task retries via _fallback_to_non_watcher_run,
it copies the producer's add_cmd_flags() output into its own dbt
invocation. The producer hardcodes log_format="json" so it can parse
dbt's structured event stream, but that internal choice was leaking
into the consumer's user-facing retry dbt command, producing raw JSON
in task logs (one structured event per line) instead of dbt's
default formatted text.
Add --log-format to the _filter_flags skip list so the producer's
internal log_format choice does not propagate. Users who want JSON
output on retry can still opt in via
operator_args={"dbt_cmd_flags": ["--log-format", "json"]} —
dbt_cmd_flags is appended by build_cmd outside of this flag pipeline.
Reported when validating example_watcher_downstream_not_skipped's
model_downstream_run retry on the 1.14.2a3 release candidate.
Mirrors example_watcher_downstream_not_skipped but adds
operator_args={"dbt_cmd_flags": ["--log-format", "json"]} so the
consumer's retry dbt command emits JSON-formatted logs. Reuses the
same watcher_downstream_not_skipped dbt project and the same fail-once
sequence, so the two DAGs must not run concurrently.
Address @pankajkoti's follow-up nit on #2713: the tuple of flags the consumer's retry must drop -- ``--select``, ``--exclude``, ``--log-format`` -- is now named ``_PRODUCER_ONLY_FLAGS`` at module scope rather than inlined in ``_filter_flags``. The constant carries the rationale for each flag in its docstring, so a future contributor adding (or removing) one only needs to touch the tuple plus the docstring rather than hunt for the literal. Behaviour is unchanged; the existing watcher unit tests still pass. 🤖 Generated with Claude Code (https://claude.com/claude-code)
tatiana
requested review from
a team,
corsettigyg,
dwreeves and
jbandoro
as code owners
tatiana
requested review from
pankajastro and
pankajkoti
and removed request for
a team
pankajkoti
approved these changes
May 21, 2026
tatiana
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Refactors dbt producer-only flag filtering by introducing a module-level constant and simplifying the _filter_flags docstring.
Changes:
- Added
_PRODUCER_ONLY_FLAGSconstant to centralize the list of filtered dbt flags. - Updated
_filter_flagsdocumentation to reference the centralized constant. - Replaced the inline tuple literal with
_PRODUCER_ONLY_FLAGSin the filtering logic.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| """Filter out dbt flags that should not propagate from the producer to the consumer's retry. | ||
|
|
||
| The set of stripped flags is defined by ``_PRODUCER_ONLY_FLAGS`` at module | ||
| scope; see that constant's docstring for the rationale per flag. |
Comment on lines
+404
to
+408
| # default to text. Each entry consumes the following token as its value (e.g. | ||
| # ``--log-format json``). | ||
| _PRODUCER_ONLY_FLAGS: tuple[str, ...] = ("--select", "--exclude", "--log-format") | ||
|
|
||
|
|
| else: | ||
| continue # skip value of previous flag | ||
| if token in ("--select", "--exclude", "--log-format"): | ||
| if token in _PRODUCER_ONLY_FLAGS: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2713 review feedback from @pankajkoti: extract the tuple of dbt flags the watcher consumer must strip from a retry command into a module-level constant.
The constant lives in
cosmos/operators/_watcher/base.py, one block aboveBaseConsumerSensor. Its docstring carries the per-flag rationale (previously buried in_filter_flags's own docstring), so a future contributor adding or removing a flag has one place to look.🤖 Generated with Claude Code (https://claude.com/claude-code)