Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 100 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,85 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)

### Fixed

- Added the missing CLI on-ramp for the issue #568 / ADR 0021 per-role
reasoning-effort catalog: the new `--role-effort-catalog default` flag now
loads `default_role_effort_catalog()` and passes it into `TaskOrchestrator`.
Previously `role_effort_catalog` had no caller anywhere in `__main__.py`, so
`apply_effort_profile` always ran its `profile=None` no-op branch in the
shipped CLI/server and the catalog's temperature/top_p/seed/reasoning_effort
injection and replayable `reasoning_effort_snapshot` were unreachable in
production. Omitting the flag keeps every payload byte-for-byte unchanged;
this does not touch the locked route/conduct selection defaults. Research
grounding for `reasoning_effort_profile` (Fugu's latency-versus-quality
split, TRINITY roles, Conductor steps/access lists) is already cited in
[`docs/architecture.md`](docs/architecture.md#implementation-mapping) next to this
module; this CLI on-ramp reuses that existing catalog and cites no new
claims of its own.
- Startup now refuses `--role-effort-catalog default` when no agent in the
configured pool proves native `reasoning_effort` support (explicit
`"reasoning_effort_supported": true`, or a `mock://` agent). Every role in
the default catalog fails closed (`unsupported_provider_fallback="abstain"`,
per ADR 0021), and ordinary real-provider agent configs and auto-discovered
agents never set `reasoning_effort_supported` — without this guard the flag
would construct successfully and then raise `EffortProfileError` on every
request. `--agents examples/agents.mock.json` (the CLI default) is
unaffected; only a non-mock pool without proven support is now rejected at
startup, before any request is attempted.
- The startup guard above only proved that *some* agent in the pool supports
`reasoning_effort`; ordinary role-based selection (`route_once`, `conduct`,
`stream_route`, `batch_route`, structured-synthesis passthrough) could still
rank or select an *unsupported* agent from a mixed pool ahead of a
supported one. `route_once`/`conduct` already recovered via `_invoke`'s
generic tool-failure failover, but `stream_route` and `batch_route` call
the provider directly with no failover and would raise `EffortProfileError`
outright. `TaskOrchestrator._ranked_agents` now narrows role-based
candidates to agents that prove `reasoning_effort` support whenever the
role's `role_effort_catalog` entry fails closed
(`unsupported_provider_fallback` other than `"omit"`), falling back to the
unfiltered set only when no candidate in that narrower selection proves
support (e.g. a required tag or free-only filter excluded the pool's only
supporting agent). This mirrors the equivalent filter already applied on
`proxy_completion`'s passthrough failover, which now shares the same
`_eligible_role_effort_candidates` helper instead of a duplicated inline
check. Runtime agent patch, removal, and discovery synchronization now
revalidate the same catalog invariant before committing a pool mutation,
so a live server cannot disable or remove its last eligible supporting
agent and strand a fail-closed role after startup.
- The startup guard above still only proved that *some enabled agent
anywhere* in the pool supports `reasoning_effort`, which is weaker than
what role-based selection actually requires: `_ranked_agents`/
`_select_agent` only ever offer a role a *general-chat* agent (see
`_is_general_chat_agent`) that is not excluded from that specific role via
`provider_exclusions`. A pool whose only proving agent was non-chat (e.g.
an embedding-only model with `reasoning_effort_supported: true`), or was
excluded from every active fail-closed role, passed the old check and
still failed every request for that role (`EffortProfileError`, or
`RuntimeError("no eligible agent available for role=...")` when the sole
prover was role-excluded). `_require_eligible_role_effort_agents` now
reapplies the same general-chat and `provider_exclusions` eligibility
rules used by role selection, per active fail-closed role, reusing
`agent_proves_reasoning_effort_support`/`_eligible_role_effort_candidates`
rather than new logic. Deliberately does not reapply
`_zdr_agent_allowed`, since that gate is per-request privacy-policy state,
not a static pool property.
- The startup rejection path now closes the already-constructed
`TaskOrchestrator` (`orchestrator.close()`) before calling argparse's
`parser.error()` (which raises `SystemExit`), instead of relying on
process exit to release its optional durable resources
(`state_db`/`agents_db` connections). This only matters for an embedded
caller of `contextual_orchestrator.__main__.main()` that catches
`SystemExit` and keeps running in the same process; the CLI/process-exit
path was already unaffected.
- `TaskOrchestrator.proxy_completion`'s single-agent passthrough (the
server's `tool_loop` call site, and any other caller that omits
`effort_profile`) now defaults an unset `effort_profile` to the opted-in
`role_effort_catalog`'s `"worker"` entry — the role every selection/
failover call in that method already uses — instead of silently skipping
sampling/token/seed/reasoning-effort injection. This mirrors the
pre-existing `effort_profile or self._role_effort_profile("synthesizer")`
fallback `_orchestrated_provider_completion` already applies for its own
role. A caller that passes its own `effort_profile` is unaffected, and so
is every caller when no `role_effort_catalog` is configured.
- Batch cost retrieval now preserves cache hits, completed endpoint-race loser
usage, and strict provider usage evidence. Remote fallbacks retain the real
submitted prompt, repeated result retrieval reuses deterministic ledger ids,
Expand Down Expand Up @@ -203,6 +282,26 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
retried as if it were a network blip. Fixes the shared classifier itself
(not just the discovery retry call site), so every current and future
caller of `is_transient_error` benefits.
<<<<<<< HEAD
- (Devin review on #958) `_orchestrated_provider_completion`'s structured/
Responses synthesis path (the `single_agent=False` branch of
`proxy_completion`) resolved its caller-supplied `effort_profile` override
only for the final `apply_effort_profile` payload call, not for the
synthesizer's own selection (`_select_agent`), replica lookup
(`_ranked_agents`), or failover list (`_failover_candidates`) — those three
call sites omitted `effort_profile` entirely and so silently fell back to
the raw `role_effort_catalog` entry inside `_ranked_agents`. A fail-closed
override (`unsupported_provider_fallback` other than `"omit"`) could still
rank/select an unproven-support agent ahead of a proven one, and
`apply_effort_profile` then raised `EffortProfileError` outright — with no
failover, unlike the identical scenario on the plain passthrough path,
which already threads its override through every selection call site. Now
resolves `effort_profile or self._role_effort_profile("synthesizer")` once,
up front, and passes it to all three call sites, matching the passthrough
path's existing pattern and `_ranked_agents`' own documented intent that
every role-based selection path -- "structured synthesis" included -- stay
consistent with the effort catalog's eligibility guard.
=======
Comment on lines +285 to +304

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Unresolved changelog conflict

The release notes still contain <<<<<<<, =======, and >>>>>>> markers. Resolve the conflict before merging.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

- (CodeRabbit review on #946) **Credential leak via cross-host redirect,
doubled by #923's retry.** `_fetch_json` -- the function every standard
provider's authenticated "list models" call goes through (openai,
Expand Down Expand Up @@ -330,6 +429,7 @@ and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
otherwise. `_write_sse` relies on `_begin_sse`'s already-set marker rather
than touching it itself, since it is only ever called after a prior
successful header flush.
>>>>>>> origin/main

### Added

Expand Down
97 changes: 96 additions & 1 deletion contextual_orchestrator/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
from .privacy_policy_analysis import (
analyze_discovered_privacy_policies,
)
from .reasoning_effort_profile import default_role_effort_catalog
from .server import DEFAULT_MAX_JSON_BODY_BYTES, SecurityConfig, serve

DEFAULT_AUTH_CREDENTIAL_NAME = "CONTEXTUAL_ORCHESTRATOR_TOKEN"
Expand Down Expand Up @@ -198,6 +199,71 @@ def _positive_int(value: str) -> int:
return parsed


def _require_eligible_role_effort_agents(
orchestrator: TaskOrchestrator,
parser: argparse.ArgumentParser,
agents_source: str,
) -> None:
"""Fail fast when an opted-in role-effort catalog cannot be honored.

``default_role_effort_catalog()`` fails closed
(``unsupported_provider_fallback="abstain"``) for every role by design
(ADR 0021): native ``reasoning_effort`` is sent only to a provider that
proves support, and an unproven provider must not silently receive it
either. Ordinary real-provider agent configs (e.g.
``examples/agents.openai.json``) and every auto-discovered agent never
set ``reasoning_effort_supported``, so without this check
``--role-effort-catalog default`` would construct successfully and then
raise ``EffortProfileError`` on every subsequent request. Reject at
startup instead, with an actionable fix, rather than after the pool is
already live for ``--serve`` or a CLI prompt.

Proving support somewhere in the pool is not sufficient on its own:
role-based selection (``TaskOrchestrator._ranked_agents`` /
``_select_agent``) only ever offers one *general-chat* agent (see
``_is_general_chat_agent``) that is not excluded from that specific role
via ``provider_exclusions`` for a given role. A pool whose only proving
agent is non-chat (e.g. an embedding-only model with
``reasoning_effort_supported: true``), or is excluded from every active
fail-closed role, would pass a pool-wide "any agent anywhere proves
support" check and still fail every request for that role at request
time -- either ``EffortProfileError`` (no proving candidate was ever
offered) or ``RuntimeError("no eligible agent available for
role=...")`` (the only proving candidate was role-excluded). So this
delegates to the orchestrator's shared pool-invariant check, which applies
the same general-chat, ``provider_exclusions``, and native-support rules
both here and on later runtime pool mutations. A pool that passes this
guard has at least one eligible, proving agent for each active fail-closed
role. The invariant deliberately does NOT reapply ``_zdr_agent_allowed``: that gate
depends on per-request privacy-policy state
(``request_policy(zdr_only=...)``), not a static pool property, so it
cannot be evaluated once at startup without assuming every future
request's privacy scope.
"""
catalog = orchestrator.role_effort_catalog
if catalog is None:
return
unsupported_roles = orchestrator._unsupported_role_effort_roles()
if not unsupported_roles:
return
orchestrator.close()
parser.error(
"--role-effort-catalog default requires, for every fail-closed role "
f"({', '.join(unsupported_roles)}), at least one enabled general-chat "
f"agent in --agents {agents_source!r} that both proves native "
"reasoning-effort support and is not excluded from that role via "
"provider_exclusions: set \"reasoning_effort_supported\": true on a "
"chat-capable agent (not an embedding/rerank/transcription-only "
"model) that is not listed in that role's provider_exclusions, or "
"point --agents at a mock:// pool (e.g. examples/agents.mock.json) "
"for evaluation. Every role in the default catalog fails closed "
"(unsupported_provider_fallback='abstain') without that proof, so "
"every request for an unsupported role would otherwise raise "
"EffortProfileError or find no eligible agent. See ADR 0021 "
"(docs/planning/adrs/0021-reasoning-effort-profiles.md)."
)


def _non_negative_int(value: str) -> int:
"""Parse a non-negative integer for an argparse option (0 means "off")."""
try:
Expand Down Expand Up @@ -766,6 +832,25 @@ def main(argv: list[str] | None = None) -> None:
action="store_true",
help="discover source-declared chat-capable models at startup and activate them",
)
parser.add_argument(
"--role-effort-catalog",
choices=["default"],
default=None,
help=(
"Opt in to the issue #568 per-role reasoning-effort catalog (ADR 0021). "
"'default' loads default_role_effort_catalog(), applying each workflow "
"role's temperature/top_p/seed/max_output_tokens and (only where a provider "
"proves support) native reasoning_effort, and attaching a replayable "
"reasoning_effort_snapshot to complete/run/stream_route/batch_route "
"results. Omit to keep today's payload unchanged -- this does not "
"change route/conduct selection defaults, which stay locked until "
"production_default_change_allowed is true. Every role in 'default' "
"fails closed for a provider that has not proven support, so at "
"least one --agents entry needs \"reasoning_effort_supported\": "
"true (or a mock:// base_url) -- startup refuses the flag "
"otherwise."
),
)
_add_log_level_arguments(parser)
args = parser.parse_args(arguments)

Expand All @@ -785,9 +870,19 @@ def main(argv: list[str] | None = None) -> None:
budget_max_output_tokens=args.budget_max_output_tokens,
budget_max_cost_usd=args.budget_max_cost_usd,
cache_ttl=args.cache_ttl,
role_effort_catalog=(
default_role_effort_catalog() if args.role_effort_catalog == "default" else None
),
Comment thread
seonghobae marked this conversation as resolved.
Resolved
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
Resolved
)
if args.auto_discover_model_agents:
_auto_discover_runtime_agents(orchestrator)
try:
_auto_discover_runtime_agents(orchestrator)
except ValueError as exc:
orchestrator.close()
parser.error(str(exc))

if args.role_effort_catalog is not None:
_require_eligible_role_effort_agents(orchestrator, parser, args.agents)
Comment thread
seonghobae marked this conversation as resolved.
Resolved
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
Resolved

if args.no_realtime_judge:
orchestrator.policy = replace(orchestrator.policy, realtime_judge=False)
Expand Down
Loading
Loading