Skip to content
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 15 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,10 +175,21 @@ push or open a PR.
scheduling (e.g. LLM-cascade / model-routing and queueing/load-balancing
papers).
- **Issue #568 slice:** `contextual_orchestrator.reasoning_effort_profile`
is the provider-neutral role catalog and equal-budget true-θ ablation.
RMSE is computed from θ̂ versus known true parameters, not a rank
constant. Do not change production route/conduct defaults until
`production_default_change_allowed` is true. Temperature is not effort.
retains provider-neutral configuration and legacy synthetic diagnostics.
The synthetic estimates are constructed from supplied true theta using
hand-authored effort/access coefficients; their RMSE is not observed model
quality or evidence of unknown-parameter recovery. The diagnostic-only
`production_default_change_allowed` API always refuses authorization.
An empirical successor needs actual observations, the released Rust/
fast-mlsirm estimation contract, exact learned-policy identity, a validated
evaluation design and separate deployment approval. Moving the remaining
synthetic routines to unit-test fixtures is still parent #1000 work.
Temperature is not effort. For an explicit request profile, only literal
boolean `True` is positive native-effort capability evidence; `False` and
`None` follow the explicit unsupported-provider fallback, and malformed
types are rejected before mutation without coercion. See
`docs/doctoring/learned_policy_authority_20260910.md` and
`docs/doctoring/effort_capability_evidence_20260912.md` for scope and evidence.
<!-- END cwl-agent-guidance -->

## Tool-call handoffs
Expand Down
2 changes: 1 addition & 1 deletion contextual_orchestrator/orchestrator.py
Original file line number Diff line number Diff line change
Expand Up @@ -653,7 +653,7 @@ def __post_init__(self) -> None:
raise TypeError(
f"{field_name} must be a positive integer <= {_AGENT_POOL_INTEGER_MAX} or null"
)
if self.reasoning_effort_supported not in (None, True, False):
if self.reasoning_effort_supported is not None and type(self.reasoning_effort_supported) is not bool:
raise TypeError("reasoning_effort_supported must be true, false, or null")
if type(self.stream_usage_supported) is not bool:
raise TypeError("stream_usage_supported must be a boolean")
Expand Down
97 changes: 52 additions & 45 deletions contextual_orchestrator/reasoning_effort_profile.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
"""Provider-neutral reasoning-effort profiles and equal-budget ablation.

Issue #568: each TRINITY/Conductor workflow role gets an explicit
``reasoning_effort_profile``. Sampling temperature, top-p, and seed stay
independent fields. Production routing defaults stay locked until an
equal-budget ablation beats a predeclared true-θ RMSE threshold.

Buyer next action: parse a versioned profile, bind it to thinker / worker /
verifier / synthesizer / planner / judge, and compare route-versus-conduct
variants under the same token budget before asking to change defaults.
"""Provider-neutral profiles with legacy synthetic diagnostics.

The compatibility ablation below transforms supplied true parameters directly;
it does not fit observations or execute Fugu, TRINITY, or Conductor. Its fixed
role/effort coefficients and token counts are not empirical model evidence.
No report from this module can authorize a production default change. A future
promotion path must validate an independently governed empirical evaluation,
its exact policy/artifact identity, and separate deployment authorization.
"""

from __future__ import annotations
Expand All @@ -20,6 +18,7 @@
from typing import Any

PROFILE_VERSION = "reasoning_effort_profile.v1"
# Historical import compatibility only; this value has no authorization role.
PRODUCTION_RMSE_IMPROVEMENT_THRESHOLD = 0.55
WORKFLOW_ROLES = (
"thinker",
Expand Down Expand Up @@ -62,16 +61,18 @@ def apply_request_profile(
payload: dict[str, Any],
profile: ReasoningEffortProfile | None,
*,
supports_reasoning_effort: bool,
supports_reasoning_effort: bool | None,
default_max_output_tokens: int | None,
) -> dict[str, Any]:
"""Apply one validated profile to an upstream request body.

Sampling controls remain separate from provider-native reasoning effort.
When provider support is not proven, ``abstain`` and ``error`` fail closed;
``omit`` sends only the independently valid sampling and output-token
controls. The helper never writes prompts, credentials, or private
reasoning traces.
controls. Only literal ``True`` is positive capability evidence; ``None``
means unknown. Other types fail before mutation, without truth/equality
coercion or rendering caller-controlled values. The helper never writes
prompts, credentials, or private reasoning traces.

``default_max_output_tokens`` is the caller-resolved output ceiling for the
selected model (the request-scoped/client cap, else the agent's published
Expand All @@ -86,7 +87,9 @@ def apply_request_profile(
if not isinstance(profile, ReasoningEffortProfile):
raise EffortProfileError("effort profile must be a ReasoningEffortProfile")
validated = parse_reasoning_effort_profile(profile.as_dict())
if not supports_reasoning_effort and validated.unsupported_provider_fallback != "omit":
if supports_reasoning_effort is not None and type(supports_reasoning_effort) is not bool:
raise EffortProfileError("supports_reasoning_effort must be a boolean or null")
Comment thread
seonghobae marked this conversation as resolved.
Resolved
if supports_reasoning_effort is not True and validated.unsupported_provider_fallback != "omit":
raise EffortProfileError(
"provider reasoning_effort support is unproven; profile requested "
f"{validated.unsupported_provider_fallback!r}"
Expand All @@ -96,8 +99,19 @@ def apply_request_profile(
payload["top_p"] = validated.top_p
if validated.seed is not None:
payload["seed"] = validated.seed
if supports_reasoning_effort:
if supports_reasoning_effort is True:
payload["reasoning_effort"] = validated.reasoning_effort
else:
# Remove both native spellings without mutating shared nested options.
payload.pop("reasoning_effort", None)
Comment thread
seonghobae marked this conversation as resolved.
Resolved
reasoning_options = payload.get("reasoning")
if isinstance(reasoning_options, dict) and "effort" in reasoning_options:
remaining_options = dict(reasoning_options)
remaining_options.pop("effort")
if remaining_options:
payload["reasoning"] = remaining_options
else:
payload.pop("reasoning", None)
return payload


Expand Down Expand Up @@ -146,11 +160,10 @@ class EffortCatalogSnapshot:

@dataclass(frozen=True)
class ThetaEstimate:
"""Deterministic θ̂ and its RMSE against known true parameters.
"""Legacy synthetic output, not a fitted ability or model-quality estimate.

Buyer next action: compare ``estimated_theta`` to the true vector you
supplied. A lower RMSE from higher effort is evidence; a temperature-only
change is not.
The true parameter enters the construction itself. The reported RMSE only
describes that construction; it cannot validate an actual learned policy.
"""

estimated_theta: tuple[float, ...]
Expand Down Expand Up @@ -245,12 +258,12 @@ def parse_reasoning_effort_profile(raw: Mapping[str, Any] | None) -> ReasoningEf


def default_role_effort_catalog() -> dict[str, ReasoningEffortProfile]:
"""Return the issue #568 role catalog. This is evidence, not a production default.
"""Return the historical issue #568 example, not empirical routing evidence.

Thinker, planner, verifier, and judge use high effort. Worker and synthesizer
use medium effort under the same call/depth/token budget. Buyer next action:
run ``run_equal_budget_ablation`` before asking to install this catalog as
the live ``OrchestrationPolicy``.
obtain governed, observed evaluation evidence before using this example
as a live policy. The synthetic ablation cannot provide that evidence.
"""
shared = {
"max_output_tokens": 256,
Expand Down Expand Up @@ -348,7 +361,8 @@ def estimate_theta(
``θ̂_i = (1 − λ) θ_i`` where λ shrinks as effort, Conductor steps,
recursion depth, and access-list scope increase. Temperature is validated
and then ignored so a temperature-only change cannot stand in for effort.
Buyer next action: assert RMSE uses ``θ̂ − θ``, not a rank constant.
The shrinking error is built into this compatibility construction, not
evidence that a provider or coordination policy became more accurate.
"""
theta: list[float] = []
for value in true_theta:
Expand Down Expand Up @@ -389,10 +403,10 @@ def estimate_theta_rmse(
"""Return RMSE of a deterministic θ estimator against known true parameters.

Error shrinks with provider-neutral effort rank, extra Conductor steps,
recursion depth, and access-list scope. Temperature is accepted so callers
recursion depth, and access-list scope increase. Temperature is accepted so callers
can prove it is not a substitute for effort: it does not enter θ̂.
Buyer next action: treat a lower RMSE from ``high`` effort as evidence,
and a temperature-only change as non-evidence.
Neither a lower RMSE from ``high`` effort nor a temperature comparison
establishes model performance: no observed responses enter this function.
"""
return estimate_theta(
true_theta,
Expand All @@ -415,7 +429,7 @@ def _ablation_arm(
temperature: float,
budget_tokens: int,
) -> dict[str, Any]:
"""Build one equal-budget arm with θ̂, RMSE, and measured token use."""
"""Build a legacy synthetic arm with constructed θ̂ and synthetic token use."""
estimate = estimate_theta(
theta,
reasoning_effort=reasoning_effort,
Expand Down Expand Up @@ -541,22 +555,15 @@ def run_equal_budget_ablation(true_theta: Iterable[float]) -> dict[str, Any]:


def production_default_change_allowed(report: Mapping[str, Any]) -> bool:
"""Return whether a live default change is allowed from this ablation.

Buyer next action: keep current route/conduct defaults when this is false.
A later slice may unlock only after RMSE improvement, a non-estimated
measurement, and robustness all clear the predeclared gate.
"""Keep this diagnostic-only API non-authorizing for every supplied report.

The old implementation trusted editable labels and an arbitrary improvement
threshold. Neither establishes observed-data provenance, equal-budget study
validity, an exact learned-policy artifact, or deployment approval. A GitHub
release receipt also cannot stand in for that empirical evidence. Preserve
the boolean compatibility surface without parsing or executing caller-owned
mapping values. A separately reviewed empirical authorization contract is
required before any successor may return permission to change defaults.
"""
try:
baseline = float(report["single_model_baseline"]["rmse"])
candidate = float(report["role_differentiated"]["rmse"])
except (KeyError, TypeError, ValueError):
return False
if not math.isfinite(baseline) or not math.isfinite(candidate) or baseline <= 0:
return False
if report.get("measurement_status") == "estimated":
return False
if report.get("robustness_passed") is not True:
return False
improvement = (baseline - candidate) / baseline
return improvement >= PRODUCTION_RMSE_IMPROVEMENT_THRESHOLD
del report
return False
Loading
Loading