Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/fuzz.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,9 @@ jobs:
- name: Fuzz model-judge response parser
run: python fuzz/fuzz_model_judge.py -max_total_time="${FUZZ_SECONDS}" -artifact_prefix=crash- fuzz/corpus/judge

- name: Fuzz reasoning-effort profile parser
run: python fuzz/fuzz_reasoning_effort_profile.py -max_total_time="${FUZZ_SECONDS}" -artifact_prefix=crash- fuzz/corpus/reasoning_effort_profile

- name: Upload crash artifacts
if: failure()
uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # actions/upload-artifact@v5
Expand Down
5 changes: 5 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,4 +123,9 @@ push or open a PR.
cost-optimal LLM routing, upstream load balancing, and latency/throughput
scheduling (e.g. LLM-cascade / model-routing and queueing/load-balancing
papers).
- **Issue #568 slice:** `contextual_orchestrator.reasoning_effort_profile`
is the provider-neutral role catalog and equal-budget true-θ ablation.
RMSE is computed from θ̂ versus known true parameters, not a rank
constant. Do not change production route/conduct defaults until
`production_default_change_allowed` is true. Temperature is not effort.
<!-- END cwl-agent-guidance -->
35 changes: 35 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,34 @@

All notable changes to this project are documented in this file.

The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added

- Structured tool failure categories, stable fallback actions, and public adapter exceptions.
- Secret-free `tool_fallback_decision` audit events.
- Exact regression coverage for the Strix `Tool execute_command not found in agent strix` failure.
- Versioned `reasoning_effort_profile` catalog (issue #568) with fail-closed
parse, per-role bindings, replayable snapshot hash, and an equal-budget
true-θ RMSE ablation that emits θ̂ and RMSE(θ̂, θ). Sampling temperature
is not reasoning effort. Production route/conduct defaults stay locked
until `production_default_change_allowed` is true.
Next action: run `python -m pytest -q tests/test_reasoning_effort_profile.py` and keep
live defaults unchanged while the gate is false. Pass
`role_effort_catalog=default_role_effort_catalog()` to attach the same
`reasoning_effort_snapshot` on `complete`, `run`, `stream_route`, and
`batch_route`; omit it to keep today's payload.

### Fixed

- Reject missing profiles, blank `profile_version`, and fractional seeds.
Snapshot hashing now fails closed on extra or missing roles. The
production-default gate returns false on junk reports and on
`measurement_status=estimated`. Access-list scope is a real ablation
factor, not a duplicate label.

### Changed

Expand All @@ -21,3 +42,17 @@ All notable changes to this project are documented in this file.

- Ambiguous non-idempotent outcomes, invalid arguments, permission denial, and policy denial fail closed.
- Fallback errors and audit events do not copy provider exception text, tool arguments, outputs, or credentials; fail-closed exceptions also sever the original cause chain so later traceback logging cannot recover them.

### References

- Sakana AI. (2026). *Sakana Fugu Technical Report*.
https://github.com/SakanaAI/fugu/blob/main/Fugu_technical_report.pdf
- Xu, J., Sun, Q., Schwendeman, P., Nielsen, S., Cetin, E., & Tang, Y. (2025).
*Trinity: An evolved LLM coordinator* (arXiv:2512.04695).
https://arxiv.org/abs/2512.04695
- Nielsen, S., Cetin, E., Schwendeman, P., Sun, Q., Xu, J., & Tang, Y. (2025).
*Learning to orchestrate agents in natural language with the Conductor*
(arXiv:2512.04388). https://arxiv.org/abs/2512.04388
- Baker, F. B. (2001). *The basics of item response theory* (2nd ed.).
ERIC Clearinghouse on Assessment and Evaluation.
https://eric.ed.gov/?id=ED458219
1 change: 1 addition & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ A stdlib-Python lab implementing a single OpenAI-compatible API that routes, del
### Modules (`contextual_orchestrator/`)

- `orchestrator.py` — the domain heart: `ModelAgent`, `WorkflowStep`, `OrchestrationPolicy`, `ModelClient`, `TaskOrchestrator`, secret/PII redaction, budget enforcement, spend analytics, and the commercial-readiness report generators behind `/api/v1/*`. Domain code stays here until a second implementation forces extraction (see `docs/code_conventions.md`).
- `reasoning_effort_profile.py` — issue #568 role compute catalog and equal-budget true-θ ablation that emits θ̂ and RMSE(θ̂, θ). Production route/conduct defaults stay locked until `production_default_change_allowed`. Buyer next action: `python tests/test_reasoning_effort_profile.py`.
- `server.py` — HTTP delivery adapter and `SecurityConfig`; all request validation lives here.
- `admin.py` — static HTML/CSS/JS for the `/admin` operator console (stays inline while the product is dependency-free).
- `credentials.py` / `kv_config.py` — the KV seam: `get_credential`/`register_credential` over pluggable backends (`InMemoryCredentialBackend` default; pgcrypto-encrypted `PostgresCredentialBackend`, selected via `CONTEXTUAL_ORCHESTRATOR_KV_BACKEND`).
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,7 @@ python -m pip install --require-hashes -r requirements.lock
python -m pip install --no-deps -e .
python tests/test_self_check.py
python tests/test_paper_contracts.py
python -m pytest -q tests/test_reasoning_effort_profile.py
python tests/test_admin_contract.py
python tests/test_conventions.py
python tests/test_api_contract.py
Expand Down
5 changes: 1 addition & 4 deletions conductor/tracks.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,4 @@
|---|---|---|
| 001-paper-grounded-orchestrator | active | Implement the source-backed orchestration contract with TDD, DDD, and CDD |
| 002-enterprise-design-foundation | active | Add paper-grounded screen design, user stories, REST API, code/DB conventions, and i18n |
| 003-autonomous-pr-ecosystem-loop | active | Standing autonomous review→fix→merge loop across every owned ContextualWisdomLab repo, plus org-ecosystem product-gap work |
| 003-sdk-omit-real-persist | done | #668's SDK-omit honesty behavior (args/instructions/metadata write-back, chat top_logprobs hoist before tools passthrough) integrated via the http-honesty stack merge (#740) on `integrate/http-honesty-740`, after the smaller #686/#687 re-land attempts went stale and closed unmerged. |
| 003-compatibility-honesty | done | Fail-closed ASCII `[a-zA-Z0-9_-]{1,64}` on `json_schema.name`/`tool.function.name` integrated via the same #740 merge. |
| 003-responses-text-format | done | Official Responses text.format structured types integrated via the same #740 merge. |
| 003-reasoning-effort-profiles | active | Issue #568: versioned per-role `reasoning_effort_profile`, equal-budget θ̂ RMSE ablation, snapshot on run/stream/batch, production defaults locked |
14 changes: 14 additions & 0 deletions contextual_orchestrator/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,14 @@
from .credentials import NotConfigured, get_credential, register_credential
from .kv_config import InMemoryConfigStore, get_config_store
from .orchestrator import ModelAgent, TaskOrchestrator, WorkflowStep, load_agents
from .reasoning_effort_profile import (
EffortProfileError,
ReasoningEffortProfile,
apply_request_profile,
default_role_effort_catalog,
parse_reasoning_effort_profile,
snapshot_role_effort_catalog,
)
from .token_counting import HeuristicTokenCounter, build_token_counter
from .tool_fallback import (
MAX_TOOL_RETRY_ATTEMPTS,
Expand All @@ -54,6 +62,12 @@
"TaskOrchestrator",
"WorkflowStep",
"load_agents",
"ReasoningEffortProfile",
"EffortProfileError",
"apply_request_profile",
"default_role_effort_catalog",
"parse_reasoning_effort_profile",
"snapshot_role_effort_catalog",
"get_credential",
"register_credential",
"NotConfigured",
Expand Down
Loading
Loading