diff --git a/.claude/rules/grade-layer.md b/.claude/rules/grade-layer.md index 5bca2d90..d9a27589 100644 --- a/.claude/rules/grade-layer.md +++ b/.claude/rules/grade-layer.md @@ -12,12 +12,21 @@ The grade layer sits between the prune engine (#6) and the diff renderer (#8). I - **Degraded:** `score: None, passed: False, evidence: "", reasoning: ""`. Three causes route here: 1. `LLMError` retries exhausted → `reasoning="call failed: GradeLLMError"`. **Also covers a provider-specific safety-filter / no-content response** (Gemini's `finish_reason ∈ {SAFETY, RECITATION, OTHER, ...}` with empty parts is the v0.3 example — `GeminiProvider.extract_text_blocks` raises a typed `LLMResponseFormatError` per DEC-005 of #137, which propagates as an `LLMError` and lands here). The contract is provider-neutral: a future vendor with a content-filter surface MUST route through `LLMResponseFormatError` so the conservative degrade fires uniformly — `grade-artifacts` does NOT switch on provider name. 2. `GradeOutputError` (parser failure / anchor-contract failure) → `reasoning="call failed: GradeOutputError"`. - 3. `total_budget_seconds` exceeded → `reasoning="grade budget exceeded ..."`. + 3. The **effective wall-clock budget** exceeded → `reasoning="grade budget exceeded ({effective_budget}s) before evaluation"` (the value is the *computed effective* budget, not the raw config field — see § "Scaled wall-clock budget + opt-in cost ceilings (#198)"). + 4. An **opt-in cost/calls/tokens ceiling** tripped (#198) → `reasoning` ∈ {`"grade call ceiling exceeded ({N} calls)"`, `"grade cost ceiling exceeded (${X})"`, `"grade token ceiling exceeded ({N} tokens)"`}. Ceilings **degrade, never raise** — no new typed error, no exit-code/AST-scan churn; `GradeBudgetExceededError` stays reserved. Aggregate `pass_rate` and `mean_score` are computed over the **scored** subset only. `aggregate_complete: bool` is `True` iff every result was scored. **Load-bearing invariant: graceful degrade, never silent drop.** Operators check `aggregate_complete` to know if the report is partial. The whole run only aborts when the **audit itself** fails (`GradeAuditWriteError` / `GradeAuditRecordTooLargeError`). A partial audit is worse than no audit. +## Scaled wall-clock budget + opt-in cost ceilings (#198) + +The grade wall-clock budget **scales with the work**, not a flat constant. `_compute_effective_budget(*, budget_base_seconds, budget_per_pair_seconds, total_budget_seconds, num_pairs, max_concurrent_calls) -> int` (a pure module-level helper in `signalforge.grade.engine`, unit-tested without asyncio) computes `effective = budget_base_seconds + budget_per_pair_seconds × ceil(num_pairs / max_concurrent_calls)`; when `total_budget_seconds` (now `int | None`, default `None`) is set, `effective = min(scaled, total_budget_seconds)`. `effective` is always a finite positive int (`base ≥ 1`), so `asyncio.timeout(effective)` never receives `None`/0. Defaults `budget_base_seconds=60`, `budget_per_pair_seconds=20.0` are grounded in the #179 benchmark (~10s/Sonnet-judge-call; 220 pairs @ concurrency 10 → ~223s observed → ~500s budget at ~2.25× headroom — a runaway backstop, **not** a completion target). Setting `total_budget_seconds: ` preserves exact v0.1 absolute-cap semantics for pinned configs. + +**Three opt-in ceilings, all `None`=off (DEC-002/003 of #198):** `max_grade_calls` / `max_grade_cost_usd` / `max_grade_tokens`. Soft/best-effort: all pairs dispatch into the one `TaskGroup`, so a ceiling stops scheduling **new** pairs and lets in-flight calls finish (overshoot ≤ `max_concurrent_calls − 1`). `max_grade_calls` is near-hard — the slot is **reserved by incrementing a shared counter inside `_one`, immediately after `async with semaphore:` and BEFORE the first `await`** (the single event loop makes check-then-increment race-free only with no intervening await). `cost_usd`/`tokens` accumulate **after** each call returns; cache-hit pairs (#189) bypass `_one` entirely so they never count. A tripped pair degrades via `_build_degraded`, sets its slot + `completed` counter FIRST (so the synthesis pass skips it — no double-audit), then writes the GradeEvent through the **same `run_in_executor` + `asyncio.shield` path as the happy-path per-pair write** (DEC-017) — the ceiling degrade runs *inside* the concurrent `TaskGroup` region, so a synchronous fsync would block the loop and stall sibling in-flight coroutines (unlike the sequential synthesis pass, which writes sync). Ceiling-degrades count as `completed` (preserving `completed + degraded == total_pairs`). One end-of-run WARNING fires per tripped run: `"grade ceiling exceeded: {run_id, model_unique_id, ceiling, limit, completed_count, degraded_count}"` (`ceiling ∈ {calls, cost_usd, tokens}`), recording the **first** ceiling tripped. The wall-clock budget WARNING's field was renamed `total_budget_seconds` → **`effective_budget_seconds`** (carries the computed value). No `audit_schema_version` bump (GradeEvent shape unchanged; degrade reasons ride the existing `reasoning` field; no full-config-hash on GradeEvent). + +**Cost-ceiling pricing resolves ONCE up front (#198 QG).** `_validate_model_provider_compat` checks only the SKU *prefix* (`claude-`/`gpt-`/`gemini-`), NOT membership in `pricing.PRICES` — so a prefix-valid-but-unpriced SKU (a newer Opus, a typo) is accepted at config-load. The engine resolves `lookup(resolved_config.model)` **once before the `TaskGroup`** (when `max_grade_cost_usd` is set) and reuses it per pair; an unpriced SKU therefore raises `EstimateUnknownModelError` (CLI tier 2) **fast at orchestrator entry — before any billable call** — rather than aborting mid-run from inside a coroutine (uncaught by the per-pair `except` → `BaseExceptionGroup` → abort after paid calls). **Lesson: when a per-pair loop needs a lookup that can fail on operator-supplied config, hoist it to fail fast before the paid fan-out; "validator passed" ≠ "every downstream table has the key" (mirrors #187's registered-but-absent-key lesson).** + ## Fail-closed JSONL + sidecar JSON, both end-of-write durable (DEC-006, DEC-012) Two writers in `signalforge.grade.audit`, both following the project's fail-closed pattern (fourth shipped instance — safety / draft / prune / grade): @@ -130,7 +139,7 @@ Every `extra="ignore"` production model — `GradingResult`, `GradingReport`, `G ## `signalforge.yml` top-level namespace: `grade:` (DEC-029) -The grade-stage block is `{ grade: { model, cache_ttl, max_output_tokens, max_retries_*, total_budget_seconds, min_pass_rate, min_mean_score, fail_on_below_threshold, rubric? } }`. Sibling top-level keys are reserved and silently ignored by the grade loader. `GradeConfig` uses `extra="forbid"`; `_GradeConfigFile` uses `extra="ignore"` at the top level. Mirrors the other layers' top-level-namespace pattern verbatim. +The grade-stage block is `{ grade: { model, provider, cache_ttl, max_output_tokens, max_retries_*, max_concurrent_calls, budget_base_seconds, budget_per_pair_seconds, total_budget_seconds, max_grade_calls, max_grade_cost_usd, max_grade_tokens, min_pass_rate, min_mean_score, fail_on_below_threshold, cache_enabled, rubric? } }` (`budget_base_seconds` / `budget_per_pair_seconds` / `max_grade_calls` / `max_grade_cost_usd` / `max_grade_tokens` added by #198; `total_budget_seconds` reinterpreted as an optional absolute cap). Sibling top-level keys are reserved and silently ignored by the grade loader. `GradeConfig` uses `extra="forbid"`; `_GradeConfigFile` uses `extra="ignore"` at the top level. Mirrors the other layers' top-level-namespace pattern verbatim. ## Locked defaults: per-provider fast model + 1024 output cap (DEC-026, #187) diff --git a/docs/grade-ops.md b/docs/grade-ops.md index 510f59ea..39236999 100644 --- a/docs/grade-ops.md +++ b/docs/grade-ops.md @@ -93,7 +93,7 @@ on a `↳ Remediation:` line by `__str__`. - **`GradeConfigError`** — `signalforge.yml` `grade:` block failed parse or schema validation. - **`GradeRubricError`** — Rubric YAML structurally invalid (duplicate `id`, empty rubric, malformed criterion entry). - **`GradeLLMError`** — One-level adapter wrapping `signalforge.llm.LLMError`. The original error is preserved on `__cause__` and exposed via the `cause` attribute. -- **`GradeBudgetExceededError`** — `total_budget_seconds` tripped before any criterion was graded (a hard "the run did nothing" failure). A partial run completes normally with a `GradingReport` whose `aggregate_complete` flag is `False`. +- **`GradeBudgetExceededError`** — **Reserved; NOT raised in v0.1.** The engine never raises on budget exhaustion: every un-evaluated pair degrades (`score=None`) and a partial run completes normally with a `GradingReport` whose `aggregate_complete` flag is `False`. The class is reserved for a future hard "the run did nothing" failure (the budget trips *before the first pair* is graded) — see `.claude/rules/grade-layer.md` § "Schema-version surfaces". - **`GradePromptEnvelopeBreachError`** — Artefact payload contained the literal `` close tag. Refuses to render rather than ship a degraded envelope. Mirrors the drafter's `PromptEnvelopeBreachError` (#5 DEC-007). - **`GradeOutputError`** — LLM-judge response failed parse or anchor-contract validation. Carries `violation_type: GradeOutputViolationType`. - **`GradeAuditWriteError`** — Fail-closed audit-write failure (`OSError` / `PermissionError` / encoding / `fsync` / symlink containment). Aborts the run; original cause exposed via `.cause` and `__cause__`. @@ -123,7 +123,12 @@ grade: max_retries_429: 3 # Rate-limit retry budget max_retries_5xx: 1 max_retries_conn: 1 - total_budget_seconds: 300 # Wall-clock budget across the whole run + budget_base_seconds: 60 # scaled-budget constant term (#198) + budget_per_pair_seconds: 20.0 # scaled-budget per concurrency-wave allowance (#198) + # total_budget_seconds: 300 # OPTIONAL absolute hard cap; omit (default None) to use the scaled formula alone, set to cap via min(scaled, this) + # max_grade_calls: 500 # opt-in soft ceiling: stop scheduling new pairs after N judge calls (rest degrade) + # max_grade_cost_usd: 1.50 # opt-in soft ceiling: stop once accumulated USD meets/exceeds this (rest degrade) + # max_grade_tokens: 2000000 # opt-in soft ceiling: stop once total token movement meets/exceeds this (rest degrade) max_concurrent_calls: 10 # In-flight LLM calls (range [1, 100]); 1 = v0.1 sequential min_pass_rate: 0.7 # Aggregate threshold: fraction of passed criteria min_mean_score: 0.5 # Aggregate threshold: mean score across criteria @@ -165,7 +170,10 @@ Field-by-field: - **`cache_ttl`** — `Literal["5m", "1h"]`. Default `"1h"` (vs. the drafter's `"5m"`) because 60 sequential per-criterion calls under retry backoff can stretch beyond a 5-minute window; `"1h"` gives margin at no extra cost (cache writes are one-shot regardless of TTL). - **`max_output_tokens`** — Per-criterion judge response cap. Default `1024` (#187 — raised from 256 to substantially reduce truncation risk for a verbose one-line `gemini-2.5-flash` grade JSON; the expected JSON response is still ~150 tokens, so the larger ceiling costs nothing on the happy path). 1024 reduces but does not fully eliminate Gemini truncation at scale — see the per-provider floors below; Gemini-heavy runs may want `4096`. Independent of `DraftConfig.max_output_tokens`. - **`max_retries_429` / `max_retries_5xx` / `max_retries_conn`** — Per-call retry budgets at the centralised, provider-neutral `signalforge.llm.call_llm` seam (#5 DEC-012; #135 DEC-005). Defaults `3 / 1 / 1` mirror `DraftConfig`; dial down for batch CLI mode where one retry-exhaustion is preferable to dozens of stalled calls. -- **`total_budget_seconds`** — Whole-run wall-clock budget. Default `300` (5 minutes — historically ~3× safety on 60 sequential calls × 1s p50; ~10× headroom under concurrent dispatch). Mirrors `PruneConfig.total_budget_seconds` semantics: when the budget trips, every remaining `(artefact, criterion)` pair lands as a degraded `GradingResult(score=None)` rather than silently dropped. Under the asyncio orchestrator (issue #186) the budget is enforced via `asyncio.timeout(...)` wrapping the `TaskGroup`; on trip, un-completed pairs are filled in by a synthesis pass with `reasoning="grade budget exceeded ({N}s) before evaluation"`. Tests inject deterministic timing via the module-level `_async_sleep` alias (mirrors the `_sleep` injection pattern from `llm-drafter.md` DEC-004). +- **`budget_base_seconds`** — Fixed constant term in the scaled wall-clock formula (issue #198, default `60`). Covers per-run setup (config resolution, cache priming, the first concurrency wave's ramp) that does not scale with the number of pairs. Must be positive. +- **`budget_per_pair_seconds`** — Per concurrency-*wave* wall allowance in the scaled formula (issue #198, default `20.0`). The formula multiplies this by `ceil(num_pairs / max_concurrent_calls)` — the number of concurrency waves, not the raw pair count — so it is the wall-clock allowance per wave of `max_concurrent_calls` in-flight judge calls. The default is grounded in the #179 baseline (Sonnet judge p50 ~10s/call; 220 pairs at concurrency 10 → `60 + 20.0 × ceil(220/10) = 500s` against a measured 222.9s — ~2.25× headroom). It is a **runaway backstop** sized to tolerate 429 retry storms, **NOT** a completion target; the ticket-literal `2.0` would compute 104s and degrade ~half the pairs, recreating the failure this scaling fixes. Must be positive. +- **`total_budget_seconds`** — **Optional** absolute hard ceiling on the whole-run wall-clock budget (issue #198 DEC-001; reinterpreted from the flat pre-#198 default of `300`). **Default `None`.** When `None`, the engine sizes the budget from the work via the scaled formula `effective = budget_base_seconds + budget_per_pair_seconds × ceil(num_pairs / max_concurrent_calls)` — a backstop that grows with model width and concurrency rather than a flat 300s the pre-#186 sequential era was sized for. When set to an int, the effective budget is `min(scaled, total_budget_seconds)` — i.e. an explicit value still acts as a hard cap on top of the scaled estimate, preserving exact v0.1 absolute-cap semantics for pinned `signalforge.yml` files (an operator who set `total_budget_seconds: 600` keeps that 600s ceiling). Mirrors `PruneConfig.total_budget_seconds` degrade semantics: when the budget trips, every remaining `(artefact, criterion)` pair lands as a degraded `GradingResult(score=None)` rather than silently dropped (DEC-015). Under the asyncio orchestrator (issue #186) the budget is enforced via `asyncio.timeout(effective)` wrapping the `TaskGroup`; on trip, un-completed pairs are filled in by a synthesis pass with `reasoning="grade budget exceeded ({effective}s) before evaluation"`. Tests inject deterministic timing via the module-level `_async_sleep` alias (mirrors the `_sleep` injection pattern from `llm-drafter.md` DEC-004). +- **`max_grade_calls` / `max_grade_cost_usd` / `max_grade_tokens`** — Three **opt-in soft ceilings** on the grade run (issue #198 DEC-002). **All default `None` (off).** When set, whichever ceiling trips *first* stops scheduling **new** `(artefact, criterion)` pairs; the remaining pairs **DEGRADE** (`score=None`, never raise — mirroring the DEC-015 conservative-degrade contract) with a `reasoning` string naming the tripped ceiling. **Accounting:** `max_grade_calls` counts LLM judge calls only (cache hits make no call → never counted); `max_grade_cost_usd` accumulates the full per-call USD incl. cache-read/write economics, computed from per-call token usage via `signalforge.llm.pricing`; `max_grade_tokens` accumulates all token movement (input + output + cache-creation + cache-read). Each must be positive when set. **Soft / best-effort overshoot:** because every pair is dispatched into the one `TaskGroup` and cost/tokens are known only *after* a call returns, the cost/token ceilings stop only *un-started* pairs — up to `max_concurrent_calls − 1` in-flight calls may complete past the threshold. `max_grade_calls` is near-hard: it reserves a dispatch slot (increments a shared counter immediately, before the LLM `await`) so it stops at most one call over the limit in practice. - **`max_concurrent_calls`** — Number of in-flight `(artifact × criterion)` LLM calls allowed concurrently (issue #186). Default `10` matches the typical Anthropic-tier throughput sweet-spot; bounded `[1, 100]` with `@field_validator` rejecting `< 1` or `> 100` at config-load. Setting `1` yields v0.1 sequential behaviour bit-for-bit (semaphore-of-1 serialises in dispatch order, preserving `(criterion, artifact)` JSONL ordering). Under concurrent dispatch the audit JSONL lands in **arrival order** (`audit_schema_version` unchanged at `Literal[1]`); the `tests/grade/_helpers.py::_sort_grade_events(lines)` helper restores deterministic ordering for tests that snapshot the file. CLI does not expose a `--max-concurrent-calls` flag (mirrors `min_pass_rate` / `min_mean_score` config-file-only convention). - **`min_pass_rate`** — Floor on the fraction of `(artefact, criterion)` pairs that scored `passed=True` for the rubric to count as passed overall. Default `0.7`. Bounded `[0.0, 1.0]`. Mirrors `GradeThresholds.min_pass_rate`. - **`min_mean_score`** — Floor on the mean numeric score across non-null verdicts. Default `0.5`. Bounded `[0.0, 1.0]`. Mirrors `GradeThresholds.min_mean_score`. @@ -513,7 +521,14 @@ passed=False, reasoning="..."` are unchanged: 1. `LLMError` retries exhausted (including a provider-specific safety- filter / no-content response routed via `LLMResponseFormatError`). 2. `GradeOutputError` (parser failure or anchor-contract failure). -3. `total_budget_seconds` exceeded. +3. The effective wall-clock budget (scaled formula, optionally capped by + `total_budget_seconds`) exceeded. + +(The opt-in `max_grade_calls` / `max_grade_cost_usd` / `max_grade_tokens` +ceilings — issue #198 — also degrade un-started pairs, but they are a +separate operator-chosen surface, not a fourth automatic trigger; they +degrade with a `reasoning` naming the tripped ceiling and emit a distinct +`grade ceiling exceeded` WARNING — see [Debugging](#debugging).) A fourth trigger for "vacuous bound" would conflate "we could not evaluate" with "we evaluated and the result was weak" — two different @@ -791,15 +806,35 @@ DEC-004 of the plan): ordering, exactly the kind of loose-contract surface the safety / draft layers' anchor contracts exist to avoid. -**Cost-control knobs.** Three levers operators can pull when the +**Cost-control knobs.** Levers operators can pull when the default fan-out is too expensive for their use case: -- **`total_budget_seconds`** (default `300`) — Whole-run wall-clock - cap. Tripping this routes every remaining pair to the degraded path - rather than billing for the whole rubric × every artefact. A +- **`budget_base_seconds` / `budget_per_pair_seconds`** (defaults `60` / + `20.0`) — The two terms of the scaled wall-clock backstop (issue #198): + `effective = budget_base_seconds + budget_per_pair_seconds × ceil(num_pairs / max_concurrent_calls)`. + The backstop grows with model width and concurrency. It is a runaway + guard, not a completion target (~2.25× headroom over the #179 baseline); + tripping it routes every remaining pair to the degraded path. +- **`total_budget_seconds`** (**default `None`** since #198) — Optional + absolute hard cap on top of the scaled formula. `None` → use the scaled + budget alone; set to an int → `effective = min(scaled, total_budget_seconds)`. + Tripping the effective budget routes every remaining pair to the degraded + path rather than billing for the whole rubric × every artefact. A `GradeBudgetExceededError` only fires if the budget trips before ANY criterion runs (a hard "the run did nothing" failure); a partial run completes with `aggregate_complete: false`. +- **`max_grade_calls` / `max_grade_cost_usd` / `max_grade_tokens`** + (**all default `None` = off**, issue #198) — Opt-in soft ceilings on + judge calls / USD / token movement. Whichever trips first stops + scheduling **new** pairs; the rest **degrade** (never raise) with a + `reasoning` naming the ceiling, and the run emits one distinct + `grade ceiling exceeded` WARNING. USD/token ceilings are best-effort + (up to `max_concurrent_calls − 1` in-flight calls may complete past the + threshold because cost/tokens are known only post-call); `max_grade_calls` + is near-hard via pre-call slot reservation. Cache-hit pairs (#189) make + no LLM call and never count against any ceiling. See the field-by-field + [Configuration](#configuration-signalforgeyml-grade-block) above for the + accounting detail. - **`max_output_tokens`** (default `1024`) — Per-call output cap. The expected JSON response is ~150 tokens, so the cap is a truncation guard, not a target; the default was raised from 256 to 1024 in #187 @@ -1031,7 +1066,9 @@ logging.getLogger("signalforge.grade").setLevel(logging.DEBUG) Levels: - **INFO** — One line per `grade_artifacts` invocation at the end of the run, lazy-format JSON per DEC-027 (`run_id`, `model_unique_id`, `pass_rate`, `mean_score`, `passed`, `aggregate_complete`, `duration_seconds`, `results`). Mirrors `safety-layer.md` DEC-022 / `llm-drafter.md` DEC-011 / `prune-engine.md` DEC-017 — never f-string-interpolate user-controlled strings into a logger call. -- **WARNING** — One line when `total_budget_seconds` trips, JSON-encoded `{run_id, model_unique_id, evaluated, remaining_pairs, total_budget_seconds}`. Plus the inherited `signalforge.llm` retry warnings (one per retry attempt at the LLM seam). +- **WARNING (wall-clock budget)** — One line when the effective wall-clock budget trips, JSON-encoded `{run_id, model_unique_id, completed_count, degraded_count, effective_budget_seconds}`. The field is `effective_budget_seconds` (renamed from `total_budget_seconds` in #198 DEC-008) — it carries the *computed effective* budget actually passed to `asyncio.timeout`, i.e. the scaled formula optionally capped by `total_budget_seconds`. +- **WARNING (ceiling)** — One line when any opt-in `max_grade_calls` / `max_grade_cost_usd` / `max_grade_tokens` ceiling trips (issue #198 DEC-007), distinct from the wall-clock budget WARNING. JSON-encoded `{run_id, model_unique_id, ceiling, limit, completed_count, degraded_count}` where `ceiling ∈ {"calls", "cost_usd", "tokens"}` (the first ceiling to trip) and `limit` is its configured value. +- Plus the inherited `signalforge.llm` retry warnings (one per retry attempt at the LLM seam). - **DEBUG** — Reserved for future per-criterion latency observability; v0.1 emits no DEBUG from the engine. The grade layer never logs full evidence / reasoning content. The @@ -1057,7 +1094,7 @@ as `.cause` and on `__cause__`. Common causes: | `GradeConfigError` | `signalforge.yml` `grade:` block failed parse / schema validation (`extra="forbid"`, out-of-range knob, malformed rubric override). | `load_grade_config` | Inspect the `grade:` block. Typos like `mdoel:` are caught here. | | `GradeRubricError` | The resolved rubric is empty or carries duplicate `id` values. | `validate_rubric` (called at `grade_artifacts` entry and inside `load_grade_config`'s rubric validator) | Provide at least one criterion; ensure every `id` is unique. | | `GradeLLMError` | One-level wrap of `signalforge.llm.LLMError`. Retry budget exhausted, auth failure, server error, malformed cache block. | `_grade_one` per pair (degraded by orchestrator); only escapes if the entire run can't recover. | Inspect `.cause` / `__cause__` for the underlying LLM-layer detail. Common: missing `ANTHROPIC_API_KEY`, rate-limit exhaustion. | -| `GradeBudgetExceededError` | `total_budget_seconds` tripped before ANY criterion was graded (a "the run did nothing" failure). | `grade_artifacts` (rare — the normal budget path is per-pair degrade). | Raise `total_budget_seconds`, narrow the candidate set, or reduce the rubric's criterion count. | +| `GradeBudgetExceededError` | **Reserved; NOT raised in v0.1.** Budget exhaustion always degrades per-pair (`aggregate_complete=False`); this class is held for a future hard "the run did nothing" failure (budget trips before the first pair). | _Not currently raised._ | If a partial run (`aggregate_complete=False`) is undesirable, raise `budget_base_seconds` / `budget_per_pair_seconds` (or lift the `total_budget_seconds` cap if set), narrow the candidate set, or reduce the rubric's criterion count. | | `GradePromptEnvelopeBreachError` | An artefact payload contains the literal `` close tag. | Whole-run pre-flight `_scan_envelope_breach`; per-call defence-in-depth in `render_dynamic_block`. | Inspect the offending artefact (`exc.artifact_id`); remove the literal tag from the column description / rationale. | | `GradeOutputError` | LLM-judge response failed parse / anchor-contract validation. Carries `violation_type`. | `parse_grade_response` per pair (degraded by orchestrator). | Pattern-match on `.violation_type` (`json_parse`, `criterion_id_mismatch`, `score_out_of_range`, …). Re-running typically resolves transient JSON failures; structural mismatches usually point at a prompt-template regression. | | `GradeAuditWriteError` | Fail-closed audit / sidecar write failure (`OSError`, `PermissionError`, encoding, `fsync`, symlink containment). DEC-006 / DEC-012. | `write_grade_event` / `write_grading_report` (wrapped at the orchestrator's audit-write seams). | Verify `/.signalforge/` is writable, has disk space, and is not a symlink escaping the project tree. Fix the I/O issue and re-run. | diff --git a/plans/super/198-grade-budget-scaling.md b/plans/super/198-grade-budget-scaling.md new file mode 100644 index 00000000..41564580 --- /dev/null +++ b/plans/super/198-grade-budget-scaling.md @@ -0,0 +1,348 @@ +# Super Plan — #198: Grade budget: scale wall-clock with work + optional cost ceiling + +## Meta +- **Ticket:** https://github.com/wjduenow/SignalForge/issues/198 +- **Phase:** complete +- **Branch:** feature/198-grade-budget-scaling (worktree: .claude/worktrees/198-grade-budget-scaling) +- **Base:** origin/dev @ 4ad19ef (TBD — see Q3) +- **Sessions:** 1 (2026-06-03) + +## Problem (from ticket) +`GradeConfig.total_budget_seconds` defaults to a flat **300s** wall-clock ceiling on the whole grade +stage. It is mis-calibrated three ways: (1) docstring sizes it for "60 calls × 1s p50" but a real +16-col model produces 208–228 `(artifact × criterion)` pairs; (2) sized for the pre-#186 sequential +era — post-#186 concurrent dispatch (`asyncio.TaskGroup` + `Semaphore(max_concurrent_calls)`) outruns +it but the default was never recalibrated; (3) a fixed wall-clock doesn't scale with model width. + +### Part A — scale wall-clock with work +`effective_budget = budget_base_seconds + budget_per_pair_seconds × ceil(num_pairs / max_concurrent_calls)`. +New `GradeConfig` fields: `budget_base_seconds: int = 60`, `budget_per_pair_seconds: float`, +`total_budget_seconds: int | None = None` (reinterpreted as optional absolute hard ceiling; +effective = `min(scaled, total_budget_seconds)` when both apply). Degrade contract (DEC-015) unchanged. + +### Part B — optional opt-in cost ceilings (all default None = off) +`max_grade_calls: int | None`, `max_grade_cost_usd: float | None`, `max_grade_tokens: int | None`. +Whichever ceiling trips first stops dispatch; remaining pairs degrade with a reason naming the ceiling. +Cost via `signalforge.llm.pricing.lookup(model)`. + +## Discovery findings (Session 1) + +### Code map (verified on the worktree) +- **`src/signalforge/grade/config.py`** — `GradeConfig` (frozen, `extra="forbid"`). + - `total_budget_seconds: int = 300` (L176) + docstring (the stale "60 calls × 1s" sizing). + - `max_concurrent_calls: int = 10` (L183, validated `[1,100]`). + - `@field_validator("max_output_tokens","total_budget_seconds") _positive` (L327) — rejects ≤0; + must learn to skip `None` once `total_budget_seconds` becomes `int | None`. + - `@model_validator(mode="before") _resolve_model_default` (#187 sentinel pattern, L278). + - `_GradeConfigFile` wrapper `extra="ignore"` (L475). + - **No CLI flag** exists for `max_concurrent_calls` or `total_budget_seconds` (config-file-only). +- **`src/signalforge/grade/engine.py`** — `_grade_artifacts_async_core`: + - `pairs = list(_iterate_artifacts(...))`, `total_pairs = len(pairs)` (L600–601) — the count is + known BEFORE dispatch. This is where `effective_budget` will be computed. + - `semaphore = asyncio.Semaphore(resolved_config.max_concurrent_calls)` (L626). + - `total_budget_seconds = resolved_config.total_budget_seconds` (L651) — single read site. + - `async with asyncio.timeout(total_budget_seconds): async with asyncio.TaskGroup() as tg:` (L831) — + ALL pairs created as tasks at once; semaphore bounds concurrency. + - `_one(index, artifact_id, artifact_text, criterion)` (L654) — per-pair coroutine; LLM call inside; + `result.input_tokens / output_tokens / cache_*` available after the call (L420–423). + - Synthesis pass (L870–893) fills un-completed slots with + `reasoning=f"grade budget exceeded ({total_budget_seconds}s) before evaluation"`. + - Budget WARNING (L902) — locked field set `{run_id, model_unique_id, completed_count, + degraded_count, total_budget_seconds}` (pinned by `test_grade_artifacts_concurrent_budget_warning_shape_locked`). + - `_build_degraded(... reasoning=...)` (L452) — constructs `(GradingResult score=None, GradeEvent)`, + zero tokens. `_format_degrade_reasoning(exc)` (L428) is for LLM-layer failures (not budget). +- **`src/signalforge/llm/pricing.py`** — `ModelPricing(input_per_mtok, output_per_mtok, + cache_write_5m_per_mtok, cache_read_per_mtok)`; `PRICES` (read-only); `lookup(model) -> ModelPricing` + (raises `EstimateUnknownModelError`). `PRICE_TABLE_VERSION="2026-05-28"`. +- **`src/signalforge/llm/cost/_rollup.py::_compute_record_usd`** (L258) — the exact USD formula + `(in·in_mtok + out·out_mtok + cc·cw_mtok + cr·cr_mtok)/1e6`. Private; a public cost helper can be + hoisted/shared for the ceiling. +- **`src/signalforge/grade/models.py`** — `GradingResult.score: float|None`, `reasoning`; + `GradingReport.aggregate_complete` (computed: True iff every score non-None); `pass_rate`/`mean_score` + over scored subset only. `GradeEvent.audit_schema_version: int = 2` (token fields present). + **No full-config-hash on GradeEvent** → adding GradeConfig fields perturbs NO audit hash. +- **`src/signalforge/grade/errors.py::GradeBudgetExceededError`** — defined but RESERVED (never raised; + v0.1 degrades). The ticket keeps degrade semantics → it stays reserved (no graduation, no new errors). + +### Convention constraints (from .claude/rules/) +- `grade-layer.md`: DEC-015 conservative degrade (never silent drop; `aggregate_complete` signals + partial). `extra="forbid"` on `GradeConfig` + `field_validator`s. New fields under `grade:` namespace. + Read-back drift detectors (GradingResult/Report/Event) only matter if GradeEvent shape changes (it + won't). `GradeBudgetExceededError` reserved — keep reserved. #187 frozen-config default-from-sibling + resolves in `mode="before"`. +- `cli-layer.md`: new typed errors → tier-3 + exit-code table + 7th AST scan. **But ceilings degrade, + not raise → no new typed errors → no exit-code/AST churn.** New numeric config fields are + signalforge.yml-only (precedent: `max_concurrent_calls`, `total_budget_seconds`) → no CLI flag, no + 5-surface parity (unless we choose to add a flag — see Q2). +- `testing-signal.md`: no `assert True`; pin the computed formula with a unit test; fail-loud on absent + pricing key (degrade with named reason rather than crash). +- `prune-engine.md` §5-surface parity: only if `GradeBudgetExceededError` graduates (it won't here). + +### Critical calibration finding +The ticket's proposed `budget_per_pair_seconds = 2.0` **contradicts the ticket's own baseline**: 220 +pairs at concurrency 10 took **222.9s**, i.e. ~10s effective per concurrency-wave (Sonnet judge p50 +~10s/call). `60 + 2.0·ceil(220/10) = 104s` would trip at 104s and degrade ~half the pairs — recreating +the failure. The default must be grounded in the ~10s/call reality (~15–20s/pair for ~1.5–2× headroom). +See Q1. + +### Benchmark-harness coordination finding +`tests/research/179-runtime-benchmark/benchmark_runtime.py` + `docs/research/179-runtime-benchmark.md` +live ONLY on `feature/179-runtime-benchmark` (5 commits, no open PR). The harness drives the CLI against +an **external local repo** `~/Projects/intuit_airflow/plugins/dbt` with a live Anthropic key → the +benchmark-rerun AC is a **maintainer-only manual step**, not a Ralph bead. But the harness is not on +`dev` (current base). See Q3. + +## Scoping questions — see session log below + +## Session 1 decisions (2026-06-03) + +- **DEC-001 — Scaled wall-clock formula (Part A).** Add `budget_base_seconds: int = 60` and + `budget_per_pair_seconds: float = 20.0`; reinterpret `total_budget_seconds: int | None = None`. + `effective_budget = budget_base_seconds + budget_per_pair_seconds × ceil(num_pairs / max_concurrent_calls)`; + when `total_budget_seconds` is set, `effective = min(scaled, total_budget_seconds)`. + *Rationale (Q1):* baseline shows ~10s/call (220 pairs @ c=10 → 222.9s); `per_pair=20.0` gives a + 500s backstop on the baseline (~2.25× headroom) — a true runaway guard that tolerates 429 retry + storms, not a completion constraint. The ticket-literal `2.0` would compute 104s and degrade ~half + the pairs. `total_budget_seconds: ` preserves exact v0.1 absolute-cap semantics for pinned configs. + +- **DEC-002 — Ceilings degrade, never raise.** `max_grade_calls / max_grade_cost_usd / max_grade_tokens` + (all `| None = None`, opt-in). A tripped ceiling routes un-started pairs through the existing + `_build_degraded` path with a `reasoning` string naming the ceiling. **No new typed errors** → + no exit-code-table / 7th-AST-scan churn; `GradeBudgetExceededError` stays reserved. Mirrors the + DEC-015 conservative-degrade contract and prune-engine's conservative-bias routing template. + +- **DEC-003 — Soft/best-effort ceiling semantics (Q4).** All pairs are dispatched into the one + `TaskGroup`; cost/tokens are known only post-call, so the cost/token ceiling stops *un-started* + pairs (bounded overshoot ≤ `max_concurrent_calls − 1` in-flight calls). `max_grade_calls` is made + near-hard by reserving a slot (increment a shared counter) BEFORE the LLM call inside `_one`. Single + event loop → check-and-increment is race-free as long as no `await` separates read and decision. + Cache-hit pairs (#189, prefilled before the async core) make no LLM call → never count against any + ceiling. Documented overshoot in `docs/grade-ops.md`. + +- **DEC-004 — signalforge.yml-only, no CLI flags (Q2).** New fields live under the `grade:` namespace, + `extra="forbid"` + `field_validator`s, matching `max_concurrent_calls`/`total_budget_seconds`. + No CLI flags, no 5-surface parity. Runtime-override flags deferred to a follow-up if requested. + +- **DEC-005 — Base-branch sequencing (Q3).** Merge `feature/179-runtime-benchmark` to `dev` first + (open its PR, land it), then rebase the #198 worktree onto updated `dev`, so the benchmark harness + + `docs/research/179-runtime-benchmark.md` are present for the maintainer rerun AC. Implementation + + unit tests do not depend on the harness; only the final benchmark step does. + +- **DEC-006 — No `audit_schema_version` bump.** GradeEvent shape is unchanged; degrade reasons ride the + existing free-text `reasoning` field; there is no full-config-hash on GradeEvent, so new GradeConfig + fields perturb no reproducibility hash. Read-back drift detectors (GradingResult/Report/Event) stay + valid unchanged. (If refinement decides to add a structured "tripped ceiling" enum to GradeEvent, the + bump + drift-fixture refresh come back in scope — currently out of scope.) + +## Phase 2 — Architecture review (Session 1) + +| Area | Rating | Finding | +|---|---|---| +| Concurrency / race-freedom | **pass** | Single event loop. `max_grade_calls` slot-reservation is race-free if the increment sits immediately after `async with semaphore:` and BEFORE the first `await` (the LLM call). Cost/tokens soft-accumulator (update post-call, check pre-call) is coherent under the single-threaded loop. | +| Double-audit risk | **pass** | A ceiling-degrade inside `_one` sets `results_by_index[index]` (non-None) → the post-TaskGroup synthesis pass (L874 `if … is not None: continue`) skips it. No double GradeEvent. Use an **unshielded** audit-write for the pre-call ceiling degrade (no in-flight LLM await to race); keep the shielded write only for post-LLM degrades. | +| Counters / locked WARNING shape | **concern → Q5** | Ceiling-degrades that resolve inside `_one` should count as `completed` (preserves `completed + degraded == total_pairs`; keeps the synthesis-pass `degraded` semantics for wall-clock-only). The locked budget WARNING field set is pinned by `test_grade_artifacts_concurrent_budget_warning_shape_locked` — its `total_budget_seconds` field now carries the *computed effective* value (rename vs keep — Q5). Ceiling trips need their own operator signal (Q5). | +| Config migration | **pass** | `_positive` validator (config.py L327) currently covers `max_output_tokens` + `total_budget_seconds`; split so `total_budget_seconds` allows `None`. `test_grade_config_defaults_match_dec_023_to_027` (test_config.py L230) asserts `total_budget_seconds == 300` → change to `is None` + add 5 new default asserts. Fixtures `example_config.yml` (300) / austin `signalforge.yml` (600) keep working (explicit ints). No StrictGradeConfig (config-shaped). | +| `asyncio.timeout(None)` edge | **pass** | `_compute_effective_budget` always returns a finite int (`int(scaled)` when `total_budget_seconds is None`, else `int(min(scaled, total))`), so the timeout site never receives `None`. | +| Testing strategy | **pass** | Pure `_compute_effective_budget(base, per_pair, total, num_pairs, concurrency) -> int` unit-tested directly (no asyncio). Ceiling tests mirror `_config_tiny_budget` + a fake returning known `input_tokens`/`output_tokens`; cost tests MUST use a real SKU (`claude-sonnet-4-6`/`claude-haiku-4-5`) since `claude-fake` is absent from `PRICES`. `_make_candidate_with_n_columns(40)` drives the ≥40-col "0 degradations" AC deterministically (fake client, no live API). | + +### Exact touch-points (verified) +- `grade/config.py`: field L176 (`total_budget_seconds`), `_positive` validator L327, defaults test surface. +- `grade/engine.py`: L600-601 (`total_pairs`), L651 (single read → compute effective), L831 (`asyncio.timeout`), L882 (degrade reason), L902-914 (WARNING), `_one` L654 (slot-reservation site, just inside the semaphore). +- `grade/cost`: reuse the `_compute_record_usd` formula (`_rollup.py` L258) for the cost ceiling. +- Tests: `tests/grade/test_config.py` L230 + L660, `tests/grade/test_engine.py` (`_config_tiny_budget` L433, warning-shape L1745), `tests/grade/_fake.py::expect_grade_responses` (token params). +- Docs/rules: `docs/grade-ops.md` (L126/145/168 primary), `.claude/rules/grade-layer.md` L131-133 (DEC-029 grade: key enumeration — add 5 keys). + +## Phase 3 — Refinement decisions (Session 1) + +- **DEC-007 — New `grade ceiling exceeded` WARNING (Q5=A).** When any opt-in ceiling + (`max_grade_calls`/`max_grade_cost_usd`/`max_grade_tokens`) trips, the engine emits ONE end-of-run + stderr WARNING, distinct from the wall-clock budget WARNING. Locked field set: + `{run_id, model_unique_id, ceiling, limit, completed_count, degraded_count}` (`ceiling` ∈ + `{"calls","cost_usd","tokens"}`). Lazy-format JSON (`_LOGGER.warning("grade ceiling exceeded: %s", + json.dumps({...}))`) per the ANSI-safe logger grep gate. New pinned-shape test. + +- **DEC-008 — Rename the budget WARNING field `total_budget_seconds` → `effective_budget_seconds` + (Q6=A).** It now carries the computed effective budget actually passed to `asyncio.timeout`. Update + `test_grade_artifacts_concurrent_budget_warning_shape_locked` to the new field name in lockstep. + +- **DEC-009 — Degrade-reason strings (locked verbatim).** Wall-clock (synthesis pass, value now + effective): `f"grade budget exceeded ({effective_budget_seconds}s) before evaluation"` (unchanged + wording). Ceilings: `f"grade call ceiling exceeded ({max_grade_calls} calls)"`, + `f"grade cost ceiling exceeded (${max_grade_cost_usd})"`, + `f"grade token ceiling exceeded ({max_grade_tokens} tokens)"`. Pinned by tests. + +- **DEC-010 — Accounting.** Cost = full USD incl. cache (reuse the `_compute_record_usd` formula: + `(in·in_mtok + out·out_mtok + cc·cw_mtok + cr·cr_mtok)/1e6`) via `lookup(config.model)`. Tokens = + `input + output + cache_creation + cache_read` (all token movement). Calls = LLM calls only (cache + hits make no call → never counted). `_compute_effective_budget` is computed on `total_pairs` + (includes any prefilled cache-hit slots — conservative over-budget, harmless for a backstop). + +- **DEC-011 — Ceiling-degrades count as `completed`.** A ceiling-degrade resolves INSIDE `_one` + (sets the slot, unshielded audit-write) so it counts in `counters["completed"]` — preserving the + `completed + degraded == total_pairs` invariant and reserving `degraded` for the wall-clock + synthesis pass. Cache-hit pairs (prefilled, #189) bypass `_one` entirely. + +## Phase 4 — Detailed breakdown (stories) + +> Validation command (every story's final AC): +> `uv sync --dev && uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run pytest` + +### US-001 — GradeConfig: budget + ceiling fields + validator split +**Description.** Add the Part-A/Part-B fields to `GradeConfig` and split the positivity validator so +`total_budget_seconds` accepts `None`. +**Traces to:** DEC-001, DEC-002, DEC-004, DEC-006. +**Files:** `src/signalforge/grade/config.py`; `tests/grade/test_config.py`. +**Changes:** +- `total_budget_seconds: int = 300` → `int | None = None` (docstring rewritten: now an *optional absolute + hard ceiling*; `None` → use scaled formula; effective = `min(scaled, total_budget_seconds)`). +- Add `budget_base_seconds: int = 60`, `budget_per_pair_seconds: float = 20.0` (documented: per-wave wall + allowance; default grounded in ~10s/call baseline × ~2.25× headroom). +- Add `max_grade_calls: int | None = None`, `max_grade_cost_usd: float | None = None`, + `max_grade_tokens: int | None = None` (opt-in; documented as soft ceilings, cost via `signalforge.llm.pricing`). +- Split `_positive`: keep `max_output_tokens`; add positivity for `budget_base_seconds` / + `budget_per_pair_seconds`; add allow-None-or-positive validators for `total_budget_seconds` and the three + `max_grade_*`. +**TDD:** +- `test_grade_config_defaults_*`: change `total_budget_seconds == 300` → `is None`; add asserts for the 5 new + defaults (`budget_base_seconds == 60`, `budget_per_pair_seconds == 20.0`, three `max_grade_* is None`). +- positivity: `budget_base_seconds=0` / `budget_per_pair_seconds=0.0` / `total_budget_seconds=0` raise; the three + `max_grade_*=0` raise; all `*=None` (where optional) pass. +- `extra="forbid"` still rejects a typo (e.g. `max_grade_cal:`). +**Done When:** new fields present with documented defaults; validators behave per TDD; `extra="forbid"` +intact; existing fixtures (`example_config.yml`=300, austin=600) still load; validation command passes. +**Depends on:** none. + +### US-002 — `_compute_effective_budget` pure helper + unit tests +**Description.** Add a pure module-level helper in `grade/engine.py` computing the scaled budget; unit-test in +isolation (no asyncio). +**Traces to:** DEC-001, DEC-010. +**Files:** `src/signalforge/grade/engine.py`; `tests/grade/test_engine.py`. +**Changes:** `def _compute_effective_budget(*, budget_base_seconds, budget_per_pair_seconds, total_budget_seconds, +num_pairs, max_concurrent_calls) -> int:` → `scaled = base + per_pair*math.ceil(num_pairs/concurrency)`; +return `int(scaled)` when `total_budget_seconds is None`, else `int(min(scaled, total_budget_seconds))`. +`num_pairs == 0` → returns `base` (guard the ceil divide-by-… is fine; concurrency ≥ 1 by validator). +**TDD (pin the formula):** +- 220 pairs, c=10, base=60, per_pair=20.0, total=None → **500**. +- same with total=300 → **300** (cap wins). +- same with total=900 → **500** (scaled wins). +- num_pairs=1, c=10 → 60 + 20·1 = 80. +- num_pairs=0 → 60. +- non-multiple: 221 pairs, c=10 → ceil(22.1)=23 → 60+460=520. +**Done When:** helper returns a finite int for all inputs incl. `total=None`; formula tests pass; validation passes. +**Depends on:** US-001. + +### US-003 — Engine: wire scaled budget + rename WARNING field + ≥40-col AC test +**Description.** Use `_compute_effective_budget` at the timeout site; thread the effective value into the degrade +reason; rename the budget WARNING field; add the width AC test. +**Traces to:** DEC-001, DEC-008, DEC-009. +**Files:** `src/signalforge/grade/engine.py`; `tests/grade/test_engine.py`. +**Changes:** +- Replace L651 read: compute `effective_budget = _compute_effective_budget(... num_pairs=total_pairs, + max_concurrent_calls=resolved_config.max_concurrent_calls)`. +- `asyncio.timeout(effective_budget)` (L831); synthesis-pass reason uses `effective_budget` (L882, DEC-009 wording). +- WARNING field rename `total_budget_seconds` → `effective_budget_seconds` carrying `effective_budget` (L911). +**TDD:** +- Update `test_grade_artifacts_concurrent_budget_warning_shape_locked` to the renamed field; assert the value + equals the computed effective budget for the tiny-budget config. +- **AC (≥40-col, 0 width-induced degradations under default config):** `_make_candidate_with_n_columns(40)` + + default `GradeConfig` + fast fake client → `report.aggregate_complete is True`, zero `score is None` results, + no budget WARNING emitted. +**Done When:** the timeout uses the scaled budget; WARNING field renamed + test green; ≥40-col AC test green; +validation passes. +**Depends on:** US-002. + +### US-004 — Engine: cost/calls/tokens ceilings (degrade path) + ceiling WARNING +**Description.** Add the three opt-in ceilings to the `_one` dispatch with soft/best-effort semantics, degrading +un-started pairs and emitting one end-of-run ceiling WARNING. +**Traces to:** DEC-002, DEC-003, DEC-007, DEC-009, DEC-010, DEC-011. +**Files:** `src/signalforge/grade/engine.py`; `tests/grade/test_engine.py`; (maybe) `tests/grade/_fake.py`. +**Changes:** +- Closure accumulators: `calls_made`, `cost_usd`, `tokens` (+ a `tripped: {"ceiling": str|None, "limit": ...}` cell). +- In `_one`, immediately after `async with semaphore:` and BEFORE the LLM `await`: if a ceiling is configured and + already met/exceeded → build degraded (DEC-009 reason), **unshielded** audit-write, set slot, `completed += 1`, + record `tripped`, return. `max_grade_calls`: reserve a slot (increment) pre-call, no `await` between check and + increment (DEC-003). After a successful call: add usage to `cost_usd` (via `lookup(config.model)` USD formula) and + `tokens`, increment `calls_made` if not pre-reserved. +- End-of-run: if `tripped["ceiling"]` set, emit the DEC-007 `grade ceiling exceeded` WARNING (locked shape). +- Cache-hit (prefilled) pairs never enter `_one` → excluded from all ceilings (DEC-010). +**TDD (each ceiling degrades; never silent drop):** +- `max_grade_calls=K`: exactly ~K pairs scored, rest degraded with `"grade call ceiling exceeded (K calls)"`, + `aggregate_complete False`, one ceiling WARNING `ceiling="calls"`. +- `max_grade_cost_usd`: **real SKU** (`claude-sonnet-4-6`), fake returns known `input/output_tokens`; cap chosen so + ~N pairs fit; degraded reason `"grade cost ceiling exceeded ($…)"`, WARNING `ceiling="cost_usd"`. +- `max_grade_tokens`: known per-call tokens, cap → degrade `"grade token ceiling exceeded (… tokens)"`, + WARNING `ceiling="tokens"`. +- ceiling WARNING shape pinned (`{run_id, model_unique_id, ceiling, limit, completed_count, degraded_count}`). +- cache-hit pairs don't count against a tiny `max_grade_calls` (prefilled slots bypass the ceiling). +**Done When:** all three ceilings degrade un-started pairs with the locked reasons; one ceiling WARNING per tripped +run; `aggregate_complete` reflects partial; no double-audit (slot set inside `_one`); validation passes. +**Depends on:** US-003. + +### US-005 — Docs + example fixture parity (worker-writable surfaces) +**Description.** Update `docs/grade-ops.md` and the example config fixture for the new budget model + ceilings. +**Traces to:** DEC-001, DEC-002, DEC-004, DEC-007, DEC-008, DEC-009. +**Files:** `docs/grade-ops.md`; `tests/fixtures/grade/example_config.yml` (+ its round-trip test if fields added). +**Changes:** +- `docs/grade-ops.md`: rewrite the `total_budget_seconds` field doc (L168 area) → optional absolute cap; document + `budget_base_seconds`/`budget_per_pair_seconds` + the formula + the ~2.25× headroom rationale; document the three + ceilings + soft/best-effort overshoot (≤ `max_concurrent_calls − 1`); update the budget-WARNING field name; add the + new ceiling WARNING; refresh the per-provider cost-guidance mentions. +- `example_config.yml`: add the new keys as explicit/commented examples; keep round-trip test green (update it if keys + are added to the asserted set). +**Done When:** docs describe the scaled budget + ceilings accurately; example fixture round-trips; validation passes. +**Depends on:** US-004. +**Note:** `.claude/rules/grade-layer.md` (DEC-029 key enumeration + taxonomy note) is **orchestrator-writable only** +(Ralph workers can't write `.claude/`), so that edit lives in the Patterns & Memory story. + +### US-098 — Quality Gate (code review ×4 + CodeRabbit) +**Description.** Run the code reviewer 4× across the full changeset, fixing all real bugs each pass; run CodeRabbit if +available. Validation must pass after fixes. +**Depends on:** US-005 (all implementation complete). + +### US-099 — Patterns & Memory (priority 99, orchestrator-run) +**Description.** Update `.claude/rules/grade-layer.md` (DEC-029 grade: key enumeration — add `budget_base_seconds`, +`budget_per_pair_seconds`, `max_grade_calls`, `max_grade_cost_usd`, `max_grade_tokens`; note the scaled-budget + +ceiling degrade as part of the DEC-015 taxonomy; note the renamed `effective_budget_seconds` WARNING field + the new +`grade ceiling exceeded` WARNING). Record any reusable pattern (e.g. "scale a wall-clock backstop with work × not +flat"; "opt-in ceilings degrade, never raise → no exit-code/AST churn") in memory. +**Depends on:** US-098. + +## Maintainer-only steps (NOT Ralph beads — live API / external repo / `.claude` writes) +- **M-1 (prerequisite, DEC-005).** Open the PR for `feature/179-runtime-benchmark` → merge to `dev`, then + rebase the #198 worktree onto updated `dev` so `tests/research/179-runtime-benchmark/` + + `docs/research/179-runtime-benchmark.md` are present. +- **M-2 (closing AC).** Re-run `tests/research/179-runtime-benchmark/benchmark_runtime.py` against + `~/Projects/intuit_airflow/plugins/dbt` for `weekly_query_cost` AND a ≥40-col model (live Anthropic key, + cold grade cache, `prune.enabled: false`). Record the new per-stage numbers in + `docs/research/179-runtime-benchmark.md` and confirm **0 width-induced budget degradations** on the wide model. + +## Beads manifest (devolved 2026-06-04) +- **Epic:** `SignalForge-xfg` +- **Tasks (dependency chain US-001 → … → Patterns & Memory):** + - `SignalForge-xfg.1` — US-001 GradeConfig fields + validator split (ready) + - `SignalForge-xfg.2` — US-002 `_compute_effective_budget` helper + formula tests (← xfg.1) + - `SignalForge-xfg.3` — US-003 engine wire scaled budget + WARNING rename + ≥40-col AC (← xfg.2) + - `SignalForge-xfg.4` — US-004 cost/calls/tokens ceilings + ceiling WARNING (← xfg.3) + - `SignalForge-xfg.5` — US-005 docs/grade-ops.md + example fixture parity (← xfg.4) + - `SignalForge-xfg.6` — Quality Gate ×4 + CodeRabbit (← xfg.5) + - `SignalForge-xfg.7` — Patterns & Memory incl. grade-layer.md DEC-029 (← xfg.6) +- **Worktree:** `.claude/worktrees/198-grade-budget-scaling` (branch `feature/198-grade-budget-scaling`, base `dev`). +- **PR:** #199 (draft → ready on devolve). Maintainer-only M-2 benchmark rerun is the closing AC. + +## Status: Complete (2026-06-04) + +All 7 beads (epic `SignalForge-xfg`) landed via `/ralph-run`; epic auto-closed. Full suite green +(3592 passed). Implementation commits `581b19a..ce14143` on `feature/198-grade-budget-scaling`. + +- **PR:** #199 +- **Quality Gate finding (real bug, fixed):** the cost ceiling looked up `pricing.lookup` per-pair + inside the `TaskGroup`; a prefix-valid-but-unpriced SKU + `max_grade_cost_usd` raised + `EstimateUnknownModelError` mid-run (uncaught by the per-pair `except`), aborting after billable + calls. Fixed by resolving pricing once up front (fail-fast at entry) + regression test + (`ce14143`/`ddc3ca6`). Lesson recorded in `grade-layer.md` + bd memory. +- **Compounding update:** `.claude/rules/grade-layer.md` (scaled-budget + ceilings contract, DEC-029 + enumeration), `docs/grade-ops.md`, bd memory `grade-runtime-budgets-198-scale-a-wall-clock`. +- **Maintainer-only remaining (M-2 closing AC):** re-run `tests/research/179-runtime-benchmark/benchmark_runtime.py` + on `weekly_query_cost` + a ≥40-col model (live key), record numbers in `docs/research/179-runtime-benchmark.md`. + The deterministic ≥40-col "0 width-induced degradations" AC already passes + (`test_grade_artifacts_wide_model_completes_with_zero_budget_degradations`). diff --git a/src/signalforge/_demo/signalforge.yml b/src/signalforge/_demo/signalforge.yml index 5f260842..b122f2c5 100644 --- a/src/signalforge/_demo/signalforge.yml +++ b/src/signalforge/_demo/signalforge.yml @@ -11,4 +11,4 @@ grade: min_pass_rate: 0.95 min_mean_score: 0.95 fail_on_below_threshold: false - total_budget_seconds: 600 # default 300 is tight at p99 latency × ~12 calls. + total_budget_seconds: 600 # absolute hard cap; #198 default (None) uses the scaled formula. diff --git a/src/signalforge/grade/config.py b/src/signalforge/grade/config.py index a6d8236a..d5ed3844 100644 --- a/src/signalforge/grade/config.py +++ b/src/signalforge/grade/config.py @@ -33,9 +33,13 @@ ``max_output_tokens=1024`` (#187 DEC-004 — raised from 256 so a one-line ``gemini-2.5-flash`` grade JSON is substantially less likely to truncate), ``max_retries_429=3``, ``max_retries_5xx=1``, - ``max_retries_conn=1``, ``total_budget_seconds=300``, - ``min_pass_rate=0.7``, ``min_mean_score=0.5``, ``rubric=None``, - ``fail_on_below_threshold=False``. + ``max_retries_conn=1``, ``total_budget_seconds=None`` (reinterpreted by + #198 DEC-001 as an *optional* absolute hard ceiling; ``None`` → use the + scaled-budget formula via ``budget_base_seconds=60`` / + ``budget_per_pair_seconds=20.0``), the three opt-in soft ceilings + ``max_grade_calls=None`` / ``max_grade_cost_usd=None`` / + ``max_grade_tokens=None`` (off), ``min_pass_rate=0.7``, + ``min_mean_score=0.5``, ``rubric=None``, ``fail_on_below_threshold=False``. * **#187 US-002 / DEC-006** — when ``model`` is set explicitly, a SKU-prefix/provider mismatch (e.g. ``provider="openai"`` with ``model="claude-sonnet-4-6"``) fails loud at config-load. The @@ -173,12 +177,83 @@ class GradeConfig(BaseModel): registry is a plugin point designed to grow. The field validator fails loud on an unknown value — listing the registered provider names.""" - total_budget_seconds: int = 300 - """Whole-run wall-clock budget (DEC-023). 5 minutes default — ~3× - safety on 60 calls × 1s p50. Mirrors :attr:`signalforge.prune.PruneConfig.total_budget_seconds` - semantics: when the budget trips, un-evaluated ``(artifact, criterion)`` - pairs land as a degraded :class:`signalforge.grade.models.GradingResult` - rather than silently dropped.""" + total_budget_seconds: int | None = None + """Optional absolute hard ceiling on the whole-run wall-clock budget + (#198 DEC-001; reinterpreted from the flat DEC-023 default). + + ``None`` (the new default) means the engine sizes the budget from the + work via the scaled formula + ``budget_base_seconds + budget_per_pair_seconds * ceil(num_pairs / max_concurrent_calls)`` + — a backstop that grows with model width and concurrency rather than a + flat 300s that the pre-#186 sequential era was sized for. When set to an + int, the effective budget is ``min(scaled, total_budget_seconds)`` — i.e. + an explicit value still acts as a hard cap on top of the scaled estimate, + preserving exact v0.1 absolute-cap semantics for pinned ``signalforge.yml`` + files (e.g. an operator who set ``total_budget_seconds: 600`` keeps that + 600s ceiling). + + Mirrors :attr:`signalforge.prune.PruneConfig.total_budget_seconds` + degrade semantics: when the budget trips, un-evaluated + ``(artifact, criterion)`` pairs land as a degraded + :class:`signalforge.grade.models.GradingResult` rather than silently + dropped (DEC-015).""" + + budget_base_seconds: int = 60 + """Fixed startup / overhead allowance in the scaled wall-clock formula + (#198 DEC-001). + + The constant term in + ``budget_base_seconds + budget_per_pair_seconds * ceil(num_pairs / max_concurrent_calls)``. + Covers per-run setup (config resolution, cache priming, the first + concurrency wave's ramp) that does not scale with the number of pairs. + Must be positive.""" + + budget_per_pair_seconds: float = 20.0 + """Per concurrency-wave wall allowance in the scaled formula (#198 + DEC-001). + + The scaled formula multiplies this by + ``ceil(num_pairs / max_concurrent_calls)`` — i.e. the number of + concurrency *waves*, not the raw pair count — so it is the wall-clock + allowance per wave of ``max_concurrent_calls`` in-flight judge calls. + + Default ``20.0`` is grounded in the #179 baseline (Sonnet judge p50 + ~10s/call; 220 pairs at concurrency 10 → ``60 + 20.0 * ceil(220/10) = + 500s`` against a measured 222.9s — ~2.25× headroom). It is a runaway + backstop sized to tolerate 429 retry storms, NOT a completion target; + the ticket-literal ``2.0`` would compute 104s and degrade ~half the + pairs, recreating the failure this scaling fixes. Must be positive.""" + + max_grade_calls: int | None = None + """Opt-in soft ceiling on the number of LLM judge calls (#198 DEC-002). + + ``None`` (default) → off. When set, dispatch stops once this many + judge calls have been made and the remaining ``(artifact, criterion)`` + pairs DEGRADE (never raise) — mirroring the DEC-015 conservative-degrade + contract. Cache-hit pairs (#189) make no LLM call and never count + against this ceiling. Whichever of the three ``max_grade_*`` ceilings + trips first stops dispatch. Must be positive when set.""" + + max_grade_cost_usd: float | None = None + """Opt-in soft ceiling on the total USD cost of the grade run (#198 + DEC-002). + + ``None`` (default) → off. When set, dispatch stops once the accumulated + per-call cost (computed from per-call token usage via + :mod:`signalforge.llm.pricing`, including cache-read/write economics) + meets or exceeds this budget; remaining pairs DEGRADE (never raise). + Whichever of the three ``max_grade_*`` ceilings trips first stops + dispatch. Must be positive when set.""" + + max_grade_tokens: int | None = None + """Opt-in soft ceiling on the total token movement of the grade run + (#198 DEC-002). + + ``None`` (default) → off. When set, dispatch stops once the accumulated + token count (input + output + cache-creation + cache-read across the + judge calls) meets or exceeds this budget; remaining pairs DEGRADE + (never raise). Whichever of the three ``max_grade_*`` ceilings trips + first stops dispatch. Must be positive when set.""" max_concurrent_calls: int = 10 """Asyncio dispatch concurrency cap for the per-``(artifact, criterion)`` @@ -324,9 +399,55 @@ def _model_non_empty(cls, v: str | None) -> str | None: raise ValueError("must be a non-empty, non-whitespace string") return v - @field_validator("max_output_tokens", "total_budget_seconds") + @field_validator("max_output_tokens", "budget_base_seconds", "budget_per_pair_seconds") @classmethod - def _positive(cls, v: int) -> int: + def _positive(cls, v: int | float) -> int | float: + """Positive-only knobs (#198 DEC-001 split). + + Covers :attr:`max_output_tokens` (zero/negative would make the LLM + refuse output) plus the two always-on scaled-budget terms + :attr:`budget_base_seconds` / :attr:`budget_per_pair_seconds` (a + non-positive term would size the wall-clock backstop to ``0`` and + degrade every pair before any call). ``total_budget_seconds`` and the + three ``max_grade_*`` ceilings are now optional and live on the + separate :meth:`_optional_positive` validator below.""" + # Reject non-finite floats up front: ``yaml.safe_load`` parses + # ``.nan`` / ``.inf``, and ``nan <= 0`` / ``inf <= 0`` are both + # ``False`` so they would slip past the positivity check — a NaN + # ``budget_per_pair_seconds`` then crashes ``int(nan)``/``math.ceil(nan)`` + # in ``_compute_effective_budget`` (Pydantic floats allow inf/nan by + # default). Int fields can't carry inf/nan — coercion rejects them earlier. + if isinstance(v, float) and not math.isfinite(v): + raise ValueError("must be a finite number") + if v <= 0: + raise ValueError("must be positive") + return v + + @field_validator( + "total_budget_seconds", + "max_grade_calls", + "max_grade_cost_usd", + "max_grade_tokens", + ) + @classmethod + def _optional_positive(cls, v: int | float | None) -> int | float | None: + """Allow-``None``-or-positive knobs (#198 DEC-001 / DEC-002). + + :attr:`total_budget_seconds` (optional absolute cap) and the three + opt-in soft ceilings :attr:`max_grade_calls` / + :attr:`max_grade_cost_usd` / :attr:`max_grade_tokens` all default to + ``None`` (off). ``None`` passes through untouched; a *present* value + must be positive — a zero/negative cap would trip immediately and + degrade the whole run, the silent-no-op failure mode the strict + validator exists to prevent.""" + if v is None: + return v + # Reject non-finite floats (``max_grade_cost_usd: .inf`` would make the + # cost ceiling never trip — ``cost_usd >= inf`` is always ``False`` — + # i.e. a silent no-op; ``.nan`` is likewise never ``>=``). Same rationale + # as :meth:`_positive`. + if isinstance(v, float) and not math.isfinite(v): + raise ValueError("must be a finite number") if v <= 0: raise ValueError("must be positive") return v diff --git a/src/signalforge/grade/engine.py b/src/signalforge/grade/engine.py index 3430f700..abe471f8 100644 --- a/src/signalforge/grade/engine.py +++ b/src/signalforge/grade/engine.py @@ -66,6 +66,7 @@ import hashlib import json import logging +import math import time import uuid from collections.abc import Iterator @@ -138,6 +139,7 @@ LLMProviderAsyncUnsupportedError, LLMResponseFormatError, ) +from signalforge.llm.pricing import lookup as _lookup_pricing from signalforge.llm.providers import provider_for from signalforge.manifest.models import Model from signalforge.prune.models import PruneResult @@ -449,6 +451,57 @@ def _format_degrade_reasoning(exc: BaseException) -> str: return base +def _compute_effective_budget( + *, + budget_base_seconds: int, + budget_per_pair_seconds: float, + total_budget_seconds: int | None, + num_pairs: int, + max_concurrent_calls: int, +) -> int: + """Scale the grade wall-clock budget with the work to be done (DEC-001). + + The effective budget is a *runaway guard*, not a completion + constraint: it backstops 429 retry storms and pathological slow + calls rather than pacing normal completion. + + scaled = budget_base_seconds + + budget_per_pair_seconds * ceil(num_pairs / max_concurrent_calls) + + ``ceil(num_pairs / max_concurrent_calls)`` is the number of + serial *waves* of LLM calls (each wave runs ``max_concurrent_calls`` + pairs in parallel under the semaphore), so the per-pair term scales + with wall-clock depth, not raw pair count. + + When ``total_budget_seconds`` is ``None`` the scaled value is + returned verbatim; when it is set it acts as an absolute hard + ceiling (``min(scaled, total_budget_seconds)``) — preserving the + exact v0.1 absolute-cap semantics for pinned configs (DEC-010). + + Pure function: no asyncio, no I/O, no logging. Always returns a + finite ``int`` so the ``asyncio.timeout(...)`` site never receives + ``None``. + + ``num_pairs == 0`` short-circuits to ``budget_base_seconds`` (no + work to scale). ``max_concurrent_calls`` is ``>= 1`` by the config + validator, but the zero-pair guard also sidesteps any division + concern defensively. + """ + if num_pairs <= 0: + scaled: float = float(budget_base_seconds) + else: + waves = math.ceil(num_pairs / max_concurrent_calls) + scaled = budget_base_seconds + budget_per_pair_seconds * waves + # ``math.ceil`` (not ``int``) on the final conversion so a fractional + # ``budget_per_pair_seconds`` never rounds the wall-clock backstop DOWN + # (``int(121.5)`` would shave 0.5s and trip earlier than intended). The + # absolute-cap branch ceils the post-``min`` value for the same reason. + # Both are no-ops for the default integer-valued config. + if total_budget_seconds is None: + return math.ceil(scaled) + return math.ceil(min(scaled, total_budget_seconds)) + + def _build_degraded( *, artifact_id: str, @@ -562,7 +615,8 @@ async def _grade_artifacts_async_core( from the v0.1 ``grade_artifacts``. Orchestrates concurrent dispatch via :class:`asyncio.TaskGroup` throttled by an :class:`asyncio.Semaphore(max_concurrent_calls)`, bounded by - :func:`asyncio.timeout(total_budget_seconds)`. Each pair runs as a + :func:`asyncio.timeout(effective_budget)` (the scaled wall-clock + backstop from :func:`_compute_effective_budget`, DEC-001). Each pair runs as a coroutine; per-coroutine ``try/except`` isolates LLM-layer failures so one bad pair doesn't abort siblings (DEC-004 retry isolation). @@ -648,7 +702,60 @@ async def _grade_artifacts_async_core( if prefilled_results is not None: counters["completed"] = sum(1 for r in prefilled_results if r is not None) - total_budget_seconds = resolved_config.total_budget_seconds + # Scale the wall-clock backstop with the work to be done (DEC-001). + # ``effective_budget`` is a finite int (never ``None``) so the + # ``asyncio.timeout(...)`` site below always receives a valid value. + effective_budget = _compute_effective_budget( + budget_base_seconds=resolved_config.budget_base_seconds, + budget_per_pair_seconds=resolved_config.budget_per_pair_seconds, + total_budget_seconds=resolved_config.total_budget_seconds, + num_pairs=total_pairs, + max_concurrent_calls=resolved_config.max_concurrent_calls, + ) + + # Opt-in cost/calls/tokens ceilings (US-004 / DEC-002/003/010/011). + # All three default ``None`` → off; a non-None value caps the + # respective accumulator and degrades un-started pairs once met. + # + # Semantics are SOFT / best-effort (DEC-003): every pair is dispatched + # into the one ``TaskGroup``; a tripped ceiling only stops *scheduling + # new LLM calls*, so up to ``max_concurrent_calls - 1`` in-flight calls + # may complete past the threshold. The accumulators are mutable closure + # cells (mirroring ``budget_state`` / ``counters``) so the per-pair + # ``_one`` coroutine reads + updates them; the single event loop makes + # check-then-reserve race-free as long as no ``await`` separates the + # read from the slot-reservation increment (DEC-003). + max_grade_calls = resolved_config.max_grade_calls + max_grade_cost_usd = resolved_config.max_grade_cost_usd + max_grade_tokens = resolved_config.max_grade_tokens + # Resolve the cost-ceiling pricing ONCE, up front, before any LLM call + # is dispatched. ``GradeConfig._validate_model_provider_compat`` only + # checks the SKU *prefix* (``claude-`` / ``gpt-`` / ``gemini-``), NOT + # membership in ``pricing.PRICES`` — so a prefix-valid-but-unpriced SKU + # (a newer Opus, a typo passing the prefix check) is accepted at + # config-load. Looking the price up per-pair inside the ``TaskGroup`` + # would raise ``EstimateUnknownModelError`` from inside a coroutine — + # uncaught by the per-pair ``except`` below — aborting the whole run via + # a ``BaseExceptionGroup`` AFTER billable calls + audit writes. Resolving + # here fails fast at orchestrator entry (the typed error surfaces cleanly, + # CLI tier 2, with its remediation) and the resolved object is reused for + # every pair (no redundant per-pair lookup). ``None`` when no cost ceiling + # is configured. + cost_pricing = None + if max_grade_cost_usd is not None: + # ``resolved_config.model`` is invariantly concrete post-construction + # (#187 US-002); the lookup raises only on an unknown SKU. + assert resolved_config.model is not None + cost_pricing = _lookup_pricing(resolved_config.model) + # ``calls_made`` is the near-hard call counter: reserved (incremented) + # BEFORE the LLM await so it bounds dispatch tightly. ``cost_usd`` and + # ``tokens`` are soft accumulators updated only AFTER a successful call + # (their values are unknown until the response returns), so the cost / + # token ceilings degrade pairs that START after the accumulator already + # crossed the cap (DEC-003 bounded overshoot). + accumulators: dict[str, float] = {"calls_made": 0.0, "cost_usd": 0.0, "tokens": 0.0} + # Records the FIRST ceiling that tripped (DEC-007 WARNING source). + tripped: dict[str, object | None] = {"ceiling": None, "limit": None} async def _one(index: int, artifact_id: str, artifact_text: str, criterion: Criterion) -> None: async with semaphore: @@ -657,6 +764,86 @@ async def _one(index: int, artifact_id: str, artifact_text: str, criterion: Crit # ``started_at`` separately. crit_hash = crit_hash_by_id[criterion.id] per_call_ts = datetime.now(UTC) + + # --- Opt-in cost/calls/tokens ceilings (US-004) ------------- + # CHECK-then-RESERVE with NO ``await`` between the read and the + # decision: this whole block runs synchronously inside the + # single event loop, immediately after the semaphore acquire + # and BEFORE the LLM ``await`` below, so the accumulator reads + # and the ``calls_made`` increment are race-free (DEC-003). A + # tripped ceiling degrades THIS un-started pair (DEC-002) with + # the locked reason (DEC-009), writes its audit record + # UNSHIELDED (no in-flight LLM await precedes it — the + # synchronous ``_write_event_or_abort_kw`` mirrors the + # synthesis pass), sets the slot (so the synthesis pass skips + # it — no double audit), counts it as ``completed`` (DEC-011), + # records the first ceiling that tripped, and returns without + # making the LLM call. + ceiling_reason: str | None = None + ceiling_name: str | None = None + ceiling_limit: object | None = None + if max_grade_calls is not None and accumulators["calls_made"] >= max_grade_calls: + ceiling_reason = f"grade call ceiling exceeded ({max_grade_calls} calls)" + ceiling_name = "calls" + ceiling_limit = max_grade_calls + elif max_grade_cost_usd is not None and accumulators["cost_usd"] >= max_grade_cost_usd: + ceiling_reason = f"grade cost ceiling exceeded (${max_grade_cost_usd})" + ceiling_name = "cost_usd" + ceiling_limit = max_grade_cost_usd + elif max_grade_tokens is not None and accumulators["tokens"] >= max_grade_tokens: + ceiling_reason = f"grade token ceiling exceeded ({max_grade_tokens} tokens)" + ceiling_name = "tokens" + ceiling_limit = max_grade_tokens + + if ceiling_reason is not None: + grading_result, event = _build_degraded( + artifact_id=artifact_id, + criterion=criterion, + reasoning=ceiling_reason, + config=resolved_config, + rubric_hash=rubric_hash, + template_hash=template_hash, + crit_hash=crit_hash, + run_id=run_id, + timestamp=per_call_ts, + model_unique_id=model_unique_id, + ) + # Slot + counters FIRST (DEC-011: ceiling-degrades count as + # completed), then the audit write — mirroring the happy-path + # ordering so a cancellation during the write await leaves the + # synthesis pass correctly skipping this index (no double audit). + results_by_index[index] = grading_result + counters["completed"] += 1 + if tripped["ceiling"] is None: + tripped["ceiling"] = ceiling_name + tripped["limit"] = ceiling_limit + # Audit-write via the executor + shield (DEC-017), exactly like + # the happy-path write below: this degrade runs INSIDE the + # concurrent TaskGroup region (unlike the sequential synthesis + # pass), so a synchronous fsync here would block the event loop + # and stall sibling in-flight coroutines. The shield lets the + # write finish even if a budget timeout cancels this task + # mid-flight; on CancelledError we await the future so a + # GradeAuditWriteError / GradeAuditRecordTooLargeError still + # propagates and aborts the run (fail-closed, DEC-006). + loop = asyncio.get_running_loop() + ceiling_audit_future = loop.run_in_executor( + None, _write_event_or_abort_kw, event, resolved_audit_path + ) + try: + await asyncio.shield(ceiling_audit_future) + except asyncio.CancelledError: + await ceiling_audit_future + raise + return + + # Reserve a call slot for the near-hard ``max_grade_calls`` + # ceiling BEFORE the LLM await (DEC-003). The check above + # already rejected the over-cap case, so this increment can + # only push ``calls_made`` up to exactly ``max_grade_calls``. + if max_grade_calls is not None: + accumulators["calls_made"] += 1 + try: grading_result, event = await _grade_one_async( artifact_id=artifact_id, @@ -714,6 +901,36 @@ async def _one(index: int, artifact_id: str, artifact_text: str, criterion: Crit results_by_index[index] = grading_result counters["completed"] += 1 + # Update the soft cost / token accumulators from this call's + # actual usage (DEC-010). Read off ``event`` (the GradeEvent + # built from ``result.{input,output,cache_*}_tokens`` in + # ``_grade_one_async``; a degraded pair carries zeros — harmless + # to accumulate). ``tokens`` is ALL token movement (input + + # output + cache write + cache read); ``cost_usd`` is full USD + # incl. cache via the frozen pricing table. Cache-hit pairs + # (#189) bypass ``_one`` entirely so they're already excluded. + # The cost lookup uses ``resolved_config.model`` (the SKU the + # operator selected), never ``event.model`` — the live config + # is the single source of truth for which price table applies. + if max_grade_tokens is not None: + accumulators["tokens"] += ( + event.input_tokens + + event.output_tokens + + event.cache_creation_input_tokens + + event.cache_read_input_tokens + ) + if max_grade_cost_usd is not None: + # ``cost_pricing`` was resolved once up front (an unpriced SKU + # already failed fast at orchestrator entry), so reuse it here + # rather than looking up per pair. + assert cost_pricing is not None + accumulators["cost_usd"] += ( + event.input_tokens * cost_pricing.input_per_mtok + + event.output_tokens * cost_pricing.output_per_mtok + + event.cache_creation_input_tokens * cost_pricing.cache_write_5m_per_mtok + + event.cache_read_input_tokens * cost_pricing.cache_read_per_mtok + ) / 1_000_000 + # Audit-write per pair (DEC-006 fail-closed; DEC-017 # executor-wrap so the fsync doesn't block the loop). On # GradeAuditRecordTooLargeError / GradeAuditWriteError we @@ -828,7 +1045,7 @@ async def _one(index: int, artifact_id: str, artifact_text: str, criterion: Crit ) try: - async with asyncio.timeout(total_budget_seconds): + async with asyncio.timeout(effective_budget): async with asyncio.TaskGroup() as tg: for index, (artifact_id, artifact_text, criterion) in enumerate(pairs): # Skip cache-hit slots already populated by the sync @@ -879,7 +1096,7 @@ async def _one(index: int, artifact_id: str, artifact_text: str, criterion: Crit grading_result, event = _build_degraded( artifact_id=artifact_id, criterion=criterion, - reasoning=(f"grade budget exceeded ({total_budget_seconds}s) before evaluation"), + reasoning=(f"grade budget exceeded ({effective_budget}s) before evaluation"), config=resolved_config, rubric_hash=rubric_hash, template_hash=template_hash, @@ -908,7 +1125,29 @@ async def _one(index: int, artifact_id: str, artifact_text: str, criterion: Crit "model_unique_id": model_unique_id, "completed_count": counters["completed"], "degraded_count": counters["degraded"], - "total_budget_seconds": total_budget_seconds, + "effective_budget_seconds": effective_budget, + } + ), + ) + + # Emit ONE ceiling WARNING (distinct from the wall-clock budget + # WARNING above) when any opt-in cost/calls/tokens ceiling tripped + # (DEC-007). Locked field set: ``run_id``, ``model_unique_id``, + # ``ceiling`` (∈ {"calls","cost_usd","tokens"}), ``limit``, + # ``completed_count``, ``degraded_count``. Lazy-format JSON per the + # ANSI-safe logger grep gate — no f-string interpolation of + # user/config data into the logger call. + if tripped["ceiling"] is not None: + _LOGGER.warning( + "grade ceiling exceeded: %s", + json.dumps( + { + "run_id": run_id, + "model_unique_id": model_unique_id, + "ceiling": tripped["ceiling"], + "limit": tripped["limit"], + "completed_count": counters["completed"], + "degraded_count": counters["degraded"], } ), ) @@ -971,9 +1210,10 @@ def grade_artifacts( 4. Generate ``run_id`` (uuid4 hex, DEC-020). Compute the run's ``rubric_hash`` (DEC-014), ``prompt_version_template``, ``rubric_block`` (cached prefix for every call). - 5. Iterate every ``(criterion, artifact)`` pair. At the top of - each loop iteration, check the wall-clock against - ``config.total_budget_seconds``; once exceeded, every + 5. Iterate every ``(criterion, artifact)`` pair, bounded by the + scaled wall-clock backstop ``_compute_effective_budget(...)`` + (DEC-001 — base + per-pair × waves, optionally capped by + ``config.total_budget_seconds``); once exceeded, every remaining pair lands as a degraded ``GradingResult(score=None, ...)`` plus matching :class:`GradeEvent` (DEC-015). Per-pair LLM failures @@ -1264,7 +1504,7 @@ def grade_artifacts( # 5. Iterate ``(criterion, artifact)`` pairs via the async core # (issue #186, US-009 / DEC-002 + DEC-004). The async core wraps - # a ``TaskGroup`` in ``asyncio.timeout(total_budget_seconds)`` + # a ``TaskGroup`` in ``asyncio.timeout(effective_budget)`` # and dispatches up to ``max_concurrent_calls`` coroutines via a # ``Semaphore``. Per-coroutine ``try/except`` handles LLM-layer # failures and budget-cancellation; the public sync entry-point diff --git a/tests/fixtures/dbt_project_austin/signalforge.yml b/tests/fixtures/dbt_project_austin/signalforge.yml index 5f260842..b122f2c5 100644 --- a/tests/fixtures/dbt_project_austin/signalforge.yml +++ b/tests/fixtures/dbt_project_austin/signalforge.yml @@ -11,4 +11,4 @@ grade: min_pass_rate: 0.95 min_mean_score: 0.95 fail_on_below_threshold: false - total_budget_seconds: 600 # default 300 is tight at p99 latency × ~12 calls. + total_budget_seconds: 600 # absolute hard cap; #198 default (None) uses the scaled formula. diff --git a/tests/fixtures/grade/example_config.yml b/tests/fixtures/grade/example_config.yml index 88a0a92b..a13e39e0 100644 --- a/tests/fixtures/grade/example_config.yml +++ b/tests/fixtures/grade/example_config.yml @@ -6,7 +6,12 @@ grade: max_retries_429: 3 # Rate-limit retry budget max_retries_5xx: 1 max_retries_conn: 1 - total_budget_seconds: 300 # Wall-clock budget across the whole run + budget_base_seconds: 60 # scaled-budget constant term (#198) + budget_per_pair_seconds: 20.0 # scaled-budget per concurrency-wave allowance (#198) + total_budget_seconds: 300 # optional ABSOLUTE hard cap; effective = min(scaled, this). Omit (None) to use the scaled formula alone. + # max_grade_calls: 500 # opt-in soft ceiling: stop scheduling new pairs after N judge calls (remaining pairs degrade) + # max_grade_cost_usd: 1.50 # opt-in soft ceiling: stop once accumulated USD meets/exceeds this (degrade rest) + # max_grade_tokens: 2000000 # opt-in soft ceiling: stop once total token movement meets/exceeds this (degrade rest) min_pass_rate: 0.7 # Aggregate threshold: fraction of passed criteria min_mean_score: 0.5 # Aggregate threshold: mean score across criteria fail_on_below_threshold: false # v0.1 always false (flag-only); v0.2 will gate exit code diff --git a/tests/grade/test_config.py b/tests/grade/test_config.py index ccf4bd54..26c2b610 100644 --- a/tests/grade/test_config.py +++ b/tests/grade/test_config.py @@ -242,7 +242,16 @@ def test_grade_config_defaults_match_dec_023_to_027() -> None: assert cfg.max_retries_429 == 3 assert cfg.max_retries_5xx == 1 assert cfg.max_retries_conn == 1 - assert cfg.total_budget_seconds == 300 + # #198 DEC-001: total_budget_seconds is now an OPTIONAL absolute hard + # ceiling; None (the new default) routes the engine to the scaled formula. + assert cfg.total_budget_seconds is None + # #198 DEC-001: scaled-budget terms (always on). + assert cfg.budget_base_seconds == 60 + assert cfg.budget_per_pair_seconds == 20.0 + # #198 DEC-002: three opt-in soft ceilings, all default off (None). + assert cfg.max_grade_calls is None + assert cfg.max_grade_cost_usd is None + assert cfg.max_grade_tokens is None assert cfg.max_concurrent_calls == 10 assert cfg.min_pass_rate == 0.7 assert cfg.min_mean_score == 0.5 @@ -484,7 +493,8 @@ def test_grade_config_max_output_tokens_negative_rejected(tmp_path: Path) -> Non def test_grade_config_total_budget_seconds_zero_rejected(tmp_path: Path) -> None: """A zero total budget would route every criterion to the degraded - path before any LLM call; refuse at config-load time.""" + path before any LLM call; refuse at config-load time. (#198 DEC-001: + the field is now optional, but a *present* value must still be positive.)""" config_path = tmp_path / "signalforge.yml" config_path.write_text( "grade:\n total_budget_seconds: 0\n", @@ -494,6 +504,144 @@ def test_grade_config_total_budget_seconds_zero_rejected(tmp_path: Path) -> None load_grade_config(tmp_path) +# ----- #198 DEC-001/DEC-002: scaled-budget + soft-ceiling validators ----- + + +def test_grade_config_total_budget_seconds_none_accepted() -> None: + """#198 DEC-001: ``None`` is the new default and means "use the scaled + formula" — the allow-None-or-positive validator passes it through.""" + cfg = GradeConfig(total_budget_seconds=None) + assert cfg.total_budget_seconds is None + + +def test_grade_config_total_budget_seconds_explicit_int_accepted() -> None: + """#198 DEC-001: an explicit int still validates — it acts as an absolute + hard cap (``min(scaled, total_budget_seconds)``), preserving v0.1 pinned + configs.""" + cfg = GradeConfig(total_budget_seconds=600) + assert cfg.total_budget_seconds == 600 + + +def test_grade_config_budget_base_seconds_zero_rejected() -> None: + """#198 DEC-001: ``budget_base_seconds`` is an always-on scaled-budget + term; a non-positive value would size the wall-clock backstop to ~0 and + degrade every pair. Fail loud.""" + from pydantic import ValidationError + + with pytest.raises(ValidationError): + GradeConfig(budget_base_seconds=0) + + +def test_grade_config_budget_per_pair_seconds_zero_rejected() -> None: + """#198 DEC-001: ``budget_per_pair_seconds`` must be positive (the + per-wave wall allowance can't be zero).""" + from pydantic import ValidationError + + with pytest.raises(ValidationError): + GradeConfig(budget_per_pair_seconds=0.0) + + +def test_grade_config_budget_per_pair_seconds_negative_rejected() -> None: + """#198 DEC-001: a negative per-wave allowance is rejected.""" + from pydantic import ValidationError + + with pytest.raises(ValidationError): + GradeConfig(budget_per_pair_seconds=-1.0) + + +def test_grade_config_max_grade_calls_zero_rejected() -> None: + """#198 DEC-002: the opt-in soft ceilings accept ``None`` but reject a + present ``<= 0`` value (a zero ceiling would trip immediately and degrade + the whole run).""" + from pydantic import ValidationError + + with pytest.raises(ValidationError): + GradeConfig(max_grade_calls=0) + + +def test_grade_config_max_grade_cost_usd_zero_rejected() -> None: + """#198 DEC-002: a present ``max_grade_cost_usd`` must be positive.""" + from pydantic import ValidationError + + with pytest.raises(ValidationError): + GradeConfig(max_grade_cost_usd=0.0) + + +def test_grade_config_max_grade_tokens_zero_rejected() -> None: + """#198 DEC-002: a present ``max_grade_tokens`` must be positive.""" + from pydantic import ValidationError + + with pytest.raises(ValidationError): + GradeConfig(max_grade_tokens=0) + + +def test_grade_config_optional_positive_fields_reject_negative() -> None: + """#198 DEC-001/002: the optional-positive validator rejects a present + NEGATIVE value (not just zero) for every field it guards. Zero-rejection + alone would still pass if a future refactor slipped from ``<= 0`` to + ``== 0``/``!= 0`` while letting negatives through; pin negatives too.""" + from pydantic import ValidationError + + for kwargs in ( + {"total_budget_seconds": -1}, + {"budget_base_seconds": -1}, + {"max_grade_calls": -1}, + {"max_grade_cost_usd": -0.01}, + {"max_grade_tokens": -1}, + ): + with pytest.raises(ValidationError): + GradeConfig(**kwargs) # pyright: ignore[reportArgumentType] + + +def test_grade_config_float_fields_reject_non_finite() -> None: + """#198 (PR review): the float-bearing knobs reject NaN / +/-inf. + + ``yaml.safe_load`` parses ``.nan`` / ``.inf``, and Pydantic floats allow + them by default. ``nan <= 0`` / ``inf <= 0`` are both ``False``, so without + an explicit finiteness guard a NaN ``budget_per_pair_seconds`` would slip + through and later crash ``math.ceil(nan)`` in ``_compute_effective_budget``; + ``max_grade_cost_usd: .inf`` would silently make the cost ceiling a no-op.""" + from pydantic import ValidationError + + for kwargs in ( + {"budget_per_pair_seconds": float("nan")}, + {"budget_per_pair_seconds": float("inf")}, + {"budget_per_pair_seconds": float("-inf")}, + {"max_grade_cost_usd": float("nan")}, + {"max_grade_cost_usd": float("inf")}, + ): + with pytest.raises(ValidationError): + GradeConfig(**kwargs) # pyright: ignore[reportArgumentType] + + +def test_grade_config_soft_ceilings_accept_none() -> None: + """#198 DEC-002: all three opt-in soft ceilings accept ``None`` (off) — + the explicit-None path mirrors the default.""" + cfg = GradeConfig(max_grade_calls=None, max_grade_cost_usd=None, max_grade_tokens=None) + assert cfg.max_grade_calls is None + assert cfg.max_grade_cost_usd is None + assert cfg.max_grade_tokens is None + + +def test_grade_config_soft_ceilings_accept_positive() -> None: + """#198 DEC-002: a present positive value for each soft ceiling + validates.""" + cfg = GradeConfig(max_grade_calls=50, max_grade_cost_usd=1.25, max_grade_tokens=500_000) + assert cfg.max_grade_calls == 50 + assert cfg.max_grade_cost_usd == 1.25 + assert cfg.max_grade_tokens == 500_000 + + +def test_grade_config_typo_max_grade_cal_fails_loud() -> None: + """#198 / safety-layer.md DEC-015: ``extra="forbid"`` still rejects a typo + on a new ceiling key (``max_grade_cal`` missing ``ls``) rather than + silently leaving the ceiling off.""" + from pydantic import ValidationError + + with pytest.raises(ValidationError): + GradeConfig(max_grade_cal=5) # pyright: ignore[reportCallIssue] + + def test_grade_config_max_retries_negative_rejected(tmp_path: Path) -> None: """Retries are non-negative; ``-1`` would silently become "no retries" if not validated.""" @@ -661,6 +809,15 @@ def test_load_grade_config_doc_example_round_trips(tmp_path: Path) -> None: assert config.min_pass_rate == 0.7 assert config.min_mean_score == 0.5 assert config.fail_on_below_threshold is False + # #198: the fixture carries the two always-on scaled-budget terms; pin + # fixture<->loader parity explicitly (a fixture typo that happened to match + # another valid key would otherwise pass via the defaults test alone). The + # three opt-in ceilings are commented out in the fixture, so they load None. + assert config.budget_base_seconds == 60 + assert config.budget_per_pair_seconds == 20.0 + assert config.max_grade_calls is None + assert config.max_grade_cost_usd is None + assert config.max_grade_tokens is None def test_load_grade_config_full_well_formed_block(tmp_path: Path) -> None: diff --git a/tests/grade/test_engine.py b/tests/grade/test_engine.py index 6074a81b..f44a9791 100644 --- a/tests/grade/test_engine.py +++ b/tests/grade/test_engine.py @@ -49,7 +49,7 @@ ) from signalforge.grade.models import GradeEvent, GradingReport from signalforge.grade.rubric import DEFAULT_RUBRIC, Criterion, Rubric -from signalforge.llm.errors import LLMRateLimitError +from signalforge.llm.errors import EstimateUnknownModelError, LLMRateLimitError from signalforge.manifest.models import Column, Manifest, Model from signalforge.prune.models import PruneResult from tests.grade._fake import expect_grade_responses @@ -505,6 +505,15 @@ def test_grade_artifacts_budget_exceeded_marks_remaining_pairs_score_none( assert all(r.score is None for r in report.results) assert all(r.passed is False for r in report.results) assert report.aggregate_complete is False + # DEC-009: the wall-clock degrade reason embeds the EFFECTIVE budget, not + # the raw config field. The tiny-budget config sets total_budget_seconds=1, + # which caps effective = min(scaled, 1) = 1, so the locked string reads + # "(1s)". Pin it verbatim — this is the one DEC-009 reason string the other + # tests don't `==`-pin, so a wording drift (or a regression that reverts to + # interpolating the raw total instead of effective) would otherwise slip. + assert all( + r.reasoning == "grade budget exceeded (1s) before evaluation" for r in report.results + ) def test_grade_artifacts_budget_exceeded_aggregate_complete_is_false( @@ -716,6 +725,129 @@ def test_format_degrade_reasoning_preserves_bare_shape_for_non_response_format_c assert _format_degrade_reasoning(parser) == "call failed: GradeOutputError" +# --------------------------------------------------------------------------- +# Scaled wall-clock budget formula (#198 DEC-001 / DEC-010) +# --------------------------------------------------------------------------- + + +def test_compute_effective_budget_scaled_no_cap() -> None: + """``total_budget_seconds=None`` returns the scaled value verbatim. + + base=60, per_pair=20.0, 220 pairs @ concurrency=10 → + ceil(220/10)=22 waves → 60 + 20*22 = 500. + """ + result = engine_module._compute_effective_budget( + budget_base_seconds=60, + budget_per_pair_seconds=20.0, + total_budget_seconds=None, + num_pairs=220, + max_concurrent_calls=10, + ) + assert result == 500 + assert isinstance(result, int) + + +def test_compute_effective_budget_cap_wins() -> None: + """When the absolute ceiling is below the scaled value, the cap wins.""" + result = engine_module._compute_effective_budget( + budget_base_seconds=60, + budget_per_pair_seconds=20.0, + total_budget_seconds=300, + num_pairs=220, + max_concurrent_calls=10, + ) + assert result == 300 + assert isinstance(result, int) + + +def test_compute_effective_budget_scaled_wins() -> None: + """When the scaled value is below the absolute ceiling, scaled wins.""" + result = engine_module._compute_effective_budget( + budget_base_seconds=60, + budget_per_pair_seconds=20.0, + total_budget_seconds=900, + num_pairs=220, + max_concurrent_calls=10, + ) + assert result == 500 + assert isinstance(result, int) + + +def test_compute_effective_budget_single_pair() -> None: + """One pair still costs one full wave: 60 + 20*ceil(1/10) = 80.""" + result = engine_module._compute_effective_budget( + budget_base_seconds=60, + budget_per_pair_seconds=20.0, + total_budget_seconds=None, + num_pairs=1, + max_concurrent_calls=10, + ) + assert result == 80 + assert isinstance(result, int) + + +def test_compute_effective_budget_zero_pairs_returns_base() -> None: + """``num_pairs == 0`` short-circuits to ``budget_base_seconds``.""" + result = engine_module._compute_effective_budget( + budget_base_seconds=60, + budget_per_pair_seconds=20.0, + total_budget_seconds=None, + num_pairs=0, + max_concurrent_calls=10, + ) + assert result == 60 + assert isinstance(result, int) + + +def test_compute_effective_budget_non_multiple_rounds_up() -> None: + """A non-multiple pair count rounds the wave count up via ceil. + + base=60, per_pair=20.0, 221 pairs @ concurrency=10 → + ceil(221/10)=ceil(22.1)=23 waves → 60 + 20*23 = 520. + """ + result = engine_module._compute_effective_budget( + budget_base_seconds=60, + budget_per_pair_seconds=20.0, + total_budget_seconds=None, + num_pairs=221, + max_concurrent_calls=10, + ) + assert result == 520 + assert isinstance(result, int) + + +def test_compute_effective_budget_fractional_per_pair_never_rounds_down() -> None: + """#198 (PR review): a fractional ``budget_per_pair_seconds`` must round the + final budget UP, not truncate it — a smaller backstop would trip earlier + than the operator intended. base=60, per_pair=20.5, 220 pairs @ c=10 → + 22 waves → 60 + 20.5*22 = 511.0 (already whole); use 21.5 to force a + fraction: 60 + 21.5*22 = 533.0 ... pick per_pair=20.3 → 60 + 20.3*22 = + 506.6 → ceil = 507 (``int`` would give 506).""" + result = engine_module._compute_effective_budget( + budget_base_seconds=60, + budget_per_pair_seconds=20.3, + total_budget_seconds=None, + num_pairs=220, + max_concurrent_calls=10, + ) + assert result == 507 # ceil(506.6), NOT int(506.6)=506 + assert isinstance(result, int) + + +def test_compute_effective_budget_cap_fractional_never_rounds_down() -> None: + """#198 (PR review): the absolute-cap branch also ceils (never truncates) + the post-``min`` value. cap=250.5 below scaled → min=250.5 → ceil=251.""" + result = engine_module._compute_effective_budget( + budget_base_seconds=60, + budget_per_pair_seconds=20.0, + total_budget_seconds=250, # int cap below scaled (500) → 250 + num_pairs=220, + max_concurrent_calls=10, + ) + assert result == 250 + assert isinstance(result, int) + + # --------------------------------------------------------------------------- # Whole-run pre-flight envelope-breach (DEC-013) # --------------------------------------------------------------------------- @@ -1748,13 +1880,15 @@ def test_grade_artifacts_concurrent_budget_warning_shape_locked( caplog: pytest.LogCaptureFixture, ) -> None: """On budget trip the engine emits exactly one WARNING with the - locked JSON field set (DEC-018 of #186): ``run_id``, - ``model_unique_id``, ``completed_count``, ``cancelled_count``, - ``degraded_count``, ``total_budget_seconds``. + locked JSON field set (DEC-018 of #186; ``effective_budget_seconds`` + rename per DEC-008 of #198): ``run_id``, ``model_unique_id``, + ``completed_count``, ``degraded_count``, ``effective_budget_seconds``. External operator dashboards key on these field names; locking the shape via a pinned test is what makes the audit corpus a stable - contract. + contract. ``effective_budget_seconds`` carries the value actually + passed to ``asyncio.timeout`` — the scaled budget capped by + ``total_budget_seconds`` when set (DEC-001/DEC-008 of #198). """ project_dir = _project(tmp_path) model = _make_model() @@ -1795,16 +1929,383 @@ def test_grade_artifacts_concurrent_budget_warning_shape_locked( "model_unique_id", "completed_count", "degraded_count", - "total_budget_seconds", + "effective_budget_seconds", } assert payload["model_unique_id"] == model.unique_id - assert payload["total_budget_seconds"] == 1 + # The tiny-budget config sets ``total_budget_seconds=1``, so the + # effective budget is ``min(scaled, 1) == 1`` (the absolute cap + # wins). DEC-001/DEC-008 of #198. + config = _config_tiny_budget() + expected_effective = engine_module._compute_effective_budget( + budget_base_seconds=config.budget_base_seconds, + budget_per_pair_seconds=config.budget_per_pair_seconds, + total_budget_seconds=config.total_budget_seconds, + num_pairs=len(rubric) * len(_stable_artifact_pairs(candidate)), + max_concurrent_calls=config.max_concurrent_calls, + ) + assert expected_effective == 1 + assert payload["effective_budget_seconds"] == expected_effective # Every pair accounted for: completed + degraded == total. candidate_pairs = len(_stable_artifact_pairs(candidate)) total_pairs = len(rubric) * candidate_pairs assert payload["completed_count"] + payload["degraded_count"] == total_pairs +def _make_candidate_with_n_columns(n: int) -> CandidateSchema: + """Build a :class:`CandidateSchema` with ``n`` columns, each carrying + a description, a rationale, and one ``not_null`` test. + + Drives the ≥40-column acceptance-criterion test below: a wide model + produces many ``(artifact × criterion)`` pairs, exercising the + scaled wall-clock budget (DEC-001 of #198). + """ + columns = tuple( + CandidateColumn( + name=f"col_{i}", + description=f"column {i} description", + rationale=f"column {i} rationale", + tests=(CandidateTestNotNull(column=f"col_{i}"),), + ) + for i in range(n) + ) + return CandidateSchema( + name="orders", + description="wide model description", + rationale="wide model rationale", + columns=columns, + tests=(), + ) + + +def test_grade_artifacts_wide_model_completes_with_zero_budget_degradations( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + """A ≥40-column model completes under the DEFAULT GradeConfig with + ZERO width-induced budget degradations (the #198 acceptance criterion). + + With the scaled wall-clock budget (DEC-001 — ``budget_base_seconds`` + + ``budget_per_pair_seconds`` × waves, no absolute cap by default), + a 40-column candidate (122 artifacts × 4 default-rubric criteria = + 488 pairs) finishes well inside the backstop when every call returns + instantly via the fake. The proof: ``aggregate_complete is True``, + every result scored (no ``score is None``), and NO budget WARNING. + """ + project_dir = _project(tmp_path) + model = _make_model() + candidate = _make_candidate_with_n_columns(40) + # Sanity: a wide model produces many pairs (40 cols × 3 artifacts + # + 2 model-level = 122 artifacts × 4 criteria = 488 pairs). + artifact_count = len(_stable_artifact_pairs(candidate)) + assert artifact_count == 122 + + fake = FakeAnthropicClient() + expect_grade_responses(fake, rubric=DEFAULT_RUBRIC, candidate=candidate) + + caplog.set_level(logging.WARNING, logger="signalforge.grade.engine") + report = grade_artifacts( + model, + candidate, + _empty_prune_result(model), + # No rubric / config args → DEFAULT_RUBRIC + default GradeConfig + # (no ceilings, scaled budget). + client=fake, + project_dir=project_dir, + ) + + # Every pair scored — zero width-induced budget degradations. + assert report.aggregate_complete is True + assert all(r.score is not None for r in report.results) + assert len(report.results) == artifact_count * len(DEFAULT_RUBRIC) + + # No budget WARNING was emitted. + budget_warns = [ + r + for r in caplog.records + if r.name == "signalforge.grade.engine" + and r.levelno == logging.WARNING + and "grade budget exceeded" in r.getMessage() + ] + assert budget_warns == [] + + +# --------------------------------------------------------------------------- +# Opt-in cost/calls/tokens ceilings (US-004 / DEC-002/003/007/009/010/011) +# --------------------------------------------------------------------------- + + +def _ceiling_warns(caplog: pytest.LogCaptureFixture) -> list[dict[str, Any]]: + """Collect every ``grade ceiling exceeded`` WARNING payload.""" + payloads: list[dict[str, Any]] = [] + for r in caplog.records: + if ( + r.name == "signalforge.grade.engine" + and r.levelno == logging.WARNING + and "grade ceiling exceeded" in r.getMessage() + ): + payload_json = r.getMessage().split("grade ceiling exceeded: ", 1)[1] + payloads.append(json.loads(payload_json)) + return payloads + + +def test_grade_artifacts_max_grade_calls_ceiling_degrades_remaining( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + """``max_grade_calls=K`` scores exactly K pairs and degrades the rest + with the locked reason; one ceiling WARNING with ``ceiling="calls"``. + + ``max_concurrent_calls=1`` serialises dispatch so the first K pairs + (in iteration order) reserve the K slots and the remainder degrade — + making the "exactly K scored" assertion deterministic (the + check-then-reserve is race-free under any concurrency, but the WHICH-K + is only deterministic when serialised). + """ + project_dir = _project(tmp_path) + model = _make_model() + candidate = _load_sample_candidate() + rubric = _two_criteria() + fake = FakeAnthropicClient() + expect_grade_responses(fake, rubric=rubric, candidate=candidate) + + total_pairs = len(rubric) * len(_stable_artifact_pairs(candidate)) + assert total_pairs == 14 # 7 artifacts × 2 criteria + k = 5 + config = GradeConfig( + model="claude-fake", + cache_ttl="1h", + max_output_tokens=64, + max_retries_429=0, + max_retries_5xx=0, + max_retries_conn=0, + max_concurrent_calls=1, + max_grade_calls=k, + ) + + caplog.set_level(logging.WARNING, logger="signalforge.grade.engine") + report = grade_artifacts( + model, + candidate, + _empty_prune_result(model), + rubric=rubric, + config=config, + client=fake, + project_dir=project_dir, + ) + + scored = [r for r in report.results if r.score is not None] + degraded = [r for r in report.results if r.score is None] + assert len(scored) == k + assert len(degraded) == total_pairs - k + assert report.aggregate_complete is False + for r in degraded: + assert r.reasoning == f"grade call ceiling exceeded ({k} calls)" + + warns = _ceiling_warns(caplog) + assert len(warns) == 1 + payload = warns[0] + assert set(payload.keys()) == { + "run_id", + "model_unique_id", + "ceiling", + "limit", + "completed_count", + "degraded_count", + } + assert payload["model_unique_id"] == model.unique_id + assert payload["ceiling"] == "calls" + assert payload["limit"] == k + # Ceiling-degrades count as completed (DEC-011); the synthesis pass + # never runs (no budget trip), so degraded_count stays 0. + assert payload["completed_count"] == total_pairs + assert payload["degraded_count"] == 0 + + +def test_grade_artifacts_max_grade_cost_usd_ceiling_degrades_remaining( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + """``max_grade_cost_usd`` degrades pairs once the accumulated USD + crosses the cap; locked reason + ``ceiling="cost_usd"`` WARNING. + + Uses a REAL SKU (``claude-sonnet-4-6``) so ``pricing.lookup`` resolves. + Each call's usage (input=1700, output=140, cache_read=1500) prices to + $0.00765, so a $0.03 cap fits exactly 4 calls + (4 × 0.00765 = 0.0306 ≥ 0.03 trips the 5th pair's check). + """ + project_dir = _project(tmp_path) + model = _make_model() + candidate = _load_sample_candidate() + rubric = _two_criteria() + fake = FakeAnthropicClient() + expect_grade_responses(fake, rubric=rubric, candidate=candidate) + + total_pairs = len(rubric) * len(_stable_artifact_pairs(candidate)) + cap = 0.03 + config = GradeConfig( + model="claude-sonnet-4-6", + cache_ttl="1h", + max_output_tokens=64, + max_retries_429=0, + max_retries_5xx=0, + max_retries_conn=0, + max_concurrent_calls=1, + max_grade_cost_usd=cap, + ) + + caplog.set_level(logging.WARNING, logger="signalforge.grade.engine") + report = grade_artifacts( + model, + candidate, + _empty_prune_result(model), + rubric=rubric, + config=config, + client=fake, + project_dir=project_dir, + ) + + scored = [r for r in report.results if r.score is not None] + degraded = [r for r in report.results if r.score is None] + assert len(scored) == 4 + assert len(degraded) == total_pairs - 4 + assert report.aggregate_complete is False + for r in degraded: + assert r.reasoning == f"grade cost ceiling exceeded (${cap})" + + warns = _ceiling_warns(caplog) + assert len(warns) == 1 + assert warns[0]["ceiling"] == "cost_usd" + assert warns[0]["limit"] == cap + assert warns[0]["completed_count"] == total_pairs + assert warns[0]["degraded_count"] == 0 + + +def test_grade_artifacts_cost_ceiling_unpriced_model_fails_fast_before_any_call( + tmp_path: Path, +) -> None: + """A cost ceiling on a prefix-valid-but-unpriced SKU fails fast at + orchestrator entry — BEFORE any (billable) LLM call — rather than aborting + mid-run from inside the TaskGroup after paid calls. + + ``GradeConfig._validate_model_provider_compat`` checks only the SKU prefix + (``claude-``), so ``claude-opus-4-8`` (absent from ``pricing.PRICES``) is + accepted at config-load. With ``max_grade_cost_usd`` set, the engine + resolves pricing once up front; an unknown SKU raises + ``EstimateUnknownModelError`` before dispatch. The fake client is given NO + queued responses: if the engine reached a grade call it would raise a + different ("unexpected call") error, so asserting ``EstimateUnknownModelError`` + proves the failure preceded every LLM call. + """ + project_dir = _project(tmp_path) + model = _make_model() + candidate = _load_sample_candidate() + rubric = _two_criteria() + fake = FakeAnthropicClient() # deliberately no expectations queued + + config = GradeConfig( + model="claude-opus-4-8", # claude- prefix (valid) but NOT in PRICES + max_concurrent_calls=1, + max_grade_cost_usd=0.01, + ) + + with pytest.raises(EstimateUnknownModelError): + grade_artifacts( + model, + candidate, + _empty_prune_result(model), + rubric=rubric, + config=config, + client=fake, + project_dir=project_dir, + ) + + +def test_grade_artifacts_max_grade_tokens_ceiling_degrades_remaining( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + """``max_grade_tokens`` degrades pairs once accumulated token movement + crosses the cap; locked reason + ``ceiling="tokens"`` WARNING. + + Each call moves 1700+140+0+1500 = 3340 tokens, so a 10000-token cap + fits exactly 3 calls (3 × 3340 = 10020 ≥ 10000 trips the 4th pair). + """ + project_dir = _project(tmp_path) + model = _make_model() + candidate = _load_sample_candidate() + rubric = _two_criteria() + fake = FakeAnthropicClient() + expect_grade_responses(fake, rubric=rubric, candidate=candidate) + + total_pairs = len(rubric) * len(_stable_artifact_pairs(candidate)) + cap = 10000 + config = GradeConfig( + model="claude-fake", + cache_ttl="1h", + max_output_tokens=64, + max_retries_429=0, + max_retries_5xx=0, + max_retries_conn=0, + max_concurrent_calls=1, + max_grade_tokens=cap, + ) + + caplog.set_level(logging.WARNING, logger="signalforge.grade.engine") + report = grade_artifacts( + model, + candidate, + _empty_prune_result(model), + rubric=rubric, + config=config, + client=fake, + project_dir=project_dir, + ) + + scored = [r for r in report.results if r.score is not None] + degraded = [r for r in report.results if r.score is None] + assert len(scored) == 3 + assert len(degraded) == total_pairs - 3 + assert report.aggregate_complete is False + for r in degraded: + assert r.reasoning == f"grade token ceiling exceeded ({cap} tokens)" + + warns = _ceiling_warns(caplog) + assert len(warns) == 1 + assert warns[0]["ceiling"] == "tokens" + assert warns[0]["limit"] == cap + + +def test_grade_artifacts_no_ceiling_emits_no_ceiling_warning( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, +) -> None: + """A default-config run (all ceilings ``None``) emits NO + ``grade ceiling exceeded`` WARNING and degrades nothing (regression + guard for the opt-in default-off contract). + """ + project_dir = _project(tmp_path) + model = _make_model() + candidate = _load_sample_candidate() + rubric = _two_criteria() + fake = FakeAnthropicClient() + expect_grade_responses(fake, rubric=rubric, candidate=candidate) + + caplog.set_level(logging.WARNING, logger="signalforge.grade.engine") + report = grade_artifacts( + model, + candidate, + _empty_prune_result(model), + rubric=rubric, + config=_config_no_audit_in_path(), + client=fake, + project_dir=project_dir, + ) + + assert report.aggregate_complete is True + assert all(r.score is not None for r in report.results) + assert _ceiling_warns(caplog) == [] + + def test_grade_artifacts_module_level_async_sleep_alias_present() -> None: """The ``_async_sleep`` alias is module-scoped and reassignable for deterministic budget tests, mirroring :data:`signalforge.llm.client._async_sleep`