diff --git a/.claude/rules/llm-drafter.md b/.claude/rules/llm-drafter.md index ccbe6641..32c71e9f 100644 --- a/.claude/rules/llm-drafter.md +++ b/.claude/rules/llm-drafter.md @@ -14,14 +14,32 @@ Every `# pyright: ignore[...]` and `# type: ignore[...]` comment for the Anthrop `call_llm(*, system, cached_block, dynamic_block, model, max_tokens, cache_ttl="5m", prompt_version, max_retries_*, provider="anthropic", client=None) -> LLMResult` (renamed from `call_anthropic` by #135, which dropped the old name) is the single shared seam for **both** the drafter and grader. It owns the generic machinery — retry loop + backoff (`2**attempt*_rand_uniform(0.75,1.25)`), per-class budgets, WARNING/INFO logs, the min/cap token validation, and `LLMResult` assembly — and dispatches the vendor-specific bits to a provider strategy resolved from a registry. When `client is None`, `call_llm` builds it via `strategy.make_client()` (DEC-006 — client construction lives at the seam, not the CLI). -- **`LLMProvider` ABC + registry** in `signalforge.llm.providers`: `register_provider(provider)` / `provider_for(name) -> LLMProvider`; unknown name → `UnknownProviderError(LLMError)` listing available keys (CLI tier 2). A provider supplies `make_client`, `build_create_kwargs`, `build_count_tokens_kwargs`, `extract_text_blocks`, `extract_usage` (→ `UsageMetrics`), `classify_exception` (→ `ExceptionCategory`), plus capability flags `supports_prompt_caching` / `supports_token_count`. Wiring a new provider = that class + `register_provider` + a config enum value. `AnthropicProvider` (`name="anthropic"`, both flags `True`) is the only one registered in v0.x. +- **`LLMProvider` ABC + registry** in `signalforge.llm.providers`: `register_provider(provider)` / `provider_for(name) -> LLMProvider`; unknown name → `UnknownProviderError(LLMError)` listing available keys (CLI tier 2). A provider supplies `make_client`, `build_create_kwargs`, `build_count_tokens_kwargs`, `extract_text_blocks`, `extract_usage` (→ `UsageMetrics`), `classify_exception` (→ `ExceptionCategory`), `estimate_input_tokens(model, text, *, system="", client=None) -> int` (#136 US-005; powers `--estimate`), plus capability flags `supports_prompt_caching` / `supports_token_count`. Wiring a new provider = that class + `register_provider` + a config enum value. Registered in v0.3: `AnthropicProvider` (`name="anthropic"`, both flags `True`); `OpenAIProvider` (`name="openai"`, both flags `False`, #136). - **Neutral value objects:** `UsageMetrics` + the `ExceptionCategory` enum (`AUTH`, `RATE_LIMIT`, `SERVER_ERROR`, `CONNECTION`, `NO_RETRY`) keep the orchestrator off vendor-shaped dicts. - **Capability-gated behaviour (DEC-008):** `supports_prompt_caching=False` ⇒ no `cache_control` marker, no `extended-cache-ttl` beta header, 0 cache tokens, no dual-zero anomaly WARNING. `supports_token_count=False` ⇒ skip the pre-send count gate (no pre-send `LLMCacheTooLargeError`). Anthropic sets both `True`, so its emitted bytes/control flow are unchanged — the byte-identity gate (fixtures + prompt-cache snapshot + drift detectors) is the regression guard. - **`provider` config field (DEC-007):** `DraftConfig.provider` (`llm:` block) and `GradeConfig.provider` (`grade:` block), both registry-validated `str` defaulting to `"anthropic"` — **deliberately NOT a `Literal`** (a registry is a plugin point that grows; #136/#137 register a provider instead of editing a Literal in two configs). The validator raises `UnknownProviderError` (an `LLMError`, so Pydantic v2 does NOT wrap it into `ValidationError` — it propagates raw with the available-keys remediation). **Gate the cache marker on BOTH capability flags, not just `supports_prompt_caching` (#135 QG lesson).** `call_llm` sets `cache_marker_active = supports_prompt_caching AND supports_token_count`. The pre-send count gate is what enforces the sub-minimum drop + the 8000-token oversize cap; attaching a `cache_control` marker without that gate having run would send an *unvalidated* marker (a sub-minimum block silently no-ops the marker — paying the input premium with no discount; an oversize block bypasses `LLMCacheTooLargeError`). Anthropic is `True/True` so the default path is unaffected, but a future provider that caches yet has no token-count API (`True/False`) must degrade to no-caching rather than send an unguarded marker. A new provider's capability flags are load-bearing — set them honestly, and don't assume "supports caching" alone is sufficient to attach a marker. -When a new vendor lands (#136 OpenAI / #137 Gemini), add a `__client.py` shim + a `LLMProvider` subclass + `register_provider`; don't pool SDK ignores into a generic util module, and don't reach into `call_llm` — extend via the strategy. +When a new vendor lands (#137 Gemini next), add a `__client.py` shim + a `LLMProvider` subclass + `register_provider`; don't pool SDK ignores into a generic util module, and don't reach into `call_llm` — extend via the strategy. + +### OpenAI provider shape (#136 — the second concrete provider, the no-cache precedent) + +`OpenAIProvider` ships under `provider="openai"` for both stages; the shim at `src/signalforge/llm/_openai_client.py` confines every `# pyright: ignore` / `# type: ignore` for the `openai` SDK (DEC-012 of #5 generalised). Three load-bearing patterns established by #136 that the next no-cache vendor should mirror: + +1. **`.messages.create` façade adapter.** The orchestrator hard-calls `llm_client.messages.create(**kwargs)`, but OpenAI's SDK exposes `client.chat.completions.create(...)`. `_OpenAIClientAdapter.messages` is a `SimpleNamespace` instance whose `.create` callable delegates to `chat.completions.create(**kwargs)` and whose `.count_tokens` raises `NotImplementedError` defensively (orchestrator never calls it when `supports_token_count=False`, but the protocol surface is uniform). Any vendor whose SDK uses a different call shape gets the same shim adaptation rather than a special-case branch in `call_llm`. +2. **`response_format={"type":"json_object"}` belt-and-braces with the tolerant JSON parser (DEC-006 of #136).** `OpenAIProvider.build_create_kwargs` attaches the JSON-mode flag at every call. The grade + draft system prompts already contain "json" (case-insensitive), satisfying OpenAI's prompt-requirement check. The tolerant `extract_json_payload` parser (issue #144) remains the fallback if a future model strips the flag; server-side enforcement eliminates the prose-preamble drift class entirely for the providers that support it. **A future vendor with an equivalent JSON-mode (e.g. Gemini's `response_mime_type="application/json"`) MUST set it for the same reason — don't lean on the parser alone when the API exposes a server-side gate.** +3. **`tiktoken` model-id fallback for `--estimate` (DEC-012 of #136).** `_count_openai_tokens(model, text)` calls `tiktoken.encoding_for_model(model)` inside a try/except, falling back to `tiktoken.get_encoding("cl100k_base")` on `KeyError` for unknown ids (newer model SKUs released after the installed `tiktoken` ship out of the registry). **Don't raise on unknown id** — `--estimate` is a calibration signal, not a billing guarantee (mirrors the `EXPLAIN`-based planner-estimate caveat in `warehouse-adapters.md`). + +### `estimate_input_tokens(*, system=...)` — separate the system envelope to preserve real-API byte-identity (#136 US-005 / DEC-013, refined by US-008 QG) + +The `--estimate` path generalised in US-005 via `LLMProvider.estimate_input_tokens(model, text, *, system="", client=None)`. The `system` parameter is keyword-only with an empty default, but **threading it as its own kwarg is load-bearing for Anthropic byte-identity:** + +- **AnthropicProvider** passes `system=system` to `messages.count_tokens(...)` when non-empty. The pre-refactor inline call did the same — and Anthropic's server-side tokenizer applies its own system-envelope tokens to that block. Dropping the kwarg (concatenating system into the user-content text) silently under-reports real-API counts by the system-envelope size, while a fake-driven byte-identity snapshot still passes because the fake returns canned `input_tokens` regardless of kwargs. **Lesson — fake-driven byte-identity tests pin rendered-output identity, NOT real-API call shape.** If a refactor changes the SDK call shape, the snapshot won't catch it; a real-API `@pytest.mark.anthropic` test against `--estimate` (or an explicit kwargs-shape assertion on the fake) is the only way. This is the same "snapshot/sqlglot tier vs live tier" gap documented in `warehouse-adapters.md` for the Snowflake compiler. +- **OpenAIProvider** concatenates `system + text` before `tiktoken` — there's no system-envelope distinction at the BPE level, so every token contributes to the same total. Matches what OpenAI's chat-completion endpoint bills. +- **Grade-side estimate (`_count_grade_criterion_tokens`) deliberately double-counts the rubric.** The pre-refactor shape passed `system=system_and_rubric` AND embedded `system_and_rubric` inside the cached user-content block, so the rubric was counted twice. This was already the behaviour; preserving byte-identity required reproducing it. **Don't "fix" this double-count in a future tidy-pass — it's the pre-refactor shape, and a quiet repair would silently shift estimate figures by ~the rubric size.** A real future cleanup would need a tied snapshot regeneration + operator-visible "estimate calibration changed" CHANGELOG entry. + +A new provider added under #137+ MUST implement `estimate_input_tokens` (it's `@abstractmethod`; missing impls fail at instantiation time). The implementation can ignore `system` (FakeNoCacheProvider stub concatenates for word-count; `_DummyProvider` returns `0`); but if the vendor's API distinguishes system from user tokens (like Anthropic), thread it separately or the same drift surfaces. ## Module-level `_sleep` / `_rand_uniform` aliases (DEC-004) @@ -99,12 +117,15 @@ Same rule as the other layers (`safety-layer.md` DEC-022 / `prune-engine.md` DEC ## AST audit-completeness scans (DEC-013) -`tests/test_audit_completeness.py` runs four AST scans: +`tests/test_audit_completeness.py` runs five AST scans relevant to the LLM seam: - `LLMRequest` constructed only in `signalforge.safety.request` (existing from #4). - `AuditEvent` constructed only in `signalforge.safety.request`. - `anthropic.Anthropic(...)` constructed only in `signalforge.llm._anthropic_client` (DEC-012 — the SDK seam; module renamed from `_client` by #135). - `LLMResponseEvent` constructed only in `signalforge.draft.audit` — every event flows through `_build_response_event`. +- **`openai.OpenAI(...)` constructed only in `signalforge.llm._openai_client`** (the 9th project AST scan, #136 US-001/DEC-010). Mirrors Scan 3 shape exactly — reuses `_AttributeCallFinder`, catches all three bypass patterns (bare via `from openai import OpenAI`, import-alias via `from openai import OpenAI as O`, module-attribute via `import openai; openai.OpenAI(...)`). + +When a new vendor lands (#137 Gemini bumps the tally to 10 for `genai.Client(...)`), add a **new** scan rather than extending an existing one — Scan 3 is Anthropic-specific (it hunts `anthropic.Anthropic`), so the OpenAI / Gemini scans are siblings, not extensions. The companion per-vendor `tests/llm/test__client_confinement.py` mirrors the Snowflake-shaped line-based scan for `# type: ignore` / `# pyright: ignore` confinement. If a new module legitimately needs to construct one of these gated names, update the scan's exclusion list AND document the audit-write seam. Don't suppress the test. @@ -114,4 +135,4 @@ The drafter's config block is `{ llm: { provider, model, cheap_model, max_output ## Reference -`plans/super/5-llm-draft-pipeline.md` — DEC-001 … DEC-027. `plans/super/135-provider-neutral-llm-seam.md` — DEC-001 … DEC-012 (the provider-neutral seam: `call_llm`, `LLMProvider` ABC + registry, capability flags, `provider` config field). `src/signalforge/llm/` (incl. `providers.py` + `_anthropic_client.py`), `src/signalforge/draft/` — current implementation. `tests/llm/_fake.py::FakeAnthropicClient` — `expect_*` API; `tests/llm/_fake_provider.py::FakeNoCacheProvider` + `tests/grade/test_provider_neutrality.py` — the no-cache provider-neutrality proof. `docs/draft-ops.md` — operational reference. `tests/fixtures/draft/llm_response_*.json` — fixture set exercising happy + each error path. +`plans/super/5-llm-draft-pipeline.md` — DEC-001 … DEC-027. `plans/super/135-provider-neutral-llm-seam.md` — DEC-001 … DEC-012 (the provider-neutral seam: `call_llm`, `LLMProvider` ABC + registry, capability flags, `provider` config field). `plans/super/136-openai-grading-provider.md` — DEC-001 … DEC-014 (OpenAI as the second concrete provider: shim + `OpenAIProvider` + four pricing SKUs + `--estimate` strategy refactor + JSON-mode enforcement; the no-cache real-vendor precedent for #137 Gemini). `src/signalforge/llm/` (incl. `providers.py` + `_anthropic_client.py` + `_openai_client.py`), `src/signalforge/draft/` — current implementation. `tests/llm/_fake.py::FakeAnthropicClient` + `tests/llm/_fake_openai.py::FakeOpenAIClient` — `expect_*` API; `tests/llm/_fake_provider.py::FakeNoCacheProvider` + `tests/grade/test_provider_neutrality.py` + `tests/grade/test_provider_neutrality_openai.py` — the no-cache provider-neutrality proofs (synthetic + real OpenAI). `docs/draft-ops.md` / `docs/grade-ops.md` / `docs/cost-estimate-ops.md` — operational references. `tests/fixtures/draft/llm_response_*.json` / `tests/fixtures/estimate/anthropic_byte_identity_golden.txt` — fixture sets exercising happy + each error path + the DEC-013 Anthropic byte-identity floor. diff --git a/CHANGELOG.md b/CHANGELOG.md index c58a1bdd..209f0fd4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,14 @@ All notable changes to SignalForge are documented here. The format is loosely ba ## [Unreleased] -_Nothing yet — entries land here on `dev` and get promoted to a dated section at release time._ +### Added + +- **OpenAI as a grading + drafting provider (#136).** Set `grade.provider: openai` or `llm.provider: openai` in `signalforge.yml`; requires the `[openai]` install extra and `OPENAI_API_KEY`. Ships four pricing SKUs (`gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, `gpt-4-turbo`); `--estimate` works via tiktoken (no extra API round-trip). Server-side JSON enforcement via `response_format={"type": "json_object"}`. v0.3 ships without prompt caching (no Anthropic-style cache discount); follow-up to evaluate OpenAI prompt caching. + +### Fixed + +- **`--estimate` grader-side token counts no longer double-count the rubric (#136 US-008 QG).** The pre-US-005 inline Anthropic call passed the rubric in BOTH the `system=` kwarg AND embedded in the cached user-content block, counting it twice per criterion. The first QG fix preserved that for Anthropic byte-identity, which then triple-counted the rubric for OpenAI (system→`system + text` concat → rubric prefix in text). Corrected to match the runtime grader call: rubric in `system=` once, artifact envelope in user content. Real-API `--estimate` figures for the grade-side shift down by ~one rubric per criterion (was: bug → over-report; now: matches what gets billed). Fake-driven byte-identity golden unchanged (canned token counts are call-shape-agnostic). +- **`estimate(...)` engine parameter renamed `anthropic_client` → `client` (#136 US-008 QG).** Post-US-005 the slot was already typed `object | None` and forwarded verbatim to whichever provider strategy is active; the old name implied Anthropic-only and would mislead a future #137 Gemini wiring. CLI in `generate.py` already passed `None` for non-Anthropic providers; the rename surfaces that without behaviour change. ## [0.3.0] — 2026-05-27 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 59cced4a..e6698051 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -34,10 +34,11 @@ uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run ## Pre-release coverage audit The default `pytest` run — and therefore the coverage badge — measures only the -default marker set. Tests gated behind `bigquery`, `anthropic`, `cli_subprocess`, -`e2e`, and `wheel_smoke` are filtered out by `addopts` (see -`.claude/rules/testing-signal.md` § "Known gap: excluded markers"), so the -real-network and packaging paths are not instrumented in the badge number. +default marker set. Tests gated behind `bigquery`, `anthropic`, `openai`, +`cli_subprocess`, `e2e`, `snowflake`, and `wheel_smoke` are filtered out by +`addopts` (see `.claude/rules/testing-signal.md` § "Known gap: excluded +markers"), so the real-network and packaging paths are not instrumented in the +badge number. Run this audit against the **matrix ceiling** (currently Python 3.13 — the highest version CI exercises) by prefixing `uv run --python 3.13`, so the @@ -56,9 +57,11 @@ uv run pytest # --cov-append combines with run 1 so the term report shows the COMBINED total. # --cov-fail-under=0 overrides the 80% gate inherited from addopts — gated # markers alone never clear it, and this is a measurement, not a gate. -# (bigquery/anthropic/e2e need creds; cli_subprocess/wheel_smoke do not.) -SF_RUN_BQ=1 ANTHROPIC_API_KEY=sk-... GOOGLE_CLOUD_PROJECT= \ - uv run pytest -m 'bigquery or anthropic or e2e or cli_subprocess or wheel_smoke' \ +# (bigquery/anthropic/openai/snowflake/e2e need creds; cli_subprocess/wheel_smoke do not.) +SF_RUN_BQ=1 ANTHROPIC_API_KEY=sk-... OPENAI_API_KEY=sk-... \ + SF_RUN_OPENAI=1 SF_RUN_SNOWFLAKE=1 GOOGLE_CLOUD_PROJECT= \ + SNOWFLAKE_ACCOUNT=... SNOWFLAKE_USER=... SNOWFLAKE_PASSWORD=... SNOWFLAKE_WAREHOUSE=... \ + uv run pytest -m 'bigquery or anthropic or openai or snowflake or e2e or cli_subprocess or wheel_smoke' \ --cov=signalforge --cov-append --cov-fail-under=0 --cov-report=term ``` @@ -129,3 +132,34 @@ SF_RUN_BQ)`. The tests query `bigquery-public-data.samples.shakespeare` (164K rows, free under the 1 TB/month BigQuery tier). They are maintainer-only for v0.1; no CI job runs them. + +## OpenAI live-API smoke tests + +Three tests gated by `@pytest.mark.openai` exercise the OpenAI provider +end-to-end (issue #136): + +- `tests/grade/test_smoke_real_api_openai.py` — `grade_artifacts` against + a tiny in-test fixture, single criterion. +- `tests/draft/test_smoke_real_api_openai.py` — `draft_schema` against a + small in-test manifest; honours DEC-005's "scope both stages" commitment. +- `tests/cli/test_e2e_estimate_openai.py` — `signalforge generate + --estimate` with `llm.provider: openai` + `grade.provider: openai`. + +All three are skipped by default (filtered out by `addopts = -m 'not +openai'`) and additionally self-skip via a runtime gate if either env var +is missing — the belt-and-suspenders pattern from +`.claude/rules/testing-signal.md` § "End-to-end gated tests". + +Run with credentials: + +```bash +SF_RUN_OPENAI=1 OPENAI_API_KEY=sk-... uv run pytest -m openai --no-cov +``` + +The `--estimate` test additionally honours `GOOGLE_CLOUD_PROJECT` when +present (lets the warehouse-bytes leg compute instead of degrading to +``); absent, the warehouse half degrades cleanly per +DEC-005 of #36 and the test still passes. They are maintainer-only; no +CI job runs them. Each run hits the real OpenAI API and incurs a small +cost (the grade smoke is 5 `gpt-4o` calls at ~$0.005 each; the draft +smoke is 1 call; `--estimate` is local tiktoken only, no API call). diff --git a/README.md b/README.md index 8b53d63a..8e1b726a 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ without adding it to a project environment. **Working from a clone (contributing)?** Install the dev toolchain with `uv sync --dev` — see [CONTRIBUTING.md](CONTRIBUTING.md) for the full workflow. -### 2. Authenticate to BigQuery and Anthropic +### 2. Authenticate to BigQuery and your LLM provider ```bash gcloud auth application-default login @@ -104,6 +104,14 @@ export ANTHROPIC_API_KEY=sk-ant-... Use a fresh shell session (or `unset ANTHROPIC_API_KEY` after the run) so the key doesn't persist in your bash history. +Anthropic is the default LLM provider. OpenAI is also supported — +`pip install signalforge-dbt[openai]`, set `OPENAI_API_KEY`, and +switch via `llm.provider: openai` / `grade.provider: openai` in +`signalforge.yml`. See +[docs/draft-ops.md § OpenAI provider](docs/draft-ops.md#openai-provider) +and [docs/grade-ops.md § OpenAI provider](docs/grade-ops.md#openai-provider) +for the per-stage config + cost notes. + ### 3. Minimum `signalforge.yml` The fixture ships a working config; a minimum that exercises the full diff --git a/docs/cost-estimate-ops.md b/docs/cost-estimate-ops.md new file mode 100644 index 00000000..cd998c88 --- /dev/null +++ b/docs/cost-estimate-ops.md @@ -0,0 +1,156 @@ +# `--estimate` — operations guide + +Operational reference for `signalforge generate --estimate`, the +pre-flight cost-preview path. Companion to +[`docs/cli-ops.md`](cli-ops.md) (where `--estimate` is registered as a +`generate` flag), [`docs/draft-ops.md`](draft-ops.md) (the LLM seam the +estimate counts tokens against), and +[`docs/grade-ops.md`](grade-ops.md) (the per-criterion grading fan-out +the estimate projects). + +## What it does + +`signalforge generate --estimate` runs the full pipeline +prelude (manifest load, safety policy resolve, draft + grade + diff +config load, warehouse profile load, adapter construction) so that any +typo in `--profiles-dir` / `signalforge.yml` surfaces BEFORE the +estimate is computed (DEC-009 of +[`plans/super/36-estimate-cost-preview.md`](../plans/super/36-estimate-cost-preview.md)). +It then issues a small set of cheap calls to project the cost of the +billable pipeline that `signalforge generate` *without* `--estimate` +would perform: + +- **Drafter half** — one token-count for the drafter prompt plus the + per-criterion judge token-counts (`1 + len(rubric)` calls). The + full `messages.create` LLM call is never invoked. +- **Warehouse half** — one BigQuery `dryRun` (or the warehouse's + equivalent — see + [`docs/warehouse-adapter-ops.md`](warehouse-adapter-ops.md)) to + project the bytes the prune step will scan, multiplied by the + `3.5 tests/column` heuristic (DEC-012 of + [`plans/super/36-estimate-cost-preview.md`](../plans/super/36-estimate-cost-preview.md)). + +Output goes to stdout as plain text with three sections (Draft / +Grade / Warehouse) followed by totals and a footer listing the +price-table version +(`signalforge.llm.pricing.PRICE_TABLE_VERSION`). + +`--estimate` reports a **billing ceiling** — actual scans usually +come in lower because cache hits, sampled rows, and shorter LLM +responses all trim the projected numbers. Treat the figure as a +calibration signal, not a billing guarantee (mirrors the +planner-estimate caveats in +[`docs/warehouse-adapter-ops.md`](warehouse-adapter-ops.md) for +Snowflake `EXPLAIN`). + +## Provider-aware token counting (issue #136 US-005) + +The drafter and grader token counts are computed via the +`LLMProvider.estimate_input_tokens(model, text) -> int` ABC method +(issue #135's provider-neutral seam, extended in issue #136 DEC-003). +Each registered provider supplies its own implementation: + +- **Anthropic (default)** — calls the SDK's + `client.messages.count_tokens(...)` (one real round-trip per + count). `ANTHROPIC_API_KEY` is required because this is a live API + call, not a local computation (DEC-006 of #36). +- **OpenAI** — uses `tiktoken` locally (BPE tokeniser, no extra API + round-trip — DEC-012 of #136). Resolves the model id via + `tiktoken.encoding_for_model(model)` with a graceful `cl100k_base` + fallback for unknown ids. + +Anthropic stdout is byte-identical before and after the #136 refactor +— pinned via a snapshot test in `tests/cli/test_estimate.py` per +DEC-013. Selecting a different provider routes the token count +through that provider's strategy method without touching the +Anthropic path. + +## OpenAI provider — `[openai]` install extra + +Selecting `grade.provider: openai` and/or `llm.provider: openai` in +`signalforge.yml` requires the `openai` install extra so both the SDK +and `tiktoken` are available: + +```bash +pip install signalforge-dbt[openai] +# or, in a contributor checkout +uv sync --dev # the dev group already pulls openai + tiktoken +``` + +`tiktoken` is OpenAI's local BPE tokeniser (MIT-licensed, no native +build; wheels for CPython 3.11–3.13). It runs entirely client-side — +no API round-trip per count — which is the main reason the OpenAI +estimate path is meaningfully faster than the Anthropic one on +multi-criterion grading runs. + +### Registered OpenAI pricing SKUs + +`signalforge.llm.pricing._PRICES_MUTABLE` ships four OpenAI SKUs +(issue #136 US-004): + +| Model id | Notes | +|---|---| +| `gpt-4o` | Default judge model for the OpenAI provider (DEC-004). | +| `gpt-4o-mini` | Budget tier — cheapest OpenAI SKU registered. | +| `gpt-4.1` | Newer flagship variant. | +| `gpt-4-turbo` | Back-compat for projects pinned to the prior generation. | + +Each SKU carries `input_per_mtok` and `output_per_mtok` rates; the +cache fields are `0.0` because OpenAI's Chat Completions surface does +not expose Anthropic-style prompt caching (see +[`docs/grade-ops.md` § OpenAI provider](grade-ops.md#openai-provider) +and [`docs/draft-ops.md` § OpenAI provider](draft-ops.md#openai-provider) +for the no-cache cost note). + +### `EstimateUnknownModelError` for unknown SKUs + +Setting `grade.model` or `llm.model` to an id that is **not** in the +pricing table raises `EstimateUnknownModelError` from the +`--estimate` path at config-load resolution time (the live draft / +grade calls themselves still run; only `--estimate` requires a +pricing row). Common cases: + +- A model id that hasn't been added to `_PRICES_MUTABLE` yet (file an + issue with the public pricing for the SKU). +- A typo (e.g. `gpt-4o-min` instead of `gpt-4o-mini`). + +Maps to CLI exit-code tier 2 (`INPUT`) — see +[`docs/cli-ops.md`](cli-ops.md) for the full exit-code taxonomy. + +## Maintainer-only live smoke tests + +Three `@pytest.mark.openai` gated tests exercise the OpenAI half of +the estimate path against the real API (DEC-008 of #136): + +```bash +SF_RUN_OPENAI=1 OPENAI_API_KEY=sk-... uv run pytest -m openai --no-cov +``` + +The marker is excluded from the default CI run via +`addopts -m 'not openai'`. Both env vars are required (each missing +var produces a clear skip reason naming the var). Mirrors the +`@pytest.mark.anthropic` precedent: + +```bash +ANTHROPIC_API_KEY=sk-... uv run pytest -m anthropic --no-cov +``` + +The Anthropic suite also includes the byte-identity snapshot for the +estimate stdout that DEC-013 of #136 pins as the refactor floor. + +## References + +- Design records: + [`plans/super/36-estimate-cost-preview.md`](../plans/super/36-estimate-cost-preview.md) + (the original `--estimate` design), + [`plans/super/135-provider-neutral-llm-seam.md`](../plans/super/135-provider-neutral-llm-seam.md) + (the provider-neutral seam), + [`plans/super/136-openai-grading-provider.md`](../plans/super/136-openai-grading-provider.md) + (the OpenAI provider + tiktoken estimate path). +- CLI flag reference: [`docs/cli-ops.md`](cli-ops.md) `--estimate`. +- Per-provider config / cost notes: + [`docs/draft-ops.md`](draft-ops.md) and + [`docs/grade-ops.md`](grade-ops.md) (OpenAI provider sections). +- Warehouse-side estimate (BigQuery `dryRun`, Snowflake `EXPLAIN + USING JSON`): + [`docs/warehouse-adapter-ops.md`](warehouse-adapter-ops.md). diff --git a/docs/draft-ops.md b/docs/draft-ops.md index a827a674..4e2b5c9f 100644 --- a/docs/draft-ops.md +++ b/docs/draft-ops.md @@ -520,7 +520,7 @@ other stages and silently ignored by the draft loader. ```yaml # signalforge.yml llm: - provider: anthropic # registry-validated; only "anthropic" registered today + provider: anthropic # registry-validated; "anthropic" + "openai" are registered (see OpenAI provider below) model: claude-sonnet-4-6 cheap_model: claude-haiku-4-5-20251001 max_output_tokens: 4096 @@ -539,8 +539,9 @@ Field-by-field: value fails loud at config-load, listing the registered provider names. Deliberately a registry-validated `str`, not a `Literal` — the provider registry is a forward-looking plugin point (#136 OpenAI / - #137 Gemini register more providers); today only `anthropic` is - registered. + #137 Gemini register more providers). Today `anthropic` and `openai` + are registered; see [OpenAI provider](#openai-provider) below for the + `openai` option. - **`model`** — the model id used by every `call_llm` invocation. Default `claude-sonnet-4-6`. Any string the SDK accepts is allowed. - **`cheap_model`** — informational; not selected automatically. @@ -579,6 +580,58 @@ If `signalforge.yml` is missing entirely (or the `llm:` key is absent), `load_draft_config(project_dir)` returns the built-in defaults silently — same behaviour as `load_safety_config`. +## OpenAI provider + +Issue #136 registered `OpenAIProvider` as the second +`signalforge.llm.providers.LLMProvider`. Select it by setting +`llm.provider: openai` in `signalforge.yml`: + +```yaml +llm: + provider: openai + model: gpt-4o # default drafter model for the OpenAI provider; any model id the SDK accepts is allowed + max_output_tokens: 4096 + # cache_ttl, max_retries_*, exclude_tests — same shape as the anthropic provider (cache_ttl is ignored, see below) +``` + +Requirements: + +- **Install extra:** `pip install signalforge-dbt[openai]` (or `uv sync --dev` in a contributor checkout). Pulls `openai>=1.40` plus `tiktoken` for the `--estimate` cost-preview path. +- **Env var:** `OPENAI_API_KEY` (mirrors `ANTHROPIC_API_KEY` for the default provider). +- **Pricing SKUs registered:** `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, `gpt-4-turbo`. Other model ids raise `EstimateUnknownModelError` from the `--estimate` path (the live draft call still runs; `--estimate` is the only surface that requires a pricing row). See [`docs/cost-estimate-ops.md`](cost-estimate-ops.md). + +**No prompt caching (cost note).** OpenAI's Chat Completions surface +does not expose Anthropic-style prompt caching. `OpenAIProvider` +reports `supports_prompt_caching=False` and `supports_token_count=False`, +which means the orchestrator (`signalforge.llm.call_llm`) skips both +the `cache_control` marker and the pre-send `count_tokens` gate +(issue #135 DEC-008). **The `cache_ttl` config knob is silently +ignored** for the OpenAI provider — every drafting call ships the full +system + cached manifest summary on every invocation, with no read +discount. For batch-CLI usage (`signalforge generate --select`) the +absence of caching is the main cost delta vs. the Anthropic provider; +budget input-token spend at full per-call rates. v0.3 ships without +OpenAI prompt caching; their recent prompt-cache mechanism is a +candidate for a follow-up. + +**Server-enforced JSON.** `OpenAIProvider.build_create_kwargs` +attaches `response_format={"type": "json_object"}` so the drafter +model is forced to emit valid JSON server-side (DEC-006). The tolerant +`extract_json_payload` parser (issue #144) remains as defence-in-depth. + +**Live smoke gating.** A gated `@pytest.mark.openai` real-API +end-to-end test exercises drafting against `gpt-4o`. Run it with: + +```bash +SF_RUN_OPENAI=1 OPENAI_API_KEY=sk-... uv run pytest -m openai --no-cov +``` + +Mirrors the `@pytest.mark.anthropic` precedent — excluded from the +default CI run via `addopts -m 'not openai'`; both env vars are +required (each missing var produces a clear skip reason). See +[`docs/cost-estimate-ops.md`](cost-estimate-ops.md) for the +maintainer's three-test smoke set (drafter + grader + `--estimate`). + ## Error hierarchy ### `signalforge.llm.errors` diff --git a/docs/grade-ops.md b/docs/grade-ops.md index 78c03735..deb5f18c 100644 --- a/docs/grade-ops.md +++ b/docs/grade-ops.md @@ -115,7 +115,7 @@ loader cannot drift): ```yaml # signalforge.yml — grade stage configuration (v0.1) grade: - provider: anthropic # registry-validated; only "anthropic" registered today + provider: anthropic # registry-validated; "anthropic" + "openai" are registered (see OpenAI provider below) model: claude-sonnet-4-6 # model id (default) cache_ttl: 1h # Prompt-cache TTL ('5m' or '1h') max_output_tokens: 256 # Per-criterion JSON response cap @@ -157,7 +157,7 @@ grade: Field-by-field: -- **`provider`** — The LLM provider strategy name (issue #135 DEC-007), resolved against the `signalforge.llm.providers` registry and threaded into `call_llm` from the per-criterion judge call, independently of the drafter's `DraftConfig.provider`. Default `"anthropic"`. An unknown value fails loud at config-load, listing the registered provider names. Deliberately a registry-validated `str`, not a `Literal` — the provider registry is a forward-looking plugin point (#136 OpenAI / #137 Gemini register more providers); today only `anthropic` is registered. +- **`provider`** — The LLM provider strategy name (issue #135 DEC-007), resolved against the `signalforge.llm.providers` registry and threaded into `call_llm` from the per-criterion judge call, independently of the drafter's `DraftConfig.provider`. Default `"anthropic"`. An unknown value fails loud at config-load, listing the registered provider names. Deliberately a registry-validated `str`, not a `Literal` — the provider registry is a forward-looking plugin point (#136 OpenAI / #137 Gemini register more providers). Today `anthropic` and `openai` are registered; see [OpenAI provider](#openai-provider) below for the `openai` option. - **`model`** — The model id used by every per-pair judge call. Default `claude-sonnet-4-6`. Mirrors `DraftConfig.model` default. Haiku 4.5 is documented as a v0.2 cost-conscious option but not exposed in v0.1. - **`cache_ttl`** — `Literal["5m", "1h"]`. Default `"1h"` (vs. the drafter's `"5m"`) because 60 sequential per-criterion calls under retry backoff can stretch beyond a 5-minute window; `"1h"` gives margin at no extra cost (cache writes are one-shot regardless of TTL). - **`max_output_tokens`** — Per-criterion judge response cap. Default `256`. The expected JSON response is ~150 tokens; 256 gives 2× safety. Independent of `DraftConfig.max_output_tokens`. @@ -497,6 +497,58 @@ operators (DEC-014). The current architecture preserves the option: each criterion has its own prompt seam already, so a `cost_mode: batched` flag is additive rather than a rewrite. +## OpenAI provider + +Issue #136 registered `OpenAIProvider` as the second +`signalforge.llm.providers.LLMProvider`. Select it by setting +`grade.provider: openai` in `signalforge.yml`: + +```yaml +grade: + provider: openai + model: gpt-4o # default judge model for the OpenAI provider; any model id the SDK accepts is allowed + # cache_ttl, max_retries_*, total_budget_seconds, thresholds — same shape as the anthropic provider +``` + +Requirements: + +- **Install extra:** `pip install signalforge-dbt[openai]` (or `uv sync --dev` in a contributor checkout). Pulls `openai>=1.40` plus `tiktoken` for the `--estimate` cost-preview path. +- **Env var:** `OPENAI_API_KEY` (mirrors `ANTHROPIC_API_KEY` for the default provider). +- **Pricing SKUs registered:** `gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, `gpt-4-turbo`. Other model ids raise `EstimateUnknownModelError` from the `--estimate` path (the live judge call still runs; `--estimate` is the only surface that requires a pricing row). See [`docs/cost-estimate-ops.md`](cost-estimate-ops.md). + +**No prompt caching (cost note).** OpenAI's Chat Completions surface +does not expose Anthropic-style prompt caching. `OpenAIProvider` +reports `supports_prompt_caching=False` and `supports_token_count=False`, +which means the orchestrator (`signalforge.llm.call_llm`) skips both +the `cache_control` marker and the pre-send `count_tokens` gate +(issue #135 DEC-008). **Every grading call ships the full system + +rubric block** — there is no cached read discount on subsequent +criteria, so the per-`(artefact × criterion)` fan-out (see +[Cost guidance](#cost-guidance-dec-014) above) costs a flat +input-token bill on every call. Budget accordingly: a 4-criterion × +12-artefact run is 48 full system+rubric sends, not one write + 47 +reads. v0.3 ships without prompt caching; OpenAI's recent prompt-cache +mechanism is a candidate for a follow-up. + +**Server-enforced JSON.** `OpenAIProvider.build_create_kwargs` +attaches `response_format={"type": "json_object"}` so the judge model +is forced to emit valid JSON server-side (DEC-006). The tolerant +`extract_json_payload` parser (issue #144) remains as defence-in-depth +for the same prose-preamble drift class the Anthropic path handles. + +**Live smoke gating.** A gated `@pytest.mark.openai` real-API +end-to-end test exercises grading against `gpt-4o`. Run it with: + +```bash +SF_RUN_OPENAI=1 OPENAI_API_KEY=sk-... uv run pytest -m openai --no-cov +``` + +Mirrors the `@pytest.mark.anthropic` precedent — excluded from the +default CI run via `addopts -m 'not openai'`; both env vars are +required (each missing var produces a clear skip reason). See +[`docs/cost-estimate-ops.md`](cost-estimate-ops.md) for the +maintainer's three-test smoke set (grader + drafter + `--estimate`). + ## Prompt-injection mitigation The grader's only LLM-prompt defence is the diff --git a/mkdocs.yml b/mkdocs.yml index 6227c229..7151a167 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -55,6 +55,7 @@ nav: - Prune Engine: prune-ops.md - Quality Grader: grade-ops.md - Diff Renderer: diff-ops.md + - Cost Estimate: cost-estimate-ops.md - Audits & Sidecars: audits.md - End-to-End Smoke Test: e2e-smoke-test.md - Snowflake E2E Setup: snowflake-e2e-setup.md diff --git a/plans/super/136-openai-grading-provider.md b/plans/super/136-openai-grading-provider.md new file mode 100644 index 00000000..4f7a1ddf --- /dev/null +++ b/plans/super/136-openai-grading-provider.md @@ -0,0 +1,356 @@ +# Super Plan — #136: OpenAI model support for grading + +## Meta + +- **Ticket:** https://github.com/wjduenow/SignalForge/issues/136 +- **Parent epic:** #134 (pluggable LLM provider for grading — OpenAI/Gemini). Milestone v0.3. +- **Depends on:** #135 (provider-neutral LLM seam) — **merged**, PR #148. +- **Sibling:** #137 (Gemini grading; mirrors the same shape). +- **Phase:** published (awaiting approval — PR #152) +- **Branch:** `feature/136-openai-grading` +- **PR:** https://github.com/wjduenow/SignalForge/pull/152 + +## What / Why + +Register **OpenAI** as the second LLM provider behind the #135 provider-neutral seam, selectable via `grade.provider: openai` in `signalforge.yml`. Validates that the seam is genuinely vendor-pluggable — Anthropic stays byte-identical (the regression floor), and a real-world non-caching vendor wires in cleanly through the same `LLMProvider` ABC + registry that #135 established. Closes the v0.3 epic's "OpenAI grading" deliverable; #137 (Gemini) ships next as the third provider with the same pattern. + +The seam already validates the abstract case: `tests/grade/test_provider_neutrality.py` registers `FakeNoCacheProvider` (both capability flags `False`) and proves `grade_artifacts` end-to-end. **OpenAI is the same shape with a real SDK** — `supports_prompt_caching=False`, `supports_token_count=False` (no Anthropic-equivalent count_tokens API; OpenAI offers a one-shot completions call). The work is mechanical SDK plumbing + a confinement shim + tests, with one genuine design choice — how `make_client()` reconciles OpenAI's `client.chat.completions.create(...)` with the orchestrator's hard-coded `client.messages.create(...)` protocol. + +## Discovery findings + +### The seam after #135 (all in `src/signalforge/llm/`) + +- **`providers.py`** — `LLMProvider` ABC + `register_provider(provider)` / `provider_for(name)` registry + `AnthropicProvider` (registered at import time). Capability flags: `name`, `supports_prompt_caching`, `supports_token_count`. Six abstract methods: `make_client`, `build_create_kwargs`, `build_count_tokens_kwargs`, `extract_text_blocks`, `extract_usage`, `classify_exception`. +- **`client.py` `call_llm(*, system, cached_block, dynamic_block, model, max_tokens, cache_ttl="5m", prompt_version, max_retries_*, provider="anthropic", client=None) -> LLMResult`** — generic orchestrator. Retry loop dispatches on `ExceptionCategory` (AUTH / RATE_LIMIT / SERVER_ERROR / CONNECTION / NO_RETRY). Pre-send token-count gate is **gated on `supports_token_count`** — skipped entirely when `False`. Cache marker + dual-zero WARNING gated on `supports_prompt_caching`. +- **`_anthropic_client.py`** — sole home of every Anthropic `# pyright: ignore`. Lazy SDK import inside `_make_anthropic_client` and `_load_anthropic_exception_classes`. Confinement enforced by `tests/test_audit_completeness.py` Scan 3 (`anthropic.Anthropic(...)` constructions allowed only here). +- **`pricing.py`** — `_PRICES_MUTABLE` dict of `ModelPricing(input_per_mtok, output_per_mtok, cache_write_5m_per_mtok, cache_read_per_mtok)`. Three Anthropic SKUs (sonnet-4-6, opus-4-7, haiku-4-5). `PRICE_TABLE_VERSION = "2026-05-11"`. Consumed by `cli/_estimate.py:457,473`. + +### Grade / draft consumption sites + +- `src/signalforge/grade/engine.py:306-310` — `call_llm(system=_SYSTEM_PROMPT, cached_block=rubric_block, dynamic_block=dynamic_block, ..., provider=config.provider, client=client)`. +- `src/signalforge/grade/config.py:127` — `provider: str = "anthropic"` with the `_provider_registered` validator (lines 209–230) calling `provider_for(v)`. +- `src/signalforge/draft/config.py:113` — same shape, parity-mirrored. +- `src/signalforge/draft/schema.py:197` — `call_llm(..., provider=config.provider, ...)`. + +### `--estimate` cost-preview path (`src/signalforge/cli/_estimate.py`) + +- **Anthropic-coupled today.** Threads a single `anthropic_client: AnthropicClientProtocol` through `_count_draft_tokens` (calls `anthropic_client.messages.count_tokens(...)`) and the grader-side equivalent. +- **`pricing_grade = _pricing.lookup(grade_config.model)`** (line 473) — raises `EstimateUnknownModelError` on unknown model. +- **Scope answer:** SD-3 says **fully wire OpenAI to `--estimate`** — generalise the estimate path's token-counting to be provider-aware (Anthropic = `messages.count_tokens`; OpenAI = `tiktoken` local), add OpenAI pricing entries, keep Anthropic byte-identical. + +### Provider-neutrality test infrastructure (already in place) + +- `tests/llm/_fake.py` — `FakeAnthropicClient` with `expect_count_tokens` / `expect_messages_create` FIFO queues + `assert_all_expectations_met()`. This is the shape to mirror as `FakeOpenAIClient`. +- `tests/llm/_fake_provider.py` — `FakeNoCacheProvider` (synthetic no-cache provider) + `FakeNoCacheClient` with `create_calls` inspector + a `count_tokens` that raises if invoked. Already proves the no-cache path works end-to-end. +- `tests/grade/test_provider_neutrality.py` — three tests: AC #1 registry validation, AC #2 GradeConfig validator accepts a registered provider, AC #3 `grade_artifacts` drives the no-cache provider end-to-end and verifies cache_*=0 in audit JSONL + 16-hex blake2b-8 reproducibility hashes + sidecar round-trip + **no dual-zero WARNING**. +- AST confinement precedents: `tests/test_audit_completeness.py` Scan 3 for `anthropic.Anthropic(...)`; `tests/warehouse/test_snowflake_client_confinement.py` for the line-based `# type: ignore` confinement (Snowflake-shaped). + +### `openai` SDK availability + +- **Not a declared dependency.** `pyproject.toml:19` pins `anthropic>=0.50,<2.0`; no `openai` entry. (`openai 1.3.7` happens to be in this env but isn't required by SignalForge.) Need a new optional extra `[openai]` mirroring `[snowflake]`, with a lazy SDK import inside `_openai_client.py` (the same pattern as Snowflake — `tests/warehouse/test_snowflake_client_confinement.py` enforces it). + +## Scoping decisions (Phase 1) + +- **SD-1 — OpenAI API surface: Chat Completions.** `client.chat.completions.create(...)`. Stable, universal, simplest. Read text from `response.choices[0].message.content`. Pin `openai>=1.40` (covers stable Chat Completions + structured outputs). Responses API deferred. +- **SD-2 — No cross-validation of provider/model.** Keep `GradeConfig.model` a free string. Model-naming evolves fast (gpt-4o → gpt-4.1 → …); a hard allowlist rots. Registry validates the provider name only; an obviously-wrong model errors at API call time with a typed `LLMError`. +- **SD-3 — Fully wire OpenAI to `--estimate`.** Generalise the estimate path's token-counting to be provider-aware: Anthropic keeps `messages.count_tokens`; OpenAI uses `tiktoken` (local BPE counter, no API call). Add OpenAI pricing SKUs. Anthropic estimate bytes stay identical. +- **SD-4 — Default judge model: `gpt-4o`.** Used in docs examples + the gated live test. `GradeConfig.model` default stays `claude-sonnet-4-6` — operators choosing OpenAI explicitly set `grade.model: gpt-4o`. + +## Architecture review (Phase 2) + +| Area | Rating | Finding | +|---|---|---| +| **SDK confinement** | concern | Every `# pyright: ignore` / `import openai` must live in `_openai_client.py`. Extend Scan 3 in `tests/test_audit_completeness.py` to also exclude `_openai_client.py` for `openai.OpenAI(...)` constructions. Mirrors Anthropic precedent exactly. **Resolution:** add the new AST scan + add `openai` to the per-SDK exclusion lists. | +| **`.messages` adapter shape** | concern | OpenAI SDK exposes `client.chat.completions.create(...)`, NOT `client.messages.create(...)`. The orchestrator hard-calls `llm_client.messages.create(**kwargs)`. **Resolution:** `OpenAIProvider.make_client()` returns a thin adapter object whose `.messages.create(**kwargs)` delegates to the underlying `openai.OpenAI().chat.completions.create(**kwargs)`. `messages.count_tokens` is never called (`supports_token_count=False`); the adapter raises `NotImplementedError` on that path defensively. | +| **`build_count_tokens_kwargs` ABC contract** | pass | A provider with `supports_token_count=False` never sees the method called by the orchestrator. Precedent: `FakeNoCacheProvider.build_count_tokens_kwargs` raises `NotImplementedError`. Match that. | +| **Anthropic byte-identity** | concern | The estimate refactor (SD-3) threads a provider strategy through `_count_draft_tokens` + the grader-side equivalent. **Anthropic estimate output must stay byte-identical.** Resolution: extract the SDK-call into a per-strategy `count_input_tokens(client, ...) -> int` method; Anthropic impl is the existing `client.messages.count_tokens(...)` call verbatim. Pin a snapshot test on Anthropic estimate stdout before refactor + verify after. | +| **`tiktoken` dependency** | concern | Adding a local-tokeniser dependency for OpenAI estimate. `tiktoken` is OpenAI-published, MIT, no native build (wheels for cpython 3.11–3.13). **Resolution:** add to the `[openai]` optional extra, lazy-import inside `_openai_client.py`. `OpenAIProvider.count_input_tokens` uses tiktoken's `encoding_for_model` with a graceful fallback to `cl100k_base` for unknown model ids. | +| **Pricing table churn** | pass | `_PRICES_MUTABLE` gains OpenAI SKUs (at minimum `gpt-4o`); `PRICE_TABLE_VERSION` bumps. Additive change. Cache fields set to `0.0` (OpenAI has no equivalent cache discount). | +| **Drafter provider symmetry** | concern | #135 gave BOTH `grade.provider` and `draft.provider` a `str` field. Once `OpenAIProvider` is registered, both stages naturally accept `provider: openai`. **Decision needed (refinement):** scope #136 to "grade only" with `draft.provider: openai` documented as untested-but-permitted, OR explicitly cover both stages. The work is identical; the doc message differs. | +| **GradeEvent / drift detector** | pass | `GradeEvent.cache_creation_input_tokens` / `cache_read_input_tokens` default to 0 — already proven by `FakeNoCacheProvider` round-trip in `tests/grade/test_drift_detector.py`. No schema bump. | +| **Reproducibility hashes** | pass | `rubric_hash`, `prompt_version_template`, `criterion_prompt_hash`, `response_text_hash` are LLM-content-agnostic. OpenAI responses produce a different `response_text_hash` (different judge model output) but the same 16-hex blake2b-8 shape. | +| **Exception taxonomy** | pass | OpenAI SDK exceptions map cleanly: `AuthenticationError`/`PermissionDeniedError` → AUTH; `RateLimitError` → RATE_LIMIT; `APIConnectionError` → CONNECTION; `APIStatusError` with 5xx → SERVER_ERROR; 4xx-non-auth + anything else → NO_RETRY. Mirrors `AnthropicProvider.classify_exception`. | +| **JSON parser tolerance** | pass | `parse_grade_response` already routes through `extract_json_payload` (issue #144) which strips prose preambles. No prefill needed (OpenAI Chat Completions doesn't support assistant-turn prefill either — same constraint as `claude-sonnet-4-6`). Optional refinement: set OpenAI's `response_format={"type":"json_object"}` to enforce JSON server-side. | +| **Live gated test** | pass | Add `@pytest.mark.openai` marker; gate on `SF_RUN_OPENAI=1` + `OPENAI_API_KEY`; register in `pyproject.toml` `[tool.pytest.ini_options].markers` + add to `addopts -m 'not ...'` exclusion. Mirrors the `anthropic` marker precedent. | +| **`--estimate` parity test** | concern | Need a unit-level estimate test with `grade.provider: openai` driving a `FakeOpenAIClient` + faked-or-real tiktoken count, asserting the report renders correctly. | +| **Observability / logger gate** | pass | Lazy-format JSON logger gate (`tests/llm/test_logger_grep_gate.py`) already scans `src/signalforge/llm`. New `_openai_client.py` falls under the gate automatically. No new logger calls planned in the shim — logging stays in `client.py`. | +| **Documentation surfaces** | concern | `docs/grade-ops.md` needs an OpenAI section (config snippet, no-cache caveat, model id guidance, env var). `docs/cost-estimate-ops.md` (or wherever `--estimate` ops live) needs the tiktoken note. `CLAUDE.md` "Related projects" doesn't need a change. `.claude/rules/llm-drafter.md` adds a sub-section on the OpenAI shim + Chat Completions adapter pattern as the precedent for #137. **Resolution:** dedicated docs story. | +| **CHANGELOG** | pass | Add a `0.3.0.dev` entry under "Added" for OpenAI grading. | + +**Blockers:** none. **Concerns:** 6 listed — all routed to refinement or absorbed into specific stories. No architectural blockers. + +## Refinement log + +### Phase 1 scoping decisions (operator-facing) + +- **DEC-001 — OpenAI API surface: Chat Completions.** `client.chat.completions.create(...)` is the stable, universal surface; read text from `response.choices[0].message.content`. Pin `openai>=1.40`. Responses API deferred — no operator-visible feature in v0.3 needs it. From SD-1. +- **DEC-002 — No cross-validation of provider/model.** `GradeConfig.model` / `DraftConfig.model` stay free strings. The provider registry validates the provider name; the model id is checked at API call time. Model naming evolves too fast (gpt-4o → gpt-4.1 → next) for a hard allowlist to be worth maintaining. From SD-2. +- **DEC-003 — `--estimate` fully wired for OpenAI.** Generalise the estimate-path token-counting through a new `LLMProvider.estimate_input_tokens(model, text) -> int` ABC method. Anthropic impl calls `client.messages.count_tokens(...)` (preserves byte-identity); OpenAI impl uses `tiktoken` (local BPE, no API call). Add OpenAI pricing SKUs. From SD-3. +- **DEC-004 — Default judge model: `gpt-4o`.** Used in docs examples + gated live tests. `GradeConfig.model` / `DraftConfig.model` keep their `claude-sonnet-4-6` default — operators selecting OpenAI explicitly set the model. From SD-4. + +### Phase 3 refinement decisions + +- **DEC-005 — Scope both grade AND draft explicitly.** `OpenAIProvider` is global once registered (provider field on both configs is symmetric since #135). Ship both stages with tests, docs sections, and live smokes. The work is mechanical; asymmetric documentation would imply a non-existent guard. The two stages share one provider class, one shim, one set of pricing SKUs — the *test/docs surface* doubles, not the implementation. +- **DEC-006 — Server-enforce JSON via `response_format={"type":"json_object"}`.** `OpenAIProvider.build_create_kwargs` attaches the JSON-mode flag. Belt-and-braces with the existing tolerant `extract_json_payload` parser: server-side enforcement eliminates the prose-preamble drift class (mirrors issue #144's fix for `claude-sonnet-4-6`), and the parser remains the fallback if a future model strips the flag. The grade system prompt already names "JSON" so OpenAI's prompt-requirement check passes. +- **DEC-007 — Ship four OpenAI SKUs in `pricing.py`.** `gpt-4o` (default judge per DEC-004), `gpt-4o-mini` (budget tier), `gpt-4.1` (newer flagship), `gpt-4-turbo` (back-compat). Each carries `input_per_mtok` + `output_per_mtok`; cache fields are `0.0` (OpenAI has no equivalent cache discount). Bump `PRICE_TABLE_VERSION` to the ship date. +- **DEC-008 — Live gated smoke covers `grade_artifacts`, `draft_schema`, AND `--estimate`.** Three `@pytest.mark.openai` tests gated on `SF_RUN_OPENAI=1` + `OPENAI_API_KEY`: one drives end-to-end grading against the real API; one drives end-to-end drafting (honors DEC-005's both-stages scope at live level too); one runs `signalforge generate --estimate` with `grade.provider: openai` and asserts the report renders. Mirrors the maintainer-only `anthropic` marker precedent and #137's three-live-test breadth. + +### Phase 2 architecture-concern resolutions + +- **DEC-009 — `OpenAIProvider.make_client()` returns a thin `.messages`-shaped adapter.** OpenAI SDK exposes `client.chat.completions.create(...)`; the orchestrator hard-calls `client.messages.create(**kwargs)`. The adapter pattern: `_OpenAIClientAdapter` has a `.messages` namespace whose `.create(**kwargs)` delegates to the underlying `openai.OpenAI().chat.completions.create(**kwargs)`. `.messages.count_tokens` raises `NotImplementedError` defensively (orchestrator never calls it for a `supports_token_count=False` provider). +- **DEC-010 — `_openai_client.py` is the sole home of every OpenAI SDK ignore; add a new 9th AST scan.** Scan 3 in `tests/test_audit_completeness.py` is Anthropic-specific (`anthropic.Anthropic(...)`); adding the OpenAI confinement requires a **new** AST scan, not an extension — bumping the project tally from 8 → 9 (and to 10 once #137's Gemini scan lands; whichever vendor merges first owns the 8 → 9 bump and the second owns 9 → 10). The new scan reuses the existing `_QualifiedNameCallFinder` helper per `testing-signal.md` § "AST single-construction-seam scans must catch all three bypass patterns" (bare / import-alias / module-attribute). Excludes `_openai_client.py`; sanity check asserts ≥1 legitimate `openai.OpenAI(...)` construction lives in the shim. Companion per-file confinement test `tests/llm/test_openai_client_confinement.py` mirrors the Snowflake-shaped `# type: ignore` line scan. +- **DEC-011 — `OpenAIProvider.build_count_tokens_kwargs` raises `NotImplementedError`.** Matches `FakeNoCacheProvider.build_count_tokens_kwargs` precedent. The orchestrator never invokes it (`supports_token_count=False`), but the ABC requires the method present; raising is the honest behaviour. +- **DEC-012 — `tiktoken` lives in the `[openai]` extra, lazy-imported in the shim; dual-listed across all three dev slots.** Mirrors Snowflake's `[snowflake]` precedent verbatim per `python-build.md` § "uv-managed dev environment": `openai>=1.40,<3.0` AND `tiktoken>=0.7,<1.0` appear in **three** places in lockstep — `[project.optional-dependencies].openai` (operator install: `pip install signalforge-dbt[openai]`), `[project.optional-dependencies].dev` (pip back-compat for `pip install -e ".[dev]"`), and `[dependency-groups].dev` (uv-native, what CI uses). Missing any one slot drifts the install surfaces. `uv.lock` refreshes in the same commit. `_count_openai_tokens(model, text)` uses `tiktoken.encoding_for_model(model)` with a `cl100k_base` fallback for unknown ids. +- **DEC-013 — Anthropic estimate byte-identity is the floor.** Before the estimate refactor, capture a golden snapshot of `signalforge generate --estimate` stdout for an Anthropic-config fixture. After the refactor (DEC-003 — strategy-driven token counting), the snapshot must reproduce byte-for-byte. Pin via `tests/cli/test_estimate.py`. +- **DEC-014 — `_load_openai_exception_classes()` returns empty tuples on `ImportError`.** Mirrors `_load_anthropic_exception_classes`'s `pragma: no cover` branch exactly. If a base install ships without the `[openai]` extra, `import openai` raises and the loader returns a frozen `_OpenAIExceptionClasses` with empty tuples in every category. `OpenAIProvider.classify_exception` then routes every exception to `NO_RETRY` cleanly — the operator never gets `provider: openai` to resolve a real call (the registry validator at config load would fail first, since the registration also runs lazily), but import-time behaviour is graceful. **Refusal / content-filter symmetry note:** OpenAI returns refusals as model-generated text (e.g. "I cannot help with that") rather than via a typed exception. The grade parser's tolerant JSON extraction (issue #144) treats unparseable refusal text as `GradeOutputError(violation_type="json_parse")` → standard degrade. No Gemini-style `safety_filter → typed degrade` DEC is needed; the existing pipeline handles it. + +## Detailed breakdown + +Stories follow the natural ordering: dependency wiring → shim → provider strategy → fakes/tests → pricing → estimate refactor → live smokes → docs → QG → P&M. The canonical validation command (`uv sync --dev && uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run pytest`) is implicit in every AC. + +### US-001 — `_openai_client.py` shim + dependency + AST confinement + +**Description:** Create the single shim where every `openai` SDK type ignore lives, add the optional `[openai]` extra (openai + tiktoken) in lockstep across the three pyproject slots, and add a **new 9th AST scan** confining `openai.OpenAI(...)` constructions to the shim. + +**Traces to:** DEC-001, DEC-009, DEC-010, DEC-012, DEC-014. + +**Files:** +- `src/signalforge/llm/_openai_client.py` (new) — `OpenAIClientProtocol`, `_OpenAIMessagesAdapter`, `_OpenAIClientAdapter`, `_make_openai_client(api_key=None)`, `_load_openai_exception_classes()` (DEC-014 empty-tuple fallback on `ImportError`), `_OpenAIExceptionClasses` frozen dataclass, `_count_openai_tokens(model, text)`. All `# pyright: ignore` / `# type: ignore` confined here. +- `pyproject.toml` — **three-slot dual listing** per DEC-012: add `openai = ["openai>=1.40,<3.0", "tiktoken>=0.7,<1.0"]` under `[project.optional-dependencies]`; append both packages to `[project.optional-dependencies].dev` AND `[dependency-groups].dev`. Regenerate `uv.lock`. +- `tests/test_audit_completeness.py` — **add a new 9th AST scan** (NOT an extension of Scan 3 — Scan 3 is Anthropic-specific) reusing `_QualifiedNameCallFinder` to detect `openai.OpenAI(...)` constructions, excluding `_openai_client.py`. Add the three-pattern planted-violation regression test (bare / import-alias / module-attribute) per `testing-signal.md` § "AST single-construction-seam scans must catch all three bypass patterns." Sanity test asserts ≥1 legitimate `openai.OpenAI(...)` in the shim. +- `tests/llm/test_openai_client_confinement.py` (new) — line-based scan rejecting `openai`-mentioning `# type: ignore` / `# pyright: ignore` outside the shim (mirrors `tests/warehouse/test_snowflake_client_confinement.py`). + +**TDD:** Write the 9th scan + planted-violation test + per-file confinement test first; all should fail (no shim) → red. Add the shim with one legitimate `openai.OpenAI(...)` construction → green. Then plant each of the three bypass patterns (bare, import-alias, module-attribute) and re-run to confirm the scan catches each; revert. + +**Acceptance criteria:** +- `uv sync --dev` installs both `openai` and `tiktoken` (dev group includes the extra). +- `_make_openai_client(api_key=None)` lazy-imports the SDK and returns a `_OpenAIClientAdapter`. +- `_OpenAIClientAdapter.messages.create(**kwargs)` delegates to `chat.completions.create(**kwargs)`; `.messages.count_tokens(...)` raises `NotImplementedError`. +- `_count_openai_tokens("gpt-4o", "hello world")` returns a positive int; an unknown model id falls back to `cl100k_base` without raising. +- AST Scan 3 still passes; a planted `openai.OpenAI(...)` outside the shim fails the scan. +- Line-based confinement test asserts every `openai`-tagged `# type: ignore` lives only in `_openai_client.py`. + +**Done when:** Above ACs all pass; canonical validation command is green. + +**Depends on:** none. + +### US-002 — `OpenAIProvider` + registration + config-validator coverage + +**Description:** Add `OpenAIProvider(LLMProvider)` to `providers.py`, register at import time, and pin that both `GradeConfig` and `DraftConfig` accept `provider="openai"` after registration. + +**Traces to:** DEC-001, DEC-005, DEC-006, DEC-009, DEC-011. + +**Files:** +- `src/signalforge/llm/providers.py` — add `OpenAIProvider` class (mirrors `AnthropicProvider` shape) with `name="openai"`, `supports_prompt_caching=False`, `supports_token_count=False`. Six ABC method impls: `make_client()` → `_make_openai_client()`; `build_create_kwargs()` → returns `{"model", "max_tokens", "messages": [{"role":"system",...},{"role":"user","content": cached_block+dynamic_block}], "response_format":{"type":"json_object"}}` (cache_marker_active / cache_ttl ignored); `build_count_tokens_kwargs()` raises `NotImplementedError`; `extract_text_blocks()` reads `response.choices[0].message.content`; `extract_usage()` reads `response.usage.{prompt_tokens, completion_tokens}` mapped to `UsageMetrics(input_tokens, output_tokens, cache_creation_input_tokens=0, cache_read_input_tokens=0)`; `classify_exception()` maps SDK exceptions via `_load_openai_exception_classes()` to `ExceptionCategory`. `register_provider(OpenAIProvider())` at module end. +- `src/signalforge/llm/__init__.py` — export `OpenAIProvider` in `__all__`. +- `tests/llm/test_providers.py` — extend (or add) tests: `provider_for("openai")` returns an `OpenAIProvider`; `UnknownProviderError("xyz")` message lists both "anthropic" and "openai"; each ABC method has a focused unit test against synthetic inputs/exceptions. +- `tests/grade/test_config.py` + `tests/draft/test_config.py` — pin that `GradeConfig(provider="openai", model="gpt-4o")` validates; `DraftConfig(provider="openai", model="gpt-4o")` validates. + +**TDD:** For each ABC method, write the unit test first (e.g. `classify_exception(openai.RateLimitError(...))` returns `ExceptionCategory.RATE_LIMIT`); fill in impl until green. Cover all five `ExceptionCategory` branches (AUTH / RATE_LIMIT / SERVER_ERROR / CONNECTION / NO_RETRY) — each maps from a real `openai.*` exception class. + +**Acceptance criteria:** +- `provider_for("openai")` returns an `OpenAIProvider` instance with `supports_prompt_caching=False`, `supports_token_count=False`. +- `OpenAIProvider().build_create_kwargs(...)` returns a dict containing `model`, `max_tokens`, `messages` (a list with a system role + a user role), and `response_format={"type":"json_object"}`. No `cache_control` marker anywhere. +- `OpenAIProvider().build_count_tokens_kwargs(...)` raises `NotImplementedError`. +- `OpenAIProvider().classify_exception(...)` returns the correct `ExceptionCategory` for at least one concrete SDK exception per category. +- `GradeConfig(provider="openai", model="gpt-4o")` and `DraftConfig(provider="openai", model="gpt-4o")` validate without error. +- `provider_for("xyz")` raises `UnknownProviderError` listing `("anthropic", "openai")` (order-insensitive). + +**Done when:** Above ACs pass; validation green. + +**Depends on:** US-001. + +### US-003 — `FakeOpenAIClient` + grade end-to-end provider-neutrality test + +**Description:** Build the test fake mirroring `FakeAnthropicClient`'s `expect_*` API and add an end-to-end `grade_artifacts(provider="openai")` integration test that proves cache_*=0, reproducibility hashes, and no dual-zero WARNING — the OpenAI analogue of the existing `FakeNoCacheProvider` proof. + +**Traces to:** DEC-001, DEC-005, DEC-006, DEC-009, DEC-011. + +**Files:** +- `tests/llm/_fake_openai.py` (new) — `FakeOpenAIUsage(prompt_tokens, completion_tokens)`, `FakeOpenAIMessage(content, role="assistant")`, `FakeOpenAIChoice(message, index=0, finish_reason="stop")`, `FakeOpenAICompletion(choices, usage, model, id, object="chat.completion")`. `_MessagesAdapter` with FIFO `_create_queue: list[_CreateExpectation]` + `create_calls: list[dict]` inspector. `FakeOpenAIClient` exposes `.messages` (delegating to `chat.completions` for parity with real SDK adapter); `expect_messages_create(matching, returns)` + `assert_all_expectations_met()`. +- `tests/grade/test_provider_neutrality_openai.py` (new) — three tests mirroring the no-cache provider neutrality suite: (1) `provider_for("openai")` resolves and capability flags are False/False; (2) `GradeConfig(provider="openai", model="gpt-4o")` validates; (3) `grade_artifacts(..., provider="openai", client=FakeOpenAIClient())` drives the engine end-to-end against canned JSON judge responses and asserts: JSONL `cache_creation_input_tokens == 0` and `cache_read_input_tokens == 0`, 16-hex blake2b-8 reproducibility hashes, sidecar round-trips, no dual-zero cache-anomaly WARNING in caplog. + +**TDD:** Write the end-to-end test first; it fails because no `FakeOpenAIClient` exists. Build the fake until the test passes. Then plant edge cases (Exception in `returns`, mismatched `matching`) and confirm the fake's `assert_all_expectations_met()` catches under-consumption. + +**Acceptance criteria:** +- `FakeOpenAIClient` exposes `.messages.create(**kwargs)` consuming one matching expectation from the FIFO queue; raises `AssertionError` on no-match. +- The end-to-end test passes: `grade_artifacts(provider="openai", client=FakeOpenAIClient())` produces a valid `GradingReport`, JSONL audit, and sidecar. +- `caplog` contains no `"cache marker no-op"` WARNING in the OpenAI path. +- `assert_all_expectations_met()` after the run reports zero un-consumed expectations. + +**Done when:** Above ACs pass; validation green. + +**Depends on:** US-002. + +### US-004 — Pricing entries (`gpt-4o`, `gpt-4o-mini`, `gpt-4.1`, `gpt-4-turbo`) + +**Description:** Add four OpenAI SKUs to `_PRICES_MUTABLE` in `pricing.py`; bump `PRICE_TABLE_VERSION`. + +**Traces to:** DEC-003, DEC-004, DEC-007. + +**Files:** +- `src/signalforge/llm/pricing.py` — extend `_PRICES_MUTABLE` with the four SKUs (input/output per-Mtok USD from OpenAI's public price page at PR-prep time; cache fields = 0.0). Bump `PRICE_TABLE_VERSION` to today (e.g. `"2026-05-27"`). +- `tests/llm/test_pricing.py` — assert `lookup("gpt-4o")`, `lookup("gpt-4o-mini")`, `lookup("gpt-4.1")`, `lookup("gpt-4-turbo")` each return a non-zero `input_per_mtok` and `output_per_mtok` and zero cache fields. Assert `lookup("gpt-9-unicorn")` still raises `EstimateUnknownModelError`. + +**TDD:** Pricing-lookup tests first (red); add SKU entries (green); pin the version bump. + +**Acceptance criteria:** +- All four OpenAI SKUs resolve via `lookup()` with non-zero input/output rates and zero cache rates. +- `PRICE_TABLE_VERSION` bumped. +- Unknown model still raises. + +**Done when:** Above ACs pass; validation green. + +**Depends on:** none (parallel-safe). + +### US-005 — `--estimate` provider-aware token counting + +**Description:** Generalise the estimate path's token-counting through a new `LLMProvider.estimate_input_tokens(model, text) -> int` abstract method. Anthropic impl preserves byte-identity (calls existing SDK `count_tokens`); OpenAI impl uses tiktoken; `FakeNoCacheProvider` impl returns a constant. Refactor `cli/_estimate.py` to thread the strategy. + +**Traces to:** DEC-003, DEC-007, DEC-012, DEC-013. + +**Files:** +- `src/signalforge/llm/providers.py` — add `LLMProvider.estimate_input_tokens(model, text) -> int` abstract method. Implement on `AnthropicProvider` (delegates to its SDK; reuses the client construction path used by `_estimate`); implement on `OpenAIProvider` (delegates to `_count_openai_tokens`). +- `src/signalforge/cli/_estimate.py` — refactor `_count_draft_tokens` and the grader-side equivalent to dispatch through `provider_for(config.provider).estimate_input_tokens(model, text)`. Remove the hard-coded `anthropic_client.messages.count_tokens(...)` callsite; thread the resolved client (or `None` for clients the strategy builds itself) through the strategy. +- `tests/llm/_fake_provider.py` — add `FakeNoCacheProvider.estimate_input_tokens(model, text) -> int` returning a constant (e.g. `len(text.split())`) so existing neutrality tests still pass. +- `tests/cli/test_estimate.py` — (a) pin Anthropic byte-identity: capture a golden snapshot of estimate stdout for an Anthropic-config fixture BEFORE the refactor in the same commit (via a new fixture) and assert it after; (b) add a test driving `--estimate` with `grade.provider: openai` + `grade.model: gpt-4o` (and `draft.provider: openai` + `draft.model: gpt-4o`) against `FakeOpenAIClient`, asserting the report renders with non-zero token counts and a non-zero USD estimate. + +**TDD:** Capture Anthropic golden first; refactor; verify identity. Then write the OpenAI estimate test; implement until green. + +**Acceptance criteria:** +- `LLMProvider.estimate_input_tokens(model, text) -> int` is an abstract method on the ABC. +- `AnthropicProvider.estimate_input_tokens` reproduces the pre-refactor token count for the same input. +- `OpenAIProvider.estimate_input_tokens` returns a positive int for `gpt-4o`. +- Anthropic estimate stdout snapshot is byte-identical before and after the refactor (pinned by `tests/cli/test_estimate.py`). +- `signalforge generate --estimate` with `grade.provider: openai` produces an `EstimateReport` with non-zero grader token counts and non-zero USD figures. + +**Done when:** Above ACs pass; validation green; no `--cov-fail-under` regression. + +**Depends on:** US-002, US-004. + +### US-006 — Live gated smoke tests (`grade_artifacts` + `draft_schema` + `--estimate`) + +**Description:** Add the `openai` pytest marker, register three gated tests against the real OpenAI API (grader + drafter + estimate, per DEC-005 + DEC-008), document the env-var gate. Mirrors the `anthropic` marker precedent. + +**Traces to:** DEC-001, DEC-004, DEC-005, DEC-008. + +**Files:** +- `pyproject.toml` — register `"openai: real-API smoke test (requires OPENAI_API_KEY; excluded from default CI)"` under `[tool.pytest.ini_options].markers`; extend `addopts -m 'not ...'` exclusion to include `not openai`. +- `tests/grade/test_smoke_real_api_openai.py` (new) — `pytestmark = pytest.mark.openai`; env-gate `SF_RUN_OPENAI=1` + `OPENAI_API_KEY`. Drives `grade_artifacts(..., provider="openai", config=GradeConfig(model="gpt-4o", ...), client=None)` and asserts shape-only (positive scores, valid JSONL, no dual-zero WARNING). +- `tests/draft/test_smoke_real_api_openai.py` (new) — `pytestmark = pytest.mark.openai`; same env gates. Drives `draft_schema(..., provider="openai", config=DraftConfig(model="gpt-4o", ...), client=None)` against a small in-test manifest fixture; asserts `CandidateSchema` validates + `LLMResponseEvent` JSONL is written with `cache_*_input_tokens == 0`. Honours DEC-005's "scope both stages" commitment that US-003's grade-only neutrality test alone doesn't cover live-side. +- `tests/cli/test_e2e_estimate_openai.py` (new) — `pytestmark = pytest.mark.openai`; same env gates. Runs `signalforge generate --estimate ...` with `grade.provider: openai` + `grade.model: gpt-4o`; asserts the rendered report includes a non-zero grader USD estimate, exit code 0, no traceback. +- `CONTRIBUTING.md` (or `docs/cost-estimate-ops.md`) — document the `SF_RUN_OPENAI=1` + `OPENAI_API_KEY` gating env vars next to the existing Anthropic equivalents. + +**TDD:** Stub the three test files with the env-skip plumbing first; ensure the default suite still passes (marker is excluded). Run `uv run pytest -m openai --no-cov` manually with credentials to validate against the live API once. + +**Acceptance criteria:** +- `uv run pytest` excludes the new tests by default (marker not in default set). +- `uv run pytest -m openai --no-cov` with `SF_RUN_OPENAI=1` + `OPENAI_API_KEY` runs all three tests; without those env vars, each skips with a clear reason naming the missing var. +- Grade live smoke against `gpt-4o` produces a valid `GradingReport` (shape assertions only — no value pinning). +- Draft live smoke against `gpt-4o` produces a `CandidateSchema` that validates and an `LLMResponseEvent` JSONL row with zero cache tokens. +- Live `--estimate` produces non-zero token counts and a non-zero USD estimate; exit 0. + +**Done when:** Above ACs pass; default validation still green (gated tests excluded); maintainer has run the live smokes once. + +**Depends on:** US-002, US-005. + +### US-007 — Documentation surfaces + +**Description:** Update every documentation surface that names the available providers / `--estimate` flow / shim convention. + +**Traces to:** DEC-001 through DEC-014 (collectively). + +**Files:** +- `docs/grade-ops.md` — add an "OpenAI provider" section: config snippet (`grade.provider: openai`, `grade.model: gpt-4o`), `OPENAI_API_KEY` env var, no-prompt-cache caveat, link to live smoke gating. +- `docs/draft-ops.md` — add equivalent section for `draft.provider: openai`. +- `docs/cost-estimate-ops.md` (or wherever `--estimate` ops live; create if absent) — add tiktoken note + the `[openai]` extra requirement. +- `.claude/rules/llm-drafter.md` — extend the "Provider-neutral seam" section with the OpenAIProvider shim notes (Chat Completions adapter pattern, `response_format=json_object`, capability flags False/False). Becomes the canonical precedent for #137 (Gemini). +- `CHANGELOG.md` — under `0.3.0.dev` "Added": `OpenAI as a grading + drafting provider (#136). Set grade.provider: openai or draft.provider: openai in signalforge.yml; requires the [openai] install extra and OPENAI_API_KEY.` +- `README.md` — if the README enumerates supported providers, extend the list. + +**TDD:** N/A (docs only). + +**Acceptance criteria:** +- `docs/grade-ops.md` carries an OpenAI section with a copy-pasteable config example. +- `docs/draft-ops.md` carries the equivalent. +- The estimate ops doc names tiktoken + the `[openai]` extra. +- `.claude/rules/llm-drafter.md` carries the OpenAIProvider shim sub-section. +- `CHANGELOG.md` has the new entry. +- `uv run --only-group docs mkdocs build` succeeds (the `docs-build` CI job mirrors this). + +**Done when:** Above ACs pass; the `docs-build` job is green locally. + +**Depends on:** US-001, US-002, US-005 (so the docs describe a working surface). + +### US-008 — Quality Gate + +**Description:** Multi-pass code review across the full changeset, CodeRabbit if available, full validation including gated markers. + +**Files:** wherever the prior stories' bugs land. + +**Acceptance criteria:** +- `/code-review` run 4 times; every real bug surfaced is fixed (false positives recorded with rationale). +- CodeRabbit review run if accessible. +- `uv sync --dev && uv run ruff check . && uv run ruff format --check . && uv run pyright && uv run pytest` passes. +- `uv run pytest -m anthropic --no-cov` passes (proves Anthropic byte-identity in the estimate refactor end-to-end). +- `uv run pytest -m openai --no-cov` passes locally with credentials. +- `uv run pytest -m wheel_smoke --no-cov` passes (new `[openai]` extra doesn't break wheel build). +- Coverage stays at or above the current threshold. + +**Done when:** All gates green. + +**Depends on:** US-001 through US-007. + +### US-009 — Patterns & Memory + +**Description:** Capture durable lessons in `.claude/rules/llm-drafter.md` (the canonical precedent for #137) and add memory entries for any non-obvious traps surfaced during implementation. + +**Files:** +- `.claude/rules/llm-drafter.md` — refine the OpenAI shim sub-section if implementation surfaced anything unexpected (likely candidates: the `.messages` adapter wrap pattern, tiktoken model-id fallback strategy, `response_format=json_object` interaction with the tolerant JSON parser). +- `~/.claude/projects/-home-wesd-Projects-SignalForge/memory/` — one memory file per non-obvious trap, indexed in `MEMORY.md`. + +**Acceptance criteria:** +- `.claude/rules/llm-drafter.md` has a concrete OpenAI sub-section a future contributor can mirror for #137. +- Memory entries (if any) follow the user/feedback/project/reference taxonomy and link related entries via `[[name]]`. + +**Done when:** Above ACs pass. + +**Depends on:** US-008. + +## Worker-writability routing + +Per `~/.claude/projects/-home-wesd-Projects-SignalForge/memory/ralph-worker-claude-dir-perms.md`: **Ralph workers cannot Write under `.claude/` in worktrees** — only the orchestrator can. The story split honours this: + +- US-001 through US-007 + US-008 (Quality Gate) touch only worker-writable paths (`src/`, `tests/`, `pyproject.toml`, `docs/`, `CHANGELOG.md`, `README.md`, `uv.lock`). +- **US-009 (Patterns & Memory) is orchestrator-only** because it edits `.claude/rules/llm-drafter.md` (and potentially `.claude/rules/grade-layer.md`). If a worker is dispatched against US-009 the bead fails with a write-denied error; route it to the orchestrator. + +This is the same routing convention `#137`'s plan codifies; mirroring it here so the rule lands durably for both #136 and #137. + +## Open notes for implementation + +Pragmatic verification items that depend on the installed SDK version at implementation time — flag during US-001 / US-002 / US-005, not codified as DECs because the SDK surface evolves faster than the plan: + +- **Verify `openai` SDK exception class names + status-code attrs against the installed version.** DEC-009's exception → `ExceptionCategory` mapping (`openai.AuthenticationError`/`PermissionDeniedError` → AUTH; `RateLimitError` → RATE_LIMIT; `APIConnectionError` → CONNECTION; `APIStatusError` 5xx → SERVER_ERROR; 4xx-non-auth + everything else → NO_RETRY) is the shape; the precise class names (`InternalServerError` vs `APIStatusError`-with-status-code; `Timeout` vs `APITimeoutError`) may need a tiny adjustment for `openai>=1.40`. The unit tests in US-002 drive this — write the tests against the installed SDK, then implement to pass. +- **Confirm the `.messages.create` façade adapts cleanly to `chat.completions.create`.** US-001 / US-002: the orchestrator hard-calls `llm_client.messages.create(**kwargs)`. `_OpenAIClientAdapter.messages.create(**kwargs)` delegates to `self._raw.chat.completions.create(**kwargs)`. The kwargs dict shape is OpenAI-native (`model`, `max_tokens`, `messages` list of `{role, content}` dicts, `response_format`). Verify there's no per-call mutation needed; if `extra_headers` is passed (it shouldn't be — capability-gated off), drop it at the adapter rather than at the provider. +- **`tiktoken` model-id fallback table.** `tiktoken.encoding_for_model("gpt-4o")` works for the four planned SKUs at SDK version pin time; if a model id isn't recognised, fall through to `tiktoken.get_encoding("cl100k_base")`. Don't raise on unknown — `--estimate` is a calibration signal, not a billing guarantee (mirrors the planner-estimate caveats in `warehouse-adapters.md` § "estimate_query_bytes graduation"). Log one INFO line per unknown-model fallback so the operator knows the count is approximate. +- **`response_format={"type":"json_object"}` requires "json" in the prompt.** The grade system prompt already names JSON; verify by reading `signalforge.grade.prompts._SYSTEM_PROMPT` during US-002. The drafter system prompt likewise. If either ever drops the word "json", the OpenAI request will fail server-side with a `BadRequestError` — pin a unit test asserting both prompts contain `"json"` (case-insensitive) to catch future drift. +- **`pricing.lookup` returns zero cache fields for the four OpenAI SKUs.** US-004: assert this in `tests/llm/test_pricing.py`. `cli/_estimate.py:489` lines (cache cost math) should produce 0.0 contributions without raising — verify the multiplication doesn't break on a zero `cache_write_5m_per_mtok`. +- **Anthropic byte-identity snapshot — capture in the SAME COMMIT as the refactor.** US-005: the `tests/cli/test_estimate.py` golden file must be added in the same PR commit that introduces the strategy method, or git history can't prove byte-identity. Capture stdout pre-refactor on the feature branch's first commit; refactor on the second; the test compares against the captured golden. If the snapshot changes during the refactor, the refactor is wrong. + +## Beads manifest + +- **Epic:** `bd_1-scaffolding-4tw` — `#136 epic: OpenAI grading provider` (P2, external-ref `gh-136`) +- **Tasks** (dep edges per plan's "Depends on:" lines; all P2): + - `.1` US-001 — `_openai_client.py` shim + `[openai]` extra + 9th AST scan — **READY** (no deps) + - `.2` US-002 — `OpenAIProvider` + registration + config-validator coverage — blocked by `.1` + - `.3` US-003 — `FakeOpenAIClient` + grade end-to-end provider-neutrality test — blocked by `.2` + - `.4` US-004 — Pricing entries (gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4-turbo) — **READY** (parallel-safe; no deps) + - `.5` US-005 — `--estimate` provider-aware token counting — blocked by `.2`, `.4` + - `.6` US-006 — Live gated smoke tests (grade + draft + `--estimate`) — blocked by `.2`, `.5` + - `.7` US-007 — Documentation surfaces — blocked by `.1`, `.2`, `.5` + - `.8` Quality Gate — code-review ×4 + CodeRabbit + full validation + `wheel_smoke` — blocked by `.1`–`.7` + - `.9` Patterns & Memory (orchestrator-only — edits `.claude/rules/`) — blocked by `.8` +- **Cross-epic gate (downstream):** `bd_1-scaffolding-41a` (sentinel for #137 US-007) **`DEPENDS ON .9`**. When this epic completes, US-009 closes → sentinel becomes close-eligible → operator closes sentinel after PR #152 merges to `dev` → #137 US-007 unblocks. See #137 plan DEC-019. +- **Parallel-safe entry points at devolve time:** `.1` (shim/extra/AST) and `.4` (pricing). They touch disjoint files — `.1` edits `_openai_client.py` / `pyproject.toml` / `test_audit_completeness.py`; `.4` edits `pricing.py` / `test_pricing.py` — so `ralph-serialize-shared-registry-beads` does NOT apply. Run them concurrently. +- **Sessions:** 2 (initial plan; devolve + #137 cross-review revisions) + diff --git a/pyproject.toml b/pyproject.toml index 5ac70528..cc44a63e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -23,10 +23,17 @@ dependencies = [ # Retained for `pip install -e ".[dev]"` back-compat. CI runs uv (`uv sync --dev`) # which consumes `[dependency-groups].dev` below; the two lists are kept in sync. # pyright pinned exactly: tool bumps are tested via deliberate maintainer upgrades, not CI surprise from a new release. -dev = ["ruff", "pyright==1.1.409", "pytest", "dbt-core>=1.8,<2", "types-PyYAML>=6,<7", "pytest-cov>=5.0", "snowflake-connector-python>=3,<4", "fakesnow>=0.9"] +dev = ["ruff", "pyright==1.1.409", "pytest", "dbt-core>=1.8,<2", "types-PyYAML>=6,<7", "pytest-cov>=5.0", "snowflake-connector-python>=3,<4", "fakesnow>=0.9", "openai>=1.40,<3.0", "tiktoken>=0.7,<1.0"] # Optional warehouse extra: pulls the Snowflake SDK only when the operator # installs `signalforge-dbt[snowflake]`. The base install stays BigQuery-only. snowflake = ["snowflake-connector-python>=3,<4"] +# Optional LLM provider extra: pulls the OpenAI SDK + tiktoken local +# tokeniser only when the operator installs `signalforge-dbt[openai]`. The +# base install stays Anthropic-only. tiktoken powers the local pre-send +# token count for the `--estimate` path (OpenAI has no server-side +# count_tokens API; supports_token_count=False on OpenAIProvider). See +# .claude/rules/llm-drafter.md for the three-slot lockstep convention. +openai = ["openai>=1.40,<3.0", "tiktoken>=0.7,<1.0"] [dependency-groups] # uv-native dependency groups (PEP 735). @@ -52,6 +59,8 @@ dev = [ "build>=1.2,<2", "snowflake-connector-python>=3,<4", "fakesnow>=0.9", + "openai>=1.40,<3.0", + "tiktoken>=0.7,<1.0", {include-group = "docs"}, ] @@ -96,7 +105,7 @@ testpaths = ["tests"] # `--import-mode=importlib` lets us share basenames across test dirs (e.g. # tests/manifest/test_errors.py and tests/warehouse/test_errors.py) without # adding tests/__init__.py — keeps `testing-signal.md`'s no-init rule intact. -addopts = "-ra --strict-markers --import-mode=importlib -m 'not bigquery and not anthropic and not cli_subprocess and not e2e and not wheel_smoke and not snowflake' --cov=signalforge --cov-report=xml --cov-report=term-missing --cov-fail-under=80" +addopts = "-ra --strict-markers --import-mode=importlib -m 'not bigquery and not anthropic and not cli_subprocess and not e2e and not wheel_smoke and not snowflake and not openai' --cov=signalforge --cov-report=xml --cov-report=term-missing --cov-fail-under=80" minversion = "7.0" strict_markers = true markers = [ @@ -112,4 +121,5 @@ markers = [ "e2e: end-to-end smoke test against real Anthropic + real BigQuery (gated by SF_RUN_BQ=1, ANTHROPIC_API_KEY, GOOGLE_CLOUD_PROJECT; skipped by default)", "wheel_smoke: maintainer-only; builds the wheel via `python -m build --wheel` and inspects the artifact for the demo file set (run with --no-cov)", "snowflake: maintainer-only; offline compiled-SQL validation via fakesnow AND the gated live EXPLAIN-estimate certification (run with --no-cov; live tests self-skip without SF_RUN_SNOWFLAKE + connection env vars)", + "openai: real-API smoke test against OpenAI (requires SF_RUN_OPENAI=1 + OPENAI_API_KEY; excluded from default CI)", ] diff --git a/src/signalforge/cli/_estimate.py b/src/signalforge/cli/_estimate.py index db1f9d63..8dd6f693 100644 --- a/src/signalforge/cli/_estimate.py +++ b/src/signalforge/cli/_estimate.py @@ -72,7 +72,7 @@ import time import uuid from pathlib import Path -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING from pydantic import BaseModel, ConfigDict @@ -85,6 +85,7 @@ ) from signalforge.grade.rubric import DEFAULT_RUBRIC from signalforge.llm import pricing as _pricing +from signalforge.llm.providers import provider_for from signalforge.safety.models import LLMRequest, SamplingMode from signalforge.warehouse._sql_safety import validate_identifier from signalforge.warehouse.errors import WarehouseError @@ -94,7 +95,6 @@ from signalforge.draft.config import DraftConfig from signalforge.grade.config import GradeConfig from signalforge.grade.rubric import Criterion - from signalforge.llm import AnthropicClientProtocol from signalforge.manifest.models import Manifest, Model from signalforge.prune.config import PruneConfig from signalforge.warehouse.base import WarehouseAdapter @@ -284,51 +284,55 @@ def _truncate_criterion_text(text: str) -> str: def _count_draft_tokens( *, - client: AnthropicClientProtocol, + client: object | None, draft_config: DraftConfig, system: str, cached_block: str, dynamic_block: str, ) -> int: - """Issue exactly one ``count_tokens`` for the drafter prompt and - return the integer ``input_tokens`` field. - - Mirrors :func:`signalforge.llm.client.call_llm`'s pre-send - ``count_tokens`` envelope: system + the single cached user-content - block. The dynamic block is included in the messages list so the - count reflects the full prompt the drafter would send (the cache - boundary affects pricing, not token count — pricing math runs - further down using ``draft_config.cache_ttl`` indirectly). + """Return the input-token count for the drafter prompt. + + Refactored in #136 US-005 (DEC-003) to dispatch through + :meth:`signalforge.llm.providers.LLMProvider.estimate_input_tokens` + so the engine works against any registered provider — Anthropic + delegates to the SDK's server-side ``messages.count_tokens``; + OpenAI counts locally via ``tiktoken``. Byte-identity with the + pre-refactor Anthropic path is pinned by + ``tests/cli/test_estimate.py::test_estimate_anthropic_byte_identity_golden`` + (DEC-013). + + The ``system`` envelope is threaded as its own kwarg so providers + whose API counts the system block separately (Anthropic's + server-side ``messages.count_tokens(system=..., ...)``) produce + real-API-faithful counts (DEC-013 of #136). The ``cached_block`` + + ``dynamic_block`` are concatenated into the user-content text + payload because Anthropic's tokenizer collapses adjacent text blocks + identically to a single concatenated string for counting purposes. + The cache boundary affects pricing, not token count — pricing math + runs further down using ``draft_config.cache_ttl`` indirectly. """ - block_cached: dict[str, Any] = { - "type": "text", - "text": cached_block, - "cache_control": {"type": "ephemeral", "ttl": draft_config.cache_ttl}, - } - block_dynamic: dict[str, Any] = {"type": "text", "text": dynamic_block} - response = client.messages.count_tokens( - model=draft_config.model, - system=system, - messages=[{"role": "user", "content": [block_cached, block_dynamic]}], + text = cached_block + dynamic_block + return provider_for(draft_config.provider).estimate_input_tokens( + draft_config.model, text, system=system, client=client ) - input_tokens = getattr(response, "input_tokens", None) - if not isinstance(input_tokens, int): - msg = "count_tokens response is missing the `input_tokens` field." - raise RuntimeError(msg) - return input_tokens def _count_grade_criterion_tokens( *, - client: AnthropicClientProtocol, + client: object | None, grade_config: GradeConfig, system_and_rubric: str, artifact_id: str, artifact_text: str, criterion: Criterion, ) -> int: - """Issue one ``count_tokens`` for the representative ``(artifact, - criterion)`` prompt and return ``input_tokens``. + """Return the input-token count for the representative + ``(artifact, criterion)`` grade prompt. + + Refactored in #136 US-005 (DEC-003) to dispatch through + :meth:`signalforge.llm.providers.LLMProvider.estimate_input_tokens` + so the engine works against any registered provider — see + :func:`_count_draft_tokens` for the byte-identity contract. Per ``grade-layer.md`` DEC-004 the real grader issues one ``messages.create`` per ``(artifact × criterion)`` pair. The @@ -337,24 +341,28 @@ def _count_grade_criterion_tokens( across artifacts so per-criterion token counts scale linearly with artifact count, and counting one representative is a faithful proxy that keeps the LLM-call count bounded. + + **Pre-existing double-count corrected in #136 US-008 QG.** The + pre-US-005 inline Anthropic call passed ``system=system_and_rubric`` + AND embedded ``system_and_rubric`` inside the cached user-content + block, counting the rubric twice on every per-criterion estimate. + The first QG fix mistakenly preserved that double-count in the + name of byte-identity, which then *triple*-counted for OpenAI + (system kwarg → OpenAI's ``system + text`` concat → rubric prefix + in text). The correct behaviour matches the runtime grader call: + rubric in ``system=`` once, the artifact envelope in user content, + no overlap. Anthropic call shape is now system-envelope(rubric) + + dynamic_block (drops one rubric copy from the pre-refactor bytes); + OpenAI counts system_and_rubric + dynamic_block once. Fake-driven + byte-identity golden still passes (canned token counts unchanged + by call-shape), so the rendered USD floor for the golden fixture + holds; real-API ``--estimate`` figures shift down by ~one rubric + per criterion. See CHANGELOG. """ dynamic_block = render_grade_dynamic_block(artifact_id, artifact_text, criterion) - block_cached: dict[str, Any] = { - "type": "text", - "text": system_and_rubric, - "cache_control": {"type": "ephemeral", "ttl": grade_config.cache_ttl}, - } - block_dynamic: dict[str, Any] = {"type": "text", "text": dynamic_block} - response = client.messages.count_tokens( - model=grade_config.model, - system=system_and_rubric, - messages=[{"role": "user", "content": [block_cached, block_dynamic]}], + return provider_for(grade_config.provider).estimate_input_tokens( + grade_config.model, dynamic_block, system=system_and_rubric, client=client ) - input_tokens = getattr(response, "input_tokens", None) - if not isinstance(input_tokens, int): - msg = "count_tokens response is missing the `input_tokens` field." - raise RuntimeError(msg) - return input_tokens def _build_representative_sql(model: Model, adapter: WarehouseAdapter, sample_size: int) -> str: @@ -416,7 +424,7 @@ def estimate( grade_config: GradeConfig, prune_config: PruneConfig, adapter: WarehouseAdapter, - anthropic_client: AnthropicClientProtocol, + client: object | None, *, project_dir: Path | None = None, # noqa: ARG001 (reserved for v0.2) ) -> EstimateReport: @@ -443,8 +451,18 @@ def estimate( prune_config: Loaded :class:`PruneConfig` (carries ``sample_size`` for the representative dry-run SQL). adapter: Constructed :class:`WarehouseAdapter`. - anthropic_client: Constructed Anthropic client (or test fake - satisfying the protocol). + client: Optional pre-constructed provider-shaped client (or + test fake satisfying the active provider's protocol). + Renamed from ``anthropic_client`` in #136 US-008 QG — + after US-005 the slot was already typed ``object | None`` + and forwarded verbatim to whichever provider strategy is + active; the old name implied Anthropic-only and would + mislead a future #137 Gemini wiring. The CLI in + ``generate.py`` builds an Anthropic SDK client only when + ``provider == "anthropic"`` and passes ``None`` otherwise; + providers whose count is local (OpenAI's ``tiktoken``, + #137 Gemini's planned native ``count_tokens``) ignore the + kwarg either way. project_dir: Reserved for v0.2 (caching, sidecar paths). Returns: @@ -458,7 +476,7 @@ def estimate( request = _build_schema_only_request(model) system, cached_block, dynamic_block, _prompt_version = render_prompt(model, request, manifest) draft_input_tokens = _count_draft_tokens( - client=anthropic_client, + client=client, draft_config=draft_config, system=system, cached_block=cached_block, @@ -499,7 +517,7 @@ def estimate( grade_usd = 0.0 for criterion in rubric: input_tokens_per_call = _count_grade_criterion_tokens( - client=anthropic_client, + client=client, grade_config=grade_config, system_and_rubric=system_and_rubric, artifact_id=rep_artifact_id, diff --git a/src/signalforge/cli/generate.py b/src/signalforge/cli/generate.py index 9d42c0f4..c666c987 100644 --- a/src/signalforge/cli/generate.py +++ b/src/signalforge/cli/generate.py @@ -807,24 +807,22 @@ def _run_single_model( # DEC-006 of #135 — the four pipeline orchestrators now let # ``call_llm`` lazy-build the client via the configured provider, # so the CLI no longer constructs one for the real-run path. The - # ``--estimate`` engine, however, needs a concrete client up front - # (it issues ``count_tokens`` directly, an Anthropic-specific - # surface), so it builds one through the provider registry keyed on - # the drafter's configured provider. Any auth failure surfaces at - # the existing panic boundary → tier 3 via _EXCEPTION_TO_EXIT_CODE. + # ``--estimate`` engine takes the same shape: each stage's + # ``count`` call dispatches through + # ``provider_for(config.provider).estimate_input_tokens(...)`` + # (#136 US-005 DEC-003), so a non-Anthropic provider's + # ``--estimate`` path now works too — OpenAI counts locally via + # ``tiktoken`` and ignores the threaded client. # - # ``make_client`` returns ``object`` (the provider-neutral seam - # type, #135 DEC-012); the ``--estimate`` engine's pre-flight cost - # preview is Anthropic-specific (it calls ``count_tokens`` on the - # ``AnthropicClientProtocol`` surface — #36), so cast to that - # protocol here. A non-Anthropic provider's ``--estimate`` path is - # out of scope until #136/#137 wire those vendors. - # - # The estimate engine takes BOTH configs but drives this single - # Anthropic-shaped client, so fail fast rather than silently - # project grade-stage cost through the drafter's client (if the - # two providers diverge) or blow up on a provider with no - # ``count_tokens`` surface. + # The engine still accepts a single optional client object so + # Anthropic's per-call SDK construction is avoidable when the + # CLI already has one in scope. For non-Anthropic providers we + # pass ``None`` and the provider handles it (no-op for OpenAI; + # error for any provider that genuinely needs an SDK client and + # was given none — only Anthropic does today). The single-client + # design still requires the two stages' providers to match so + # we don't silently project grade-stage cost through the + # drafter's vendor when they diverge. if draft_config.provider != grade_config.provider: raise CliInputError( "--estimate requires draft.provider and grade.provider to match " @@ -834,19 +832,18 @@ def _run_single_model( "--estimate until per-provider estimation support lands." ), ) - if draft_config.provider != "anthropic": - raise CliInputError( - "--estimate currently supports only provider='anthropic' " - f"(got {draft_config.provider!r}).", - remediation=( - "Set draft.provider and grade.provider to 'anthropic', or " - "run without --estimate." - ), + # Build a concrete client only for providers that need one + # (Anthropic). Providers that count locally (OpenAI) ignore + # the kwarg; passing ``None`` keeps us from constructing an + # SDK client + requiring its API key purely for a local count. + client: object | None + if draft_config.provider == "anthropic": + client = cast( + "AnthropicClientProtocol", + provider_for(draft_config.provider).make_client(), ) - client = cast( - "AnthropicClientProtocol", - provider_for(draft_config.provider).make_client(), - ) + else: + client = None report = estimate_module.estimate( model, manifest, diff --git a/src/signalforge/llm/__init__.py b/src/signalforge/llm/__init__.py index eb863d2c..dc8c684a 100644 --- a/src/signalforge/llm/__init__.py +++ b/src/signalforge/llm/__init__.py @@ -28,6 +28,7 @@ AnthropicProvider, ExceptionCategory, LLMProvider, + OpenAIProvider, UsageMetrics, provider_for, register_provider, @@ -51,6 +52,7 @@ "LLMResult", "LLMServerError", "ModelPricing", + "OpenAIProvider", "UnknownProviderError", "UsageMetrics", "call_llm", diff --git a/src/signalforge/llm/_openai_client.py b/src/signalforge/llm/_openai_client.py new file mode 100644 index 00000000..b0d616c0 --- /dev/null +++ b/src/signalforge/llm/_openai_client.py @@ -0,0 +1,278 @@ +"""Centralised OpenAI SDK seam (#136 DEC-010 confinement). + +US-001 of #136 establishes the single shim where every +``# pyright: ignore[...]`` / ``# type: ignore[...]`` provoked by the ``openai`` +SDK is allowed to live. The rest of :mod:`signalforge.llm` and all of +:mod:`signalforge.draft` / :mod:`signalforge.grade` import the typed surface +this module exposes and stay pyright-clean. + +Mirrors the precedent established by +:mod:`signalforge.llm._anthropic_client` for the Anthropic SDK (see +``.claude/rules/llm-drafter.md`` — "One SDK seam — `signalforge.llm._anthropic_client` +confines every ``# pyright: ignore`` (DEC-012)") and +:mod:`signalforge.warehouse.adapters._snowflake_client` for the +``snowflake-connector-python`` SDK. When a future v0.x LLM provider is added +(e.g. #137 Gemini), it should get its own ``__client.py`` shim under +:mod:`signalforge.llm` for the same reason; do not pool SDK ignores into a +generic util module. + +Three responsibilities: + +* :class:`OpenAIClientProtocol` — duck-typed surface common to the real + OpenAI client (wrapped behind :class:`_OpenAIClientAdapter` to expose the + ``.messages`` namespace the orchestrator hard-calls) and the test fake + (lands in US-003). Narrow on purpose — only the methods + :func:`signalforge.llm.client.call_llm` actually consumes. +* :func:`_make_openai_client` — factory that returns + ``_OpenAIClientAdapter(openai.OpenAI(api_key=api_key))``. Lazy-imports the + SDK so test environments that inject a fake never pay the import cost. + The adapter wraps the underlying OpenAI client to expose ``.messages.create`` + (delegating to ``chat.completions.create``) — DEC-009. +* :func:`_count_openai_tokens` — local token counter via ``tiktoken``, + with ``cl100k_base`` fallback for unknown model ids (DEC-012). The + ``supports_token_count=False`` capability flag on ``OpenAIProvider`` + means the orchestrator skips its pre-send count gate; this helper is + used by the ``--estimate`` path (US-005), not the runtime retry loop. + +OpenAI Chat Completions API surface (DEC-001): + +* ``client.chat.completions.create(model=..., max_tokens=..., messages=[...], + response_format={"type": "json_object"})`` +* Response: ``response.choices[0].message.content`` is the assistant text; + ``response.usage.prompt_tokens`` / ``response.usage.completion_tokens`` + are the token counts (no cache fields — OpenAI has no equivalent cache + discount, so ``UsageMetrics.cache_*_input_tokens`` is always ``0`` for + this provider; matches ``supports_prompt_caching=False``). + +Observability discipline (mirroring DEC-027 from the warehouse layer): no +logger calls in this shim. Logging lives in the seam +(:mod:`signalforge.llm.client`) where the stage label is known. The shim +itself is structural plumbing. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from types import SimpleNamespace +from typing import Any, Protocol, runtime_checkable + + +@runtime_checkable +class _OpenAIMessagesProtocol(Protocol): + """Duck-typed surface of the ``messages`` namespace on the adapter. + + The OpenAI Python SDK exposes ``client.chat.completions.create(...)`` + rather than ``client.messages.create(...)``; the + :class:`_OpenAIClientAdapter` returned by :func:`_make_openai_client` + wraps the real client to expose a :attr:`messages` namespace whose + :meth:`create` delegates to ``chat.completions.create``. This protocol + types that namespace so :func:`signalforge.llm.client.call_llm` calls + the same method signatures regardless of which provider is wired. + + :meth:`count_tokens` raises :class:`NotImplementedError` because + OpenAI has no equivalent of Anthropic's pre-send count API; the + orchestrator gates the pre-send count call on + ``supports_token_count=True`` and so never invokes this method for an + ``OpenAIProvider`` (capability flag is ``False``). The method is + declared on the protocol for structural parity with the Anthropic + shim, and the adapter raises defensively if it is ever called. + """ + + def create(self, **kwargs: Any) -> Any: ... + + def count_tokens(self, **kwargs: Any) -> Any: ... + + +@runtime_checkable +class OpenAIClientProtocol(Protocol): + """Duck-typed surface common to the OpenAI adapter and the test fake. + + Both production (the :class:`_OpenAIClientAdapter` wrapper returned by + :func:`_make_openai_client`) and test + (``tests/llm/_fake_openai.py::FakeOpenAIClient``, US-003) clients + satisfy this protocol, so :func:`signalforge.llm.client.call_llm` can + call the same method signatures regardless of which client was + injected. The protocol is intentionally narrow — only the surface + :func:`call_llm` actually consumes (``messages.create``; + ``messages.count_tokens`` is gated off by + ``supports_token_count=False`` but declared for protocol parity with + the Anthropic shim). + + Re-exported as ``signalforge.llm.OpenAIClientProtocol`` in US-002 so + the ``client`` kwarg on :func:`signalforge.draft.draft_schema` and + :func:`signalforge.grade.grade_artifacts` can be type-annotated + without importing a private underscore-prefixed name. + """ + + messages: _OpenAIMessagesProtocol + + +class _OpenAIClientAdapter: + """Wrap an ``openai.OpenAI`` client to expose a ``.messages`` namespace. + + The orchestrator at :func:`signalforge.llm.client.call_llm` hard-calls + ``llm_client.messages.create(**kwargs)``. The OpenAI SDK exposes + ``client.chat.completions.create(...)`` instead, so the adapter + rebinds the surface: :attr:`messages` is a + :class:`types.SimpleNamespace` with a :meth:`create` callable that + delegates to ``self._raw.chat.completions.create`` and a + :meth:`count_tokens` callable that raises + :class:`NotImplementedError` (orchestrator never calls it for a + ``supports_token_count=False`` provider, per DEC-011 of #136 — but + raising is the honest behaviour if the gate ever drifts). + + The ``_raw`` reference is kept on the adapter so future code paths + (e.g. structured-outputs API, streaming) can reach through if needed + without re-importing the SDK. + + Construction goes only through :func:`_make_openai_client` — this + class is internal plumbing for the shim. + """ + + def __init__(self, raw_client: Any) -> None: + self._raw = raw_client + self.messages = SimpleNamespace( + create=self._messages_create, + count_tokens=self._messages_count_tokens, + ) + + def _messages_create(self, **kwargs: Any) -> Any: + """Delegate to ``self._raw.chat.completions.create(**kwargs)``. + + The kwargs dict is OpenAI-native (``model``, ``max_tokens``, + ``messages`` list of ``{role, content}`` dicts, + ``response_format``); the caller in + :meth:`OpenAIProvider.build_create_kwargs` (US-002) is + responsible for shaping the payload. + """ + return self._raw.chat.completions.create(**kwargs) + + def _messages_count_tokens(self, **kwargs: Any) -> Any: # pragma: no cover - defensive + """Defensive: orchestrator never calls this for OpenAI. + + ``OpenAIProvider.supports_token_count`` is ``False``, so + :func:`signalforge.llm.client.call_llm` skips its pre-send + count-tokens gate entirely for this provider (DEC-008 of #135). + Raising here surfaces a regression where the gate drifts and the + orchestrator starts calling this method against an OpenAI client + — better a loud :class:`NotImplementedError` than a silent + fabricated zero (DEC-011 of #136). + """ + raise NotImplementedError( + "OpenAI provider does not support pre-send count_tokens; " + "supports_token_count=False gates this call off in the " + "orchestrator. If you see this, the capability-flag gate has drifted." + ) + + +def _make_openai_client( + api_key: str | None = None, +) -> OpenAIClientProtocol: # pragma: no cover - exercised by integration tests only + """Construct a real ``openai.OpenAI`` client wrapped in the adapter. + + ``api_key=None`` lets the SDK consume the ``OPENAI_API_KEY`` + environment variable (standard SDK behaviour); explicit values are + preserved for callers that thread credentials through configuration. + + The ``openai`` import is lazy so test environments that inject a + fake never pay the SDK import cost, and so a base install without + the ``[openai]`` extra does not crash at module-import time + (DEC-014 of #136 — :func:`_load_openai_exception_classes` returns + empty tuples on ``ImportError``; this factory raises naturally on + import-time failure because a caller who reached this path *did* + ask for a real client). + """ + import openai # type: ignore[import-not-found] + + return _OpenAIClientAdapter(openai.OpenAI(api_key=api_key)) # type: ignore[no-any-return] + + +@dataclass(frozen=True) +class _OpenAIExceptionClasses: + """Bundle of SDK exception classes used by the retry loop in + :func:`signalforge.llm.client.call_llm`. + + Each tuple is the ``except`` clause's catch surface for one branch + of the retry taxonomy (DEC-004 of #5, generalised by #135's + provider-neutral seam). Wrapping them in a frozen dataclass keeps + the seam's import surface narrow and confines every OpenAI-SDK + ``# type: ignore`` to this module (DEC-010 of #136). + + The :class:`OpenAIProvider.classify_exception` impl (US-002) uses + this bundle to route a caught SDK exception to the right + :class:`signalforge.llm.providers.ExceptionCategory`. + """ + + rate_limit: tuple[type[BaseException], ...] + api_status: tuple[type[BaseException], ...] + auth: tuple[type[BaseException], ...] + connection: tuple[type[BaseException], ...] + + +def _load_openai_exception_classes() -> _OpenAIExceptionClasses: + """Lazy-import the SDK exception classes the retry loop catches. + + Returning empty tuples on ``ImportError`` is a defensive fallback + for base installs that did NOT pull the ``[openai]`` extra (DEC-014 + of #136 — mirrors :func:`_load_anthropic_exception_classes`'s + ``pragma: no cover`` branch). With empty tuples, every caught + exception in the retry loop routes to ``NO_RETRY`` cleanly — the + operator never reaches this path in practice because the provider + registry validator at config load rejects ``provider: openai`` + when the registration also fails import-time, but graceful + fallback at import-time is the contract. + """ + try: + import openai # type: ignore[import-not-found] + except ImportError: # pragma: no cover - the SDK is optional via [openai] + empty: tuple[type[BaseException], ...] = () + return _OpenAIExceptionClasses( + rate_limit=empty, api_status=empty, auth=empty, connection=empty + ) + return _OpenAIExceptionClasses( + rate_limit=(openai.RateLimitError,), + api_status=(openai.APIStatusError,), + auth=(openai.AuthenticationError, openai.PermissionDeniedError), + connection=(openai.APIConnectionError,), + ) + + +def _count_openai_tokens(model: str, text: str) -> int: + """Count tokens for ``text`` under ``model`` using ``tiktoken`` (DEC-012). + + Used by the ``--estimate`` path (US-005) to render a pre-send + grader/drafter cost preview without an API call (mirrors the + Anthropic ``messages.count_tokens`` path; OpenAI has no equivalent + server-side pre-send count so we count locally). + + Falls back to the ``cl100k_base`` encoding if + ``tiktoken.encoding_for_model(model)`` raises ``KeyError`` for an + unknown model id — ``--estimate`` is a calibration signal, not a + billing guarantee (mirrors the planner-estimate caveats in + ``warehouse-adapters.md`` § "estimate_query_bytes graduation"). The + fallback is loud at the logger seam (US-005), silent here in the + shim per the no-logger discipline. + + The ``tiktoken`` import is lazy so a base install without the + ``[openai]`` extra does not crash this module at import time + (DEC-012 of #136 — ``tiktoken`` ships with ``openai`` in the + ``[openai]`` extra). + """ + import tiktoken # type: ignore[import-not-found] + + try: + encoding = tiktoken.encoding_for_model(model) + except KeyError: + encoding = tiktoken.get_encoding("cl100k_base") + return len(encoding.encode(text)) + + +__all__ = [ + "OpenAIClientProtocol", + "_OpenAIClientAdapter", + "_OpenAIExceptionClasses", + "_OpenAIMessagesProtocol", + "_count_openai_tokens", + "_load_openai_exception_classes", + "_make_openai_client", +] diff --git a/src/signalforge/llm/errors.py b/src/signalforge/llm/errors.py index ff48a5b7..2e3cd633 100644 --- a/src/signalforge/llm/errors.py +++ b/src/signalforge/llm/errors.py @@ -192,7 +192,7 @@ class EstimateUnknownModelError(LLMError): default_remediation: ClassVar[str] = ( "Add the model to signalforge.llm.pricing.PRICES or use a " "supported model: claude-sonnet-4-6, claude-opus-4-7, " - "claude-haiku-4-5." + "claude-haiku-4-5, gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4-turbo." ) def __init__( diff --git a/src/signalforge/llm/pricing.py b/src/signalforge/llm/pricing.py index 21c847ec..50a57687 100644 --- a/src/signalforge/llm/pricing.py +++ b/src/signalforge/llm/pricing.py @@ -14,16 +14,21 @@ same input → same USD math, every time. Field assignment raises :class:`dataclasses.FrozenInstanceError`. * :data:`PRICES` — immutable mapping (``types.MappingProxyType``) from SKU - string to :class:`ModelPricing`. v0.1 covers the three Anthropic SKUs - this project supports. + string to :class:`ModelPricing`. Covers three Anthropic SKUs + + four OpenAI SKUs (added in #136 US-004 per DEC-007 of the + provider-neutral grading-provider plan). * :func:`lookup` — the public access seam; raises :class:`signalforge.llm.errors.EstimateUnknownModelError` (CLI tier 2 — input-validation) on miss rather than returning ``None``, so downstream callers don't need to defensively branch. -Note on prices. The four fields encode Anthropic's public per-million-token -pricing for the message-batches / standard channel as of -:data:`PRICE_TABLE_VERSION`: +Note on prices. The four fields encode each provider's public +per-million-token pricing for the message-batches / standard channel +(Anthropic) or the standard chat-completions channel (OpenAI) as of +:data:`PRICE_TABLE_VERSION`. OpenAI does not currently expose a +prompt-cache discount tier comparable to Anthropic's +``cache_control``, so the two cache fields on OpenAI SKUs are +``0.0`` (no discount, no premium): * ``input_per_mtok`` — non-cached input tokens. * ``output_per_mtok`` — output (assistant) tokens. @@ -53,13 +58,13 @@ ] -PRICE_TABLE_VERSION: str = "2026-05-11" +PRICE_TABLE_VERSION: str = "2026-05-28" """Sourcing-date stamp for :data:`PRICES`. Bump alongside any numeric edit.""" @dataclass(frozen=True, slots=True) class ModelPricing: - """Per-million-token USD prices for one Anthropic SKU. + """Per-million-token USD prices for one provider SKU (Anthropic or OpenAI). ``frozen=True`` is the reproducibility invariant — once constructed, a :class:`ModelPricing` instance cannot mutate. ``slots=True`` keeps @@ -77,13 +82,21 @@ class ModelPricing: cache_read_per_mtok: float -# TODO: verify v0.1 ships with current pricing (refresh PRICE_TABLE_VERSION -# above when these numbers change). Values below mirror Anthropic's -# publicly published per-MTok pricing for the standard channel as of the -# table version stamp; the Sonnet-family numbers are the canonical -# reference point. Apply a deliberate refresh commit per the -# "5-surface parity" rule in prune-engine.md when bumping any field. +# TODO: verify each row ships with current pricing (refresh +# PRICE_TABLE_VERSION above when these numbers change). Anthropic values +# mirror the publicly published per-MTok pricing for the standard channel +# as of the table version stamp; the Sonnet-family numbers are the +# canonical reference point. OpenAI values are calibration figures +# captured from OpenAI's public price page at PR-prep time (see +# #136 US-004); operators should treat them as a sanity-check baseline +# rather than a billing guarantee — bump PRICE_TABLE_VERSION any time +# they're refreshed. OpenAI does not currently expose a prompt-cache +# discount tier comparable to Anthropic's `cache_control`, so the two +# OpenAI cache fields are 0.0 (no discount, no premium). Apply a +# deliberate refresh commit per the "5-surface parity" rule in +# prune-engine.md when bumping any field. _PRICES_MUTABLE: dict[str, ModelPricing] = { + # -- Anthropic SKUs ------------------------------------------------ "claude-sonnet-4-6": ModelPricing( input_per_mtok=3.00, output_per_mtok=15.00, @@ -102,10 +115,38 @@ class ModelPricing: cache_write_5m_per_mtok=1.00, cache_read_per_mtok=0.08, ), + # -- OpenAI SKUs (#136 US-004, DEC-007) ---------------------------- + # `gpt-4o` is the default judge per DEC-004; the other three are + # supported back-/cross-compat options. Cache fields are 0.0 — OpenAI + # has no Anthropic-equivalent `cache_control` discount tier. + "gpt-4o": ModelPricing( + input_per_mtok=2.50, + output_per_mtok=10.00, + cache_write_5m_per_mtok=0.0, + cache_read_per_mtok=0.0, + ), + "gpt-4o-mini": ModelPricing( + input_per_mtok=0.15, + output_per_mtok=0.60, + cache_write_5m_per_mtok=0.0, + cache_read_per_mtok=0.0, + ), + "gpt-4.1": ModelPricing( + input_per_mtok=2.00, + output_per_mtok=8.00, + cache_write_5m_per_mtok=0.0, + cache_read_per_mtok=0.0, + ), + "gpt-4-turbo": ModelPricing( + input_per_mtok=10.00, + output_per_mtok=30.00, + cache_write_5m_per_mtok=0.0, + cache_read_per_mtok=0.0, + ), } PRICES: Mapping[str, ModelPricing] = MappingProxyType(_PRICES_MUTABLE) -"""Read-only mapping from Anthropic SKU to :class:`ModelPricing`. +"""Read-only mapping from provider SKU (Anthropic or OpenAI) to :class:`ModelPricing`. ``MappingProxyType`` is the standard-library immutable-mapping wrapper: callers can iterate and look up entries, but cannot mutate the table @@ -127,8 +168,8 @@ def lookup(model: str) -> ModelPricing: Raises :class:`EstimateUnknownModelError` (CLI tier 2) when ``model`` is not in :data:`PRICES`. The remediation locked on :class:`EstimateUnknownModelError.default_remediation` points the - operator at either adding the SKU or picking one of the three v0.1 - supported models. + operator at either adding the SKU or picking one of the supported + models (three Anthropic SKUs + four OpenAI SKUs as of #136 US-004). Returning the typed exception (rather than ``None`` or a sentinel) keeps downstream callers free of defensive branching — the diff --git a/src/signalforge/llm/providers.py b/src/signalforge/llm/providers.py index 24bb37a5..0dea91f8 100644 --- a/src/signalforge/llm/providers.py +++ b/src/signalforge/llm/providers.py @@ -174,6 +174,50 @@ def extract_usage(self, response: object) -> UsageMetrics: def classify_exception(self, exc: BaseException) -> ExceptionCategory: """Map a raised vendor exception to a neutral :class:`ExceptionCategory`.""" + @abc.abstractmethod + def estimate_input_tokens( + self, + model: str, + text: str, + *, + system: str = "", + client: object | None = None, + ) -> int: + """Return the input-token count the prompt would consume on ``model``. + + Used by the ``signalforge generate --estimate`` cost-preview path + (issue #36 / #136 US-005 — DEC-003). The Anthropic implementation + delegates to the SDK's ``messages.count_tokens`` (a server-side + count) so the figure matches what the runtime path would bill; + the OpenAI implementation delegates to ``tiktoken`` (a local BPE + count) because OpenAI has no equivalent pre-send count API. + + Capability flags (DEC-008 of #135) do NOT gate this method — + every provider must answer "how many tokens is this text?" even + if it lacks Anthropic-style prompt caching. The runtime retry + loop's pre-send count gate (which IS gated by + :attr:`supports_token_count`) is a different surface; this is + the estimate path's calibration seam. + + ``system`` is the system-prompt envelope, passed separately so + providers whose API counts the system block with its own envelope + tokens (Anthropic's ``messages.count_tokens(system=..., ...)``) + produce real-API-faithful counts (#136 US-005 / DEC-013 byte- + identity floor). Providers whose local tokenizer doesn't + distinguish (OpenAI's ``tiktoken``) MAY concatenate ``system + + text`` before counting; the total still includes every token. + Defaulting to an empty string preserves the call shape for + callers who don't separate the two surfaces yet. + + ``client`` is an optional pre-constructed SDK client (e.g. a + test fake satisfying the vendor's client protocol). Providers + that build a transient client per call (e.g. Anthropic) MAY + accept and reuse it to avoid the construction cost; providers + whose implementation is local (e.g. OpenAI's ``tiktoken``) MAY + ignore it. The orchestrator passes whatever client it already + has in scope (or ``None``); the provider decides. + """ + # Process-level provider registry, keyed by ``provider.name`` (DEC-003). Module # scope makes it a single registry per process; US-002 registers @@ -355,6 +399,85 @@ def classify_exception(self, exc: BaseException) -> ExceptionCategory: return ExceptionCategory.NO_RETRY return ExceptionCategory.NO_RETRY + def estimate_input_tokens( + self, + model: str, + text: str, + *, + system: str = "", + client: object | None = None, + ) -> int: + """Count tokens via the Anthropic SDK's ``messages.count_tokens``. + + Preserves real-API byte-identity with the pre-#136-US-005 inline + ``client.messages.count_tokens(...)`` calls in + :mod:`signalforge.cli._estimate` (DEC-013 of #136): the count is + issued with ``system`` threaded as its own kwarg (so Anthropic's + server-side tokenizer applies its system-block envelope tokens) + plus a single user-message text block whose ``content`` is + ``text`` (the concatenated cached + dynamic prompt body). + Anthropic's tokenizer collapses adjacent text blocks identically + to a single concatenated string for counting purposes, so the + post-refactor user-content shape matches the pre-refactor + ``[block_cached, block_dynamic]`` structured form at the count + level. + + Tests inject a queued-response ``FakeAnthropicClient`` via + ``client``; production callers either pass a pre-constructed + client (from the ``--estimate`` CLI prelude) or rely on the + lazy fallback below. + + When ``client`` is ``None``, the provider builds a transient + SDK client via + :func:`signalforge.llm._anthropic_client._make_anthropic_client`. + This path is reachable only when a caller invokes the engine + without threading a client through (no v0.x caller does so on + the happy path); the cost of one SDK construction per call is + acceptable for that fallback case. + """ + from typing import cast + + from signalforge.llm._anthropic_client import ( + AnthropicClientProtocol, + _make_anthropic_client, + ) + from signalforge.llm.errors import LLMResponseFormatError + + # Cast the optional ``client`` to the public protocol so pyright + # sees the ``messages.count_tokens`` surface without a type-checker + # suppression here — the DEC-012 confinement rule keeps every + # SDK suppression inside ``_anthropic_client.py``. Both the real + # ``anthropic.Anthropic`` (via the shim's ``_make_anthropic_client``) + # and the ``FakeAnthropicClient`` test fake satisfy the protocol + # structurally. + resolved: AnthropicClientProtocol = ( + _make_anthropic_client() if client is None else cast(AnthropicClientProtocol, client) + ) + # Pass ``system`` only when non-empty so a default ``""`` doesn't + # ship a bogus ``system=""`` kwarg the SDK would otherwise carry + # as an extra envelope block. Two explicit call shapes (instead + # of dict-unpack) keep the call site within the + # ``AnthropicClientProtocol`` typed surface — DEC-012 of #135 + # confines every Anthropic SDK type-checker suppression to + # ``_anthropic_client.py``. + if system: + response = resolved.messages.count_tokens( + model=model, + system=system, + messages=[{"role": "user", "content": text}], + ) + else: + response = resolved.messages.count_tokens( + model=model, + messages=[{"role": "user", "content": text}], + ) + input_tokens = getattr(response, "input_tokens", None) + if not isinstance(input_tokens, int): + raise LLMResponseFormatError( + "Anthropic count_tokens response is missing the `input_tokens` field.", + ) + return input_tokens + # Register the Anthropic strategy at import time so ``provider_for("anthropic")`` # resolves it. The registry is a plugin point designed to grow — #136/#137 @@ -362,10 +485,238 @@ def classify_exception(self, exc: BaseException) -> ExceptionCategory: register_provider(AnthropicProvider()) +class OpenAIProvider(LLMProvider): + """OpenAI strategy behind the generic LLM orchestrator (#136 DEC-001/005/006/009/011). + + OpenAI has no Anthropic-style prompt-caching primitive and no server-side + pre-send ``count_tokens`` API; both capability flags are therefore + ``False`` (DEC-008 of #135). The orchestrator consequently: + + * emits no ``cache_control`` marker and no extended-cache beta header, + * reports ``cache_creation_input_tokens=0`` and ``cache_read_input_tokens=0``, + * skips the pre-send count-tokens gate entirely (the ``--estimate`` path + uses a local ``tiktoken`` count instead — US-005). + + The OpenAI SDK exposes ``client.chat.completions.create(...)`` rather than + ``client.messages.create(...)``; the orchestrator hard-calls + ``client.messages.create(**kwargs)``. The shim's + :class:`signalforge.llm._openai_client._OpenAIClientAdapter` wraps the + real OpenAI client so ``.messages.create`` delegates to the underlying + ``chat.completions.create`` (DEC-009). Every OpenAI-SDK type-checker + suppression lives in :mod:`signalforge.llm._openai_client` (DEC-010); + this provider reaches the SDK only through that shim's typed surfaces. + + ``build_create_kwargs`` attaches ``response_format={"type": "json_object"}`` + to enforce JSON output server-side (DEC-006). This is belt-and-braces with + the existing tolerant :func:`signalforge.llm.json_payload.extract_json_payload` + parser; server-side enforcement eliminates the prose-preamble drift class + (mirrors issue #144's fix for ``claude-sonnet-4-6``). The grade and drafter + system prompts both already name "JSON" so OpenAI's prompt-requirement check + passes. + + .. note:: + :meth:`build_count_tokens_kwargs` raises :class:`NotImplementedError` + (DEC-011 of #136). The orchestrator gates the pre-send count call on + :attr:`supports_token_count` and so never invokes this method — + mirrors the ``FakeNoCacheProvider`` precedent. + """ + + name = "openai" + supports_prompt_caching = False + supports_token_count = False + + def make_client(self) -> object: + """Construct the real OpenAI client (wrapped in the shim adapter).""" + from signalforge.llm._openai_client import _make_openai_client + + return _make_openai_client() + + def build_create_kwargs( + self, + *, + system: str, + cached_block: str, + dynamic_block: str, + model: str, + max_tokens: int, + cache_ttl: str, + cache_marker_active: bool, + ) -> dict[str, Any]: + """Build the kwargs for ``client.chat.completions.create``. + + Returns the OpenAI-native Chat Completions kwargs shape: ``model``, + ``max_tokens``, a two-message ``messages`` list (system + user where the + user content is the cached block followed by the dynamic block), and + ``response_format={"type": "json_object"}`` for server-side JSON + enforcement (DEC-006). + + ``cache_ttl`` and ``cache_marker_active`` are ignored: OpenAI has no + prompt-caching primitive, both capability flags are ``False`` (DEC-008 + of #135), and the orchestrator already resolves ``cache_marker_active`` + to ``False`` for a non-caching provider. There is no ``cache_control`` + marker and no ``extra_headers`` attached. + """ + return { + "model": model, + "max_tokens": max_tokens, + "messages": [ + {"role": "system", "content": system}, + {"role": "user", "content": cached_block + dynamic_block}, + ], + "response_format": {"type": "json_object"}, + } + + def build_count_tokens_kwargs( + self, + *, + system: str, + cached_block: str, + model: str, + ) -> dict[str, Any]: + """Never invoked — ``supports_token_count`` is ``False`` (DEC-011). + + The orchestrator skips the pre-send count gate entirely for a provider + that cannot count tokens server-side; the ``--estimate`` path uses + :func:`signalforge.llm._openai_client._count_openai_tokens` (a local + ``tiktoken`` count) instead. Raising here makes a future regression in + the capability-flag gate loud rather than silent. + """ + raise NotImplementedError( + "build_count_tokens_kwargs is unreachable when supports_token_count=False" + ) + + def extract_text_blocks(self, response: object) -> tuple[str, ...]: + """Extract the assistant text from an OpenAI Chat Completions response. + + OpenAI returns ``response.choices[0].message.content`` as a single + string (no per-block typing — unlike Anthropic's typed-block array). + Returns a single-element tuple so the orchestrator's downstream + ``"".join(blocks)`` is the same shape across providers. Raises + :class:`signalforge.llm.errors.LLMResponseFormatError` if the + structure is missing or the content is ``None``. + """ + from signalforge.llm.errors import LLMResponseFormatError + + choices = getattr(response, "choices", None) + if not choices: + raise LLMResponseFormatError( + "OpenAI response is missing the `choices` attribute or it is empty.", + ) + first = choices[0] + message = getattr(first, "message", None) + if message is None: + raise LLMResponseFormatError( + "OpenAI response choice is missing the `message` attribute.", + ) + content = getattr(message, "content", None) + if not isinstance(content, str): + raise LLMResponseFormatError( + "OpenAI response message `content` is missing or not a string.", + ) + return (content,) + + def extract_usage(self, response: object) -> UsageMetrics: + """Extract token economics from an OpenAI Chat Completions response. + + OpenAI reports ``usage.prompt_tokens`` and ``usage.completion_tokens`` + (no cache fields — OpenAI has no equivalent cache discount). Returns + :class:`UsageMetrics` with ``cache_creation_input_tokens=0`` and + ``cache_read_input_tokens=0``; matches ``supports_prompt_caching=False``. + """ + from signalforge.llm.client import _extract_usage_field + from signalforge.llm.errors import LLMResponseFormatError + + usage = getattr(response, "usage", None) + if usage is None: + raise LLMResponseFormatError( + "OpenAI response is missing the `usage` attribute.", + ) + return UsageMetrics( + input_tokens=_extract_usage_field(usage, "prompt_tokens"), + output_tokens=_extract_usage_field(usage, "completion_tokens"), + cache_creation_input_tokens=0, + cache_read_input_tokens=0, + ) + + def classify_exception(self, exc: BaseException) -> ExceptionCategory: + """Map a raised OpenAI SDK exception to a neutral category (DEC-009 of #136). + + Dispatch order mirrors :meth:`AnthropicProvider.classify_exception`: + auth (401 / 403) → rate-limit (429) → connection → API-status (5xx → + SERVER_ERROR; 4xx-non-auth → NO_RETRY; any other status → NO_RETRY). + Anything unrecognised maps to :attr:`ExceptionCategory.NO_RETRY` so the + orchestrator surfaces it without retrying. + + Reads the SDK exception classes through the shim's + :func:`signalforge.llm._openai_client._load_openai_exception_classes` + so the DEC-010 SDK-ignore confinement holds. + """ + from signalforge.llm._openai_client import _load_openai_exception_classes + from signalforge.llm.client import _is_4xx_non_auth, _is_5xx + + exc_classes = _load_openai_exception_classes() + if isinstance(exc, exc_classes.auth): + return ExceptionCategory.AUTH + if isinstance(exc, exc_classes.rate_limit): + return ExceptionCategory.RATE_LIMIT + if isinstance(exc, exc_classes.connection): + return ExceptionCategory.CONNECTION + if isinstance(exc, exc_classes.api_status): + if _is_5xx(exc): + return ExceptionCategory.SERVER_ERROR + if _is_4xx_non_auth(exc): + return ExceptionCategory.NO_RETRY + return ExceptionCategory.NO_RETRY + return ExceptionCategory.NO_RETRY + + def estimate_input_tokens( + self, + model: str, + text: str, + *, + system: str = "", + client: object | None = None, + ) -> int: + """Count tokens locally via ``tiktoken`` (DEC-003/DEC-012 of #136). + + OpenAI has no server-side ``count_tokens`` API (and the provider + declares ``supports_token_count=False`` so the runtime retry loop + skips its pre-send count gate entirely — DEC-008 of #135). The + ``--estimate`` calibration path counts tokens locally instead by + delegating to + :func:`signalforge.llm._openai_client._count_openai_tokens`, + which uses ``tiktoken.encoding_for_model(model)`` with a + ``cl100k_base`` fallback for unknown model ids. + + ``system`` is concatenated with ``text`` before counting — + ``tiktoken`` does not distinguish a "system" envelope from + regular tokens (unlike Anthropic's server-side counter), so + every token contributes to the same total. The combined count + matches what OpenAI's chat-completion endpoint will bill at + runtime (system prompt + user content). + + ``client`` is ignored — the count is a pure local BPE pass with + no SDK or network involvement. The kwarg is declared for + protocol parity with :meth:`AnthropicProvider.estimate_input_tokens` + so the orchestrator can call every provider the same way. + """ + from signalforge.llm._openai_client import _count_openai_tokens + + del client # tiktoken needs no SDK client + return _count_openai_tokens(model, system + text) + + +# Register the OpenAI strategy at import time so ``provider_for("openai")`` +# resolves it and both ``GradeConfig`` / ``DraftConfig`` validators accept +# ``provider="openai"`` (DEC-003 of #135; US-002 of #136). +register_provider(OpenAIProvider()) + + __all__ = ( "AnthropicProvider", "ExceptionCategory", "LLMProvider", + "OpenAIProvider", "UsageMetrics", "provider_for", "register_provider", diff --git a/tests/cli/test_e2e_estimate_openai.py b/tests/cli/test_e2e_estimate_openai.py new file mode 100644 index 00000000..fb4ce3e2 --- /dev/null +++ b/tests/cli/test_e2e_estimate_openai.py @@ -0,0 +1,197 @@ +"""End-to-end ``signalforge generate --estimate`` smoke against the real OpenAI API. + +Issue #136 / US-006 — DEC-001, DEC-004, DEC-005, DEC-008. Drives the +``--estimate`` short-circuit with ``llm.provider: openai`` + +``grade.provider: openai`` (both ``gpt-4o`` per DEC-004) and asserts the +rendered report includes a non-zero grader USD figure and the run exits +cleanly with no traceback leak. + +Gated by the ``openai`` marker — excluded from default CI by +:file:`pyproject.toml`'s ``addopts = "... -m 'not openai'"``. Requires +``SF_RUN_OPENAI=1`` + ``OPENAI_API_KEY``. The warehouse-bytes leg +(``estimate_query_bytes`` via BigQuery dry-run) gracefully degrades to +```` if Application Default Credentials / a billing +project aren't available locally — DEC-005 of #36 (the +multi-source-CLI degrade pattern, ``cli-layer.md``); the LLM-cost half +of the report still computes, exit code stays 0, and that's what this +test pins. To exercise the warehouse path too, set +``GOOGLE_CLOUD_PROJECT=`` + ``gcloud auth +application-default login`` first. + +What this proves end-to-end: + +* The OpenAI tiktoken token counter (DEC-012 of #136 / ``llm-drafter.md`` + § Open notes) plus the four ``gpt-4o`` / ``gpt-4o-mini`` / ``gpt-4.1`` / + ``gpt-4-turbo`` price-table entries (DEC-007 of #136) compose into a + non-zero grader USD figure. +* ``OpenAIProvider.estimate_input_tokens`` is wired correctly for both + the drafter and the grader (per DEC-005 — "scope both stages + explicitly"). +* The CLI's ``--estimate`` short-circuit handles the + ``supports_token_count=False`` capability flag without raising or + leaking a traceback (no Anthropic-only ``messages.count_tokens`` + call path). + +What this deliberately does NOT assert: + +* Specific token counts or USD figures — tiktoken-based estimates are + deterministic for a given input, but pinning exact values would + couple the test to the SDK / price-table version. Shape only. +""" + +from __future__ import annotations + +import os +from pathlib import Path + +import pytest + +from signalforge.cli import main +from tests.cli._e2e_helpers import copy_fixture_to_tmp + +pytestmark = pytest.mark.openai + +_FIXTURE_DIR = Path(__file__).parent.parent / "fixtures" / "dbt_project_austin" +_TRUTHY = frozenset({"1", "true", "yes", "on"}) + + +def _openai_runs_enabled() -> bool: + """``SF_RUN_OPENAI`` is set to a truthy value (mirrors ``SF_RUN_BQ`` / ``SF_RUN_SNOWFLAKE``).""" + return os.environ.get("SF_RUN_OPENAI", "").lower() in _TRUTHY + + +def _skip_reason() -> str | None: + """Return a skip-reason string if any required env var is missing. + + Returns ``None`` only when both gates are satisfied — the test then + proceeds to run ``signalforge generate --estimate`` against the real + OpenAI client. Each missing prerequisite yields its own distinct + reason so a maintainer running ``pytest -m openai`` sees exactly + what to set. Treat an empty / whitespace-only ``OPENAI_API_KEY`` as + "unset" (an empty value would otherwise reach the client and + produce a noisy auth failure rather than a skip). + """ + if not _openai_runs_enabled(): + return "SF_RUN_OPENAI=1 required (live --estimate test instantiates a real OpenAI client)" + if not os.environ.get("OPENAI_API_KEY", "").strip(): + return "OPENAI_API_KEY required (live --estimate test instantiates a real OpenAI client)" + return None + + +def test_generate_estimate_openai_provider_renders_nonzero_grader_usd( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + """Run ``signalforge generate --estimate`` end-to-end with the OpenAI provider. + + Copies the Austin bikeshare fixture into ``tmp_path`` (the audit + JSONLs / sidecars must land in temp — DEC-008 of #10) and rewrites + its ``signalforge.yml`` so the drafter AND grader both target + ``gpt-4o`` via the ``openai`` provider (DEC-005 of #136 — scope + both stages). The warehouse-bytes path is allowed to degrade to + ```` if no BQ creds are present; the LLM-cost + half still computes, which is what this test pins. + + Asserts: + + 1. Exit code 0 — the four-tier exit-code taxonomy (cli-layer.md + DEC-008) reserves 0 for clean runs; ``--estimate`` is no-API-call + on the LLM side (count-tokens / tiktoken only) and the + warehouse degrade is fail-soft. + 2. ``Estimated grade cost:`` header appears in stdout (the OpenAI + grader section computed without raising). + 3. At least one non-zero USD figure under the grader block — pins + the DEC-004 / DEC-007 wiring (gpt-4o pricing entry × tiktoken + counts × four-criterion fan-out). + 4. ``Traceback`` does NOT appear in stderr (cli-layer.md DEC-016 — + no traceback ever leaks). + """ + if reason := _skip_reason(): + pytest.skip(reason) + + project_dir = copy_fixture_to_tmp(_FIXTURE_DIR, tmp_path) + + # Override signalforge.yml so BOTH drafter and grader target OpenAI + # (DEC-005 — scope both stages explicitly). gpt-4o is the DEC-004 + # default judge model and is shipped in the price table (DEC-007). + # We retain the fixture's existing grade thresholds + safety mode + # so the rest of the pipeline (which --estimate doesn't actually + # exercise) stays consistent with the fixture's documented shape. + (project_dir / "signalforge.yml").write_text( + "llm:\n" + " provider: openai\n" + " model: gpt-4o\n" + "safety:\n" + " mode: aggregate-only\n" + "prune:\n" + " sample_strategy: materialised\n" + "grade:\n" + " provider: openai\n" + " model: gpt-4o\n" + " min_pass_rate: 0.95\n" + " min_mean_score: 0.95\n" + " fail_on_below_threshold: false\n" + " total_budget_seconds: 600\n" + ) + + # Bill the maintainer's GCP project if available — lets the + # warehouse-bytes leg succeed instead of degrading; absent, it + # degrades to ```` per DEC-005 of #36. Either + # path is acceptable for this test. + billing_project = os.environ.get("GOOGLE_CLOUD_PROJECT") + if billing_project: + (project_dir / "profiles.yml").write_text( + "austin:\n" + " target: dev\n" + " outputs:\n" + " dev:\n" + " type: bigquery\n" + " method: oauth\n" + f" project: {billing_project}\n" + " dataset: austin_bikeshare\n" + " location: US\n" + " maximum_bytes_billed: 1000000000\n" + ) + + exit_code = main( + [ + "generate", + "--estimate", + "models/staging/stg_bikeshare_trips.sql", + "--project-dir", + str(project_dir), + ] + ) + + captured = capsys.readouterr() + + # 1. Exit code 0 — clean run. + assert exit_code == 0, ( + f"expected --estimate to exit 0; got {exit_code}. stderr={captured.err!r}" + ) + + # 2. Grader section rendered. + assert "Estimated grade cost:" in captured.out, ( + f"expected the OpenAI grader cost section in stdout; got:\n{captured.out}" + ) + + # 3. At least one non-zero grader USD figure — pins the + # tiktoken × gpt-4o price-table wiring (DEC-004 / DEC-007). + # Captured under the grader block: lines like + # `` cost: $0.1152`` or per-criterion ``$0.0288``. + grade_section = captured.out.split("Estimated grade cost:", 1)[1] + # Truncate at the next blank-line-separated section so we don't + # match a draft-cost figure that happens to also be non-zero. + grade_section = grade_section.split("\n\n", 1)[0] + nonzero_usd_lines = [ + line + for line in grade_section.splitlines() + if "$" in line and not line.strip().endswith("$0.0000") + ] + assert nonzero_usd_lines, ( + f"expected a non-zero grader USD figure in the grade section; got:\n{grade_section}" + ) + + # 4. No traceback ever leaks (cli-layer.md DEC-016). + assert "Traceback" not in captured.err, ( + f"stderr leaked a Python traceback (DEC-016 violation):\n{captured.err}" + ) diff --git a/tests/cli/test_estimate.py b/tests/cli/test_estimate.py new file mode 100644 index 00000000..f4e1ab4d --- /dev/null +++ b/tests/cli/test_estimate.py @@ -0,0 +1,345 @@ +"""Provider-neutral ``--estimate`` tests (US-005 of issue #136). + +Two load-bearing invariants this file pins (DEC-003, DEC-007, DEC-012, +DEC-013): + +1. **Anthropic byte-identity (DEC-013).** The pre-refactor inline + ``client.messages.count_tokens(...)`` calls in + :mod:`signalforge.cli._estimate` were generalised in US-005 to + dispatch through + :meth:`signalforge.llm.providers.LLMProvider.estimate_input_tokens`. + The Anthropic implementation must continue to produce the same + rendered estimate stdout for the same canned token counts — the + golden ``tests/fixtures/estimate/anthropic_byte_identity_golden.txt`` + pins the bytes. A refactor that changes the rendered shape (or the + USD math) for the Anthropic path breaks this test loudly. + +2. **OpenAI ``--estimate`` works end-to-end.** With + ``draft.provider: openai`` + ``grade.provider: openai`` and an + ``OpenAIProvider`` registered, the engine produces an + :class:`signalforge.cli._estimate.EstimateReport` with non-zero + token counts and non-zero USD figures. The token count is the + local ``tiktoken`` figure (no API call), and the pricing math + uses the OpenAI SKU table added in #136 US-004. + +The two tests are deliberately co-located so a future refactor that +breaks either path surfaces both regressions in one file. +""" + +from __future__ import annotations + +from pathlib import Path +from unittest.mock import patch + +from signalforge.cli._estimate import estimate, render +from signalforge.draft.config import DraftConfig +from signalforge.grade.config import GradeConfig +from signalforge.grade.rubric import DEFAULT_RUBRIC +from signalforge.prune.config import PruneConfig +from signalforge.warehouse import BigQueryAdapter +from tests.cli.test_estimate_engine import _make_manifest, _make_model +from tests.llm._fake import FakeAnthropicClient, FakeCountTokensResponse +from tests.warehouse._fake import FakeBigQueryClient + +_FIXTURES = Path(__file__).resolve().parents[1] / "fixtures" / "estimate" +_GOLDEN_PATH = _FIXTURES / "anthropic_byte_identity_golden.txt" + + +# --------------------------------------------------------------------------- +# DEC-013 — Anthropic byte-identity floor +# --------------------------------------------------------------------------- + + +def test_estimate_anthropic_byte_identity_golden() -> None: + """The Anthropic ``--estimate`` rendered output is byte-identical to + the golden captured BEFORE the US-005 strategy-dispatch refactor. + + Inputs: + * default :class:`DraftConfig` / :class:`GradeConfig` / + :class:`PruneConfig` (Anthropic provider, ``claude-sonnet-4-6``, + default rubric of 4 criteria). + * Two-column ``model.shop.customers`` model (``id`` + ``email``). + * Canned token counts: 1000 for the drafter; 500 for each of the + 4 grade criteria (queued on the fake client). + * Canned warehouse dry-run bytes: 10,000. + + Captured 2026-05-28 from the pre-refactor engine; reproduced + verbatim by the post-refactor strategy-dispatch path. A drift in + the rendered shape (column widths, label wording, decimal + precision) OR the USD math (price table, output-token estimate, + artifact-count formula) breaks this test. The fix is to align the + refactor with the golden, NOT to regenerate the golden — that is + the entire point of DEC-013. + + If the price table is intentionally refreshed (a deliberate + decision committed alongside a ``PRICE_TABLE_VERSION`` bump per + ``python-build.md`` § "5-surface parity"), regenerate the golden + in the same commit so the relationship between "what the operator + sees" and "what the provider charges" remains explicit. + """ + model = _make_model() + mf = _make_manifest(model) + draft_config = DraftConfig() # provider="anthropic" default + grade_config = GradeConfig() # provider="anthropic" default + prune_config = PruneConfig() + fc = FakeBigQueryClient(project="fake_project") + adapter = BigQueryAdapter( + project="fake_project", + location="US", + max_bytes_billed=100_000_000, + client=fc, + ) + fa = FakeAnthropicClient(project="fake_project") + fa.expect_count_tokens( + matching=lambda kw: True, + returns=FakeCountTokensResponse(input_tokens=1000), + ) + for _ in range(len(DEFAULT_RUBRIC)): + fa.expect_count_tokens( + matching=lambda kw: True, + returns=FakeCountTokensResponse(input_tokens=500), + ) + fc.expect_dry_run(sql_matching=r"SELECT", returns_bytes=10_000) + + report = estimate( + model, + mf, + draft_config, + grade_config, + prune_config, + adapter, + fa, + ) + rendered = render(report) + + golden = _GOLDEN_PATH.read_text(encoding="utf-8") + assert rendered == golden, ( + "Anthropic --estimate rendered output drifted from the captured " + f"golden. Diff against {_GOLDEN_PATH}. DEC-013 of #136 mandates " + "byte-identity across the US-005 strategy refactor; align the " + "implementation to the golden rather than regenerating the golden, " + "unless this commit also intentionally bumps PRICE_TABLE_VERSION." + ) + + +# --------------------------------------------------------------------------- +# OpenAI provider-aware estimate (DEC-003 / DEC-007 / DEC-012) +# --------------------------------------------------------------------------- + + +def test_estimate_openai_provider_produces_nonzero_tokens_and_usd() -> None: + """``--estimate`` with ``draft.provider: openai`` + ``grade.provider: + openai`` produces a report with non-zero token counts AND non-zero + USD figures. + + OpenAI counts tokens locally via ``tiktoken`` (DEC-012); the + engine does NOT consult ``anthropic_client`` for the OpenAI path + (``client=None`` is the canonical CLI shape after the US-005 + refactor of :mod:`signalforge.cli.generate`). The Anthropic test + fake is passed in only to satisfy the engine's positional + parameter; the fake's ``count_tokens`` is NOT consumed — pinned by + ``len(fake.count_calls) == 0`` at the end. + + Pricing math uses the ``gpt-4o`` row from + :data:`signalforge.llm.pricing.PRICES` (#136 US-004 / DEC-007); + even with tiny token counts the per-MTok rates produce strictly + positive USD figures. + """ + model = _make_model() + mf = _make_manifest(model) + draft_config = DraftConfig.model_validate( + {**DraftConfig().model_dump(), "provider": "openai", "model": "gpt-4o"} + ) + grade_config = GradeConfig.model_validate( + {**GradeConfig().model_dump(), "provider": "openai", "model": "gpt-4o"} + ) + prune_config = PruneConfig() + fc = FakeBigQueryClient(project="fake_project") + adapter = BigQueryAdapter( + project="fake_project", + location="US", + max_bytes_billed=100_000_000, + client=fc, + ) + fc.expect_dry_run(sql_matching=r"SELECT", returns_bytes=10_000) + + # OpenAI counts locally; no Anthropic client is needed. The CLI + # passes ``None`` for the OpenAI path (see :mod:`signalforge.cli.generate`), + # which the engine forwards verbatim through the strategy. + report = estimate( + model, + mf, + draft_config, + grade_config, + prune_config, + adapter, + None, + ) + + # Token counts are strictly positive (the rendered drafter prompt + # is non-trivial; tiktoken returns a non-zero count for any + # non-empty text). Engineered determinism: ``len(tiktoken.encode(x)) + # > 0`` for any non-empty ``x``. + assert report.draft_input_tokens > 0 + assert all(c.total_input_tokens > 0 for c in report.grade_per_criterion) + + # USD math: the OpenAI ``gpt-4o`` row charges $2.50/MTok input and + # $10/MTok output (#136 US-004 DEC-007); any positive token count + # therefore produces a strictly positive USD figure. + assert report.draft_usd > 0 + assert report.grade_usd > 0 + assert report.total_llm_usd > 0 + + # The rendered output names the OpenAI models in the prelude and + # carries the same totals + warehouse sections. + rendered = render(report) + assert "drafter: gpt-4o" in rendered + assert "grader: gpt-4o" in rendered + assert "Total estimated LLM cost: $" in rendered + # The renderer's USD prefix on the totals line includes the + # computed value; assert the literal "$0.0000" placeholder is NOT + # the rendered total (i.e. the figure rounded to 4 decimals is + # non-zero). + assert "Total estimated LLM cost: $0.0000" not in rendered + + +def test_estimate_openai_provider_ignores_threaded_client() -> None: + """The OpenAI ``estimate_input_tokens`` impl ignores the ``client`` + kwarg — it counts via local ``tiktoken``. + + Drives the engine with a :class:`FakeAnthropicClient` AS the + ``anthropic_client`` positional (a deliberately wrong-shape + client for OpenAI) and asserts: + * the engine completes successfully, + * ``len(fake.count_calls) == 0`` — the Anthropic surface is + never touched on the OpenAI path. + + This is the load-bearing proof that the strategy dispatch in + ``_count_draft_tokens`` / ``_count_grade_criterion_tokens`` routes + by ``config.provider``, NOT by inspecting the client type. A + regression that fell through to the old hard-coded + ``client.messages.count_tokens(...)`` call would consume a queued + expectation from the fake (and fail loudly when no expectation is + queued). + """ + model = _make_model() + mf = _make_manifest(model) + draft_config = DraftConfig.model_validate( + {**DraftConfig().model_dump(), "provider": "openai", "model": "gpt-4o"} + ) + grade_config = GradeConfig.model_validate( + {**GradeConfig().model_dump(), "provider": "openai", "model": "gpt-4o"} + ) + prune_config = PruneConfig() + fc = FakeBigQueryClient(project="fake_project") + adapter = BigQueryAdapter( + project="fake_project", + location="US", + max_bytes_billed=100_000_000, + client=fc, + ) + fc.expect_dry_run(sql_matching=r"SELECT", returns_bytes=2048) + + # No ``expect_count_tokens`` queued on the Anthropic fake; a + # regression that called through would raise loudly inside + # ``FakeAnthropicClient.messages.count_tokens``. + fa = FakeAnthropicClient(project="fake_project") + + report = estimate( + model, + mf, + draft_config, + grade_config, + prune_config, + adapter, + fa, + ) + + assert len(fa.count_calls) == 0 + assert report.draft_input_tokens > 0 + assert report.total_llm_usd > 0 + + +def test_estimate_openai_uses_count_openai_tokens_for_local_count() -> None: + """The OpenAI estimate path delegates to + :func:`signalforge.llm._openai_client._count_openai_tokens` (and + therefore ``tiktoken``) — not to any Anthropic SDK surface. + + Patches the underlying ``_count_openai_tokens`` helper and asserts + every dispatch reaches it. The patch returns a sentinel positive + value so the engine's pricing math still produces non-zero USD + figures (degenerate inputs would short-circuit the test rather + than exercising the call surface). + + Without this, a refactor that re-pointed ``OpenAIProvider.estimate_input_tokens`` + at a stub (or stubbed it out for "v0.2") would pass the + nonzero-USD test above silently — every count still comes from + somewhere, just not necessarily from tiktoken. + """ + model = _make_model() + mf = _make_manifest(model) + draft_config = DraftConfig.model_validate( + {**DraftConfig().model_dump(), "provider": "openai", "model": "gpt-4o"} + ) + grade_config = GradeConfig.model_validate( + {**GradeConfig().model_dump(), "provider": "openai", "model": "gpt-4o"} + ) + prune_config = PruneConfig() + fc = FakeBigQueryClient(project="fake_project") + adapter = BigQueryAdapter( + project="fake_project", + location="US", + max_bytes_billed=100_000_000, + client=fc, + ) + fc.expect_dry_run(sql_matching=r"SELECT", returns_bytes=2048) + + sentinel = 4242 + with patch( + "signalforge.llm._openai_client._count_openai_tokens", + return_value=sentinel, + ) as mocked: + report = estimate( + model, + mf, + draft_config, + grade_config, + prune_config, + adapter, + None, + ) + + # 1 drafter call + N per-criterion calls, all routed through + # tiktoken. + n_criteria = len(DEFAULT_RUBRIC) + assert mocked.call_count == 1 + n_criteria + # The drafter token count IS the sentinel (one call, one value). + assert report.draft_input_tokens == sentinel + # Per-criterion input tokens are sentinel * artifact_count; the + # multiplication itself is engine logic, but the underlying + # ``input_tokens_per_call`` is exactly the sentinel. + for crit in report.grade_per_criterion: + assert crit.input_tokens_per_call == sentinel + # And every mocked call was issued for ``gpt-4o``. + for call in mocked.call_args_list: + assert call.args[0] == "gpt-4o" + + +def test_fakenocache_provider_estimate_input_tokens_returns_word_count() -> None: + """:class:`tests.llm._fake_provider.FakeNoCacheProvider` answers + ``estimate_input_tokens`` with ``len(text.split())``. + + The neutrality test in ``tests/grade/test_provider_neutrality.py`` + instantiates the fake provider via :class:`LLMProvider`'s ABC, so + the impl must exist (an abstract method without an override would + raise :class:`TypeError` at instantiation time). This test pins + the trivial answer shape so the neutrality test's "shape only" + assertions stay deterministic. + """ + from tests.llm._fake_provider import FakeNoCacheProvider + + provider = FakeNoCacheProvider() + assert provider.estimate_input_tokens("any-model", "one two three four") == 4 + assert provider.estimate_input_tokens("any-model", "") == 0 + # The ``client`` kwarg is accepted but ignored. + assert provider.estimate_input_tokens("any-model", "hello world", client=object()) == 2 diff --git a/tests/cli/test_estimate_engine.py b/tests/cli/test_estimate_engine.py index bf0611a6..18d7f258 100644 --- a/tests/cli/test_estimate_engine.py +++ b/tests/cli/test_estimate_engine.py @@ -726,13 +726,16 @@ def test_estimate_uses_first_artifact_alphabetical_as_grade_rep( grade_calls = fake_anthropic.count_calls[1:] assert len(grade_calls) == n_criteria for call in grade_calls: - # The dynamic block carries ``artifact_id: column.aaa_first.description`` - # in its body. Walk the messages content block to find it. + # #136 US-005: ``AnthropicProvider.estimate_input_tokens`` sends + # a single user-message with a string ``content`` payload (the + # concatenated ``system + cached + dynamic`` text); the previous + # two-block envelope is gone. The representative artifact_id + # still appears verbatim in the rendered dynamic block. messages = call["messages"] assert len(messages) == 1 - blocks = messages[0]["content"] - dynamic_block_text = blocks[1]["text"] - assert "artifact_id: column.aaa_first.description" in dynamic_block_text + content = messages[0]["content"] + assert isinstance(content, str) + assert "artifact_id: column.aaa_first.description" in content def test_estimate_returns_frozen_estimate_report( diff --git a/tests/cli/test_generate_estimate.py b/tests/cli/test_generate_estimate.py index 2ca75fea..cf2487a7 100644 --- a/tests/cli/test_generate_estimate.py +++ b/tests/cli/test_generate_estimate.py @@ -24,6 +24,7 @@ from __future__ import annotations from pathlib import Path +from typing import Any from unittest.mock import MagicMock import pytest @@ -469,9 +470,13 @@ def test_from_profile_dispatches_snowflake_to_snowflake_adapter() -> None: # --------------------------------------------------------------------------- -# Provider guard (#135 closeout) — --estimate is Anthropic-shaped (it casts to -# AnthropicClientProtocol + calls count_tokens), so it fails fast rather than -# project grade cost through a divergent / non-Anthropic provider's client. +# Provider guard (#135 closeout, narrowed by #136 US-005) — --estimate +# now dispatches its count_tokens calls through +# LLMProvider.estimate_input_tokens, so a non-Anthropic provider's +# --estimate path is supported (OpenAI counts locally via tiktoken). The +# remaining guard is the divergent-providers check: the engine takes one +# optional client object, so the two stages MUST use the same provider +# or we'd silently project grade-stage cost through the drafter's vendor. # --------------------------------------------------------------------------- @@ -505,32 +510,90 @@ def test_generate_estimate_divergent_providers_fails_fast( providers_mod._REGISTRY.update(saved) -def test_generate_estimate_non_anthropic_provider_fails_fast( +def test_generate_estimate_openai_provider_passes_client_none_to_engine( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], ) -> None: - """Both stages on a non-Anthropic provider → CliInputError (exit 2).""" - from signalforge.llm import providers as providers_mod - from tests.llm._fake_provider import FakeNoCacheProvider + """When ``llm.provider == "openai"`` (and grade matches), the + ``--estimate`` short-circuit MUST take the non-Anthropic branch in + ``cmd_generate``: skip building an Anthropic SDK client and pass + ``client=None`` into ``estimate_module.estimate(...)``. OpenAI's + ``estimate_input_tokens`` counts locally via ``tiktoken`` and + needs no SDK client (DEC-012 of #136). + + Closes PR #152 codecov gap on ``generate.py``'s + ``else: client = None`` arm — the existing + ``test_generate_estimate_*`` suite all uses the default Anthropic + provider, so the openai branch ships uncovered at the CLI level + until this test. + + The ``@pytest.mark.openai`` live smoke at + ``tests/cli/test_e2e_estimate_openai.py`` exercises the same path + against the real API, but it's excluded from default CI; this is + the offline-driven coverage equivalent. + """ + from signalforge.cli import generate as gen_mod + from signalforge.llm.providers import AnthropicProvider - saved = dict(providers_mod._REGISTRY) - providers_mod.register_provider(FakeNoCacheProvider()) - try: - project_dir = make_fake_dbt_project(tmp_path) - (project_dir / "signalforge.yml").write_text( - "llm:\n provider: fake-nocache\ngrade:\n provider: fake-nocache\n", - encoding="utf-8", - ) - monkeypatch.chdir(project_dir) - _install_estimate_patches(monkeypatch) + project_dir = make_fake_dbt_project(tmp_path) + # Both stages on openai so the divergent-providers gate doesn't + # fire; cli/_estimate.py then dispatches token counting through + # OpenAIProvider, which delegates to tiktoken (no SDK call). + (project_dir / "signalforge.yml").write_text( + "llm:\n provider: openai\n model: gpt-4o\ngrade:\n provider: openai\n model: gpt-4o\n", + encoding="utf-8", + ) + monkeypatch.chdir(project_dir) - code = main(["generate", "--estimate", "model.shop.customers"]) - captured = capsys.readouterr() + # Standard manifest + warehouse fakes from _install_estimate_patches, + # but the AnthropicProvider.make_client patch it installs MUST NOT + # be invoked on the openai path — pin that via a spy. + fa, fb = _install_estimate_patches(monkeypatch) + spy_calls: list[None] = [] + real_anthropic_make = AnthropicProvider.make_client + + def _spy(self: AnthropicProvider) -> object: + spy_calls.append(None) + return real_anthropic_make(self) + + monkeypatch.setattr(AnthropicProvider, "make_client", _spy) + + # Capture the kwargs handed to ``estimate(...)`` so we can pin that + # client=None (the load-bearing assertion for line 846). + captured: dict[str, object] = {} + real_estimate = gen_mod.estimate_module.estimate + + def _capturing_estimate(*args: Any, **kwargs: Any) -> Any: + # ``estimate(...)`` is positional in cmd_generate: (model, manifest, + # draft_config, grade_config, prune_config, adapter, client, *, + # project_dir). The 7th positional is the client. Args/kwargs are + # typed ``Any`` so pyright sees the ``*args`` forward as + # type-compatible with the real ``estimate(...)`` signature; this + # is a test-side spy, not a re-exported function. + captured["client_arg"] = args[6] if len(args) > 6 else kwargs.get("client") + return real_estimate(*args, **kwargs) + + monkeypatch.setattr(gen_mod.estimate_module, "estimate", _capturing_estimate) + + # Warehouse dry-run still runs for the bytes leg; the LLM-side is + # tiktoken-local so no fake_anthropic expectations are queued. + fb.expect_dry_run(sql_matching=r"SELECT", returns_bytes=10_000) - assert code == 2, f"stderr={captured.err}" - assert "only provider='anthropic'" in captured.err - assert "Traceback" not in captured.err - finally: - providers_mod._REGISTRY.clear() - providers_mod._REGISTRY.update(saved) + code = main(["generate", "--estimate", "model.shop.customers"]) + captured_out = capsys.readouterr() + + assert code == 0, f"stderr={captured_out.err}" + assert "Traceback" not in captured_out.err + assert spy_calls == [], ( + "AnthropicProvider.make_client MUST NOT be invoked on the openai " + f"path; got {len(spy_calls)} calls" + ) + assert captured["client_arg"] is None, ( + "cmd_generate MUST pass client=None to estimate(...) for non-" + f"Anthropic providers; got {captured['client_arg']!r}" + ) + # FakeAnthropicClient was never touched (no count_tokens expectations + # were queued, and the OpenAI path doesn't go through it). + assert len(fa.count_calls) == 0 + assert len(fa.create_calls) == 0 diff --git a/tests/draft/test_config.py b/tests/draft/test_config.py index 7cedab93..444fffe8 100644 --- a/tests/draft/test_config.py +++ b/tests/draft/test_config.py @@ -95,6 +95,15 @@ def test_draft_config_provider_accepts_registered_name() -> None: assert cfg.provider == "anthropic" +def test_draft_config_provider_accepts_openai() -> None: + """US-002 of #136: after ``OpenAIProvider`` is registered at import time, + ``DraftConfig(provider="openai", model="gpt-4o")`` validates without + error (DEC-005 of #136 — both stages accept ``provider: openai``).""" + cfg = DraftConfig(provider="openai", model="gpt-4o") + assert cfg.provider == "openai" + assert cfg.model == "gpt-4o" + + def test_draft_config_provider_rejects_unknown_with_available_keys() -> None: """DEC-007: an unknown provider fails loud with a typed :class:`UnknownProviderError` that names the registered providers. diff --git a/tests/draft/test_schema.py b/tests/draft/test_schema.py index acfce1c8..7523e4f8 100644 --- a/tests/draft/test_schema.py +++ b/tests/draft/test_schema.py @@ -503,6 +503,7 @@ def test_public_api_imports_match_dec_020() -> None: "LLMResult", "LLMServerError", "ModelPricing", + "OpenAIProvider", "UnknownProviderError", "UsageMetrics", "call_llm", diff --git a/tests/draft/test_smoke_real_api_openai.py b/tests/draft/test_smoke_real_api_openai.py new file mode 100644 index 00000000..841cf97f --- /dev/null +++ b/tests/draft/test_smoke_real_api_openai.py @@ -0,0 +1,170 @@ +"""Real-API smoke test for :func:`signalforge.draft.draft_schema` via OpenAI. + +Issue #136 / US-006 — DEC-001, DEC-005, DEC-008. Honours DEC-005's +"scope both grade AND draft explicitly" commitment that +``tests/grade/test_smoke_real_api_openai.py`` alone wouldn't cover at the +live level. Gated by the ``openai`` marker — excluded from default CI by +:file:`pyproject.toml`'s ``addopts = "... -m 'not openai'"``. Requires +``SF_RUN_OPENAI=1`` + ``OPENAI_API_KEY``. + +What this proves end-to-end: + +* The OpenAIProvider's Chat Completions adapter (DEC-001 / DEC-009) + reaches the real ``gpt-4o`` and returns JSON parseable as + :class:`CandidateSchema` under the drafter system prompt (whose + ``response_format={"type": "json_object"}`` requirement, per + ``llm-drafter.md`` § Open notes for implementation, must include the + word "json" — verified at the LLM-seam side). +* The tolerant JSON extractor (`extract_json_payload`, llm-drafter.md + issue #144) handles whatever prose preamble the OpenAI judge model + emits before the JSON object. +* The response-audit JSONL is written under + ``policy.audit_path.with_name("llm_responses.jsonl")`` and round-trips + through :class:`LLMResponseEvent`. +* OpenAI's ``cache_*_input_tokens`` audit fields are ``0`` because + ``OpenAIProvider.supports_prompt_caching`` is ``False`` (#135 / + #136 capability gate). The seam does NOT emit the dual-zero + cache-anomaly WARNING. + +What this deliberately does NOT assert: + +* Specific column descriptions / rationale wording — LLM output is not + deterministic enough; the test would be flaky if pinned. +""" + +from __future__ import annotations + +import logging +import os +from pathlib import Path + +import pytest + +from signalforge.draft import draft_schema +from signalforge.draft.audit import LLMResponseEvent +from signalforge.draft.config import DraftConfig +from signalforge.draft.models import CandidateSchema +from signalforge.draft.schema import DraftOutcome +from signalforge.manifest.models import Column, Manifest, Model +from signalforge.safety.policy import SafetyPolicy +from tests.safety._fake_adapter import FakeAdapter + +pytestmark = pytest.mark.openai + +_TRUTHY = frozenset({"1", "true", "yes", "on"}) + + +def _openai_runs_enabled() -> bool: + """``SF_RUN_OPENAI`` is set to a truthy value (mirrors ``SF_RUN_BQ`` / ``SF_RUN_SNOWFLAKE``).""" + return os.environ.get("SF_RUN_OPENAI", "").lower() in _TRUTHY + + +def _skip_reason() -> str | None: + """Return a skip-reason string if any required env var is missing. + + Returns ``None`` only when both gates are satisfied — the test then + proceeds to make a real OpenAI call. Each missing prerequisite + yields its own distinct reason so a maintainer running + ``pytest -m openai`` sees exactly what to set. Treat an empty / + whitespace-only ``OPENAI_API_KEY`` as "unset" (an empty value would + otherwise reach the API and produce a noisy auth failure). + """ + if not _openai_runs_enabled(): + return "SF_RUN_OPENAI=1 required (live test calls the real OpenAI API)" + if not os.environ.get("OPENAI_API_KEY", "").strip(): + return "OPENAI_API_KEY required (live test authenticates against the real OpenAI API)" + return None + + +def test_draft_schema_real_openai_api_smoke( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """One real OpenAI draft round-trip against a tiny in-test manifest. + + Builds a minimal :class:`Model` / :class:`Manifest`, runs the safety + layer in schema-only mode (no adapter calls — see + ``safety-layer.md`` DEC-012(c) / safety/request.py) and issues a + single ``draft_schema`` call against ``gpt-4o``. Asserts the typed + :class:`CandidateSchema` validates and the per-call + :class:`LLMResponseEvent` JSONL row carries zero cache tokens (the + #136 capability-gate contract). + """ + if reason := _skip_reason(): + pytest.skip(reason) + + # Minimal in-test fixture — one column with an obvious data type so the + # LLM has something to draft against. Schema-only mode means the + # adapter is never invoked, so a do-nothing ``FakeAdapter()`` is fine. + model = Model( + unique_id="model.sf_smoke.dim_users", + name="dim_users", + resource_type="model", + package_name="sf_smoke", + original_file_path="models/marts/dim_users.sql", + path="marts/dim_users.sql", + database="sf-smoke-proj", + schema="main", # type: ignore[call-arg] + columns={ + "user_id": Column(name="user_id", data_type="INT64"), + "email": Column(name="email", data_type="STRING"), + }, + raw_code="select 1 as user_id, 'a@b.com' as email", + ) + manifest = Manifest( + metadata={"dbt_schema_version": "v12"}, + nodes={model.unique_id: model}, + ) + + # Schema-only policy with a per-run audit path under tmp_path. The + # safety layer writes ``.../audit.jsonl``; the draft layer derives + # ``.../llm_responses.jsonl`` (DEC-006) next to it. + audit_path = tmp_path / "audit.jsonl" + policy = SafetyPolicy(audit_path=audit_path) + adapter = FakeAdapter() + + # OpenAI drafter: gpt-4o (the DEC-004 default for live smokes). The + # provider field on DraftConfig is the #135 plug-in seam. + config = DraftConfig(provider="openai", model="gpt-4o") + + with caplog.at_level(logging.WARNING): + outcome = draft_schema(model, adapter, policy, manifest, config=config) + + # Shape-only assertions on the typed outcome. + assert isinstance(outcome, DraftOutcome) + assert isinstance(outcome.candidate, CandidateSchema) + # The drafter is anchored to the model's columns — every drafted + # column must be a real one. The parser already enforces this via + # the anchor contract (``llm-drafter.md`` DEC-003); the assertion + # here documents the contract for the smoke-test reader. + drafted_names = {c.name for c in outcome.candidate.columns} + assert drafted_names, "expected the drafter to emit at least one column" + assert drafted_names.issubset({"user_id", "email"}) + + # Response-audit JSONL exists next to the safety audit (DEC-006). + response_audit = audit_path.with_name("llm_responses.jsonl") + assert response_audit.exists() + audit_lines = response_audit.read_text(encoding="utf-8").strip().splitlines() + assert len(audit_lines) >= 1 + # Last line is this run's record — round-trip it through the typed + # model and assert the OpenAI capability-gate contract. + event = LLMResponseEvent.model_validate_json(audit_lines[-1]) + assert event.model == "gpt-4o" + assert event.model_unique_id == model.unique_id + # OpenAI has no equivalent of Anthropic's prompt-cache discount — + # ``OpenAIProvider.supports_prompt_caching`` is ``False`` so both + # cache-token fields must land at zero (#136 DEC-008 capability-gate). + assert event.cache_creation_input_tokens == 0 + assert event.cache_read_input_tokens == 0 + + # The dual-zero cache-anomaly WARNING (``llm-drafter.md`` DEC-014) + # is capability-gated off for providers with + # ``supports_prompt_caching=False`` and must NEVER fire here. + cache_warning_messages = [ + record.getMessage() + for record in caplog.records + if record.levelno >= logging.WARNING and "cache marker no-op" in record.getMessage() + ] + assert not cache_warning_messages, ( + f"unexpected cache-anomaly WARNING(s) on OpenAI path " + f"(supports_prompt_caching=False should gate this off): {cache_warning_messages}" + ) diff --git a/tests/fixtures/estimate/anthropic_byte_identity_golden.txt b/tests/fixtures/estimate/anthropic_byte_identity_golden.txt new file mode 100644 index 00000000..e3fb969b --- /dev/null +++ b/tests/fixtures/estimate/anthropic_byte_identity_golden.txt @@ -0,0 +1,28 @@ +Estimate for model.shop.customers + drafter: claude-sonnet-4-6 + grader: claude-sonnet-4-6 + +Estimated draft cost: + input tokens: 1,000 + output tokens: ~4,096 (estimated) + cost: $0.0644 + +Estimated grade cost: + artifacts: 13 criteria: 4 calls: 52 + per criterion: + clarity 13 calls 6,500 tokens $0.0292 + consistency 13 calls 6,500 tokens $0.0292 + rationale 13 calls 6,500 tokens $0.0292 + no-redundant 13 calls 6,500 tokens $0.0292 + cost: $0.1170 + +Estimated warehouse cost: + bytes-per-row: ~1 (BigQuery dryRun) + test count est: 7 (3.5 tests/col x 2 cols) + sample size: 100,000 rows + total bytes: ~68.4 KB + +Total estimated LLM cost: $0.1814 +Total estimated warehouse: ~68.4 KB + +Price table: 2026-05-28 | Heuristic: ~3.5 tests/column (canonical fixture average) diff --git a/tests/grade/test_config.py b/tests/grade/test_config.py index dd46a99c..01173834 100644 --- a/tests/grade/test_config.py +++ b/tests/grade/test_config.py @@ -214,6 +214,15 @@ def test_grade_config_provider_accepts_registered_name() -> None: assert GradeConfig(provider="anthropic").provider == "anthropic" +def test_grade_config_provider_accepts_openai() -> None: + """US-002 of #136: after ``OpenAIProvider`` is registered at import time, + ``GradeConfig(provider="openai", model="gpt-4o")`` validates without + error (DEC-005 of #136 — both stages accept ``provider: openai``).""" + cfg = GradeConfig(provider="openai", model="gpt-4o") + assert cfg.provider == "openai" + assert cfg.model == "gpt-4o" + + def test_grade_config_provider_rejects_unknown_with_available_keys() -> None: """DEC-007: an unknown provider fails loud with a typed :class:`UnknownProviderError` naming the registered providers. diff --git a/tests/grade/test_provider_neutrality_openai.py b/tests/grade/test_provider_neutrality_openai.py new file mode 100644 index 00000000..f12dcbbc --- /dev/null +++ b/tests/grade/test_provider_neutrality_openai.py @@ -0,0 +1,310 @@ +"""OpenAI provider-neutrality proof — #136 US-003 (mirrors US-005 of #135). + +The OpenAI analogue of :mod:`tests.grade.test_provider_neutrality`. Drives +:func:`signalforge.grade.grade_artifacts` end-to-end with the real +:class:`signalforge.llm.providers.OpenAIProvider` (selected via +``GradeConfig(provider="openai")``) plus the hand-rolled +:class:`tests.llm._fake_openai.FakeOpenAIClient`, and pins the +capability-degrade invariants from DEC-008 of #135 / DEC-005, DEC-006, +DEC-009, DEC-011 of #136: + +* ``provider_for("openai")`` resolves an :class:`OpenAIProvider` with both + capability flags ``False``. +* ``GradeConfig(provider="openai", model="gpt-4o")`` validates without + error (the registry-validated ``provider`` field accepts it because the + provider was registered at module-import time). +* ``grade_artifacts(..., client=FakeOpenAIClient())`` writes a valid audit + JSONL + sidecar JSON; every produced ``GradeEvent`` records + ``cache_*_input_tokens == 0`` and the four reproducibility blake2b-8 + hashes are well-formed; the JSONL round-trips through the strict + ``extra="forbid"`` drift mirror; the sidecar JSON round-trips through + :class:`GradingReport`; NO dual-zero cache-anomaly WARNING fires; and + :meth:`FakeOpenAIClient.assert_all_expectations_met` reports zero + unconsumed expectations. + +Registry isolation mirrors :mod:`tests.grade.test_provider_neutrality` — +snapshot + restore ``signalforge.llm.providers._REGISTRY`` so any in-test +registry mutation does not leak. +""" + +from __future__ import annotations + +import json +import re +from pathlib import Path +from typing import TYPE_CHECKING, Any, cast + +import pytest + +from signalforge.draft.models import CandidateSchema +from signalforge.grade.config import GradeConfig +from signalforge.grade.engine import grade_artifacts +from signalforge.grade.models import GradeEvent, GradingReport +from signalforge.grade.rubric import Criterion, Rubric +from signalforge.llm.providers import OpenAIProvider, provider_for + +if TYPE_CHECKING: + from signalforge.llm import AnthropicClientProtocol +from signalforge.manifest.models import Column, Model +from signalforge.prune.models import PruneResult +from tests.grade.test_drift_detector import StrictGradeEvent +from tests.llm._fake_openai import ( + FakeOpenAIChoice, + FakeOpenAIClient, + FakeOpenAICompletion, + FakeOpenAIMessage, + FakeOpenAIUsage, +) + +_FIXTURE_PATH = Path(__file__).resolve().parent.parent / "fixtures" / "grade" + +# A 16-hex blake2b-8 fingerprint (the reproducibility-hash recipe across the +# audit corpus). Used to assert the produced events carry well-formed hashes. +_BLAKE2B8_HEX = re.compile(r"^[0-9a-f]{16}$") + + +@pytest.fixture +def _isolate_registry() -> Any: + """Snapshot + restore the process-level provider registry. + + Mirrors :mod:`tests.grade.test_provider_neutrality`'s fixture — keeps + any in-test ``register_provider`` mutation from leaking across tests. + The default :class:`OpenAIProvider` is registered at import time, so + after restore ``provider_for("openai")`` still resolves cleanly. + """ + from signalforge.llm import providers as providers_module + + saved = dict(providers_module._REGISTRY) + try: + yield + finally: + providers_module._REGISTRY.clear() + providers_module._REGISTRY.update(saved) + + +def _make_model() -> Model: + return Model( + unique_id="model.shop.orders", + name="orders", + resource_type="model", + package_name="shop", + original_file_path="models/orders.sql", + path="orders.sql", + database="fake_project", + schema="dataset", # type: ignore[call-arg] + columns={ + "order_id": Column(name="order_id"), + "customer_id": Column(name="customer_id"), + }, + raw_code="select 1", + ) + + +def _empty_prune_result(model: Model) -> PruneResult: + return PruneResult( + model_unique_id=model.unique_id, + decisions=(), + elapsed_ms=0, + signalforge_version="0.0.0-test", + ) + + +def _load_sample_candidate() -> CandidateSchema: + raw = (_FIXTURE_PATH / "sample_candidate.json").read_text(encoding="utf-8") + return CandidateSchema.model_validate_json(raw) + + +def _single_criterion() -> Rubric: + """A one-criterion rubric so every judge call shares one ``criterion_id``. + + The grade parser anchors on ``returned.criterion_id == sent.criterion_id``; + a single canned response carrying this id satisfies every call. + """ + return (Criterion(id="clarity", criterion="Is it clear?"),) + + +def _canned_judge_response_text() -> str: + """A grade-judge JSON payload the parser accepts for the ``clarity`` call.""" + return json.dumps( + { + "criterion_id": "clarity", + "score": 0.8, + "passed": True, + "evidence": "concise and unambiguous", + "reasoning": "reads clearly", + } + ) + + +def _build_canned_completion() -> FakeOpenAICompletion: + """Build a fresh canned :class:`FakeOpenAICompletion` for one judge call. + + A new instance per call keeps the JSONL ``response_text_hash`` stable + (hash is over the text payload only) while avoiding accidental shared + mutable state between expectations. + """ + return FakeOpenAICompletion( + choices=[ + FakeOpenAIChoice( + message=FakeOpenAIMessage(content=_canned_judge_response_text()), + ) + ], + usage=FakeOpenAIUsage(prompt_tokens=120, completion_tokens=60), + ) + + +def _project(tmp_path: Path) -> Path: + project_dir = tmp_path / "project" + project_dir.mkdir(parents=True, exist_ok=True) + (project_dir / ".signalforge").mkdir(parents=True, exist_ok=True) + return project_dir + + +def _fast_config() -> GradeConfig: + return GradeConfig( + provider="openai", + model="gpt-4o", + max_output_tokens=64, + max_retries_429=0, + max_retries_5xx=0, + max_retries_conn=0, + total_budget_seconds=60, + ) + + +def _read_jsonl(path: Path) -> list[dict[str, Any]]: + return [json.loads(line) for line in path.read_text().splitlines() if line] + + +def test_provider_for_openai_resolves_with_correct_capability_flags( + _isolate_registry: None, +) -> None: + """``provider_for("openai")`` returns an :class:`OpenAIProvider` with both + capability flags ``False`` (DEC-008 of #135 / DEC-001 of #136).""" + provider = provider_for("openai") + assert isinstance(provider, OpenAIProvider) + assert provider.supports_prompt_caching is False + assert provider.supports_token_count is False + + +def test_grade_config_provider_openai_validates_after_registration( + _isolate_registry: None, +) -> None: + """``GradeConfig(provider="openai", model="gpt-4o")`` validates without + error (DEC-007 of #135 — the registry-validated ``provider`` field + accepts a registered key).""" + config = GradeConfig(provider="openai", model="gpt-4o") + assert config.provider == "openai" + assert config.model == "gpt-4o" + + +def test_grade_artifacts_drives_openai_provider_end_to_end( + tmp_path: Path, + caplog: pytest.LogCaptureFixture, + _isolate_registry: None, +) -> None: + """End-to-end OpenAI grade run: zero cache tokens in the JSONL, well-formed + reproducibility hashes, strict drift-mirror round-trip on the JSONL, + :class:`GradingReport` round-trip on the sidecar, no dual-zero + cache-anomaly WARNING, and all expectations on the fake consumed + (DEC-005, DEC-006, DEC-009, DEC-011 of #136).""" + project_dir = _project(tmp_path) + model = _make_model() + candidate = _load_sample_candidate() + rubric = _single_criterion() + + # The sample candidate has 2 columns + 1 column test + 0 model tests = 7 + # artifacts × 1 criterion = 7 judge calls (matches the upstream + # FakeNoCacheProvider neutrality test). Queue one expectation per call. + client = FakeOpenAIClient() + + def _is_openai_create_kwargs(kwargs: dict[str, Any]) -> bool: + # Callable matcher: assert the orchestrator handed the provider + # OpenAI-shaped kwargs and never an Anthropic ``cache_control`` / + # ``extra_headers`` cache header. Returning False here would surface + # as "unexpected messages.create call: did not match expectation". + if kwargs.get("model") != "gpt-4o": + return False + if "response_format" not in kwargs: + return False + # No Anthropic-shaped fields ever attached. + if "system" in kwargs: + return False + if kwargs.get("extra_headers"): + return False + # No cache_control marker on any content block. + messages = kwargs.get("messages") or [] + for message in messages: + content = message.get("content") + if isinstance(content, list): + for block in content: + if isinstance(block, dict) and "cache_control" in block: + return False + return True + + for _ in range(7): + client.expect_messages_create( + matching=_is_openai_create_kwargs, + returns=_build_canned_completion(), + ) + + audit_path = project_dir / ".signalforge" / "grade.jsonl" + sidecar_path = project_dir / ".signalforge" / "grade.json" + + with caplog.at_level("WARNING", logger="signalforge.llm.client"): + # ``grade_artifacts`` types its ``client`` kwarg against the Anthropic + # injection surface (DEC-012 of #135). A non-Anthropic provider builds + # its own client and the orchestrator hands it straight to the + # strategy, so the cast is the documented seam. + report = grade_artifacts( + model, + candidate, + _empty_prune_result(model), + rubric=rubric, + config=_fast_config(), + client=cast("AnthropicClientProtocol", client), + project_dir=project_dir, + audit_path=audit_path, + sidecar_path=sidecar_path, + ) + + assert isinstance(report, GradingReport) + assert len(report.results) == 7 + assert report.aggregate_complete is True + assert all(r.score == 0.8 and r.passed for r in report.results) + + # --- Audit JSONL: one durable record per call, all cache tokens 0. --- + rows = _read_jsonl(audit_path) + assert len(rows) == len(report.results) == 7 + events = [GradeEvent.model_validate(r) for r in rows] + for event in events: + assert event.cache_creation_input_tokens == 0 + assert event.cache_read_input_tokens == 0 + assert event.run_id == report.run_id + # Reproducibility blake2b-8 fingerprints are present + well-formed. + assert _BLAKE2B8_HEX.match(event.rubric_hash) + assert _BLAKE2B8_HEX.match(event.prompt_version_template) + assert _BLAKE2B8_HEX.match(event.criterion_prompt_hash) + assert _BLAKE2B8_HEX.match(event.response_text_hash) + + # --- Drift detector: each JSONL line round-trips through the strict + # extra="forbid" mirror (zero-cache events validate cleanly). --- + for line in audit_path.read_text(encoding="utf-8").splitlines(): + if line.strip(): + StrictGradeEvent.model_validate_json(line) + + # --- Sidecar JSON: present + round-trips through GradingReport. --- + assert sidecar_path.exists() + round_tripped = GradingReport.model_validate_json(sidecar_path.read_text(encoding="utf-8")) + assert round_tripped.run_id == report.run_id + assert len(round_tripped.results) == len(report.results) + + # --- No dual-zero cache-anomaly WARNING: OpenAI's supports_prompt_caching=False + # gates this WARNING off (it would otherwise false-alarm on every call). --- + assert not any("cache marker no-op" in rec.getMessage() for rec in caplog.records), ( + "OpenAI provider (supports_prompt_caching=False) must suppress the " + "dual-zero cache-anomaly WARNING" + ) + + # --- The fake's expectation queue is exhausted (no leftover, no extras). --- + client.assert_all_expectations_met() diff --git a/tests/grade/test_smoke_real_api_openai.py b/tests/grade/test_smoke_real_api_openai.py new file mode 100644 index 00000000..b88a16e2 --- /dev/null +++ b/tests/grade/test_smoke_real_api_openai.py @@ -0,0 +1,211 @@ +"""Real-API smoke test for :func:`signalforge.grade.grade_artifacts` via OpenAI. + +Issue #136 / US-006 — DEC-001, DEC-004, DEC-005, DEC-008. Gated by the +``openai`` marker — excluded from default CI by :file:`pyproject.toml`'s +``addopts = "... -m 'not openai'"``. Requires ``SF_RUN_OPENAI=1`` + +``OPENAI_API_KEY``. Mirrors :file:`tests/grade/test_smoke_real_api.py` +(the Anthropic equivalent) in shape: ``pytestmark = pytest.mark.openai``, +env-var skip-gate, shape-only assertions (no specific scores or +``passed`` outcomes — LLM output is not deterministic enough for that +contract). + +What this proves end-to-end: + +* ``OPENAI_API_KEY`` is present and valid; the OpenAIProvider's Chat + Completions adapter (#136 DEC-001 / DEC-009) reaches OpenAI and + returns a parseable response under the grader prompt template. +* :func:`signalforge.grade.parser.parse_grade_response` validates the + LLM-judge output through :class:`GradingResult` for an OpenAI + ``gpt-4o`` response (the default judge model per DEC-004). +* :func:`signalforge.grade.engine.grade_artifacts` writes both the + fail-closed JSONL audit and the sidecar JSON. +* OpenAI's ``cache_*_input_tokens`` are zero (``supports_prompt_caching=False``) + and the seam does NOT emit the dual-zero cache-anomaly WARNING + (capability-gated off in #135 + #136). + +What this deliberately does NOT assert: + +* Specific scores or ``passed`` outcomes — LLM output is not + deterministic enough; the rubric drives the LLM's verdict and the + test would be flaky if pinned. +""" + +from __future__ import annotations + +import logging +import os +from pathlib import Path + +import pytest + +import signalforge as _sf +from signalforge.draft.models import ( + CandidateColumn, + CandidateSchema, + CandidateTestNotNull, +) +from signalforge.grade import ( + Criterion, + GradeConfig, + GradingReport, + grade_artifacts, +) +from signalforge.manifest.models import Column, Model +from signalforge.prune.models import PruneResult + +pytestmark = pytest.mark.openai + +_TRUTHY = frozenset({"1", "true", "yes", "on"}) + + +def _openai_runs_enabled() -> bool: + """``SF_RUN_OPENAI`` is set to a truthy value (mirrors ``SF_RUN_BQ`` / ``SF_RUN_SNOWFLAKE``).""" + return os.environ.get("SF_RUN_OPENAI", "").lower() in _TRUTHY + + +def _skip_reason() -> str | None: + """Return a skip-reason string if any required env var is missing. + + Returns ``None`` only when both gates are satisfied — the test then + proceeds to make real OpenAI calls. Each missing prerequisite yields + its own distinct reason so a maintainer running + ``pytest -m openai`` sees exactly what to set. Treat an empty / + whitespace-only ``OPENAI_API_KEY`` as "unset" (an empty value would + otherwise reach the API and produce a noisy auth failure). + """ + if not _openai_runs_enabled(): + return "SF_RUN_OPENAI=1 required (live test calls the real OpenAI API)" + if not os.environ.get("OPENAI_API_KEY", "").strip(): + return "OPENAI_API_KEY required (live test authenticates against the real OpenAI API)" + return None + + +def test_grade_artifacts_real_openai_api_smoke( + tmp_path: Path, caplog: pytest.LogCaptureFixture +) -> None: + """One real OpenAI round-trip against a tiny fixture; shape-only. + + Builds a minimal :class:`CandidateSchema` (1 column with + description + rationale + 1 ``not_null`` test) and a + single-criterion rubric (clarity). Issues 5 real LLM calls through + :func:`signalforge.grade.grade_artifacts` (5 artifacts × 1 criterion) + and asserts only that the typed sidecar parses cleanly, the JSONL + audit landed, OpenAI cache token fields are zero, and no dual-zero + cache-anomaly WARNING was emitted (capability gate from #135 / #136). + """ + if reason := _skip_reason(): + pytest.skip(reason) + + # Tiny manifest model — only the fields the grader's pipeline reads. + model = Model( + unique_id="model.sf_smoke.dim_users", + name="dim_users", + resource_type="model", + package_name="sf_smoke", + original_file_path="models/marts/dim_users.sql", + path="marts/dim_users.sql", + database="sf-smoke-proj", + schema="main", # type: ignore[call-arg] + columns={"user_id": Column(name="user_id")}, + raw_code="select 1 as user_id", + ) + + candidate = CandidateSchema( + name="dim_users", + description="Curated user dimension table for analytics.", + rationale=( + "Joins source.users with source.user_profiles to produce one row per active user." + ), + columns=( + CandidateColumn( + name="user_id", + description=( + "Primary key uniquely identifying each user. Sourced from source.users.id." + ), + rationale="Used as join key by every downstream fact table.", + tests=( + CandidateTestNotNull( + column="user_id", + rationale="Primary keys must never be null.", + ), + ), + ), + ), + tests=(), + ) + + rubric = ( + Criterion( + id="clarity", + criterion=( + "Is the column description clear, specific, and " + "actionable? Does it unambiguously explain the column's " + "purpose and business meaning without jargon or vagueness?" + ), + ), + ) + + # Empty prune result — the grader's no-redundant criterion is the + # only consumer of dropped tests, and we're running with a custom + # single-criterion (clarity) rubric so the empty tuple is fine. + prune_result = PruneResult( + model_unique_id=model.unique_id, + decisions=(), + elapsed_ms=0, + signalforge_version=_sf.__version__, + ) + + audit_path = tmp_path / "grade.jsonl" + sidecar_path = tmp_path / "grade.json" + + # OpenAI judge: gpt-4o (the DEC-004 default judge model). The + # provider field on GradeConfig is the #135 plug-in seam. + config = GradeConfig(provider="openai", model="gpt-4o") + + with caplog.at_level(logging.WARNING): + report = grade_artifacts( + model, + candidate, + prune_result, + rubric=rubric, + config=config, + audit_path=audit_path, + sidecar_path=sidecar_path, + project_dir=tmp_path, + ) + + # Shape-only assertions — no specific score / ``passed`` contract. + assert isinstance(report, GradingReport) + assert report.model_unique_id == model.unique_id + # 5 artifacts × 1 criterion = 5 results. + assert len(report.results) == 5 + for r in report.results: + assert r.criterion_id == "clarity" + # Score is either ``None`` (degraded path — DEC-015) or a + # finite float in [0.0, 1.0]. The model's own validator already + # enforces this at construction; the assertion documents the + # contract for the smoke test reader. + assert r.score is None or 0.0 <= r.score <= 1.0 + + # Sidecar JSON exists and round-trips through the typed model. + assert sidecar_path.exists() + sidecar_text = sidecar_path.read_text(encoding="utf-8") + GradingReport.model_validate_json(sidecar_text) + + # Audit JSONL exists and has at least one durable record per call. + assert audit_path.exists() + audit_lines = audit_path.read_text(encoding="utf-8").strip().splitlines() + assert len(audit_lines) >= 1 + + # OpenAI provider has ``supports_prompt_caching=False`` — the + # dual-zero cache-anomaly WARNING (llm-drafter.md DEC-014) is + # capability-gated off and must NEVER fire on this path. + cache_warning_messages = [ + record.getMessage() + for record in caplog.records + if record.levelno >= logging.WARNING and "cache marker no-op" in record.getMessage() + ] + assert not cache_warning_messages, ( + f"unexpected cache-anomaly WARNING(s) on OpenAI path " + f"(supports_prompt_caching=False should gate this off): {cache_warning_messages}" + ) diff --git a/tests/llm/_fake_openai.py b/tests/llm/_fake_openai.py new file mode 100644 index 00000000..1af9f1a8 --- /dev/null +++ b/tests/llm/_fake_openai.py @@ -0,0 +1,203 @@ +"""Hand-rolled fake for the OpenAI client (#136 US-003). + +Mirrors :mod:`tests.llm._fake`'s :class:`FakeAnthropicClient` shape verbatim +but speaks the OpenAI Chat Completions response surface instead of +Anthropic's typed-block array. Tests register expectations via +:meth:`FakeOpenAIClient.expect_messages_create`; the fake's +``.messages.create`` consumes one matching expectation per call (FIFO); +unexpected calls raise loudly. Unconsumed expectations at end of test are +caught by :meth:`assert_all_expectations_met`. + +The ``.messages`` namespace mirrors the production +:class:`signalforge.llm._openai_client._OpenAIClientAdapter` shape, so the +generic orchestrator's ``llm_client.messages.create(**kwargs)`` call works +unchanged. ``count_tokens`` raises :class:`NotImplementedError` (defensive +parity with the adapter — orchestrator gates it off via +``supports_token_count=False``; if it ever drifts and gets called, the +raise turns the silent regression into a loud failure, mirroring +``tests/llm/_fake_provider.py::FakeNoCacheClient``). + +Response dataclasses match the surface +:meth:`signalforge.llm.providers.OpenAIProvider.extract_text_blocks` and +:meth:`signalforge.llm.providers.OpenAIProvider.extract_usage` read: + +* ``response.choices[0].message.content`` is the assistant text (single + string, not a typed-block array). +* ``response.usage.prompt_tokens`` / ``response.usage.completion_tokens`` + are the token counts (no cache fields — OpenAI has no equivalent cache + discount). + +Hand-rolled rather than ``MagicMock``-driven because ``MagicMock`` +auto-passes everything, which would silently mask mismatches and violate +``testing-signal.md``. Lives under ``tests/llm/`` and is never imported +from production code. +""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass, field +from typing import Any + + +@dataclass +class FakeOpenAIUsage: + """Stand-in for ``openai.types.CompletionUsage``. + + Only the fields :meth:`OpenAIProvider.extract_usage` reads are exposed. + OpenAI has no cache-token fields — the provider unconditionally reports + ``cache_creation_input_tokens=0`` / ``cache_read_input_tokens=0`` (matches + ``supports_prompt_caching=False``). + """ + + prompt_tokens: int = 100 + completion_tokens: int = 40 + + +@dataclass +class FakeOpenAIMessage: + """Stand-in for ``openai.types.chat.ChatCompletionMessage``.""" + + content: str + role: str = "assistant" + + +@dataclass +class FakeOpenAIChoice: + """Stand-in for ``openai.types.chat.Choice``.""" + + message: FakeOpenAIMessage + index: int = 0 + finish_reason: str = "stop" + + +@dataclass +class FakeOpenAICompletion: + """Stand-in for ``openai.types.chat.ChatCompletion`` (the + ``chat.completions.create`` response, which the adapter exposes through + ``messages.create``). + """ + + choices: list[FakeOpenAIChoice] + usage: FakeOpenAIUsage + model: str = "openai-fake-1" + id: str = "chatcmpl_fake_001" + object: str = "chat.completion" + + +# A "matching" predicate is either a dict (subset match against the kwargs +# dict the seam passes) or a callable returning bool. +_Matcher = dict[str, Any] | Callable[[dict[str, Any]], bool] + + +@dataclass +class _MessagesCreateExpectation: + matching: _Matcher + returns: object | Exception + + +def _matches(matcher: _Matcher, kwargs: dict[str, Any]) -> bool: + """Apply a matcher to the kwargs of an SDK call. + + A dict matcher is a subset match: every key in the matcher must be + present in ``kwargs`` with an equal value. A callable matcher is + invoked with the full kwargs dict and must return bool. + """ + if callable(matcher): + return bool(matcher(kwargs)) + for key, expected in matcher.items(): + if key not in kwargs: + return False + if kwargs[key] != expected: + return False + return True + + +@dataclass +class _FakeOpenAIMessages: + """Implements the ``messages`` namespace on the fake client. + + Mirrors the production adapter's ``.messages`` surface: ``create`` + delegates here (in production it routes to + ``chat.completions.create``); ``count_tokens`` raises (the orchestrator + never calls it for a ``supports_token_count=False`` provider — a raise + here turns a capability-flag-gate regression loud). + """ + + _create_queue: list[_MessagesCreateExpectation] = field(default_factory=list) + _create_calls: list[dict[str, Any]] = field(default_factory=list) + + def create(self, **kwargs: Any) -> Any: + self._create_calls.append(kwargs) + if not self._create_queue: + raise AssertionError(f"unexpected messages.create call: {kwargs!r}") + expectation = self._create_queue[0] + if not _matches(expectation.matching, kwargs): + raise AssertionError( + f"unexpected messages.create call: {kwargs!r} did not match expectation " + f"{expectation.matching!r}" + ) + self._create_queue.pop(0) + if isinstance(expectation.returns, Exception): + raise expectation.returns + return expectation.returns + + def count_tokens(self, **kwargs: Any) -> Any: + raise NotImplementedError( + "OpenAI fake does not support pre-send count_tokens; " + "OpenAIProvider.supports_token_count=False gates this call off " + "in the orchestrator. If you see this, the capability-flag gate " + "has drifted." + ) + + +class FakeOpenAIClient: + """Explicit fake for the OpenAI client (the + :class:`signalforge.llm._openai_client._OpenAIClientAdapter` shape); + calls outside the queued expectations raise :class:`AssertionError`. + + Each :meth:`expect_messages_create` enqueues one expectation; calls + consume them FIFO. Tests must call :meth:`assert_all_expectations_met` + at the end; a non-empty queue at the end of a test is a + leftover-expectation bug. + + Exceptions queued as ``returns`` (e.g. an ``openai.RateLimitError`` + instance) are raised when the expectation is consumed — supports the + retry-loop test paths. + """ + + def __init__(self) -> None: + self._messages = _FakeOpenAIMessages() + self.messages = self._messages + + def expect_messages_create( + self, + *, + matching: _Matcher, + returns: object | Exception, + ) -> None: + self._messages._create_queue.append( + _MessagesCreateExpectation(matching=matching, returns=returns) + ) + + def assert_all_expectations_met(self) -> None: + if self._messages._create_queue: + raise AssertionError( + f"unconsumed expectations: {len(self._messages._create_queue)} " + "messages.create expectation(s)" + ) + + @property + def create_calls(self) -> list[dict[str, Any]]: + """Inspector for tests that want to assert on the kwargs the seam + passed to ``messages.create``.""" + return list(self._messages._create_calls) + + +__all__ = [ + "FakeOpenAIChoice", + "FakeOpenAIClient", + "FakeOpenAICompletion", + "FakeOpenAIMessage", + "FakeOpenAIUsage", +] diff --git a/tests/llm/_fake_provider.py b/tests/llm/_fake_provider.py index c3184743..4d6c5029 100644 --- a/tests/llm/_fake_provider.py +++ b/tests/llm/_fake_provider.py @@ -212,6 +212,30 @@ def classify_exception(self, exc: BaseException) -> ExceptionCategory: return ExceptionCategory.CONNECTION return ExceptionCategory.NO_RETRY + def estimate_input_tokens( + self, + model: str, + text: str, + *, + system: str = "", + client: object | None = None, + ) -> int: + """Return a trivial word-count proxy for ``system + text`` (#136 US-005). + + The neutrality test exercises orchestrator dispatch, not real + token counting; ``len(text.split())`` is a deterministic + non-zero positive answer that any rendered estimate will treat + as valid. Mirrors :class:`FakeNoCacheProvider`'s overall posture + of declaring the right capability flags and answering each ABC + method with a minimal honest value. + """ + del model, client # neither is consulted on the proxy path + # Join with a delimiter so the last word of ``system`` and the + # first word of ``text`` don't merge into a single token under + # ``.split()`` (boundary-word undercount; PR #152 CodeRabbit + # catch). + return len(f"{system} {text}".split()) + __all__ = [ "FAKE_NOCACHE_PROVIDER_NAME", diff --git a/tests/llm/test_client_shim.py b/tests/llm/test_client_shim.py index ec6cde19..a29104d4 100644 --- a/tests/llm/test_client_shim.py +++ b/tests/llm/test_client_shim.py @@ -35,8 +35,27 @@ def _llm_py_files_excluding_client() -> list[Path]: - """All ``.py`` under ``src/signalforge/llm/`` except ``_anthropic_client.py``.""" - return [p for p in _LLM_SRC_DIR.rglob("*.py") if p.name != "_anthropic_client.py"] + """All ``.py`` under ``src/signalforge/llm/`` except the per-vendor + SDK shims (``_anthropic_client.py`` and ``_openai_client.py``). + + Each vendor shim is the sole home for its own SDK's ``# pyright: + ignore`` / ``# type: ignore`` comments per the one-shim-per-vendor + convention (``.claude/rules/llm-drafter.md`` § "One SDK seam"). + The per-vendor *confinement* of each shim's ignores is asserted by + the per-vendor cheap-floor tests: + + * Anthropic: this module's :func:`test_no_pyright_ignores_outside_client_shim` + (but excluding the OpenAI shim — its docstring mentions the phrase + and its lazy ``import openai`` carries a legitimate ignore). + * OpenAI: ``tests/llm/test_openai_client_confinement.py``. + + Excluding the OpenAI shim from the Anthropic regex floor mirrors how + Scan 9 in ``tests/test_audit_completeness.py`` excludes + ``_anthropic_client.py`` (and vice-versa for Scan 3) — each + per-vendor seam is invisible to the others' scans by construction. + """ + excluded_names = {"_anthropic_client.py", "_openai_client.py"} + return [p for p in _LLM_SRC_DIR.rglob("*.py") if p.name not in excluded_names] def test_make_anthropic_client_returns_protocol_satisfying_object() -> None: diff --git a/tests/llm/test_openai_client_confinement.py b/tests/llm/test_openai_client_confinement.py new file mode 100644 index 00000000..88aaae70 --- /dev/null +++ b/tests/llm/test_openai_client_confinement.py @@ -0,0 +1,140 @@ +"""#136 US-001 DEC-010 — OpenAI SDK type-ignore confinement. + +Every ``# type: ignore`` / ``# pyright: ignore`` line in the +``signalforge.llm`` tree that ALSO mentions "openai" must live ONLY in +``_openai_client.py`` — the one-shim-per-vendor SDK seam. Mirrors the +spirit of the Anthropic-SDK confinement scan (Scan 3 in +``tests/test_audit_completeness.py``) and the Snowflake-shaped per-file +line scan in ``tests/warehouse/test_snowflake_client_confinement.py``; a +simple file/line scan suffices here. + +The companion Scan 9 in ``tests/test_audit_completeness.py`` enforces the +AST-level construction-call confinement (``openai.OpenAI(...)`` only in +the shim). This line-based scan is the cheap floor; Scan 9 is the +load-bearing AST one. +""" + +from __future__ import annotations + +from pathlib import Path + +_LLM_DIR = Path(__file__).resolve().parents[2] / "src" / "signalforge" / "llm" +_SHIM_FILENAME = "_openai_client.py" + + +def _openai_type_ignore_lines(path: Path) -> list[tuple[int, str]]: + """Return (lineno, text) for lines carrying an openai-mentioning + type/pyright ignore directive. + """ + hits: list[tuple[int, str]] = [] + for lineno, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1): + lowered = line.lower() + has_ignore = "type: ignore" in lowered or "pyright: ignore" in lowered + if has_ignore and "openai" in lowered: + hits.append((lineno, line.strip())) + return hits + + +def test_openai_type_ignores_only_in_shim() -> None: + """No ``.py`` under ``signalforge/llm/`` other than + ``_openai_client.py`` may carry an openai-mentioning type-ignore. + """ + offenders: list[str] = [] + # ``rglob`` (not ``glob``) so nested modules under signalforge/llm/ + # are also scanned — PR #152 CodeRabbit catch: top-level-only glob + # let openai-mentioning ignore directives in subpackages bypass the + # guard (the package is flat today but a future subpackage would + # silently un-confine the scan). + for py in sorted(_LLM_DIR.rglob("*.py")): + if py.name == _SHIM_FILENAME: + continue + for lineno, text in _openai_type_ignore_lines(py): + offenders.append(f"{py.relative_to(_LLM_DIR)}:{lineno}: {text}") + + assert not offenders, ( + "openai SDK type-ignore must live only in " + f"{_SHIM_FILENAME}, but found:\n" + "\n".join(offenders) + ) + + +def test_shim_actually_carries_openai_type_ignore() -> None: + """Sanity: the shim itself DOES carry at least one openai-mentioning + type-ignore. Without this, the confinement scan above could pass + vacuously after a refactor that dropped the seam. + """ + shim = _LLM_DIR / _SHIM_FILENAME + assert _openai_type_ignore_lines(shim), ( + f"{_SHIM_FILENAME} should confine the openai SDK type-ignore; " + "the confinement scan is only meaningful if the seam exists" + ) + + +# --------------------------------------------------------------------------- +# Coverage-closing tests for PR #152 codecov gaps on the shim internals. +# Confinement-test file is the natural home — these tests pin the per-shim +# behaviours that the production import surface depends on (the adapter +# façade + the tiktoken fallback) but that no production caller currently +# exercises in the default test set (the orchestrator drives via the +# FakeOpenAIClient, never through _OpenAIClientAdapter; tiktoken's fallback +# fires only on an unknown model id). +# --------------------------------------------------------------------------- + + +def test_openai_client_adapter_messages_create_delegates_to_chat_completions() -> None: + """``_OpenAIClientAdapter.messages.create(**kwargs)`` MUST delegate + verbatim to ``self._raw.chat.completions.create(**kwargs)``. + + The orchestrator's ``call_llm`` hard-calls + ``llm_client.messages.create(...)``; the adapter is the only thing + that maps that into OpenAI's actual SDK call shape. A regression + that breaks the delegation (e.g. a refactor that swaps the SDK call + path) would surface here, not in the integration tests (those use + ``FakeOpenAIClient`` which has its own ``.messages.create`` and + never goes through the adapter). + """ + from types import SimpleNamespace + from typing import Any + + from signalforge.llm._openai_client import _OpenAIClientAdapter + + captured: dict[str, Any] = {} + sentinel = object() + + def _create(**kwargs: Any) -> object: + captured.update(kwargs) + return sentinel + + raw = SimpleNamespace(chat=SimpleNamespace(completions=SimpleNamespace(create=_create))) + adapter = _OpenAIClientAdapter(raw) + + result = adapter.messages.create( + model="gpt-4o", + max_tokens=128, + messages=[{"role": "user", "content": "hi"}], + response_format={"type": "json_object"}, + ) + + assert result is sentinel + assert captured == { + "model": "gpt-4o", + "max_tokens": 128, + "messages": [{"role": "user", "content": "hi"}], + "response_format": {"type": "json_object"}, + } + + +def test_count_openai_tokens_falls_back_to_cl100k_base_for_unknown_model() -> None: + """``_count_openai_tokens`` MUST NOT raise on an unknown model id — + DEC-012 of #136 documents the ``cl100k_base`` fallback so a + newer-than-tiktoken OpenAI SKU still produces a usable estimate + rather than crashing the ``--estimate`` flow. ``--estimate`` is a + calibration signal, not a billing guarantee (mirrors the + planner-estimate caveat in ``warehouse-adapters.md``). + """ + from signalforge.llm._openai_client import _count_openai_tokens + + # A model id ``tiktoken.encoding_for_model`` doesn't recognise. + # Must return a positive int via the cl100k_base fallback, NOT raise. + count = _count_openai_tokens("not-a-real-openai-model-xyz", "hello world") + assert isinstance(count, int) + assert count > 0 diff --git a/tests/llm/test_pricing.py b/tests/llm/test_pricing.py index aff1a8cb..9707ea34 100644 --- a/tests/llm/test_pricing.py +++ b/tests/llm/test_pricing.py @@ -44,10 +44,12 @@ def test_lookup_raises_estimateunknownmodelerror_for_unknown_model() -> None: lookup("not-a-real-model-9999") rendered = str(exc_info.value) assert "not-a-real-model-9999" in rendered - # Locked remediation text — verbatim per US-001 AC. + # Locked remediation text — verbatim per US-001 AC, refreshed in + # #136 US-008 (QG) to enumerate the four OpenAI SKUs added by US-004. assert ( "Add the model to signalforge.llm.pricing.PRICES or use a supported " - "model: claude-sonnet-4-6, claude-opus-4-7, claude-haiku-4-5." + "model: claude-sonnet-4-6, claude-opus-4-7, claude-haiku-4-5, " + "gpt-4o, gpt-4o-mini, gpt-4.1, gpt-4-turbo." ) in rendered # Carries the model id as a structured field for typed handling. assert exc_info.value.model == "not-a-real-model-9999" @@ -64,14 +66,22 @@ def test_modelpricing_is_frozen() -> None: def test_price_table_version_is_a_nonempty_string() -> None: """``PRICE_TABLE_VERSION`` is a non-empty string. The literal value is - a sourcing-date stamp; v0.1 ships ``"2026-05-11"`` but the assertion - here is structural — that the constant exists and is non-empty — - so a price-table refresh doesn't churn the test for cosmetic reasons. + a sourcing-date stamp; the structural check here ensures the constant + exists and is non-empty, so a price-table refresh doesn't churn the + test for cosmetic reasons. """ assert isinstance(PRICE_TABLE_VERSION, str) assert PRICE_TABLE_VERSION +def test_price_table_version_pinned_to_us_004_ship_date() -> None: + """The ``#136 US-004`` ship-date stamp is pinned so a stealth bump + to the table version without a paired numeric edit / commit fails + loudly. Bump in lockstep with any future ``_PRICES_MUTABLE`` edit + (paired commit, per the module docstring).""" + assert PRICE_TABLE_VERSION == "2026-05-28" + + def test_pricing_module_exports() -> None: """All five public symbols are importable from the package top-level (``signalforge.llm``) — not just from the private @@ -87,18 +97,84 @@ def test_pricing_module_exports() -> None: assert hasattr(llm_pkg, "EstimateUnknownModelError") -def test_prices_contains_all_v01_skus() -> None: - """The v0.1 SKU set is locked: ``claude-sonnet-4-6``, - ``claude-opus-4-7``, ``claude-haiku-4-5``. Adding a fourth SKU is a - deliberate v0.2 expansion that should fail this test loudly until the - AC is updated.""" +def test_prices_contains_all_shipped_skus() -> None: + """The shipped SKU set is locked: three Anthropic + four OpenAI as of + ``#136 US-004`` (DEC-007). Adding a future SKU is a deliberate + expansion that should fail this test loudly until the AC is updated. + """ assert set(PRICES.keys()) == { + # Anthropic "claude-sonnet-4-6", "claude-opus-4-7", "claude-haiku-4-5", + # OpenAI (#136 US-004) + "gpt-4o", + "gpt-4o-mini", + "gpt-4.1", + "gpt-4-turbo", } +@pytest.mark.parametrize( + "sku", + [ + "gpt-4o", + "gpt-4o-mini", + "gpt-4.1", + "gpt-4-turbo", + ], +) +def test_lookup_returns_modelpricing_for_each_openai_sku(sku: str) -> None: + """Every OpenAI SKU added in #136 US-004 (DEC-007) ``lookup``s to a + ``ModelPricing`` with strictly positive input/output rates and + cache fields set to ``0.0`` — OpenAI has no Anthropic-equivalent + ``cache_control`` discount tier, so the cache columns are a + deliberate zero, not an unset placeholder. + """ + pricing = lookup(sku) + assert isinstance(pricing, ModelPricing) + assert pricing.input_per_mtok > 0.0 + assert pricing.output_per_mtok > 0.0 + assert pricing.cache_write_5m_per_mtok == 0.0 + assert pricing.cache_read_per_mtok == 0.0 + + +def test_lookup_raises_for_unknown_openai_flavoured_id() -> None: + """A plausible-looking but unsupported OpenAI id still raises + ``EstimateUnknownModelError`` — `lookup` does no provider sniffing + or fuzzy matching, just a strict dict lookup.""" + with pytest.raises(EstimateUnknownModelError) as exc_info: + lookup("gpt-9-unicorn") + assert exc_info.value.model == "gpt-9-unicorn" + + +def test_anthropic_skus_remain_byte_identical_after_us_004() -> None: + """Anthropic estimate byte-identity floor (DEC-013 of #135 / preserved + by #136): adding OpenAI SKUs must NOT perturb any field on the three + Anthropic ``ModelPricing`` rows. Pinning every field by value (rather + than asserting equality against a freshly-constructed dataclass) so a + silent rate drift fails loud here, not only at the cost-projection + seam. + """ + sonnet = lookup("claude-sonnet-4-6") + assert sonnet.input_per_mtok == 3.00 + assert sonnet.output_per_mtok == 15.00 + assert sonnet.cache_write_5m_per_mtok == 3.75 + assert sonnet.cache_read_per_mtok == 0.30 + + opus = lookup("claude-opus-4-7") + assert opus.input_per_mtok == 15.00 + assert opus.output_per_mtok == 75.00 + assert opus.cache_write_5m_per_mtok == 18.75 + assert opus.cache_read_per_mtok == 1.50 + + haiku = lookup("claude-haiku-4-5") + assert haiku.input_per_mtok == 0.80 + assert haiku.output_per_mtok == 4.00 + assert haiku.cache_write_5m_per_mtok == 1.00 + assert haiku.cache_read_per_mtok == 0.08 + + def test_estimateunknownmodelerror_is_in_exit_code_mapping_at_tier_2() -> None: """The 7th AST scan in ``tests/test_audit_completeness.py`` requires every concrete ``*Error`` to be registered; this test pins the diff --git a/tests/llm/test_providers.py b/tests/llm/test_providers.py index 0f1480f4..6d78d5e4 100644 --- a/tests/llm/test_providers.py +++ b/tests/llm/test_providers.py @@ -22,6 +22,7 @@ AnthropicProvider, ExceptionCategory, LLMProvider, + OpenAIProvider, UsageMetrics, provider_for, register_provider, @@ -71,6 +72,18 @@ def extract_usage(self, response: object) -> UsageMetrics: def classify_exception(self, exc: BaseException) -> ExceptionCategory: return ExceptionCategory.NO_RETRY + def estimate_input_tokens( + self, + model: str, + text: str, + *, + system: str = "", + client: object | None = None, + ) -> int: + # Trivial deterministic stub (#136 US-005) — the registry tests + # don't exercise the count, only the ABC instantiation path. + return 0 + @pytest.fixture def _isolate_registry() -> Any: @@ -397,3 +410,390 @@ def test_anthropic_provider_make_client_uses_shim(monkeypatch: pytest.MonkeyPatc sentinel = object() monkeypatch.setattr(shim, "_make_anthropic_client", lambda: sentinel) assert AnthropicProvider().make_client() is sentinel + + +# --------------------------------------------------------------------------- +# US-002 of issue #136 — OpenAIProvider strategy +# --------------------------------------------------------------------------- + + +_OPENAI_REQ = httpx.Request("POST", "https://api.openai.com/v1/chat/completions") + + +class _FakeChoiceMessage: + """Minimal stand-in for the OpenAI SDK's ``ChatCompletionMessage``.""" + + def __init__(self, content: str | None) -> None: + self.content = content + + +class _FakeChoice: + """Minimal stand-in for an OpenAI ``ChatCompletion.Choice``.""" + + def __init__(self, content: str | None) -> None: + self.message = _FakeChoiceMessage(content) + + +class _FakeOpenAIUsage: + """OpenAI Chat Completions usage shape (``prompt_tokens`` / + ``completion_tokens``; no cache fields).""" + + def __init__(self, prompt_tokens: int, completion_tokens: int) -> None: + self.prompt_tokens = prompt_tokens + self.completion_tokens = completion_tokens + + +class _FakeOpenAIResponse: + """Minimal stand-in for an OpenAI ``ChatCompletion`` response object.""" + + def __init__( + self, + *, + content: str | None = "ok", + prompt_tokens: int = 120, + completion_tokens: int = 45, + choices: object | None = None, + ) -> None: + if choices is None: + choices = [_FakeChoice(content)] + self.choices = choices + self.usage = _FakeOpenAIUsage(prompt_tokens, completion_tokens) + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_is_registered() -> None: + """US-002 of #136 registers ``OpenAIProvider`` at import time, so + ``provider_for("openai")`` returns it (DEC-003 of #135).""" + provider = provider_for("openai") + assert isinstance(provider, OpenAIProvider) + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_capability_flags() -> None: + """OpenAI has no prompt-caching primitive and no server-side + ``count_tokens`` API, so both capability flags are ``False`` + (DEC-008 of #135).""" + provider = OpenAIProvider() + assert provider.name == "openai" + assert provider.supports_prompt_caching is False + assert provider.supports_token_count is False + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_build_create_kwargs_shape() -> None: + """``build_create_kwargs`` returns the OpenAI-native Chat Completions + kwargs shape: ``model``, ``max_tokens``, a system + user ``messages`` + pair (user content = cached + dynamic), and ``response_format`` + enforcing JSON server-side (DEC-006).""" + kwargs = OpenAIProvider().build_create_kwargs( + system="SYS", + cached_block="CACHED", + dynamic_block="DYN", + model="gpt-4o", + max_tokens=1024, + cache_ttl="5m", + cache_marker_active=False, + ) + assert kwargs["model"] == "gpt-4o" + assert kwargs["max_tokens"] == 1024 + assert kwargs["response_format"] == {"type": "json_object"} + messages = kwargs["messages"] + assert len(messages) == 2 + assert messages[0] == {"role": "system", "content": "SYS"} + assert messages[1] == {"role": "user", "content": "CACHEDDYN"} + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_build_create_kwargs_never_emits_cache_marker_or_extra_headers() -> None: + """OpenAI has no caching primitive — there must be no ``cache_control`` + marker anywhere in the kwargs and no ``extra_headers`` field, regardless + of ``cache_marker_active`` / ``cache_ttl`` (the orchestrator already + resolves the flag to ``False`` for a non-caching provider; this is + belt-and-braces).""" + for cache_marker_active in (True, False): + for cache_ttl in ("5m", "1h"): + kwargs = OpenAIProvider().build_create_kwargs( + system="s", + cached_block="c", + dynamic_block="d", + model="gpt-4o", + max_tokens=10, + cache_ttl=cache_ttl, + cache_marker_active=cache_marker_active, + ) + assert "extra_headers" not in kwargs + # The kwargs dict carries no cache_control marker at any depth. + assert "cache_control" not in repr(kwargs) + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_build_count_tokens_kwargs_raises() -> None: + """``build_count_tokens_kwargs`` raises ``NotImplementedError`` because + ``supports_token_count`` is ``False`` (DEC-011 of #136 — mirrors + ``FakeNoCacheProvider`` precedent).""" + with pytest.raises(NotImplementedError): + OpenAIProvider().build_count_tokens_kwargs(system="s", cached_block="c", model="gpt-4o") + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_extract_text_blocks_returns_single_element_tuple() -> None: + """``extract_text_blocks`` pulls ``choices[0].message.content`` as a + single string and wraps it in a one-element tuple so the orchestrator's + downstream ``"".join(blocks)`` is provider-agnostic.""" + response = _FakeOpenAIResponse(content="hello world") + assert OpenAIProvider().extract_text_blocks(response) == ("hello world",) + + +@pytest.mark.unit +@pytest.mark.llm +@pytest.mark.parametrize( + "response", + [ + _FakeOpenAIResponse(choices=[]), # empty choices + _FakeOpenAIResponse(content=None), # message present but content None + ], +) +def test_openai_provider_extract_text_blocks_missing_structure_raises( + response: object, +) -> None: + """A response missing the expected structure surfaces a typed format + error (mirrors the ``AnthropicProvider`` guard).""" + from signalforge.llm.errors import LLMResponseFormatError + + with pytest.raises(LLMResponseFormatError): + OpenAIProvider().extract_text_blocks(response) + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_extract_text_blocks_missing_choices_attr_raises() -> None: + """A response object missing the ``choices`` attribute entirely raises + :class:`LLMResponseFormatError`.""" + from signalforge.llm.errors import LLMResponseFormatError + + class _NoChoices: + usage = _FakeOpenAIUsage(1, 1) + + with pytest.raises(LLMResponseFormatError): + OpenAIProvider().extract_text_blocks(_NoChoices()) + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_extract_usage_returns_usage_metrics() -> None: + """``extract_usage`` maps ``usage.prompt_tokens`` / + ``usage.completion_tokens`` to :class:`UsageMetrics` with both cache + fields fixed at 0 (OpenAI has no cache discount — + ``supports_prompt_caching=False``).""" + response = _FakeOpenAIResponse(prompt_tokens=120, completion_tokens=45) + usage = OpenAIProvider().extract_usage(response) + assert isinstance(usage, UsageMetrics) + assert usage.input_tokens == 120 + assert usage.output_tokens == 45 + assert usage.cache_creation_input_tokens == 0 + assert usage.cache_read_input_tokens == 0 + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_extract_usage_missing_usage_raises() -> None: + """A response missing ``usage`` surfaces a typed format error.""" + from signalforge.llm.errors import LLMResponseFormatError + + class _NoUsage: + choices = [_FakeChoice("x")] + + with pytest.raises(LLMResponseFormatError): + OpenAIProvider().extract_usage(_NoUsage()) + + +def _openai_api_status(message: str, status: int) -> BaseException: + """Construct an ``openai.APIStatusError`` with the given status code.""" + import openai + + return openai.APIStatusError( + message=message, response=httpx.Response(status, request=_OPENAI_REQ), body=None + ) + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_classify_exception_maps_each_category() -> None: + """Each OpenAI SDK exception type maps to the correct neutral category + (DEC-009 of #136); unrecognised exceptions map to NO_RETRY.""" + import openai + + provider = OpenAIProvider() + assert ( + provider.classify_exception( + openai.AuthenticationError( + message="auth", + response=httpx.Response(401, request=_OPENAI_REQ), + body=None, + ) + ) + is ExceptionCategory.AUTH + ) + assert ( + provider.classify_exception( + openai.PermissionDeniedError( + message="perm", + response=httpx.Response(403, request=_OPENAI_REQ), + body=None, + ) + ) + is ExceptionCategory.AUTH + ) + assert ( + provider.classify_exception( + openai.RateLimitError( + message="rl", + response=httpx.Response(429, request=_OPENAI_REQ), + body=None, + ) + ) + is ExceptionCategory.RATE_LIMIT + ) + assert ( + provider.classify_exception(openai.APIConnectionError(request=_OPENAI_REQ)) + is ExceptionCategory.CONNECTION + ) + # 5xx APIStatusError → SERVER_ERROR. + assert ( + provider.classify_exception(_openai_api_status("5xx", 503)) + is ExceptionCategory.SERVER_ERROR + ) + # 4xx-non-auth APIStatusError → NO_RETRY. + assert provider.classify_exception(_openai_api_status("4xx", 422)) is ExceptionCategory.NO_RETRY + # APIStatusError that is neither 5xx nor 4xx-non-auth (a 3xx) hits the + # defensive fallthrough → NO_RETRY. + assert provider.classify_exception(_openai_api_status("3xx", 302)) is ExceptionCategory.NO_RETRY + # Anything unrecognised maps to NO_RETRY. + assert provider.classify_exception(ValueError("unrecognised")) is ExceptionCategory.NO_RETRY + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_make_client_uses_shim(monkeypatch: pytest.MonkeyPatch) -> None: + """``make_client`` delegates to the shim's ``_make_openai_client`` so the + DEC-010 SDK-construction confinement holds.""" + import signalforge.llm._openai_client as shim + + sentinel = object() + monkeypatch.setattr(shim, "_make_openai_client", lambda: sentinel) + assert OpenAIProvider().make_client() is sentinel + + +@pytest.mark.unit +@pytest.mark.llm +def test_unknown_provider_error_lists_both_anthropic_and_openai() -> None: + """After US-002 registers ``OpenAIProvider``, an unknown name raises + :class:`UnknownProviderError` listing BOTH ``"anthropic"`` and + ``"openai"`` in its ``available`` tuple / message (DEC-003 of #135).""" + with pytest.raises(UnknownProviderError) as excinfo: + provider_for("xyz") + err = excinfo.value + assert err.name == "xyz" + assert "anthropic" in err.available + assert "openai" in err.available + rendered = str(err) + assert "anthropic" in rendered + assert "openai" in rendered + + +# --------------------------------------------------------------------------- +# Coverage-closing tests for #136 US-008 QG / PR #152 codecov gaps +# --------------------------------------------------------------------------- + + +@pytest.mark.unit +@pytest.mark.llm +def test_anthropic_provider_estimate_input_tokens_skips_system_kwarg_when_empty() -> None: + """``AnthropicProvider.estimate_input_tokens`` MUST omit the ``system=`` + kwarg entirely when ``system`` is the empty default — otherwise the SDK + would carry a spurious ``system=""`` block in the count, inflating the + figure by Anthropic's empty-system envelope size. + + Closes PR #152 codecov gap on the no-system branch + (``providers.py``: ``response = resolved.messages.count_tokens(...)`` + inside the ``else:`` arm). + """ + from tests.llm._fake import FakeAnthropicClient, FakeCountTokensResponse + + fake = FakeAnthropicClient(project="fake") + captured: dict[str, Any] = {} + + def _capture(kwargs: dict[str, Any]) -> bool: + captured.update(kwargs) + return True + + fake.expect_count_tokens(matching=_capture, returns=FakeCountTokensResponse(input_tokens=42)) + + tokens = AnthropicProvider().estimate_input_tokens( + "claude-sonnet-4-6", "hello world", system="", client=fake + ) + + assert tokens == 42 + assert "system" not in captured, ( + f"empty system MUST be omitted from count_tokens kwargs; got {captured.keys()}" + ) + fake.assert_all_expectations_met() + + +@pytest.mark.unit +@pytest.mark.llm +def test_anthropic_provider_estimate_input_tokens_raises_on_missing_input_tokens() -> None: + """A count_tokens response with a missing/non-int ``input_tokens`` field + raises :class:`LLMResponseFormatError` (defensive guard against an SDK + response-shape regression). + + Closes PR #152 codecov gap on the ``raise LLMResponseFormatError(...)`` + arm of ``AnthropicProvider.estimate_input_tokens``. + """ + from signalforge.llm.errors import LLMResponseFormatError + from tests.llm._fake import FakeAnthropicClient + + fake = FakeAnthropicClient(project="fake") + + # Queue a response object whose ``input_tokens`` attr is missing + # (a real SDK response always carries it as an int; a None return + # surfaces the guard). + class _NoInputTokens: + pass + + fake.expect_count_tokens(matching=lambda kwargs: True, returns=_NoInputTokens()) + + with pytest.raises(LLMResponseFormatError, match="input_tokens"): + AnthropicProvider().estimate_input_tokens( + "claude-sonnet-4-6", "hello", system="sys", client=fake + ) + + +@pytest.mark.unit +@pytest.mark.llm +def test_openai_provider_extract_text_blocks_missing_message_attr_raises() -> None: + """A choice object missing the ``message`` attribute (or with + ``message=None``) raises :class:`LLMResponseFormatError`. + + Distinct from the existing ``content=None`` / empty-choices cases + (those exercise different arms of ``extract_text_blocks``). Closes + PR #152 codecov gap on the ``raise LLMResponseFormatError(...)`` arm + that fires when ``getattr(first, "message", None) is None``. + """ + from types import SimpleNamespace + + from signalforge.llm.errors import LLMResponseFormatError + + # A choice with message=None (real SDK never produces this, but the + # guard is the contract — surface a typed error rather than crash + # later on the content attribute access). + response = SimpleNamespace(choices=[SimpleNamespace(message=None)]) + + with pytest.raises(LLMResponseFormatError, match="message"): + OpenAIProvider().extract_text_blocks(response) diff --git a/tests/llm/test_public_api.py b/tests/llm/test_public_api.py index 88474ea4..c53a3854 100644 --- a/tests/llm/test_public_api.py +++ b/tests/llm/test_public_api.py @@ -56,6 +56,8 @@ "provider_for", # Anthropic strategy (US-002 of #135) — registered at import time. "AnthropicProvider", + # OpenAI strategy (US-002 of #136) — registered at import time. + "OpenAIProvider", ) @@ -94,6 +96,7 @@ def test_each_public_name_is_importable_via_from_signalforge_llm() -> None: LLMResult, LLMServerError, ModelPricing, + OpenAIProvider, UnknownProviderError, UsageMetrics, call_llm, diff --git a/tests/test_audit_completeness.py b/tests/test_audit_completeness.py index 237fe9af..4b824c84 100644 --- a/tests/test_audit_completeness.py +++ b/tests/test_audit_completeness.py @@ -24,6 +24,15 @@ whose body issues ``os.write`` / ``os.fsync``; every writer function must use a short-write loop. The propagation IS the defence (safety-layer.md DEC-011, repeated in prune/grade/diff rules). +* **Scan 9** — ``openai.OpenAI(...)`` outside + ``signalforge.llm._openai_client`` (#136 DEC-010). NEW — not an + extension of Scan 3 (which is Anthropic-specific). Mirrors Scan 3's + shape: reuses :class:`_AttributeCallFinder` to detect the SDK + construction call regardless of import-alias bypasses; the companion + per-file ``# type: ignore`` confinement test + (``tests/llm/test_openai_client_confinement.py``) mirrors the + Snowflake shim's line-based check. Once #137 lands its Gemini + equivalent, the tally moves to 10. Each scan is its own test with an explicit, justified exclusion list. The scans are deterministic and cheap: each ``.py`` is read once via @@ -181,13 +190,38 @@ def __init__(self, obj_name: str, attr_name: str) -> None: self._direct_aliases: set[str] = set() self.calls: list[tuple[int, int]] = [] + def visit_Module(self, node: ast.Module) -> None: # noqa: N802 — ast API + # Two-pass walk on the module root: collect every alias FIRST, + # then visit normally to find Call hits. Closes the + # "call-before-import" bypass where a function body references + # an alias defined by a later top-level import (PR #152 + # CodeRabbit catch — mirrors the precedent in + # ``test_qualified_name_finder_catches_late_import_alias``). + # Without this, ``ast.NodeVisitor``'s depth-first walk visits + # the Call before the Import/ImportFrom and the alias map is + # empty when the Call is matched. + for sub in ast.walk(node): + if isinstance(sub, ast.Import): + for alias in sub.names: + if alias.name == self._obj_name: + self._obj_aliases.add(alias.asname or alias.name) + elif isinstance(sub, ast.ImportFrom) and sub.module == self._obj_name: + for alias in sub.names: + if alias.name == self._attr_name: + self._direct_aliases.add(alias.asname or alias.name) + self.generic_visit(node) + def visit_Import(self, node: ast.Import) -> None: # noqa: N802 — ast API + # Idempotent re-add on the second pass (alias already collected + # by ``visit_Module``). Kept so callers feeding a sub-module + # subtree (not the Module root) still register aliases. for alias in node.names: if alias.name == self._obj_name: self._obj_aliases.add(alias.asname or alias.name) self.generic_visit(node) def visit_ImportFrom(self, node: ast.ImportFrom) -> None: # noqa: N802 — ast API + # Same idempotent-on-second-pass note as ``visit_Import``. if node.module == self._obj_name: for alias in node.names: if alias.name == self._attr_name: @@ -882,6 +916,71 @@ def test_fail_closed_writers_use_short_write_loop() -> None: assert failures == [], "\n".join(failures) +# --------------------------------------------------------------------------- +# Scan 9 — openai.OpenAI only in llm._openai_client (#136 DEC-010) +# --------------------------------------------------------------------------- + + +# DEC-010 of #136: every OpenAI SDK ``# pyright: ignore`` / ``# type: ignore`` +# and the SDK construction call itself live in ``_openai_client.py``. This is +# a **new** scan, not an extension of Scan 3 — Scan 3 is Anthropic-specific +# (``anthropic.Anthropic(...)``). Both scans mirror the same shape because +# each vendor SDK gets its own shim per the one-shim-per-vendor convention +# (``.claude/rules/llm-drafter.md`` § "One SDK seam"). When #137 ships the +# Gemini equivalent, the tally bumps 9 → 10. +_LLM_OPENAI_EXCLUSIONS: set[str] = { + # _openai_client.py is the sole SDK seam: it lazy-imports + # ``openai`` and constructs ``openai.OpenAI(api_key=...)`` + # inside ``_make_openai_client``. + "_openai_client.py", +} + + +def test_openai_client_construction_only_in_llm_client_shim() -> None: + """DEC-010 of #136: ``openai.OpenAI(...)`` outside + ``signalforge.llm._openai_client`` violates the SDK-confinement + convention. The AST scan is stricter than the regex check in + ``tests/llm/test_openai_client_confinement.py`` (catches multi-line + / commented forms the regex would miss). + + Mirrors :func:`test_anthropic_client_construction_only_in_llm_client_shim` + verbatim with ``openai`` / ``OpenAI`` substituted. The + :class:`_AttributeCallFinder` handles all three bypass patterns + (bare ``from openai import OpenAI``, alias ``import openai as o``, + attribute ``openai.OpenAI(...)``). + """ + hits = _scan_dir_for_attribute_calls( + _LLM_DIR, + obj_name="openai", + attr_name="OpenAI", + excluded_relpaths=_LLM_OPENAI_EXCLUSIONS, + ) + formatted = "\n".join(f" {p}:{line}" for p, line in hits) + assert not hits, ( + "openai.OpenAI(...) constructed outside " + "signalforge.llm._openai_client:\n" + f"{formatted}\n" + "Construct only via _make_openai_client — DEC-010 of #136 confines " + "OpenAI-SDK noise to the shim." + ) + + +def test_openai_client_construction_in_llm_client_shim_is_present() -> None: + """Sanity: at least one ``openai.OpenAI(...)`` in + ``_openai_client.py``. If this fails the scan above is no longer + load-bearing. + """ + client_path = _LLM_DIR / "_openai_client.py" + tree = ast.parse(client_path.read_text(encoding="utf-8")) + finder = _AttributeCallFinder("openai", "OpenAI") + finder.visit(tree) + assert finder.calls, ( + "Expected openai.OpenAI(...) call in " + "signalforge.llm._openai_client — the AST-scan above is no longer " + "load-bearing if the legitimate constructor disappears." + ) + + # --------------------------------------------------------------------------- # Negative tests: confirm the AST visitors detect planted violations # --------------------------------------------------------------------------- @@ -967,6 +1066,59 @@ def test_qualified_name_finder_catches_all_three_bypass_patterns() -> None: ) +def test_attribute_call_finder_catches_all_three_openai_bypass_patterns() -> None: + """#136 DEC-010 planted-violation regression: Scan 9 must catch + each of the three bypass patterns for ``openai.OpenAI(...)``. + + Mirrors :func:`test_qualified_name_finder_catches_all_three_bypass_patterns` + in spirit but uses :class:`_AttributeCallFinder` (Scan 9's helper) + against synthetic source for each pattern. Per + ``testing-signal.md`` § "AST single-construction-seam scans must + catch all three bypass patterns" — a bare-name-only visitor is + trivially bypassable and provides false confidence. + """ + # Pattern 1: bare ``OpenAI(...)`` after ``from openai import OpenAI``. + bare_src = "from openai import OpenAI\ndef make():\n return OpenAI(api_key='x')\n" + bare = _AttributeCallFinder("openai", "OpenAI") + bare.visit(ast.parse(bare_src)) + assert len(bare.calls) == 1, ( + "Pattern 1 (bare `from openai import OpenAI; OpenAI(...)`) not detected — " + "_AttributeCallFinder regressed." + ) + + # Pattern 2: import-alias ``from openai import OpenAI as O``. + alias_src = "from openai import OpenAI as O\ndef make():\n return O(api_key='x')\n" + alias = _AttributeCallFinder("openai", "OpenAI") + alias.visit(ast.parse(alias_src)) + assert len(alias.calls) == 1, ( + "Pattern 2 (import-alias `from openai import OpenAI as O; O(...)`) not detected — " + "_AttributeCallFinder regressed." + ) + + # Pattern 3: module-attribute ``import openai; openai.OpenAI(...)``. + attr_src = "import openai\n\nx = openai.OpenAI(api_key='x')\n" + attr = _AttributeCallFinder("openai", "OpenAI") + attr.visit(ast.parse(attr_src)) + assert len(attr.calls) == 1, ( + "Pattern 3 (module-attribute `import openai; openai.OpenAI(...)`) not detected — " + "_AttributeCallFinder regressed." + ) + + # Pattern 4: late-import alias — call appears in source order BEFORE + # the import that defines its alias. Single-pass alias collection + # misses this; ``_AttributeCallFinder.visit_Module`` does a two-pass + # walk to close the bypass (PR #152 CodeRabbit catch; mirrors the + # ``_QualifiedNameCallFinder`` regression test below). + late_alias_src = "def make():\n return O(api_key='x')\nfrom openai import OpenAI as O\n" + late = _AttributeCallFinder("openai", "OpenAI") + late.visit(ast.parse(late_alias_src)) + assert len(late.calls) == 1, ( + "Pattern 4 (late-import alias: call appears before its `from openai import OpenAI " + "as O` line) not detected — _AttributeCallFinder regressed; the two-pass " + "`visit_Module` alias collection is the gate that closes this bypass." + ) + + def test_qualified_name_finder_catches_late_import_alias() -> None: """Regression for the late-import bypass CodeRabbit flagged on PR #69: a function body that references an alias defined by a later diff --git a/uv.lock b/uv.lock index cccd2dcf..c851dc20 100644 --- a/uv.lock +++ b/uv.lock @@ -1385,6 +1385,25 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/88/b2/d0896bdcdc8d28a7fc5717c305f1a861c26e18c05047949fb371034d98bd/nodeenv-1.10.0-py2.py3-none-any.whl", hash = "sha256:5bb13e3eed2923615535339b3c620e76779af4cb4c6a90deccc9e36b274d3827", size = 23438, upload-time = "2025-12-20T14:08:52.782Z" }, ] +[[package]] +name = "openai" +version = "2.38.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "anyio" }, + { name = "distro" }, + { name = "httpx" }, + { name = "jiter" }, + { name = "pydantic" }, + { name = "sniffio" }, + { name = "tqdm" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8f/12/cfa322c5f5dd8fa21aab9a7a8e979e7a11123800f86ca8d82eb68a83d213/openai-2.38.0.tar.gz", hash = "sha256:798694c6cf74145541fda94325b6f8f72d8e1fd0262cc137c8d728177a6a4ce3", size = 772764, upload-time = "2026-05-21T21:23:42.105Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0a/bf/ccff9be562e24207716d04ef9dc931c76aff0c89a7265da43e2104d7fe06/openai-2.38.0-py3-none-any.whl", hash = "sha256:ec6661c57b2dcc47414a767e6e3335c7ed3d19c9696999283a3c82e95c756a3c", size = 1344910, upload-time = "2026-05-21T21:23:39.636Z" }, +] + [[package]] name = "orderly-set" version = "5.5.0" @@ -1891,6 +1910,110 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" }, ] +[[package]] +name = "regex" +version = "2026.5.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/dc/0e/49aee608ad09480e7fd276898c99ec6192985fa331abe4eb3a986094490b/regex-2026.5.9.tar.gz", hash = "sha256:a8234aa23ec39894bfe4a3f1b85616a7032481964a13ac6fc9f10de4f6fca270", size = 416074, upload-time = "2026-05-09T23:15:19.37Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c2/dc/c1f2df4027e82fc54b5a473e4b250f5139faca49a0fbe29a48668d228f34/regex-2026.5.9-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:ccf5249114cc3e772ecdd88a98a86eca0fd74c61ce32a94743758c083fc05d48", size = 489445, upload-time = "2026-05-09T23:12:06.111Z" }, + { url = "https://files.pythonhosted.org/packages/03/d2/59f01110660081cce9c0bc30ebd0b5ee250dacf658e3248ed92f01e0e8ee/regex-2026.5.9-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:46f1326ca6e65b0879d23ca302c0f2415aad42ff0309b9c818e7949fe19a41d8", size = 291271, upload-time = "2026-05-09T23:12:07.731Z" }, + { url = "https://files.pythonhosted.org/packages/58/b6/14b2c84ff90ddb370c81d27503f4a0fcf071496416f4855f6cc8c5d81c35/regex-2026.5.9-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:ef31cbfe458e21c6122ba8150ff060e0c7789ed0d26eb423f25472584920b555", size = 289212, upload-time = "2026-05-09T23:12:09.266Z" }, + { url = "https://files.pythonhosted.org/packages/03/d0/4db86529117320de0c84afd90e70bb47434625875e34fcef9d8c127c5b16/regex-2026.5.9-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:992604d02e6d9c6d786c24a706a71ecffe1020fc1ef264044474cd81fa2c3919", size = 792310, upload-time = "2026-05-09T23:12:11.416Z" }, + { url = "https://files.pythonhosted.org/packages/07/78/fe4800cd322f862ecffd2d553409b20d80650e5ed71b9d178f853d020b82/regex-2026.5.9-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c9411dd64ca95477225734a93dfc8583b51916b8d5942f99d6cac21e09965451", size = 861721, upload-time = "2026-05-09T23:12:13.681Z" }, + { url = "https://files.pythonhosted.org/packages/b5/d0/b3618a895dd8feb897c61bb2954edd265e1767d82a01d53065d5871127a3/regex-2026.5.9-cp311-cp311-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3dd4a3ff360dfb836fecdb93a4598f9d6e2ac81e3e397125145c6221bf58cf4c", size = 906460, upload-time = "2026-05-09T23:12:15.443Z" }, + { url = "https://files.pythonhosted.org/packages/33/6f/1481597e859ef19508b345eec4afd1416ed6e6b459c75a64026ef193aecf/regex-2026.5.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2a661a7d270a61f7cf460caee8b9fa2d5ef9e5c681234bcb9e0fe14f488e7dfc", size = 799843, upload-time = "2026-05-09T23:12:16.892Z" }, + { url = "https://files.pythonhosted.org/packages/73/59/955734c803f59108deccba3597ae440c76b62a652733c0006e6243758420/regex-2026.5.9-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f079e50a0d3cc3cd5091fa9ff45869a2e6b2cd35895731edafb0327901a8d86d", size = 773610, upload-time = "2026-05-09T23:12:19.127Z" }, + { url = "https://files.pythonhosted.org/packages/68/8f/70c04a236d651c81881dac42ef8538bddda6121434509d0a22d9e601503b/regex-2026.5.9-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:4ebe8f0b5ec5a5024dc4a4c59f444c4e9afc5f2abdbb8962065b75d27fb971f9", size = 781645, upload-time = "2026-05-09T23:12:20.806Z" }, + { url = "https://files.pythonhosted.org/packages/1d/96/05c7434d88185e5d27fe54aeb74df86bd77cd79f52f0b4eae54faa8fea70/regex-2026.5.9-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:97cf3bc1b7d7d2306772ec07366c80d9df00ff79e79cea32898883a646d2fae2", size = 854473, upload-time = "2026-05-09T23:12:22.465Z" }, + { url = "https://files.pythonhosted.org/packages/4e/c1/6e3d8202d981f3117004bf341ee74893ba4ba8a9fbaf4b94615846550a08/regex-2026.5.9-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:0f9eede6a5cbdc02d4978090186390936e1776a7d1359b21e41014c609880bcf", size = 763311, upload-time = "2026-05-09T23:12:24.351Z" }, + { url = "https://files.pythonhosted.org/packages/93/c7/e7737f1526b3fb32bd4c337fd6c71c3ebb5c8296fc34d11197e0955d2e35/regex-2026.5.9-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:01f0f5f55f4b64dacec85dc116d3c05fd23ad3ff037bbc73a2085775953c2611", size = 844593, upload-time = "2026-05-09T23:12:26.341Z" }, + { url = "https://files.pythonhosted.org/packages/a5/27/0daffb1a535bb39f422c3d200f4ab023c71110ad66a32b366bee708baba0/regex-2026.5.9-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:1268eddd8486dc561d08eee1156e40aa3a8fe10f4bdec8fa653b455fcbffd12c", size = 789167, upload-time = "2026-05-09T23:12:27.975Z" }, + { url = "https://files.pythonhosted.org/packages/ce/fc/294fe4fac4f2ed67207b17471815870c1c45b3a489e08e0ac96daea16ef6/regex-2026.5.9-cp311-cp311-win32.whl", hash = "sha256:8676474c07469d6f33dd1085ca2cd45f65785f32518f2b20e36d9953ca07f994", size = 266249, upload-time = "2026-05-09T23:12:30.141Z" }, + { url = "https://files.pythonhosted.org/packages/d0/b0/8dce459f6245bcf8f6e9f23ac9569f1a0f15c131cc0745e82b43226204cf/regex-2026.5.9-cp311-cp311-win_amd64.whl", hash = "sha256:246de9d60aa3f8538b519834dd95cbf276ea263d6a7bd5a3666dc3fa0230505b", size = 278423, upload-time = "2026-05-09T23:12:31.676Z" }, + { url = "https://files.pythonhosted.org/packages/db/8d/f9aeff6ad63a3ef720386f2907e6d34a35a510a6e498ebad28b0fb3f6ab6/regex-2026.5.9-cp311-cp311-win_arm64.whl", hash = "sha256:d726ca3f0d76969bf1e8e477d160d3d666bbf999f6860bd314889e5345782046", size = 270420, upload-time = "2026-05-09T23:12:33.194Z" }, + { url = "https://files.pythonhosted.org/packages/50/9b/6550044bc44e17c84d312c031c2ec42fbdb6a4ec4e29093be3a172d08772/regex-2026.5.9-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:57eeeb05db7979413dec5438f2db21d7ecbba787cde7a711df1a6f6df672aa06", size = 490451, upload-time = "2026-05-09T23:12:34.72Z" }, + { url = "https://files.pythonhosted.org/packages/1e/95/fc7ba4303b5a0f92446a12ee6778ef2c6c799233f5060042a31bf390cfe9/regex-2026.5.9-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:398c521292f4c7fb807001dcd54694d3a1fcafc179a36ad9cc56f98df85930b6", size = 292112, upload-time = "2026-05-09T23:12:36.285Z" }, + { url = "https://files.pythonhosted.org/packages/54/4b/ee27938d1b2c443e89a9a10e00d2d19aa5ee300cd3d61140644e93bb083e/regex-2026.5.9-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f7a7c26137296beba7784de6eba69c6a93a63ccebc385e4962fe67e267a91225", size = 289599, upload-time = "2026-05-09T23:12:38.089Z" }, + { url = "https://files.pythonhosted.org/packages/d8/dd/ba103dc19614e25f3880800ca67ce093d6e21b325d72b8383c7bf906e9fa/regex-2026.5.9-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6441cc660d76107934a09c22167200839a0e89604a6297f78a974e66e931d2c0", size = 796732, upload-time = "2026-05-09T23:12:40.062Z" }, + { url = "https://files.pythonhosted.org/packages/cf/e7/f035b4fd858b050b0080bf302968dc0f59ba34e391872d54936758e6844e/regex-2026.5.9-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:91328f1c23d47595ca3ef0a7557fa129c5a23404b775c770697d2f35b33e0107", size = 865440, upload-time = "2026-05-09T23:12:42.059Z" }, + { url = "https://files.pythonhosted.org/packages/0a/51/8cd301ecc899aea28124357f729f4272f44de7806fc7ca02490bfbe253e8/regex-2026.5.9-cp312-cp312-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:93a7860539414dddaefba2b40f8771765ae17949d4c7182b876ce429e11a8309", size = 912329, upload-time = "2026-05-09T23:12:44.373Z" }, + { url = "https://files.pythonhosted.org/packages/cc/1e/3fbe2fa1e8cebd62f3bb7d3321cff1640aca2e240b51d9bd624aad949260/regex-2026.5.9-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dd2810d22146b6d838acc5ec15602cb6b47920aa4e33015df3868eedfd20bab8", size = 801239, upload-time = "2026-05-09T23:12:46.268Z" }, + { url = "https://files.pythonhosted.org/packages/17/2f/6f6008682bf2cf98040a0d3153a8e557b6ab728d7713d045cee4ce544ab8/regex-2026.5.9-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:daff2bdbaf1d23e52fdff7c0b7bc2048b68f978df6a4d107ac981f94caef2e66", size = 777054, upload-time = "2026-05-09T23:12:48.051Z" }, + { url = "https://files.pythonhosted.org/packages/19/2b/eee0d20a6842ba04df4b8847a920b57ef56853f14ef85405473e586b605a/regex-2026.5.9-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:4eeb011098fcb77af513dcef521a3dbecbf8849b1e38940759d293b7a93f5026", size = 785098, upload-time = "2026-05-09T23:12:49.851Z" }, + { url = "https://files.pythonhosted.org/packages/4a/98/6fc1e6410feefb92159edaed5041992bfe390e8d26c721865434acbca558/regex-2026.5.9-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:ea9c8ecfa1b73c73b626534d6626e5340d429630943672b8480724f44e84b962", size = 860095, upload-time = "2026-05-09T23:12:51.666Z" }, + { url = "https://files.pythonhosted.org/packages/18/a3/bd855e0f2cb1a978ecf6fa6bb69632dd9c3f6ea3b81cde62fde14c9daec7/regex-2026.5.9-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:cd2846168eb9ee3c513902bc8225409cb1caab31d04728b145171fa1625d9621", size = 765762, upload-time = "2026-05-09T23:12:53.413Z" }, + { url = "https://files.pythonhosted.org/packages/dc/66/0ae8c092e60b14c79d24f8e0b7f0aea5bfbffdcab00b5483d13404d3c3a5/regex-2026.5.9-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:39617fb0cde9c0e6306dc70e3bfc096f3da793219879f7ae7aa341a69fbdcf6d", size = 852100, upload-time = "2026-05-09T23:12:55.256Z" }, + { url = "https://files.pythonhosted.org/packages/21/de/8dfde60fc1b21c946a893ba273403b72617edb261370cb1087099a83f088/regex-2026.5.9-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:fd03c4f0e33280d15cae17159b899245d6b7c53d21def19b263b39655061f5ce", size = 789479, upload-time = "2026-05-09T23:12:57.573Z" }, + { url = "https://files.pythonhosted.org/packages/c3/1c/bdcc98f9a4af4fdd166c74941174619ccff4726d3ce32faa8e9a2ecd38dd/regex-2026.5.9-cp312-cp312-win32.whl", hash = "sha256:164eba9b755ea6f244b0d881196fbc1fac09714e9782c9e2732b813142033c8e", size = 266699, upload-time = "2026-05-09T23:12:59.14Z" }, + { url = "https://files.pythonhosted.org/packages/78/87/240d36864f9e48ace85f72e79ced97ceb7f27ce87739a947dcb834b4e6bc/regex-2026.5.9-cp312-cp312-win_amd64.whl", hash = "sha256:86f40a5d6444db30a125c9c9177e6b25dad981cbc37451fd838f145e6edac92e", size = 277783, upload-time = "2026-05-09T23:13:00.789Z" }, + { url = "https://files.pythonhosted.org/packages/4f/b5/7b30f312b0669dff5beebe5b0989dc2d1a312b1a44fab852199c387a5b96/regex-2026.5.9-cp312-cp312-win_arm64.whl", hash = "sha256:96f5f58b54a063d7ea9dca08e1cf57bfe10499c4d579ee672da284f57f5f0070", size = 270513, upload-time = "2026-05-09T23:13:02.426Z" }, + { url = "https://files.pythonhosted.org/packages/aa/da/797e91ecec6f84135da778ddce78c20e0af5d2a15c26f87a81bc3eadb6db/regex-2026.5.9-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:d626b84406444b165fc0ba981604edea39f0588ff1f92baa23fe50799ea9afdb", size = 490303, upload-time = "2026-05-09T23:13:04.382Z" }, + { url = "https://files.pythonhosted.org/packages/44/da/bf30abaaa737b58f4a4b8c4a03659e02fd92092c822e0197ed9e0daab917/regex-2026.5.9-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d7bdc0ab8f3dd7e1b4f9ab88634e13374669db86bb3c72e8292f07ae313f539f", size = 292019, upload-time = "2026-05-09T23:13:06.022Z" }, + { url = "https://files.pythonhosted.org/packages/2d/e7/d0eaf5713828417b9e5648cf81fa9bacd4961f6ab98c380c2034f8716e35/regex-2026.5.9-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a8820737949116ffff55fe18f9fc644530063ba6ebfcb8314239416e78f1347c", size = 289468, upload-time = "2026-05-09T23:13:08.214Z" }, + { url = "https://files.pythonhosted.org/packages/d3/9b/b3fdd62b003baa1a9b593cd8c8699c9651c2e80cc21a5c715707983c42d7/regex-2026.5.9-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:aa0fbdbac82cb3e4450d0ccde7d7a35607f4cb2dd9fba4b8b69bfaf8c9fa6aed", size = 796749, upload-time = "2026-05-09T23:13:10.573Z" }, + { url = "https://files.pythonhosted.org/packages/d4/30/66ab84588765f5b4b271a9ca09ef7ce2b87caa95176ec3d2ad65d7bc4902/regex-2026.5.9-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:57e8915c7986aa33d25e4d3629cef711cd2863f2961b10409f0c04cb8b7d9020", size = 865445, upload-time = "2026-05-09T23:13:12.523Z" }, + { url = "https://files.pythonhosted.org/packages/1a/89/f05169e8588aac365f35ffc7f3bc3184f095ef4cfded7cfaa3c7fd5dbd89/regex-2026.5.9-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:508f56a89ba9cb26e4168cbc37dbd60a28d82430a9e18ad1d25fe0883c314ca2", size = 912322, upload-time = "2026-05-09T23:13:14.281Z" }, + { url = "https://files.pythonhosted.org/packages/30/e1/c93444052cf41581f3c884ab3fb5823daf0992f11cd4388d4275ca610558/regex-2026.5.9-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b6d189041f15691cfa2b6c4290448ec221244d225b3f5fe9e7771b34ffcdf6e2", size = 801269, upload-time = "2026-05-09T23:13:16.569Z" }, + { url = "https://files.pythonhosted.org/packages/50/fe/0cf96b882f540e62e8b9956599798203d599c44cf4c77917ca27400ff69b/regex-2026.5.9-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e82db382b44d0111b22601c509c89f64434816c9e0eef9d1989cda8cc6ff1c04", size = 777085, upload-time = "2026-05-09T23:13:18.675Z" }, + { url = "https://files.pythonhosted.org/packages/23/5c/d78d4924e7fc875557b9e9b768423925fdfaac5549d06da7810019a9bd26/regex-2026.5.9-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2acfb48634f64996b57f90f39afa692ff362162722581921fe92239a59960f3c", size = 785153, upload-time = "2026-05-09T23:13:20.525Z" }, + { url = "https://files.pythonhosted.org/packages/bf/e0/5214774090e7b4524dcea3e3c4aa74141d43043f8beb49c1599db1c8b53a/regex-2026.5.9-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:d29eebfc9525db68cad3c97eedd7f754fa265aa5cd0cf4f863b2421e1b48fc9f", size = 860164, upload-time = "2026-05-09T23:13:22.263Z" }, + { url = "https://files.pythonhosted.org/packages/6e/e1/4a57a83350319b1271f0d7a249b8672513ed928b237a741631270de6caea/regex-2026.5.9-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:debb893095e944091c16e641a6e33c1b0f4cb61ab945ec5afbf53ce7068834d8", size = 765731, upload-time = "2026-05-09T23:13:24.277Z" }, + { url = "https://files.pythonhosted.org/packages/12/f4/499e74a20c156fc75836ee04a72a38d1a063978f600937f9760467beb1b0/regex-2026.5.9-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:d659eee77986549c9ea45b861c7567e44d6287c3dc9a4565478853f7b9fe2ff6", size = 852062, upload-time = "2026-05-09T23:13:26.125Z" }, + { url = "https://files.pythonhosted.org/packages/5b/92/7eebc0d0a01e78629695f342ba17e0deaff8fb45e79cc0d7b98287da6e3e/regex-2026.5.9-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:2efa205e6d98b24d1f3ab395c11aa15cdf10935bca283d0285e0499c284fba21", size = 789577, upload-time = "2026-05-09T23:13:27.814Z" }, + { url = "https://files.pythonhosted.org/packages/05/a4/018e71f7d2ad48c1ebe6d3ae0026f9b7cb4802fd15c7cc02fdf724355102/regex-2026.5.9-cp313-cp313-win32.whl", hash = "sha256:f3844f134e834076677dd369976e9f5068679fcb8e50102fdf6b7ac96a3ec127", size = 266691, upload-time = "2026-05-09T23:13:29.549Z" }, + { url = "https://files.pythonhosted.org/packages/e6/1d/861a93719fb9ee7dbfc3761b3797b7a3e112a5d42c6129459d2d741be9b5/regex-2026.5.9-cp313-cp313-win_amd64.whl", hash = "sha256:3527bb4942d2c14552155406cdedd906567456821848aed1cb4933a391bf5eca", size = 277747, upload-time = "2026-05-09T23:13:31.859Z" }, + { url = "https://files.pythonhosted.org/packages/d9/c6/0a2436ae4da1ba76e51cb98943c6838a9a721faa40ebe2dce07694ae34e3/regex-2026.5.9-cp313-cp313-win_arm64.whl", hash = "sha256:56a33f191f17d8c417f99945ebdc1e691d3af9605d86ec68c7e54a57e3e17af6", size = 270500, upload-time = "2026-05-09T23:13:33.525Z" }, + { url = "https://files.pythonhosted.org/packages/e8/e9/d21346f7b60ed58789371358ed66b09d00f832e1bd7c06e55d9da5679882/regex-2026.5.9-cp313-cp313t-macosx_10_13_universal2.whl", hash = "sha256:01f28d868834624c934b8d2e0aa1c8341337e37831f4a012f18a5afcba4cbaf3", size = 494172, upload-time = "2026-05-09T23:13:35.935Z" }, + { url = "https://files.pythonhosted.org/packages/c4/43/fd1177a2032037c681baecdb3422ee4e1424aec4e4f470ef47793d325274/regex-2026.5.9-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:48036f6374aaa79eb3b754ec29c61d1c6b1606749d705a13f8854fa2539671f6", size = 293952, upload-time = "2026-05-09T23:13:38.307Z" }, + { url = "https://files.pythonhosted.org/packages/f2/7d/9fbf919768368d3f8a4f6c692cf2aa61e482b2b81ec6a298ace4cbf02480/regex-2026.5.9-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:b96350aa424e79d4fd6b567b344dcbe2b2d6bfc48dfe7717587e1fa6d43da6ff", size = 292314, upload-time = "2026-05-09T23:13:40.353Z" }, + { url = "https://files.pythonhosted.org/packages/e2/6c/e41bfeecb589716843e7c4df09ba46ff2a42961457afece19059d85caeef/regex-2026.5.9-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8f3af7a4903c5c04a11a196a5aa75cdd7dd3f8508132f9fb3259d9f5908e3b88", size = 811681, upload-time = "2026-05-09T23:13:42.543Z" }, + { url = "https://files.pythonhosted.org/packages/87/83/a5c1c525fba0aa656e88ad0face0b1829788ef4c2fb6b26df58aa1151b84/regex-2026.5.9-cp313-cp313t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:7e87577720152d2caae19fe2baaf1f8d5ca12091e9e229f03915c37d1e4b9178", size = 871135, upload-time = "2026-05-09T23:13:44.326Z" }, + { url = "https://files.pythonhosted.org/packages/18/d4/80882e799e440dd878b0979cbebf8fa4d54624a332c83037c7a701649e3f/regex-2026.5.9-cp313-cp313t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c8b9b9d294cfea3cd19c718ade7cc93492b2c4991abd9a68d0b3477ae6d8e100", size = 917265, upload-time = "2026-05-09T23:13:47.295Z" }, + { url = "https://files.pythonhosted.org/packages/ae/ff/8db60211e2286e396aad7dc7725356c502bff0901ea05bd6cdc2e1a042b9/regex-2026.5.9-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:728d8bfd28a8845c8b6bc5dc7ce010453d206396786c0765c2740cb65f37791e", size = 816311, upload-time = "2026-05-09T23:13:49.885Z" }, + { url = "https://files.pythonhosted.org/packages/4c/47/742ef579c61730f8d268e5cf1f9ce0e37e2ea041ad0f5644724f2378e463/regex-2026.5.9-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:7e30b874d341fac767d7df5a0870540541c2c054b80cfaac116e8d367a8a7ff2", size = 785498, upload-time = "2026-05-09T23:13:52.25Z" }, + { url = "https://files.pythonhosted.org/packages/7f/ab/cb0999802dcb0fb95b1ab005e8d4163d8afdd67efc2cb6b6630ac13f8cb1/regex-2026.5.9-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:fd190e88a895a8901325fad284a3f74ea52b1da8525b76cc811fa9b1edf0ce2b", size = 801348, upload-time = "2026-05-09T23:13:54.127Z" }, + { url = "https://files.pythonhosted.org/packages/7d/62/8ca59a24c55bc34d166eefaf3717bd77772f329fdbf984d86581e0a3571c/regex-2026.5.9-cp313-cp313t-musllinux_1_2_ppc64le.whl", hash = "sha256:8e76e8161ad00694cfce6767d5dea860c6391ac5b83e5c3a39661e696f11fc7e", size = 866493, upload-time = "2026-05-09T23:13:56.067Z" }, + { url = "https://files.pythonhosted.org/packages/8d/3d/30f2ae62cef3278bb5bb821f467277a55fb73f01032cf85997e15e8289a8/regex-2026.5.9-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:ddda5340e6c01a293027dd46232fa79eaff1b48058ce7a98f572b6445b088041", size = 772811, upload-time = "2026-05-09T23:13:57.867Z" }, + { url = "https://files.pythonhosted.org/packages/d8/ae/7d2089bcd78ad0c0161bc684339df50032acb438a7bd3305e7ddb1193cec/regex-2026.5.9-cp313-cp313t-musllinux_1_2_s390x.whl", hash = "sha256:205109e96b3cf5adf8f4cd62bedde9487feb282b9497a3535451e5a24cd706a0", size = 856584, upload-time = "2026-05-09T23:13:59.679Z" }, + { url = "https://files.pythonhosted.org/packages/a9/29/92ff47f75990131ea4f24ba17819e5a9d141e10819807e09addd73409af6/regex-2026.5.9-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:dfbe4579b9f08036aa7d101d1835437a20783574ac66327e6b29b4018a138081", size = 803453, upload-time = "2026-05-09T23:14:01.978Z" }, + { url = "https://files.pythonhosted.org/packages/04/99/eff29f1037dcab36702c9ee5d6858cf1ce2336ea8ea2987f64245b99ea5e/regex-2026.5.9-cp313-cp313t-win32.whl", hash = "sha256:ed2c9e8068b614c574d8d30e543d617cf5379b0535d46f97ef00e904745a08b5", size = 269951, upload-time = "2026-05-09T23:14:03.661Z" }, + { url = "https://files.pythonhosted.org/packages/0e/9d/8870b8981d27b22cda77bb26a5ac7ebfa9c7d9e0dea195a834a82380e748/regex-2026.5.9-cp313-cp313t-win_amd64.whl", hash = "sha256:b46b0f094dc1d3b90356c85a0bd2c9bafc4a6a190b9d6f8ddd5a033b6e088ed4", size = 281240, upload-time = "2026-05-09T23:14:05.56Z" }, + { url = "https://files.pythonhosted.org/packages/72/b1/3379415e8f135c13ac551353397cc4fe97b4978f3cac73c5fcbcded548b8/regex-2026.5.9-cp313-cp313t-win_arm64.whl", hash = "sha256:872acc074bd29ffc9913ecdfedf6ea77502312ca44a4aa0d3779089c6069d8de", size = 272383, upload-time = "2026-05-09T23:14:07.843Z" }, + { url = "https://files.pythonhosted.org/packages/13/3e/9c3cd292d8808b3645a2ce517e200179b6d0e903f176300bd8b542e14de5/regex-2026.5.9-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:1bd7587a2948b4085195d5a3374eaf4a425dc3e55784c038175355ecf3bbbf8a", size = 490376, upload-time = "2026-05-09T23:14:09.64Z" }, + { url = "https://files.pythonhosted.org/packages/60/70/d43ee8a2ca0a8b68d167f21658b85520ac0574617c7f320367c5047f7556/regex-2026.5.9-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:dea2e88e1cce4522496cce630e11e67b98b7076620bc4336c3f674bc21a375f4", size = 291964, upload-time = "2026-05-09T23:14:11.424Z" }, + { url = "https://files.pythonhosted.org/packages/21/91/9d50b433828d8e74196904e168a43abf1e6e88b2a15d47ed742456720c37/regex-2026.5.9-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:2099f7e7ff7b6aa3192312650a56e91cc091e49d50b04e4f6f8b6e28b3b27f1c", size = 289682, upload-time = "2026-05-09T23:14:13.123Z" }, + { url = "https://files.pythonhosted.org/packages/3e/d2/b835e3cafbb9d977736912436259ff551d60919f7d7b3d37d46659c63564/regex-2026.5.9-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ecd353045824e4477562a2ac718c25799cdaaa41f7aa925a806a8a3e6848a5b9", size = 796996, upload-time = "2026-05-09T23:14:14.923Z" }, + { url = "https://files.pythonhosted.org/packages/2c/a6/9f992d00019166b9de01c546dd4549bc679f2a68df11b877740b0760b7c2/regex-2026.5.9-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:65c8c8c37377794bd5b2f3ebe51919042bf17aec802e23c833d89782ed0c78af", size = 866089, upload-time = "2026-05-09T23:14:17.757Z" }, + { url = "https://files.pythonhosted.org/packages/e0/08/4d32af657e049b19cb62b02e46e38fe1518797bfb2203ee93a510b21b0dc/regex-2026.5.9-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:5b73ab8afcf66c622db143d1c6fda4e58e4d537ee4f125229ad47b1ab80f34c0", size = 911530, upload-time = "2026-05-09T23:14:20.353Z" }, + { url = "https://files.pythonhosted.org/packages/d9/27/2af43dd1dc201d1fecefda64a45f4ad0995855b92724f795a777b402ee69/regex-2026.5.9-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0de5cf193997384ed2ca6f1cd4f78055b255d93d82d5a8cd6ba0d11c10b167e4", size = 800643, upload-time = "2026-05-09T23:14:22.265Z" }, + { url = "https://files.pythonhosted.org/packages/a4/dd/23a249047013b5321d4a60c4d2437462086f601b061776a525e5fba2a59f/regex-2026.5.9-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d641a8c9a61618047796d572a39a79b26167b0411d2c3031937b2fe2d081e2cf", size = 777223, upload-time = "2026-05-09T23:14:24.179Z" }, + { url = "https://files.pythonhosted.org/packages/94/6a/e85ed9538cd19586d0465076a4578a12e093ce776d15f3f8ce92733a8dd6/regex-2026.5.9-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:24b2355ef5cc9aa5b8f07d17704face1c166fdcc2290fa7bd6e6c925655a8346", size = 785760, upload-time = "2026-05-09T23:14:26.065Z" }, + { url = "https://files.pythonhosted.org/packages/2a/c4/f25473209438638e947c55f9156fd8f236f74169229028cc99116380868e/regex-2026.5.9-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:a24852d3c29ad9e47593593d8a247c44ccc3d0548ef12c822d6ed0810affe676", size = 860891, upload-time = "2026-05-09T23:14:28.17Z" }, + { url = "https://files.pythonhosted.org/packages/f9/f7/f4f86e3c74419c37370e91f150ae0c2ef7d34b2e0e4cdd5da046a02e4022/regex-2026.5.9-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:916714069da19329ef7de197dcbc77bb3104145c7c2c864dbfbe318f46b88b14", size = 765891, upload-time = "2026-05-09T23:14:30.06Z" }, + { url = "https://files.pythonhosted.org/packages/26/70/704d8e13765939146b1cd0ef4e2feb71d7929727d2290f026eed10095955/regex-2026.5.9-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:fa411799ca8da32a8d38d020a88faa5b6f91657d284761352940ecf9f7c3bbdd", size = 851380, upload-time = "2026-05-09T23:14:32.123Z" }, + { url = "https://files.pythonhosted.org/packages/26/29/1a13582a8460038edc38e49f64ceb0dd7c60f5caba77571f4bf6601965d9/regex-2026.5.9-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:1e6da47d679b7010ef27556b6e0f99771b744936db1792a10ceac6547ae1503e", size = 789350, upload-time = "2026-05-09T23:14:34.799Z" }, + { url = "https://files.pythonhosted.org/packages/73/56/3dcafe34fc72e271d62ad9a291801e88a1457bb251c132f15fcc2e5aad1a/regex-2026.5.9-cp314-cp314-win32.whl", hash = "sha256:98bd73080e8756255137e1bd3f3f00295bbc5aa383c0e0f973920e9134d7c4ad", size = 272130, upload-time = "2026-05-09T23:14:36.729Z" }, + { url = "https://files.pythonhosted.org/packages/d0/9c/02eebf0be95efe416c664db7fb8b6b05b7a0b06a7544f2884f2558b0526f/regex-2026.5.9-cp314-cp314-win_amd64.whl", hash = "sha256:ff8d372ac2acdc048d1c19916f27ee61bc5722728458ba6ca5052f2c72d51763", size = 280999, upload-time = "2026-05-09T23:14:39.126Z" }, + { url = "https://files.pythonhosted.org/packages/70/5a/1dd1abee76cb7a846a0bcf42fdc87e5720c3c33c24f3e37814310a513d9f/regex-2026.5.9-cp314-cp314-win_arm64.whl", hash = "sha256:e1d93bf647916292e8edcec150c07ddf3dc50179ccaf770c04a7f9e452155372", size = 273500, upload-time = "2026-05-09T23:14:41.059Z" }, + { url = "https://files.pythonhosted.org/packages/86/c1/c5f619b0057a7965cb78ec559c1d7a45ce8c99a35bea95483d64959a93d9/regex-2026.5.9-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:83d0ee4a57d1c87cb549e195ec300b8f0ec3a82eba66d835e4e2ed8634fe4499", size = 494269, upload-time = "2026-05-09T23:14:42.869Z" }, + { url = "https://files.pythonhosted.org/packages/05/2c/5d01f1aee33de4bbe60c8452945bfc8477ca7c5ae4450f6bfe711036cb36/regex-2026.5.9-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:d3d7eb5c9a7f6df82ed3cfac9beb93882a5cbcb5b8b157b56cb2b3b276574ac1", size = 293954, upload-time = "2026-05-09T23:14:44.822Z" }, + { url = "https://files.pythonhosted.org/packages/7a/fe/e8988b2ae2108c6ef71bd4aa8d87fbe257976dd0810e826cd75f701c68b6/regex-2026.5.9-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:075160bf16658e16d35233300b8453aac25de4cbea808d22348b6979668e924d", size = 292405, upload-time = "2026-05-09T23:14:47.211Z" }, + { url = "https://files.pythonhosted.org/packages/79/34/d2b0937faa7859263f7f0a3c6b103a1296306be6952dc173d0154e9a2f49/regex-2026.5.9-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:45375819235558a4ff1c4971dc32881f022613abdb180128f5cb4768c1765a1c", size = 811855, upload-time = "2026-05-09T23:14:49.21Z" }, + { url = "https://files.pythonhosted.org/packages/80/fe/daf53a47457a8486db66c66c01ceb9c2303eecee3f87197f1e77eb1a736d/regex-2026.5.9-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:ead4b163ac30a29574510cd4b3e2e985ac5290c05fc7095557d6a5f403fc31b5", size = 871189, upload-time = "2026-05-09T23:14:51.555Z" }, + { url = "https://files.pythonhosted.org/packages/1c/75/058fc4470cbfbf57d800aff1a0022b929a3f9fa553ee10a0cdf2070eb31f/regex-2026.5.9-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:8c6e4218fbdfbcd4f6c19efca40930d24a621bf4b48cb76bc6640543bd28ef20", size = 917485, upload-time = "2026-05-09T23:14:53.633Z" }, + { url = "https://files.pythonhosted.org/packages/88/e7/179cfda3a28bc843b5c6cfe7f79f23489c791ed95f151083803660878432/regex-2026.5.9-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6351571c8a42b505eb555c0dc47d740d0fb66977dc142919eea6f4325b7c56a0", size = 816369, upload-time = "2026-05-09T23:14:56.198Z" }, + { url = "https://files.pythonhosted.org/packages/41/90/6f0cc422071688266d344fca8462d787cba0a2c144acb25721f9a61ec265/regex-2026.5.9-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:002205cafd2a9e78c6290c7d1df277bf3277b3b7a30e0b4bb0dac2e2e3f7cb2d", size = 785869, upload-time = "2026-05-09T23:14:58.602Z" }, + { url = "https://files.pythonhosted.org/packages/02/67/a31f1760f09c27b251ef39e9beb541f462cf977381d067faa764c2c0e393/regex-2026.5.9-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:8abd33fef90b2a9efac5557d6033ca82d1195ed3a15fea5af15ba7b463c6a63b", size = 801427, upload-time = "2026-05-09T23:15:00.642Z" }, + { url = "https://files.pythonhosted.org/packages/e3/c4/1a80654597b6bc1e1ea0494824c31200e8a956abe290afae9b19a166a148/regex-2026.5.9-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:31037c82eccb44b7ea2e9e221d7c01429430e989a1f4b91ea5a855f6017b509a", size = 866482, upload-time = "2026-05-09T23:15:03.384Z" }, + { url = "https://files.pythonhosted.org/packages/d1/11/960724e06482c08466ff5611e242e86f80062949cdf6b4b9cc317b9dd93d/regex-2026.5.9-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:5604dfd046dc37eca90250fc3be938b076c8059fa772ac0ed6f499b0f0fb0415", size = 773022, upload-time = "2026-05-09T23:15:05.625Z" }, + { url = "https://files.pythonhosted.org/packages/50/a8/a9979c3e7918280e93159ebcab5ef1a65116dd4f3bd6091be0eae4a126e8/regex-2026.5.9-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:0e1b1b4e496afbb24f4a62aba855ee4f88f25578927697b340702e48c9ee6bc2", size = 856642, upload-time = "2026-05-09T23:15:07.966Z" }, + { url = "https://files.pythonhosted.org/packages/fe/d4/a9b732f2f0072c0ab12227483abb24fffcb9f73f8a2b203df0a6d0434735/regex-2026.5.9-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:be3372b9df6ddecff6486d37e19095a7b4973137caf5512407a89f4455361f41", size = 803552, upload-time = "2026-05-09T23:15:10.215Z" }, + { url = "https://files.pythonhosted.org/packages/d5/fe/1b3113817447a1d4155e4ac76d2e072f42c0bcba2f43fa8a0e756ea2cd91/regex-2026.5.9-cp314-cp314t-win32.whl", hash = "sha256:3ddd90103f9e5c471c49c7852ecc1fe27c7e45eb99e977aefe7caa4e779f4f58", size = 275746, upload-time = "2026-05-09T23:15:12.609Z" }, + { url = "https://files.pythonhosted.org/packages/92/73/93d42045302636c91f2e5ef588b65b84b01428f28ec77de256b1dfdfbe5c/regex-2026.5.9-cp314-cp314t-win_amd64.whl", hash = "sha256:ca518ed29c46eecba6010b15f1b9a479314d2de409536e71b6a13aa04e3b8a77", size = 285685, upload-time = "2026-05-09T23:15:15.086Z" }, + { url = "https://files.pythonhosted.org/packages/da/80/35b4c33c804a165a7f55289afda3ea9e3eb6d15800341a2d66455c0f1f30/regex-2026.5.9-cp314-cp314t-win_arm64.whl", hash = "sha256:5e41809d2683fcde7d5a8c87a6567ba1fb1ce0de9f31bff578de00a4b2d76daa", size = 275713, upload-time = "2026-05-09T23:15:16.98Z" }, +] + [[package]] name = "requests" version = "2.34.0" @@ -2065,13 +2188,19 @@ dependencies = [ dev = [ { name = "dbt-core" }, { name = "fakesnow" }, + { name = "openai" }, { name = "pyright" }, { name = "pytest" }, { name = "pytest-cov" }, { name = "ruff" }, { name = "snowflake-connector-python" }, + { name = "tiktoken" }, { name = "types-pyyaml" }, ] +openai = [ + { name = "openai" }, + { name = "tiktoken" }, +] snowflake = [ { name = "snowflake-connector-python" }, ] @@ -2083,11 +2212,13 @@ dev = [ { name = "fakesnow" }, { name = "mkdocs-include-markdown-plugin" }, { name = "mkdocs-material" }, + { name = "openai" }, { name = "pyright" }, { name = "pytest" }, { name = "pytest-cov" }, { name = "ruff" }, { name = "snowflake-connector-python" }, + { name = "tiktoken" }, { name = "types-pyyaml" }, ] docs = [ @@ -2101,6 +2232,8 @@ requires-dist = [ { name = "dbt-core", marker = "extra == 'dev'", specifier = ">=1.8,<2" }, { name = "fakesnow", marker = "extra == 'dev'", specifier = ">=0.9" }, { name = "google-cloud-bigquery", specifier = ">=3.20,<4" }, + { name = "openai", marker = "extra == 'dev'", specifier = ">=1.40,<3.0" }, + { name = "openai", marker = "extra == 'openai'", specifier = ">=1.40,<3.0" }, { name = "pydantic", specifier = ">=2.5,<3" }, { name = "pyright", marker = "extra == 'dev'", specifier = "==1.1.409" }, { name = "pytest", marker = "extra == 'dev'" }, @@ -2109,9 +2242,11 @@ requires-dist = [ { name = "ruff", marker = "extra == 'dev'" }, { name = "snowflake-connector-python", marker = "extra == 'dev'", specifier = ">=3,<4" }, { name = "snowflake-connector-python", marker = "extra == 'snowflake'", specifier = ">=3,<4" }, + { name = "tiktoken", marker = "extra == 'dev'", specifier = ">=0.7,<1.0" }, + { name = "tiktoken", marker = "extra == 'openai'", specifier = ">=0.7,<1.0" }, { name = "types-pyyaml", marker = "extra == 'dev'", specifier = ">=6,<7" }, ] -provides-extras = ["dev", "snowflake"] +provides-extras = ["dev", "openai", "snowflake"] [package.metadata.requires-dev] dev = [ @@ -2120,11 +2255,13 @@ dev = [ { name = "fakesnow", specifier = ">=0.9" }, { name = "mkdocs-include-markdown-plugin", specifier = ">=6.0" }, { name = "mkdocs-material", specifier = ">=9.0" }, + { name = "openai", specifier = ">=1.40,<3.0" }, { name = "pyright", specifier = "==1.1.409" }, { name = "pytest" }, { name = "pytest-cov", specifier = ">=5.0" }, { name = "ruff" }, { name = "snowflake-connector-python", specifier = ">=3,<4" }, + { name = "tiktoken", specifier = ">=0.7,<1.0" }, { name = "types-pyyaml", specifier = ">=6,<7" }, ] docs = [ @@ -2242,6 +2379,60 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a6/a5/c0b6468d3824fe3fde30dbb5e1f687b291608f9473681bbf7dabbf5a87d7/text_unidecode-1.3-py2.py3-none-any.whl", hash = "sha256:1311f10e8b895935241623731c2ba64f4c455287888b18189350b67134a822e8", size = 78154, upload-time = "2019-08-30T21:37:03.543Z" }, ] +[[package]] +name = "tiktoken" +version = "0.13.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "regex" }, + { name = "requests" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/e4/e5/5f3cb2159769d0f4324c0e9e87f9de3c4b1cd45848a96b2eb3566ad5ca77/tiktoken-0.13.0.tar.gz", hash = "sha256:c9435714c3a84c2319499de9a300c0e604449dd0799ff246458b3bb6a7f433c1", size = 38986, upload-time = "2026-05-15T04:51:27.153Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1a/4c/1bc81f4cd53e827c4ee67ca951b5935724716049452d8dfa09b8b82372bb/tiktoken-0.13.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:7bfe1849caa65d1e1d9871817170ec497bbb7984e182012e1bdce72f66608cdb", size = 1036353, upload-time = "2026-05-15T04:50:21.757Z" }, + { url = "https://files.pythonhosted.org/packages/75/91/10b9c7076bc02c246c853201fdbbe300a4b8c5ed7b84c25f7403f4e32655/tiktoken-0.13.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:91c180fe255bd5a86d8316210d2833a1d4d33d026cd86a67812f4773743c8d26", size = 984644, upload-time = "2026-05-15T04:50:23.256Z" }, + { url = "https://files.pythonhosted.org/packages/4e/e4/fceae98015fab47fcd49b8bd7f46145bcd187a47e0add1e5378ed67ef980/tiktoken-0.13.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:059c8ecf554eb5b41e6e054ba467b871b03277d267dee7244380aca4359747d4", size = 1119261, upload-time = "2026-05-15T04:50:24.348Z" }, + { url = "https://files.pythonhosted.org/packages/f9/39/fe42ad00de01a8c4a49ad8649a2c8a316835a9cad5961b11d21eac0020a5/tiktoken-0.13.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:36217497eaffc158607a3b26f065300db2aefd43b115263f3b9688ce38146173", size = 1138253, upload-time = "2026-05-15T04:50:25.505Z" }, + { url = "https://files.pythonhosted.org/packages/03/c4/ccee1ecccca107e9a16efcecdeeb964c325305038554d466ece65b42338f/tiktoken-0.13.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:303f7d91b4fce3baddbcde05c139091d4caa5026ac7214c1dc7ff7a71ee429ff", size = 1185747, upload-time = "2026-05-15T04:50:27.02Z" }, + { url = "https://files.pythonhosted.org/packages/9d/03/cd0cba295522b91eb55c6b2704f1df895f8226cfe60ab10d4d51d0cc9e69/tiktoken-0.13.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:5d48843bee149630eb735a99e1f4a85b47308d21868ea63163f6e87768d3cfed", size = 1241265, upload-time = "2026-05-15T04:50:28.815Z" }, + { url = "https://files.pythonhosted.org/packages/7e/25/a10efd564402d82c2ff50d12057353ace447aa8007deceaa48641f63d35c/tiktoken-0.13.0-cp311-cp311-win_amd64.whl", hash = "sha256:fc1c44cd37b43fc46bae593129164f4f281e82ea116b57a85aa81bda57eafc94", size = 876509, upload-time = "2026-05-15T04:50:30.026Z" }, + { url = "https://files.pythonhosted.org/packages/85/8e/144bde4e01df66b34bb865557c7cd754ed08b036217ebd79c9db5e9048a9/tiktoken-0.13.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:32ac870a806cfb260a02d0cb70426aef02e038297f8ad50df5040bb5af360791", size = 1034888, upload-time = "2026-05-15T04:50:31.579Z" }, + { url = "https://files.pythonhosted.org/packages/36/18/d4ac9d20956cdebca04841316660ed584c2fecdc2b81722a28bc7ad3b1e4/tiktoken-0.13.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4d9980f11429ed2d737c463bb1fb78cf330caa026adf002f714aced7849a687b", size = 982970, upload-time = "2026-05-15T04:50:32.961Z" }, + { url = "https://files.pythonhosted.org/packages/74/ed/6bb8d05b9f731f749fee5c6f5ca63e981143c826a5985877330507bd13b7/tiktoken-0.13.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:3f277ebea5edd7b8bf03c6f9431e1d67d517530115572b2dc1d465326e8f88c7", size = 1115741, upload-time = "2026-05-15T04:50:34.475Z" }, + { url = "https://files.pythonhosted.org/packages/34/de/2ca96b07a82d972b74fe4b46de055b79c904e45c7eab699354a0bfa697dc/tiktoken-0.13.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:a116178fa7e1b4065bff05214360373a65cac22f965be7b3f73d00a0dbfe7649", size = 1136523, upload-time = "2026-05-15T04:50:35.782Z" }, + { url = "https://files.pythonhosted.org/packages/ee/dc/9dafec002c2d4424378563cf4cf5c7fb93631d2a55013c8b87554ee4012c/tiktoken-0.13.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2c397ddda233208345b01bd30f2fca79ff730e55731d0108a603f9bc57f6af3b", size = 1181954, upload-time = "2026-05-15T04:50:36.99Z" }, + { url = "https://files.pythonhosted.org/packages/a1/d0/1f8578c45b2f24759b46f0b50d31878c63c73e6bf0f2227e10ec5c5408dc/tiktoken-0.13.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:95097e4f89b06403976e498abf61a0ee73a7497e73fb599cb211d8197a054d91", size = 1240069, upload-time = "2026-05-15T04:50:38.221Z" }, + { url = "https://files.pythonhosted.org/packages/aa/90/28d7f154888610aa9237e541986beb62b479df29d193a5a0617dbb1514d0/tiktoken-0.13.0-cp312-cp312-win_amd64.whl", hash = "sha256:8f2d16e7a7c783ad81f36e457d046d1f1c8af70b22aec8a13238efe531977c41", size = 874748, upload-time = "2026-05-15T04:50:39.587Z" }, + { url = "https://files.pythonhosted.org/packages/9c/83/b096c859c2a47c11731bf2f5885f4028b809dfe2396582883eed9cae372f/tiktoken-0.13.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:5df5d1507bd245f1ccad4a074698240021239e455eb0bb4ced4e3d7181872154", size = 1034228, upload-time = "2026-05-15T04:50:40.988Z" }, + { url = "https://files.pythonhosted.org/packages/53/61/c68e123b6d753e3fc2751e9b18e732c9d8bf1e1926762e736eee935d931c/tiktoken-0.13.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:8fe806a50664e83a6ffd56cbd1e4f5dcc6cd32a3e7538f70dc38b1a271384545", size = 982978, upload-time = "2026-05-15T04:50:42.195Z" }, + { url = "https://files.pythonhosted.org/packages/ef/8b/96cc178cc584e65d363134500f297790b06cd48cdeb1e8fcf7bbe60f4715/tiktoken-0.13.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:125bc05005e747f993a83dc67934249932d6e4209854452cd4c0b1d53fba3ba2", size = 1116355, upload-time = "2026-05-15T04:50:43.564Z" }, + { url = "https://files.pythonhosted.org/packages/86/f5/bab735d2c72ea55404b295d02d092644eb5f7cc6205e34d35eb9abfb9ab2/tiktoken-0.13.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:5e6358911cab4adee6712da27d65573496a4f68cf8a2b5fca6a4ad10fc5748cf", size = 1135772, upload-time = "2026-05-15T04:50:44.782Z" }, + { url = "https://files.pythonhosted.org/packages/4e/b9/6de04ebdf904edfaad87788011b3735087a0c9ea671b9027e1e4e965e8c8/tiktoken-0.13.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:975cbd78d085d75d26b59660e262736dcaed1e35f8f142cd6291025c01d25486", size = 1182415, upload-time = "2026-05-15T04:50:46.422Z" }, + { url = "https://files.pythonhosted.org/packages/0d/9c/470a05f3b1caf038f44880e334d47ab674e0c80d514c66b375d14d5afa10/tiktoken-0.13.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:75ab9bc99fa020a4c283424590ecd7f3afd70c1c281cb3fa3192a6c3af9f9615", size = 1239879, upload-time = "2026-05-15T04:50:48.052Z" }, + { url = "https://files.pythonhosted.org/packages/42/a6/c1936d16055436cb32e6c6128d68629622e00f4768562f55653752d34768/tiktoken-0.13.0-cp313-cp313-win_amd64.whl", hash = "sha256:6b1615f0ff71953d19729ceb18865429c185b0a23c5353f1bbca34a394bf60f7", size = 874829, upload-time = "2026-05-15T04:50:49.202Z" }, + { url = "https://files.pythonhosted.org/packages/d6/07/acb5992c3772b5a36284f742cfb7a5895aa4471d1848ac31464ad50d7fdf/tiktoken-0.13.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:6eb4a5bfbc6426938026b1a334e898ac53541360d62d8c689870160cc80abd67", size = 1033600, upload-time = "2026-05-15T04:50:50.4Z" }, + { url = "https://files.pythonhosted.org/packages/14/e9/742e9aec30f59b9f161f7ff7cd072e02ea836c9e1c0854a8076dfcd40d5c/tiktoken-0.13.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:43cee3e5400573b2046fbf092cc7a5bc30164f9e4c95ce20714da929df48737a", size = 982516, upload-time = "2026-05-15T04:50:52.03Z" }, + { url = "https://files.pythonhosted.org/packages/72/74/ca1541b053e7648254d2e4b42a253e1bb4359f2c91a0a8d49228c794e1a0/tiktoken-0.13.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:7de52e3f566d19b3b11bd37eea552c6c305ad74081f736882bd44d148ed4c48d", size = 1115518, upload-time = "2026-05-15T04:50:53.543Z" }, + { url = "https://files.pythonhosted.org/packages/46/e3/93825eaf5a4a504795b787e5d5dea07fbeb3dabf97aa7b450be8bde59c89/tiktoken-0.13.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:51384448aa508e4df84c0f7c1dc3211c7f7b8096325660ee5fc82f3e11b381ce", size = 1136867, upload-time = "2026-05-15T04:50:55.191Z" }, + { url = "https://files.pythonhosted.org/packages/8c/46/002b68de6827091d5ae90b048f326e8aad8d953520950e5ce1508879414f/tiktoken-0.13.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:e28157350f7ebf35008dd8e9e0fdb621f976e4230c881099c85e8cf07eaa50e2", size = 1181826, upload-time = "2026-05-15T04:50:56.296Z" }, + { url = "https://files.pythonhosted.org/packages/db/c6/d393e3185a276505182f7abd93fe714f3c444a2be9180798fa052347504e/tiktoken-0.13.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:165cf1820ea4a354985c2490a5205d4cc74661c934aca79dd0368232fff94e0f", size = 1239489, upload-time = "2026-05-15T04:50:57.918Z" }, + { url = "https://files.pythonhosted.org/packages/b7/4d/bc07d1f1635d4897a202acc0ae11c2886eaa7325c359ba4741b47bf8e225/tiktoken-0.13.0-cp313-cp313t-win_amd64.whl", hash = "sha256:6c43a675ca14f6f2749ba7f12075d37456015a24b859f2517b9beb4ef30807ec", size = 873820, upload-time = "2026-05-15T04:50:59.528Z" }, + { url = "https://files.pythonhosted.org/packages/8c/93/0dd6adca026a616c3a92974566b43381eea4b475ce1f36c062b8271a9ac5/tiktoken-0.13.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:eaaaef47c2406277181d2086484c317bf7fc433e2d5d03ff94f56b0dcec87471", size = 1034977, upload-time = "2026-05-15T04:51:00.957Z" }, + { url = "https://files.pythonhosted.org/packages/d9/77/5ec6e6bc5b30bed6d93f7f2162d8f6b32437b3ba27cb527cfe004f6109c9/tiktoken-0.13.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:ca8b310bd93b3772cb1b7922d915446864860f562bdfe4825c63a0aed3fb28cd", size = 983635, upload-time = "2026-05-15T04:51:02.629Z" }, + { url = "https://files.pythonhosted.org/packages/94/b0/c8ae9aff00d625c50659b4513e707a0462c4bf5d4d6cc1b802103225c02e/tiktoken-0.13.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:32e0c12305105002c047b3bb1070b0dd9a73b0cb3b2856a8972b810e7a4f5881", size = 1116036, upload-time = "2026-05-15T04:51:04.082Z" }, + { url = "https://files.pythonhosted.org/packages/1b/ac/6a5dddd1d0a6018ecb389bd0353e6b4a515eb4d2286611bd0ace1937b9e1/tiktoken-0.13.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:5ba5fd62507a932d1241346179e3b39bc7bf7408f03c272652d93b3bedf5db24", size = 1135544, upload-time = "2026-05-15T04:51:05.229Z" }, + { url = "https://files.pythonhosted.org/packages/f4/b8/585032b4384b2f7dcdaddcb52865c83a701a420d09e3c2b4a2be1c450c57/tiktoken-0.13.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:d108bc2d470fc53c8ecd24f2c0fd2b5f98c33e87cdb6aa2e9b8c5dced703d273", size = 1182217, upload-time = "2026-05-15T04:51:06.517Z" }, + { url = "https://files.pythonhosted.org/packages/cd/b6/993ff1ded3958215fd341a847b8e5ffeb5de473f435296870d314fc91ac4/tiktoken-0.13.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:cb99cb5127449f58d0a2d5f5ccfb390d8dbdfd919c221246caaee29d8725ed51", size = 1239404, upload-time = "2026-05-15T04:51:07.843Z" }, + { url = "https://files.pythonhosted.org/packages/dd/3d/fef7e06e3b33e7538db0ced734cf9fe23b6832d2ac4990c119c377aec55e/tiktoken-0.13.0-cp314-cp314-win_amd64.whl", hash = "sha256:115c4f26ffa11caac8b54eea35c2ad38c612c20a48d35dd15d70a02ac6f51f58", size = 918686, upload-time = "2026-05-15T04:51:08.925Z" }, + { url = "https://files.pythonhosted.org/packages/c1/82/a7fc44582bc32ab00de988a2299bf77c077f59068b233109e34b7d6ca7e6/tiktoken-0.13.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:472527e9132952f2fbf77cd290658bacf003d4d5a3fabc18e5fbd407cbae4d9b", size = 1034454, upload-time = "2026-05-15T04:51:10.035Z" }, + { url = "https://files.pythonhosted.org/packages/37/d0/24d8a890c14f432a05cea669c17bebeaa99f96a7c79523b590f564246411/tiktoken-0.13.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:4e2f67d27c9626cdd25fe33d9313c5cdb3d8d82da646b68d6eb8e7e9c20e6448", size = 982976, upload-time = "2026-05-15T04:51:11.23Z" }, + { url = "https://files.pythonhosted.org/packages/49/b7/2ab43f62788a9266187a9bfc1d3af99ad83e5eaa25fbef168a69cd5ad14f/tiktoken-0.13.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:2b920b35805cd64585a37c3dc7ce65fba4d2d36016be01e1d7942482ca29093a", size = 1115526, upload-time = "2026-05-15T04:51:12.608Z" }, + { url = "https://files.pythonhosted.org/packages/64/39/1494321ed323ce7a14d88e3cd6cb9058625977df1c6961ddc492bd10a9f3/tiktoken-0.13.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:493af3aa28a4aaf2e3d2600a2ee717252c9bf5ab38fff94eb5a02db5ab77e5ad", size = 1136466, upload-time = "2026-05-15T04:51:13.926Z" }, + { url = "https://files.pythonhosted.org/packages/96/d9/dfd086aa2d918c563a140720e0ce296cada1634efd2783d5cf51e05f984e/tiktoken-0.13.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:6644c9c2b5cf3916f5a3641d7d12fdb3f006a7b3d9ff6acdaec44e29ab1ff91e", size = 1181863, upload-time = "2026-05-15T04:51:15.025Z" }, + { url = "https://files.pythonhosted.org/packages/2f/68/a18b4f307086954fdae32714cb4f85562e34f9d34ab206e61f1816aa6018/tiktoken-0.13.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5cb65b60b9408563676d874a3a4ee573370066f0dc4e29d84e82e989c6517424", size = 1239218, upload-time = "2026-05-15T04:51:16.103Z" }, + { url = "https://files.pythonhosted.org/packages/16/5b/f2aa703a4fc5d2dff73460a7d46cc2f3f44aa0f3dd8eeb20d2a0ecf68862/tiktoken-0.13.0-cp314-cp314t-win_amd64.whl", hash = "sha256:85b78cc3a2c3d48723ca751fa981f1fedccd54194ca0471b957364353a898b07", size = 918110, upload-time = "2026-05-15T04:51:17.237Z" }, +] + [[package]] name = "tomli" version = "2.4.1" @@ -2305,6 +2496,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/6a/43/8bd850ee71a191bf072e31302c73a66be413fecdd98fdcd111ecbcce13ca/tomlkit-0.15.0-py3-none-any.whl", hash = "sha256:4dbc8f0fc024412b57ced8757ac7461305126a648ff8c2c807fcb8e133a78738", size = 41328, upload-time = "2026-05-10T07:38:23.517Z" }, ] +[[package]] +name = "tqdm" +version = "4.67.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/09/a9/6ba95a270c6f1fbcd8dac228323f2777d886cb206987444e4bce66338dd4/tqdm-4.67.3.tar.gz", hash = "sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb", size = 169598, upload-time = "2026-02-03T17:35:53.048Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/16/e1/3079a9ff9b8e11b846c6ac5c8b5bfb7ff225eee721825310c91b3b50304f/tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf", size = 78374, upload-time = "2026-02-03T17:35:50.982Z" }, +] + [[package]] name = "types-pyyaml" version = "6.0.12.20260510"