This repository was archived by the owner on May 26, 2026. It is now read-only.
feat(KR-P2-K ST2): inference-call estimator wire-in (3 chokepoints) - #67
Merged
Merged
Conversation
3 tasks
rafe-walker
changed the base branch from
feat/kora-KR-P2-K-st1-cost-state-holder
to
main
May 22, 2026 00:11
R4.1 §9.6 — feed the CostStateHolder estimator from every inference
dispatch path so the $200 Anthropic Agent SDK pool burn-down is
tracked uniformly per Joshua's billing memo ("estimator must cover
ALL inference paths").
# New module — agent/cost_ladder_wire.py
- ``record_inference_from_response(response, *, model, provider,
base_url, api_mode)`` — extracts ``response.usage``, normalizes
via ``agent.usage_pricing.normalize_usage``, feeds
``holder.record_inference``. Falls back to ``response.model``
when caller omits ``model``. Fail-soft on EVERY failure path
(no holder, no usage, normalize raises, record raises) — DEBUG
log + return; the inference response handler must never crash.
- ``record_rate_limit_pulse_from_response(response)`` — extracts
the 6 ``anthropic-ratelimit-{requests,tokens}-{limit,remaining,reset}``
headers, builds a ``RateLimitPulse``, feeds
``holder.record_rate_limit_pulse``. Best-effort: only the 2
direct-Anthropic dispatch sites surface these headers (per B1
ruling). Handles both top-level ``response.headers`` AND
``response.http_response.headers`` (httpx-style).
# Wire-in chokepoints (3 sites per B1 ruling)
**Site 1 — conversation_loop chokepoint (primary signal, covers ALL
agent.dispatch paths including direct-Anthropic + OpenAI-compat):**
``agent/conversation_loop.py`` — injected immediately after the
existing ``estimate_usage_cost(...)`` call at the normalize_usage
block. Both api_mode="anthropic_messages" AND api_mode="openai"
responses flow through this chokepoint.
**Site 2 — auxiliary_client.call_llm / async_call_llm wrap (primary
signal, covers ALL side-task inference: compression / vision /
web_extract / session_search / skills_hub / mcp / title_generation):**
``agent/auxiliary_client.py`` — module-bottom wrap. Original
``call_llm`` / ``async_call_llm`` renamed to ``_call_llm_inner`` /
``_async_call_llm_inner``. New wrappers call the inner function,
feed the holder via ``record_inference_from_response``, return the
response unchanged. Covers all 16+ internal return paths (success +
retry + fallback chains) with one wrap rather than threading
kwargs through each.
Failed inferences (raise before producing a response) correctly do
NOT contribute — only successful inferences burn the pool.
**Site 3 — direct-Anthropic rate-limit-pulse capture (secondary
signal, best-effort):**
- ``run_agent.py:_anthropic_messages_create`` — feeds
``record_rate_limit_pulse_from_response`` after the SDK call.
Primary $-burn signal lands via the conversation_loop chokepoint
after this returns; only the rate-limit pulse is captured here.
- ``agent/auxiliary_client.py:AnthropicAuxiliaryClient`` (line
~1018) — same pattern for the auxiliary-tier Anthropic-direct
path.
# Note on the "3 sites" framing
The verification report identified 3 sites:
``run_agent.py:2870`` + ``auxiliary_client.py:1018`` (both
direct-Anthropic) + ``agent_runtime_helpers.py:1275`` (OpenAI-compat).
The third is actually the **OpenAI client construction** site, not
an inference call. The inference dispatched through that client
flows back through the ``conversation_loop`` chokepoint, where Site 1
above captures it. Coverage is intact; the verification framing was
site-count vs chokepoint-count.
# Tests
- ``tests/test_cost_ladder_wire.py`` — 19 new tests:
- ``record_inference_from_response``: no-holder no-op / no-usage no-op
/ happy path Anthropic shape / happy path OpenAI-compat shape /
response.model fallback when arg omitted / normalize failure
swallowed / record_inference failure swallowed
- ``record_rate_limit_pulse_from_response``: happy path both axes
captured / reads from ``response.http_response.headers`` when
top-level absent / missing-header no-op / malformed int / malformed
timestamp / ISO-8601 with trailing ``Z`` / ISO-8601 with offset
- Header lookup helper: case sensitivity + object-with-get
- Regression sweep:
- ``tests/test_cost_state_holder.py`` (ST1) — still 33 passed
- ``tests/agent/test_auxiliary_client*.py`` — 268 passed (call_llm
wrap is transparent)
- ``tests/run_agent/test_run_agent.py`` +
``test_tool_call_guardrail_runtime.py`` — 346 passed
(_anthropic_messages_create wrap is transparent)
Total: **666 passed**. Ruff clean.
# Operator-visible behavior
- Every successful Anthropic inference (direct or via OpenAI-compat
router) updates spent_to_date_usd according to the published per-
token rates from usage_pricing.py's multi-source lookup.
- Direct-Anthropic responses additionally update latest_rate_limit_pulse
on the holder; COST-PANEL admin UI can surface this as a secondary
health signal.
- All wire-helpers are fail-soft — a malformed response or pricing
miss logs DEBUG and continues; the inference response handler
never sees the estimator's failures.
# Stacked on ST1
Branch ``feat/kora-KR-P2-K-st2-estimator-wire-in`` is based on
``feat/kora-KR-P2-K-st1-cost-state-holder``. ST1 → ST2 → ST3 → ST4
→ ST5 cascade.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rafe-walker
force-pushed
the
feat/kora-KR-P2-K-st2-estimator-wire-in
branch
from
May 22, 2026 00:12
1a18bcc to
a652dd3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
KR-P2-K ST2 — wires the
CostStateHolderestimator into every inference dispatch path per Joshua's billing memo ("estimator must cover ALL inference paths, including ancillary plugin calls"). 3 chokepoints cover the full dispatch tree.Stacked on PR #65 (ST1) — base
feat/kora-KR-P2-K-st1-cost-state-holder.New module —
agent/cost_ladder_wire.pyrecord_inference_from_response(response, *, model, provider, base_url, api_mode)— primary signal. Extractsresponse.usage, normalizes viausage_pricing.normalize_usage, feedsholder.record_inference. Falls back toresponse.modelwhen caller omits the kwarg. Fail-soft on every failure path.record_rate_limit_pulse_from_response(response)— secondary signal. Extracts the 6 Anthropicanthropic-ratelimit-*headers, builds aRateLimitPulse, feeds the holder. Handles top-level + httpx-style header lookup. Best-effort per B1.Chokepoints
agent/conversation_loop.py:~1590$-burnagent/auxiliary_client.pycall_llm+async_call_llm(16+ return paths in one wrap)$-burnrun_agent.py:_anthropic_messages_create+AnthropicAuxiliaryClientFailed inferences (which raise before producing a response) correctly do NOT contribute — only successful inferences burn the pool.
Verification framing correction
The §1.3 verification identified
agent_runtime_helpers.py:1275as "site #3" but that's actually the OpenAI client CONSTRUCTION site, not an inference call. The inference dispatched through that client flows back through theconversation_loopchokepoint (site #1), so coverage is intact — the framing was site-count vs chokepoint-count.Test plan
tests/test_cost_ladder_wire.py— 19 new tests covering:record_inference_from_response: no-holder / no-usage / happy path (both Anthropic + OpenAI-compat shape) /response.modelfallback / normalize-raise swallowed / record-raise swallowedrecord_rate_limit_pulse_from_response: happy path both axes / httpx-stylehttp_response.headers/ missing header no-op / malformed int + timestamp / ISO-8601 trailing-Z + offsettests/test_cost_state_holder.py— 33 passed (no changes)tests/agent/test_auxiliary_client*.py— 268 passed (call_llm wrap transparent)tests/run_agent/test_run_agent.py+test_tool_call_guardrail_runtime.py— 346 passed (_anthropic_messages_create wrap transparent)uv run ruff check— clean🤖 Generated with Claude Code