feat: LLM cost-review ledger + sync/batch routing hub (pg-llm-batch) - #46
Conversation
Extend the OpenAI-compatible orchestrator into the LLM cost-review and
routing control point.
Cost review:
- llm_usage_records ledger written on every completion (sync + batch) with
token counts and cost computed from a configurable llm_price_entries price
table; seven attribution dimensions catalogued in
cost_attribution_dimensions (account, service, upstream_api/provider,
model_name, team, group, company).
- Rollup/report by any dimension over any time window.
- Token counts reuse pg-llm-batch pg_tiktoken when a DSN is set; deterministic
heuristic otherwise. In-memory + PEP-249 SQL (stdlib sqlite3 / psycopg) stores.
Routing:
- RoutingPolicy decides sync vs batch from request hints + KV thresholds.
- Batch path submits to pg-llm-batch (added as a git submodule + a client)
over its OpenAI-compatible Batch API; local in-process backend preserves the
mock/standalone path. Cost-optimising upstream selection (cheapest_upstream).
Server:
- /healthz liveness; /api/v1/cost_reports/rollup, /api/v1/llm_usage_records,
/api/v1/cost_attribution_dimensions, /api/v1/batch_routing_jobs(+/{id}[/results]).
- /v1/chat/completions records usage and honours attribution + routing hints;
real token usage now reported. OpenAPI contract updated.
Config/secrets come from a KV store, never runtime os.getenv. DB objects are
2+ word snake_case. Papers (FrugalGPT, RouteLLM, Hybrid LLM) added under
docs/papers with citations. CodeGraph index built on clone and used to locate
the complete/route_once/_invoke seam before editing.
Tests: cost-ledger writes + multi-dim rollup + price computation, sync-vs-batch
decision, and batch submit/retrieve to pg-llm-batch (mocked); 174 pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P
Add POST /v1/batch/embeddings + GET /v1/batch/embeddings/{batch_id} so naruon's
batch-tolerant import embeddings route through the orchestrator (the routing /
cost hub) instead of 404-ing and silently falling back to per-item calls.
- batch_routing.py: EmbeddingBatchRequest/ResultItem, LocalEmbeddingBatchBackend
(deterministic in-process embedder for the mock/standalone path) and
PgLlmBatchEmbeddingBackend (embeddings JSONL -> pg-llm-batch).
- cost_router.py: submit/poll/complete embeddings batch; records one usage-ledger
row per vector with full attribution, returns naruon's exact shape
{batch_id, status, embeddings:[{index,embedding}], cost_micro_usd,
token_counts, total_tokens, part_count}. Idempotent poll (cost recorded once).
- server.py: inference-scoped routes; accepts inputs (naruon) or OpenAI input;
maps attribution dims from metadata into the ledger.
- api_contract.py OpenAPI + README/rest_api_design docs.
- Real end-to-end contract test over a live socket via LocalEmbeddingBatchBackend,
asserting the shared tests/fixtures/batch_embeddings_contract.json shape.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P
OpenCode Review Overview
Pull request overviewOpenCode reviewed the current-head bounded evidence and found no blocking issues. FindingsNo blocking findings. SummaryApproval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (11 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (11 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (5 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (5 files)"]
R2 --> V2["docs review"]
Evidence --> S3["Test (6 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (6 files)"]
R3 --> V3["targeted test run"]
|
|
Codex started follow-up work on 2026-07-10 KST: add token-budgeted map-reduce splitting for /v1/batch/embeddings so Azure/LiteLLM over-limit embedding requests are chunked before hitting the provider. |
Add token/char-budgeted map-reduce for /v1/batch/embeddings so Azure/LiteLLM embedding size limits are handled before provider calls. Oversized inputs are mapped to safe parts, part vectors are reduced with a token-weighted average, and the usage ledger still records one row per original vector.\n\nRefs ContextualWisdomLab/naruon#973.
|
Follow-up implemented in commit bb0360b.\n\nWhat changed:\n- /v1/batch/embeddings now maps oversized original inputs into provider-safe embedding parts before the backend/provider call.\n- Limits are KV-configured: routing.embedding_max_tokens_per_request defaults to 280000, routing.embedding_max_chars_per_part defaults to 240000.\n- Part vectors are reduced back to one vector per original input with a token-weighted average; response now includes input_part_counts and map_reduce metadata while preserving the existing naruon contract keys.\n- Usage ledger records one row per original vector, with aggregated prompt tokens and existing attribution dimensions.\n\nVerification:\n- python -m pytest tests/test_batch_embeddings.py tests/test_cost_router.py tests/test_batch_routing.py -q -> 20 passed\n- python -m pytest -q -> 178 passed\n\nThis directly addresses the Azure/LiteLLM 300000-token embeddings request failure mode by splitting before the provider call instead of surfacing a BadRequest to the caller. |
…batch-routing # Conflicts: # contextual_orchestrator/__init__.py
|
Follow-up after main merge:\n- Merged origin/main into this branch and resolved the only conflict in contextual_orchestrator/init.py by preserving both the cost/batch exports and the main-branch credential exports.\n- Re-ran full suite after the merge: python -m pytest -q -> 187 passed.\n\nCurrent head: 5612627. |
Raise the opencode review CI uv and pytest minimums past known vulnerable ranges so the central OSV scan does not fail on the review-tooling manifest. Mirrors the focused fix already validated by #50.
|
Update: PR #50 is now merged into main, and this branch has been refreshed on top of the updated main. New head: |
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
|
Update: removed the external/pg-llm-batch submodule gitlink and .gitmodules from this PR because the central Strix workflow fails closed on non-regular PR-head files. The pg-llm-batch integration remains injectable through the existing client boundary; standalone/local behavior and the map-reduce embedding split are unchanged. Verification:
New head: 97536cd. Waiting for refreshed GitHub checks. |
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including README.md, contextual_orchestrator/init.py, contextual_orchestrator/api_contract.py, contextual_orchestrator/batch_routing.py, contextual_orchestrator/cost_ledger.py, and 14 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects README.md to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
- Result: APPROVE
- Reason: All tests pass, no unresolved threads, and coverage is sufficient.
- Head SHA:
97536cdaded6c8254696682e96a48b8eb5a61047 - Workflow run: 29062255497
- Workflow attempt: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (11 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (11 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (5 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (5 files)"]
R2 --> V2["docs review"]
Evidence --> S3["Test (6 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (6 files)"]
R3 --> V3["targeted test run"]
|
CodeGraph review showed the LiteLLM P2028 pattern maps to this PR's usage ledger path, not to alert suppression. This update makes usage telemetry prompt-safe and keeps external ledger persistence out of the completion path: usage records contain generated IDs, token counts, cost, provider/model/channel/route metadata, and attribution only; raw prompt/answer text is not exported. NonBlockingLedgerStore queues external persistence, records export failures as telemetry health, and CostRoutingCoordinator.complete() is covered against P2028-like store failures.\n\nVerification: python -m pytest -q -> 190 passed. |
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including README.md, contextual_orchestrator/init.py, contextual_orchestrator/api_contract.py, contextual_orchestrator/batch_routing.py, contextual_orchestrator/cost_ledger.py, and 14 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects README.md to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
- Result: APPROVE
- Reason: No blocking issues found; tests passed and CodeGraph initialized.
- Head SHA:
50dad6daab6889d600d52bca97c0e29754c2ebe7 - Workflow run: 29062719259
- Workflow attempt: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (11 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (11 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (5 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (5 files)"]
R2 --> V2["docs review"]
Evidence --> S3["Test (6 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (6 files)"]
R3 --> V3["targeted test run"]
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including README.md, contextual_orchestrator/init.py, contextual_orchestrator/api_contract.py, contextual_orchestrator/batch_routing.py, contextual_orchestrator/cost_ledger.py, and 14 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects README.md to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
- Result: APPROVE
- Reason: No blocking issues found in the current-head bounded evidence.
- Head SHA:
06b89e58c8be08332cc2f4435f03e23eef4d440e - Workflow run: 29063350456
- Workflow attempt: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (11 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (11 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (5 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (5 files)"]
R2 --> V2["docs review"]
Evidence --> S3["Test (6 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (6 files)"]
R3 --> V3["targeted test run"]
|
진행 상태 업데이트입니다. 구현 및 검증은 완료했습니다.
남은 blocker는 target repo의 org required workflow |
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including README.md, contextual_orchestrator/init.py, contextual_orchestrator/api_contract.py, contextual_orchestrator/batch_routing.py, contextual_orchestrator/cost_ledger.py, and 14 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects README.md to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
- Result: APPROVE
- Reason: No blocking issues found in current-head evidence
- Head SHA:
06b89e58c8be08332cc2f4435f03e23eef4d440e - Workflow run: 29063173560
- Workflow attempt: 2
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file (11 files)"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file (11 files)"]
R1 --> V1["required checks"]
Evidence --> S2["Docs (5 files)"]
S2 --> I2["operator or user guidance"]
I2 --> R2["Review risk: Docs (5 files)"]
R2 --> V2["docs review"]
Evidence --> S3["Test (6 files)"]
S3 --> I3["regression suite"]
I3 --> R3["Review risk: Test (6 files)"]
R3 --> V3["targeted test run"]
github-project-automation
Bot
moved this from In Progress
to Done
in naruon Platform Roadmap
2 participants
Summary
Extends the OpenAI-compatible orchestrator (
ModelClient//v1/chat/completions/TaskOrchestrator.complete) into the LLM cost-review + routing hub — a LiteLLM-plus scope: cost optimiser + upstream load balancing + batch routing. All config/prices/thresholds come from a KV config store, never runtimeos.getenv; new DB objects are 2+ word snake_case.1. Cost review (per-request usage + multi-dimensional cost)
llm_usage_records— one row written on every completion, sync and batch, with prompt/completion/total tokens and a cost computed from a configurable price table.llm_price_entries(KV-backed, per-1K-token, provider+model with aprovider:*wildcard). Money math inDecimal.cost_attribution_dimensions: account, service, upstream API/provider, model name, team, group, company.pg-llm-batch'spg_tiktokencounter when a Postgres DSN is configured; deterministic heuristic otherwise.sqlite3andpsycopg.GET /api/v1/cost_reports/rollup?dimension=&start=&end=,GET /api/v1/llm_usage_records,GET /api/v1/cost_attribution_dimensions.2. Routing (sync-vs-batch + upstream)
RoutingPolicydecides sync vs batch from request hints ({"routing": {"latency_tolerant": true}}) plus KV thresholds (batch_enabled,batch_min_tokens,interactive_forces_sync). Interactive stays on the fast sync path; latency-tolerant/bulk go to batch.external/pg-llm-batch+ a client) over its OpenAI-compatible Batch API (upload JSONL → poll → retrieve); usage/cost recorded on retrieval. ALocalBatchBackendpreserves the mock/standalone path.cheapest_upstream) against the price table.POST /api/v1/batch_routing_jobs,GET /api/v1/batch_routing_jobs/{id},POST /api/v1/batch_routing_jobs/{id}/results.3. Standalone + submodule, health
pg_tiktoken, and the pg-llm-batch backend. All submodule imports are guarded so the package imports with the submodule absent.GET /healthzunauthenticated liveness probe.4. Tests (green)
test_cost_ledger.py,test_batch_routing.py,test_cost_router.py,test_cost_review_server.pycover ledger writes + multi-dim rollup correctness, price computation, sync-vs-batch decision, and batch submit/retrieve to pg-llm-batch (mocked async client mirroringBatchAPIClient). 174 tests pass (140 pre-existing + 34 new); docstring coverage 97%.Papers
docs/papers/adds FrugalGPT (2305.05176), RouteLLM (2406.18665), and Hybrid LLM (2404.14618) with citations; batch/load-balancing papers ship in the submodule.CodeGraph
A CodeGraph index was built on the fresh clone and used to locate the
complete → route_once/conduct → _invokeseam (andrun()'s persistence hook) before editing — the cost/routing layer wraps that seam without changing existing signatures.5. Batch embeddings — cross-PR contract reconciliation
Adds a batch embeddings endpoint so naruon's batch-tolerant import embeddings
(naruon PR #973) actually route through this hub. Previously naruon POSTed to
/v1/batch/embeddingsbut this service exposed only/api/v1/batch_routing_jobs(chat JSONL) — so real calls 404'd and silently fell back to per-item embedding;
only naruon's mocks passed. This closes that gap:
POST /v1/batch/embeddings({model, input|inputs:[...], endpoint, metadata|attribution})and
GET /v1/batch/embeddings/{batch_id}— inference-scoped, OpenAPI-documented.RoutingPolicy/ cost optimiser and a newPgLlmBatchEmbeddingBackend(embeddings JSONL → pg-llm-batch);LocalEmbeddingBatchBackendpreserves the mock/standalone path. Records one
llm_usage_recordsrow per vectorwith full attribution (service/team/group/company/provider from
metadata).{batch_id, status, embeddings:[{index, embedding}], cost_micro_usd, token_counts, total_tokens, part_count}.tests/test_batch_embeddings.py) drives the live HTTP serverend-to-end via
LocalEmbeddingBatchBackendand asserts that shape + cost attribution,against a shared
tests/fixtures/batch_embeddings_contract.jsonthat naruon keeps abyte-identical copy of. Full suite 176 pass.
🤖 Generated with Claude Code
https://claude.ai/code/session_01RTAMs4bpSZS77Xe3RQjv9P