fix(batch): gate real Batch API on declared batch_endpoint_supported

[seonghobae]

Root cause

TaskOrchestrator.batch_route() selects a worker agent per prompt via _select_agent(), which
filters only on chat-capability tags (_is_general_chat_agent() + caller required_tags/
prefer_tags, neither of which batch_route supplies) — it can return any configured
chat-capable agent: Anthropic-shaped, NVIDIA NIM, OpenRouter (proxying dozens of providers), Azure,
or a self-hosted OpenAI-compatible gateway. Many of these do not implement OpenAI's real async Batch
API (/files, /batches, /files/{id}/content) at all.

ModelClient.batch_chat()'s only pre-flight gate before routing straight into that real endpoint was
is_chat_compatible_model_id(agent.model) — confirmed (chat_capability.py) to be a pure model-id
shape heuristic (excludes obviously-non-chat ids like embedding/vision/audio models); it says
nothing about whether the provider implements a Batch API. Any agent that was neither mock:// nor
a recognized local provider URL fell into _batch_run()'s real HTTP upload → create-batch → poll →
download flow unconditionally. On a 404 (provider has no Batch API), the whole prompt group failed
hard — _local_batch_chat() already implements exactly the batch-emulation fallback needed (loop each
request through chat(), aggregate), but it was wired only to the local-provider branch, never as a
fallback for an unproven remote provider.

Grepped the codebase for any existing "does this model/provider support Batch API" signal
(supports_batch, batch_capable, batch_endpoint, etc.) — zero hits. pg-llm-batch (the real
Batch API client this repo vendors) only validates the URL path inside an already-created job; it
has no per-provider capability registry either, and its own module docstring assigns that
responsibility to "the orchestrator, not the caller."

Fix

Add ModelAgent.batch_endpoint_supported: bool | None = None, mirroring the existing
reasoning_effort_supported tri-state field exactly (this repo's own established pattern for
"provider support is unproven, fail closed unless explicitly declared" protocol-capability gaps) —
same __post_init__ validation, to_config/from_dict round-trip, and agent_pool SQLite
persistence/migration (ALTER TABLE ... ADD COLUMN ... CHECK (... IS NULL OR ... IN (0,1)), copied
verbatim from the reasoning_effort_supported migration).

ModelClient.batch_chat() now only takes the real Batch API path when
agent.batch_endpoint_supported is True. Every other remote agent (None/False, the default) falls
back to the same per-item emulation _local_batch_chat() already performs for local providers,
aggregating into the identical {custom_id: {"content", "usage"}} result shape — no misrouting into a
404, no hard failure of the whole batch, and no new heuristic or timeout introduced.

_batch_run(), is_chat_compatible_model_id(), and batch_route()'s group-by-agent persistence
logic are untouched — the fix is entirely which branch batch_chat() selects.

Tests

Extended tests/test_batch_api.py's real-HTTP fake provider (already used for _batch_run's
multipart-upload/poll/parse flow) with a /chat/completions handler, then added:

• test_batch_chat_routes_batch_capable_agent_to_real_batch_endpoint — batch_endpoint_supported=True
  still takes the real Batch API path (one /batches POST, zero /chat/completions calls).
• test_batch_chat_falls_back_to_emulation_when_batch_endpoint_unproven — for both None and
  False, batch_chat() returns a correctly aggregated result via an actual per-request HTTP round
  trip through /chat/completions (not a stubbed return value — asserts real per-item call count),
  and proves the real Batch API was never touched (zero /batches POSTs, no /files upload).

uv run --locked --extra api --extra db --extra queue --group dev python -m pytest -q \
  tests/test_batch_api.py tests/test_batch_routing.py tests/test_batch_routing_boundaries.py \
  tests/test_batch_routing_boundaries_extra.py tests/test_cost_router.py \
  tests/test_cost_router_boundaries.py tests/test_agent_pool_db.py tests/test_batch_optimizer.py \
  tests/test_provider_integration.py
# 168 passed

🤖 Generated with Claude Code

---

[coderabbitai]

Warning

Review limit reached

Next included review available in 37 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: d4a0c531-b764-41b9-9301-d9368a1549f3

📥 Commits

Reviewing files that changed from the base of the PR and between 2e414d1 and 61d5170.

📒 Files selected for processing (10)

• CHANGELOG.d/batch-endpoint-capability-gate.md
• README.md
• contextual_orchestrator/api_contract.py
• contextual_orchestrator/orchestrator.py
• contextual_orchestrator/server.py
• tests/test_agent_pool_db.py
• tests/test_api_contract.py
• tests/test_batch_api.py
• tests/test_orchestrator_client_boundaries.py
• tests/test_provider_reliability.py

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

Devin Review found 3 potential issues.

[seonghobae]

Rebased onto current main

This PR was stuck mergeable_state: behind against a stale base (8839081) while main had advanced to 212ff437 (rater-observation/criterion-binding domain module, review-gateway credential-array support, admin model-group audit refresh). Merged origin/main in — clean, no conflicts.

CI status: a real, pre-existing regression (not a merge or infra issue) — please see before merging

The Full unit and contract suite check is genuinely failing, and it is not caused by staleness or this merge — I confirmed the identical 3 failures reproduce on this PR's original head (8ace8f40) before touching anything, and reproduce identically again after the main merge:

FAILED tests/test_orchestrator_client_boundaries.py::test_batch_chat_success_on_https_provider_returns_validated_results
  - NotConfigured: batch_remote_agent requires a resolvable credential 'REMOTE_API_KEY' in the KV
FAILED tests/test_orchestrator_client_boundaries.py::test_batch_chat_wraps_provider_failures_without_provider_text
  - NotConfigured: batch_failing_agent requires a resolvable credential 'REMOTE_API_KEY' in the KV
FAILED tests/test_provider_reliability.py::test_batch_boundary_hides_raw_upload_error_text_and_cause
  - assert isinstance(NotConfigured(...), ProviderUpstreamError) is False

Root cause: these two pre-existing test files exercise the real Batch API path (_batch_run) using generic HTTPS-provider ModelAgent fixtures that don't set the new batch_endpoint_supported=True field this PR introduces. With the new gate, those agents (previously routed to the real Batch API unconditionally) now fall through to _local_batch_chat() emulation instead, which calls chat() per item and fails on KV credential resolution before ever reaching the code these two tests are asserting against. This is a direct, in-scope consequence of the PR's own behavior change that its test plan (which only touched/re-ran tests/test_batch_api.py and adjacent batch-routing/cost-router files) didn't catch, since it never re-ran the full suite.

I did not attempt to fix this myself — it's a substantive product/test decision (e.g. whether test_orchestrator_client_boundaries.py's and test_provider_reliability.py's fixture agents should now set batch_endpoint_supported=True to keep testing the real-endpoint path, or whether those tests need to move to the emulation path) that belongs to the PR author/reviewer, not to a merge-hygiene pass.

Local verification after the merge (Python 3.12 venv, requirements.lock + pip install --no-deps -e .)

• PR's own stated test command (9 files) → 168 passed, matching the PR description exactly
• Merge-diff-touched files (test_rater_observation*.py, test_review_gateway*.py, test_chat_model_capability_isolation.py) → 86 passed
• test_orchestrator_client_boundaries.py + test_provider_reliability.py → 3 failed (as above, pre-existing, unchanged by this merge), 68 passed

Pushed directly to fix/batch-endpoint-capability-gate (no force-push; merge commit on top of the existing single commit). Flagging for the author/reviewer: the 3 failures above need a decision and fix before this can pass required CI.

---

Generated by Claude Code

[seonghobae]

Fixed: two tests broke under the fail-closed default

batch_chat()'s new default (batch_endpoint_supported unset → emulate rather than hit the real Batch API) broke two boundary tests that constructed remote HTTPS agents specifically to exercise the real Batch API path:

• tests/test_orchestrator_client_boundaries.py::test_batch_chat_success_on_https_provider_returns_validated_results
• tests/test_orchestrator_client_boundaries.py::test_batch_chat_wraps_provider_failures_without_provider_text
• tests/test_provider_reliability.py::test_batch_boundary_hides_raw_upload_error_text_and_cause

Fix (ecac975d): added batch_endpoint_supported=True to the ModelAgent these tests construct, so they keep exercising the real-Batch-API branch they were written for — consistent with how the PR's new test_batch_api.py cases opt in.

Verification evidence

Full suite, run twice (once via a stray background process from an earlier turn that I killed as stale, once as a fresh blocking run against the pushed head) in the PR worktree at /home/user/work/wt/co-1021-batch-endpoint-gate:

python -m pytest tests -q
3350 passed, 2 skipped, 3 failed in 847.77s (0:14:07)

Targeted re-run of every file touched by this PR or by the test fix, in isolation:

python -m pytest tests/test_batch_api.py tests/test_orchestrator_client_boundaries.py tests/test_provider_reliability.py -q
77 passed in 7.59s

The 3 failures in the full run are pre-existing and out of scope — confirmed by git diff origin/main -- <file> being empty for all three (this branch never touches them):

• test_admin_contract.py::test_model_group_mutations_refresh_audit_events — NameError: name 'json' is not defined (missing import in the test file itself, unrelated to batch routing)
• test_psychometric_routing.py::test_fast_mlsirm_fit_uses_judge_acceptance_item_for_context_score — ModuleNotFoundError: No module named 'fast_mlsirm' (optional dependency not installed in this environment)
• test_spend_analytics.py::test_exact_output_without_prompt_usage_is_explicitly_unavailable — usage_source assertion ('tokenizer' == 'mixed'), unrelated to batch endpoint gating

Pushed to fix/batch-endpoint-capability-gate (head now ecac975d, non-force). No other changes made beyond the two test files.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

---

Generated by Claude Code

[seonghobae]

Autonomous loop note: noema-review (run 33733437527) failed with HTTP Error 502: Bad Gateway; phase=connecting, duration=2049.1s — transient upstream gateway infra, not a review verdict. All other checks on this head are green. Re-ran the failed job; no source change needed.

---

Generated by Claude Code

[seonghobae]

목표 #32 exact-head 재검증

• head: 61d5170b96a3e579ebe1aa237d0c55b67246e72c
• base: 2e414d15ba58f28597751b625a8a2f00fc9fadcf
• merge-base: base와 동일

근본 원인은 chat 호환 모델 식별자를 provider Batch API 지원 증거로 오인한 것이었습니다. 이 PR은 batch_endpoint_supported is True인 agent만 실제 /files → /batches → poll → result-file 경로로 보내고, None 또는 False인 원격 agent는 각 항목을 일반 chat endpoint로 호출해 같은 결과 형태로 집계합니다. 일반 chat 요청은 기존 sync 경로를 유지합니다.

OpenAI 공식 계약도 Batch를 모델명 기반 자동 전환으로 정의하지 않습니다. purpose=batch JSONL 파일을 올린 뒤 input_file_id, 대상 endpoint, completion_window=24h로 별도 비동기 작업을 명시적으로 생성합니다.

• 공식 가이드: https://developers.openai.com/api/docs/guides/batch
• create API: https://developers.openai.com/api/reference/resources/batches/methods/create
• canonical DB 경계: pg-llm-batch는 endpoint 검증과 upload/create/poll/retrieve를 담당하며 provider capability 판정은 호스트에 남깁니다. source는 복제하지 않았습니다.

검증:

• focused batch/routing/API/persistence 경계: 190 passed in 6.30s
• 전체 suite: 3397 passed, 2 skipped in 651.75s
• git diff --check origin/main...HEAD: 통과
• 미해결 리뷰 스레드: 0

README도 실제 Batch와 명시적 emulation 경계를 정확히 설명하도록 맞췄습니다. 제품 PR이므로 admin bypass 없이 기존 auto-merge와 보호 규칙을 따릅니다.

@coderabbitai review

[coderabbitai]

@seonghobae I will review pull request #1021 at the reported head commit.

⚠️ Action not completed

Review rate limited.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.