Skip to content

fix(server): classify provider response 502 detail - #976

Open
seonghobae wants to merge 7 commits into
mainfrom
codex/commercial-loop-20260901-issue932
Open

fix(server): classify provider response 502 detail#976
seonghobae wants to merge 7 commits into
mainfrom
codex/commercial-loop-20260901-issue932

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a stable machine-readable provider_response_failure_kind detail for ProviderResponseError
  • distinguish reasoning-without-content from missing assistant content at the orchestrator boundary
  • preserve the existing top-level 502 invalid_structured_output contract while surfacing the new bounded detail in HTTP responses

Testing

  • uv run pytest -q tests/test_provider_reliability.py tests/test_chat_response_format_http_honesty.py
  • uv run pytest -q tests/test_provider_error_taxonomy.py

Fixes #932.


Devin Review

Summary by CodeRabbit

  • 개선 사항

    • 제공업체 응답 오류가 기계 판독 가능한 실패 유형과 상세 정보를 포함하도록 개선되었습니다.
    • 추론 내용만 있거나 응답 콘텐츠가 누락·빈 값인 경우를 구분해 처리합니다.
    • 관련 오류 응답에 재시도 가능 여부 등 구조화된 정보가 포함됩니다.
    • 내부 오류 메시지는 외부 응답에 노출되지 않습니다.
  • 테스트

    • 다양한 제공업체 응답 실패 유형과 HTTP 오류 응답 형식을 검증하는 테스트가 추가되었습니다.

@coderabbitai

coderabbitai Bot commented Aug 31, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 9fc7d520-a3ee-4992-be87-964cb3c6a031

📥 Commits

Reviewing files that changed from the base of the PR and between 212ff43 and fd9edee.

📒 Files selected for processing (4)
  • contextual_orchestrator/orchestrator.py
  • contextual_orchestrator/server.py
  • tests/test_chat_response_format_http_honesty.py
  • tests/test_provider_reliability.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Provider 응답의 콘텐츠 누락 원인을 기계 판독 가능한 실패 유형으로 분류합니다. 서버는 이 유형을 502 invalid_structured_output 응답의 detail에 포함합니다. 단위 테스트와 HTTP 통합 테스트가 분류 및 정보 비노출을 검증합니다.

Changes

Provider 응답 실패 처리

Layer / File(s) Summary
Provider 실패 유형 분류
contextual_orchestrator/orchestrator.py, tests/test_provider_reliability.py
ProviderResponseErrorfailure_kinddetail을 저장합니다. reasoning만 있거나 assistant 콘텐츠가 없거나 비어 있는 응답을 각각 분류합니다. 테스트는 reasoning 내용이 예외 메시지에 포함되지 않는지 검증합니다.
HTTP 오류 상세 정보 전달
contextual_orchestrator/server.py, tests/test_chat_response_format_http_honesty.py
서버가 ProviderResponseError.detailinvalid_structured_output 응답에 전달합니다. HTTP 테스트는 두 실패 유형, retryable 정보, 내부 오류 메시지 비노출을 검증합니다.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to fe414

The PR adds a bounded provider failure classification to existing 502 responses while preserving the established status and error code, helping clients distinguish failure modes without changing authorization or response handling. No actionable merge-blocking risk remains after normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant ModelClient
  participant ProviderResponseError
  participant server.do_POST
  participant HTTP Client
  ModelClient->>ProviderResponseError: failure_kind 저장
  ProviderResponseError-->>server.do_POST: detail 전달
  server.do_POST-->>HTTP Client: 502 invalid_structured_output 응답
Loading

Suggested reviewers: claude

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 3 files. (1 skipped: 1… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed PR 제목은 서버의 502 응답에 프로바이더 응답 실패 분류 정보를 추가하는 주요 변경을 정확하고 간결하게 설명합니다.
Linked Issues check ✅ Passed 변경 사항은 Issue #932의 요구사항을 충족합니다. ProviderResponseError는 reasoning_without_content와 assistant_content_missing을 provider_response_failure_kind로 분류하고, 서버는 이를 HTTP 502 invalid_structured_output 응답의 detail에…
Out of Scope Changes check ✅ Passed 변경된 오케스트레이터 로직, 서버 응답 처리, 통합 테스트 및 신뢰성 테스트는 모두 Issue #932의 기계 판독 가능한 502 오류 분류 요구사항과 직접 관련됩니다. 라우팅, 타임아웃, OpenRouter 동작 등 범위 외 변경은 확인되지 않습니다.
Full details: Linked Issues check

Explanation

변경 사항은 Issue #932의 요구사항을 충족합니다. ProviderResponseError는 reasoning_without_content와 assistant_content_missing을 provider_response_failure_kind로 분류하고, 서버는 이를 HTTP 502 invalid_structured_output 응답의 detail에 전달합니다. 관련 테스트도 두 실패 유형과 retryable 정보를 검증합니다.

Full details: Docstring Coverage

Explanation

Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 3 files. (1 skipped: 1 too large.)

✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/commercial-loop-20260901-issue932

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

devin-ai-integration[bot]

This comment was marked as resolved.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Cross-PR integration contract: routing identity is provider-neutral model_group only; do not add or preserve a provider-family abstraction. OpenRouter discovery must retain concrete free model IDs, while the aggregate openrouter/free router is not a serving candidate. OpenCode, Noema, and Strix must call contextual-orchestrator. Do not impose fixed wall-clock deadlines on inference, initial ping, readiness/health, provider discovery, or OpenRouter ZDR-list retrieval; use explicit cancellation and evidence-backed transport failure instead. Reconcile this PR with #971 and central .github #1508 before merge.

…-20260901-issue932

# Conflicts:
#	contextual_orchestrator/server.py
#	tests/test_chat_response_format_http_honesty.py

Copy link
Copy Markdown
Contributor Author

Pushed d2f6fa0e, merging current main in (normal merge commit, no force push) to resolve the dirty mergeable state. Two conflicts:

  • contextual_orchestrator/server.py: main had added a new except BatchDownloadError as exc: handler between BatchModelSelectionError and ProviderResponseError since this branch forked. Kept both -- main's new handler, and this PR's except ProviderResponseError as exc: (needed for the exc.detail this PR adds to the 502 response).
  • tests/test_chat_response_format_http_honesty.py: main added an import (ProviderUpstreamError) this branch didn't have, alongside this PR's own new import (ProviderResponseError). Both are used later in the file, so kept both.

Validation: full suite run (uv run pytest tests -q, Python 3.11 venv): 5 failures, 3304 passed, 2 skipped. Verified all 5 are pre-existing and unrelated to this PR by reproducing them identically against vanilla origin/main before this merge:

  • 3 in test_orchestrated_responses_stream.py (test_virtual_models_stream_openai_reasoning_summaries[orchestrator/free], test_http_virtual_responses_preserves_message_array_and_sampling_controls, test_stream_failure_emits_terminal_responses_event) and 1 in test_spend_analytics.py (test_exact_output_without_prompt_usage_is_explicitly_unavailable) all fail identically on vanilla main.
  • test_psychometric_routing.py::test_fast_mlsirm_fit_uses_judge_acceptance_item_for_context_score fails with ModuleNotFoundError: No module named 'fast_mlsirm' in both venvs -- an environment-scoped gap (the dependency is conditioned on python_full_version >= '3.12' in pyproject.toml; this reproduction venv is 3.11), not a real regression.

The three targeted test files this PR's own body names (test_provider_reliability.py, test_chat_response_format_http_honesty.py, test_provider_error_taxonomy.py) pass clean: 75 passed.


Generated by Claude Code

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 new potential issues.

Devin Review

Comment on lines +223 to +230
failure_kind: str = "invalid_provider_response",
detail: Mapping[str, Any] | None = None,
) -> None:
super().__init__(message)
self.failure_kind = failure_kind
self.detail = {
**(dict(detail) if detail else {}),
"provider_response_failure_kind": failure_kind,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Legacy failures stay generically classified

Other ProviderResponseError sites receive invalid_provider_response. This avoids mislabeling file, size, schema, and structured-output failures as content-specific failures.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +228 to +231
self.detail = {
**(dict(detail) if detail else {}),
"provider_response_failure_kind": failure_kind,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Canonical classification wins detail merges

provider_response_failure_kind overwrites any same-named detail entry. Supplementary fields survive without allowing contradictory classifications.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +2168 to +2178
if isinstance(content, str) and content:
return content
if isinstance(message, dict) and message.get("reasoning"):
raise ProviderResponseError(
f"provider {agent.id} returned reasoning without content; "
"for mlx-lm set chat_template_args={\"enable_thinking\": false} or increase max_output_tokens"
"for mlx-lm set chat_template_args={\"enable_thinking\": false} or increase max_output_tokens",
failure_kind="reasoning_without_content",
)
raise ProviderResponseError(f"provider {agent.id} response did not contain assistant content")
raise ProviderResponseError(
f"provider {agent.id} response did not contain assistant content",
failure_kind="assistant_content_missing",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 Info: Empty content follows missing-content handling

An empty answer no longer succeeds. Responses with reasoning retain the reasoning-only classification; other empty answers use the missing-content classification.

Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Two follow-ups on this PR's current head (d2f6fa0e, unchanged since my last comment):

  1. CI failure notification: the "Full unit and contract suite" failure just fired again on this same head — this is the identical 5-failure set (4 pre-existing test failures + 1 environment-scoped fast_mlsirm gap) I already root-caused and confirmed against vanilla main in my earlier comment. Nothing has changed on either side (main is still at 4d143601, this branch still at d2f6fa0e), so this is a delayed/duplicate delivery of the same already-explained failure, not a new one.

  2. Reconciliation with fix(routing): select concrete free model groups #971 and central .github#1508: checked this PR's actual diff against fix(routing): select concrete free model groups #971's stated scope (model_group-based routing selection, removing fixed wall-clock deadlines, OpenRouter concrete-free-model handling). This PR (#976) touches none of that surface — its diff is entirely ProviderResponseError.failure_kind/.detail classification for the 502 response (orchestrator.py, server.py, and their two test files). There's no provider-family abstraction, model_group logic, timeout code, or OpenRouter code anywhere in this diff to reconcile. This PR is already compliant with the provider-neutral routing-identity contract simply because it doesn't touch routing at all.

No action needed on either point; noting both so the reconciliation instruction isn't left silently unaddressed in the thread.


Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

Third follow-up on this PR's current head (d2f6fa0e, still unchanged): the noema-review required check failed with Noema reviewed line 3 is not an exact changed-side line.

Traced this to scripts/ci/noema_review_gate.py::validate_substantive_verdict (central .github script, executed here via the required-workflow ruleset) — it strictly rejects any reviewed_lines/adversarial_validation.probes entry whose (path, line, side) doesn't exactly match a location parsed straight out of the unified diff. This run's own sidecar log shows the review was routed through orchestrator/free's small/free-tier model pool (google/gemma-3-*, meta/llama-3.2-*, etc., not a frontier model) — line-index hallucination is a known failure mode for smaller models, and this is exactly the evidence-gating this validator exists to enforce (per this repo's and .github's "OpenCode/Noema judges PRs... evidence-gated" governance model): reject a claimed review location it can't verify against the actual diff, rather than accept unverifiable evidence.

This isn't a code bug in #976 itself, and I don't have evidence of a bug in the validator either (I can't inspect the LLM's raw verdict JSON from the job log to rule out an off-by-one, but the design intent and the specific model pool involved both point to a genuine hallucinated line rather than a validator defect). Treating this as a rejected review attempt that should self-resolve via the scheduler's normal redispatch, not something to force a code change for. If it recurs across multiple redispatch attempts on this same PR, that pattern would be worth escalating as a possible validator bug rather than continued model hallucination.


Generated by Claude Code

@seonghobae seonghobae added bug Something isn't working priority: medium Normal-priority or P2 work status: needs-review Open pull request requiring current-head review or checks type: bug Defect or incorrect behavior labels Sep 2, 2026 — with ChatGPT Codex Connector
…-20260901-issue932

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

Copy link
Copy Markdown
Contributor Author

noema-review failure investigation + branch update

Actual noema-review failure cause (checked the job log for run 33603290234, job 100161561401, not assumed): this was a genuine review-model output failure, not pure staleness. The sidecar's LLM phase produced findings that failed schema validation (findings must be a list of objects), and the deterministic JSON-repair pass also failed on the raw output (Expecting property name enclosed in double quotes: line 1 column 2130). All 27 other checks on this head were green (CodeQL, Semgrep, Trivy, OSV, Scorecard, dependency-review, Atheris, Hypothesis, the full unit/contract suite, etc.) — only the review-model's malformed JSON output caused the failure.

Since the PR was also mergeable_state: behind (base at 8839081..., main had since moved to 212ff437...), I merged main into this branch to give the required review a fresh head to re-run against:

  • git fetch origin main && git merge origin/main --no-edit — merged cleanly, no conflicts (main's changes — rater-observation/criterion-binding modules, admin audit refresh — don't overlap this PR's orchestrator.py/server.py changes).
  • Verification (Python 3.11 venv, hash-locked requirements.lock + pip install --no-deps -e .):
    • PR's own stated coverage — test_provider_reliability.py, test_chat_response_format_http_honesty.py, test_provider_error_taxonomy.py: 75 passed.
    • Tests touched by the merge diff (test_review_gateway*.py, test_rater_observation*.py, test_admin_contract.py): all pass except one known pre-existing failure (see below).
    • Full suite (pytest tests -q --ignore=tests/test_psychometric_routing.py, skipped since this sandbox runs Python 3.11 and that file needs numpy on Python≥3.12): 3346 passed, 2 skipped, 2 failed in 836s. Both failures (test_admin_contract.py::test_model_group_mutations_refresh_audit_events — missing import json, already being fixed elsewhere; test_spend_analytics.py::test_exact_output_without_prompt_usage_is_explicitly_unavailable) were confirmed to fail identically on unmodified origin/main — pre-existing, not regressions from this merge.
  • Pushed directly (no force) to codex/commercial-loop-20260901-issue932: dfbfcf76..fd9edee5.

Generated by Claude Code

Copy link
Copy Markdown
Contributor Author

"Full unit and contract suite" failure — root-caused and resolved

What failed: run 33658571997/100343181374 failed with a single NameError: name 'json' is not defined in tests/test_admin_contract.py::test_model_group_mutations_refresh_audit_events (1 failed, 3353 passed, 2 skipped). This PR's own diff (orchestrator.py/server.py/two test files, provider-response 502 classification) does not touch that test file at all.

Root cause confirmed as already-fixed, not a defect in this PR: this branch's previous head (fd9edee5) merged main at 212ff437 — the exact commit that introduced the missing-import bug in test_admin_contract.py. That bug was independently found and fixed on main via 39a4348e (#1035), landing after this branch's last main-merge. So the failing CI run was simply exercising a stale, since-fixed test on this branch's base.

Fix applied: git fetch origin main && git merge --no-edit (clean merge, zero conflicts — the intervening main history, mostly web_search/hourly-loop/ADR changes, doesn't overlap this PR's orchestrator.py/server.py diff). Pushed non-force: fd9edee5..fe414f8b.

Verification (Python 3.11 sandbox venv, hash-locked requirements.lock + pip install --no-deps -e .):

  • tests/test_admin_contract.py: 3 passed (previously 1 failed on the json NameError).
  • This PR's own stated coverage — tests/test_provider_reliability.py tests/test_chat_response_format_http_honesty.py tests/test_provider_error_taxonomy.py: 75 passed.
  • Full suite (pytest tests -q): 3391 passed, 2 skipped, 2 failed in 903.75s. Both failures are the same pre-existing, environment-scoped gaps already documented earlier in this thread for a Python 3.11 venv (CI runs Python 3.12): test_psychometric_routing.py::test_fast_mlsirm_fit_uses_judge_acceptance_item_for_context_score (fast_mlsirm is conditioned on python_full_version >= '3.12' in pyproject.toml, absent in this 3.11 sandbox) and test_spend_analytics.py::test_exact_output_without_prompt_usage_is_explicitly_unavailable (usage_source 'tokenizer' vs 'mixed') — neither appeared in the actual Python-3.12 CI run for this PR, and both were already reproduced identically against vanilla main in this thread's earlier comments.

PR is now caught up with main (including the admin-contract fix) and ready for CI to re-run clean.


Generated by Claude Code

seonghobae added a commit that referenced this pull request Sep 5, 2026
ProviderResponseError.detail was a read-only property that rebuilt a fresh
dict on every read. Sibling changes give the class a caller-owned detail:
#1004 assigns and mutates it in __init__ (`self.detail["workflow_run_id"] =
...`) and #976 sets and reads `provider_response_failure_kind`. Merged with
either, the property raised at construction (no setter) or silently dropped
item writes (fresh dict per read).

detail is now backed by a stored dict returned on every read, with a setter
that adopts a caller-supplied dict; attempts/stop_reason are mirrored into
the stored dict on read so failover evidence stays visible and never goes
stale, matching the contract pinned by
tests/test_provider_response_error_detail_compat.py (red on the previous
head: "property 'detail' ... has no setter").

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working priority: medium Normal-priority or P2 work status: needs-review Open pull request requiring current-head review or checks type: bug Defect or incorrect behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose a machine-readable field distinguishing ProviderResponseError causes through /v1/chat/completions' 502

2 participants