Skip to content

fix: classify tool-runtime failures and fail safely - #771

Merged
seonghobae merged 24 commits into
mainfrom
fix/issue-567-tool-failure
Aug 24, 2026
Merged

fix: classify tool-runtime failures and fail safely#771
seonghobae merged 24 commits into
mainfrom
fix/issue-567-tool-failure

Conversation

@seonghobae

@seonghobae seonghobae commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Buyer-visible outcome

A missing tool no longer terminates an otherwise recoverable workflow, while uncertain state-changing outcomes are stopped safely.

Implementation

  • Adds provider-neutral ToolFailureKind, ToolFallbackAction, ToolFailureDecision, and structured ToolExecutionError.
  • Recognizes the exact Strix missing-tool error through a bounded, cycle-safe cause chain.
  • Retries only explicitly idempotent transient tool failures, with full-jitter backoff and a shared four-attempt ceiling.
  • Fails over missing/unavailable tools and preserves generic provider failover.
  • Fails closed for ambiguous outcomes, malformed arguments, authorization, and policy denials.
  • Keeps public errors and audit events free of prompts, tool arguments, provider bodies, credentials, and raw exception text.
  • Publishes JSON/SSE fail-closed error contracts and APA 7 doctoring for RFC 9110 and NIST AI 600-1.
  • Does not add a duplicate hourly scheduler; the existing provider catalog scheduler and central maintenance scheduler remain the single owners.

Exact-head evidence

  • Full suite: 1528 passed
  • Focused fallback/security/HTTP regression suite: 142 passed
  • New module coverage: 100% statement, branch, and docstring coverage
  • compileall, git diff --check, and Trivy CRITICAL/HIGH scan passed

Closes #567


Open in Devin Review

Summary by CodeRabbit

  • 새 기능

    • 도구 실행 실패를 분류해 제한적으로 재시도하거나 다른 에이전트로 전환합니다.
    • 재시도 불가 또는 결과가 불확실한 작업은 안전하게 중단합니다.
    • 일반 요청과 스트리밍 응답에 구조화된 도구 실행 오류를 제공합니다.
    • 감사 기록에 민감정보 없이 실패 유형과 처리 결과를 남깁니다.
  • 개선

    • 제공자 오류와 검색 실패에서 원시 예외 및 응답 내용이 노출되지 않습니다.
    • 도구 실행 재시도에 지수 백오프와 상한을 적용합니다.
  • 문서 및 테스트

    • 도구 오류 처리 정책과 운영 지침을 문서화했습니다.
    • 재시도, 페일오버, 오류 응답, 민감정보 비노출에 대한 회귀 테스트를 추가했습니다.

@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@seonghobae, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 27 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 1c13de93-a989-4119-a7a2-3cb23ead6c5a

📥 Commits

Reviewing files that changed from the base of the PR and between e258875 and c05ab7e.

📒 Files selected for processing (3)
  • contextual_orchestrator/orchestrator.py
  • tests/test_provider_reliability.py
  • tests/test_tool_execution_fallback.py
📝 Walkthrough

Walkthrough

도구 실행 실패를 안정적인 유형으로 분류하고, 멱등성 및 결과 불확실성에 따라 재시도, 에이전트 failover 또는 fail-closed를 적용한다. Provider 원시 오류를 외부 응답과 감사 이벤트에서 제거한다. HTTP 및 SSE 오류 계약과 회귀 테스트를 추가한다.

Changes

도구 실행 fallback 정책

Layer / File(s) Summary
실패 분류 계약과 정책
contextual_orchestrator/tool_fallback.py, contextual_orchestrator/__init__.py, docs/adr/..., docs/doctoring/..., docs/tool_execution_fallbacks.md, docs/superpowers/plans/...
도구 실패 유형과 fallback 동작을 정의한다. 명시적 멱등 실패는 제한적으로 재시도하고, 결과 불확실성·잘못된 인수·권한·정책 거부는 fail-closed로 처리한다. 공개 엔터티와 정책 문서를 추가한다.
오케스트레이터 재시도와 failover
contextual_orchestrator/orchestrator.py, tests/test_tool_execution_fallback.py, tests/test_tool_fallback_wrapped_metadata.py
TaskOrchestrator가 재시도 예산, 지수 백오프와 jitter를 적용한다. 재시도 소진 후 다음 적격 에이전트로 전환한다. fallback 결정, 회로 상태와 비밀정보 없는 감사 이벤트를 검증한다.
Provider 및 HTTP 오류 경계
contextual_orchestrator/model_discovery.py, contextual_orchestrator/server.py, contextual_orchestrator/orchestrator.py, tests/test_model_discovery.py, tests/test_model_judge.py, tests/test_provider_reliability.py, tests/*_http_honesty.py
Provider 오류를 안정적인 코드와 예외로 변환한다. 도구 실행 중단을 HTTP 409 JSON 또는 finish_reason: "error"인 SSE 응답으로 반환한다. 원시 오류와 원인을 노출하지 않는 동작을 검증한다.
회귀 검증과 변경 기록
CHANGELOG.md, README.md, tests/test_tool_execution_fallback.py, tests/test_provider_reliability.py
도구 fallback 정책, 오류 계약, 비밀정보 비노출 및 점검 명령을 변경 기록과 테스트에 반영한다.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: 🟠 High · up to e2588

A provider-originated HTTP 409 can be converted into a generic failure before terminal handling, potentially causing a state-changing operation to be retried or re-executed and preventing the documented 409/SSE response. This high-impact correctness risk should be fixed before merge.

Sequence Diagram(s)

sequenceDiagram
  participant ToolAdapter
  participant TaskOrchestrator
  participant BackupAgent
  participant HTTPServer
  participant Client
  ToolAdapter->>TaskOrchestrator: 도구 실행 결과 또는 구조화된 실패 전달
  TaskOrchestrator->>TaskOrchestrator: 실패 유형과 멱등성 판정
  TaskOrchestrator->>ToolAdapter: 허용된 동일 에이전트 재시도
  TaskOrchestrator->>BackupAgent: 재시도 소진 또는 도구 부재 후 failover
  TaskOrchestrator->>HTTPServer: fail-closed 오류 전달
  HTTPServer->>Client: 409 JSON 또는 오류 SSE 프레임 전송
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 34.74% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 95 functions across 12 files. (3 skipped: 3 unsupported.) Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed 제목은 도구 런타임 실패 분류와 안전한 폴백이라는 주요 변경을 간결하게 설명합니다.
Linked Issues check ✅ Passed 변경 사항은 #567의 실패 분류, 제한된 재시도, 순차적 폴백, fail-closed 처리와 보안·문서·테스트 요구사항을 충족합니다.
Out of Scope Changes check ✅ Passed 코드, 문서, 변경 로그, README 및 테스트 변경은 모두 도구 실패 폴백과 오류 경계 강화 목표에 관련됩니다.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/issue-567-tool-failure

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Review exact current HEAD 6cc4466dcbc8a9ba8367b8a0ed0775207b7edf7e only.

The PR is limited to issue #567's provider-neutral tool-runtime failure contract and bounded fallback integration. Fresh evidence for this SHA: full suite 1528 passed, focused fallback/security/HTTP regression suite 142 passed, new tool_fallback.py statement/branch/docstring coverage 100%, compileall, diff check, and Trivy CRITICAL/HIGH scan passed. The hourly scheduler is intentionally not duplicated because existing scheduler owners remain in the provider catalog and central maintenance workflows.

Please publish a formal current-head review verdict using only same-head Checks and evidence. Do not push unrelated changes.

@seonghobae

Copy link
Copy Markdown
Contributor Author

@opencode-agent Exact current head 2351cab adds the root-cause hardening found in review: the bounded exception-chain classifier now honors explicit raise ... from None suppression, and non-string tool names fail closed at construction. New focused fallback suite: 94 passed; diff check clean. Review this exact SHA only and preserve the single-owner hourly scheduler design.

@seonghobae

Copy link
Copy Markdown
Contributor Author

@opencode-agent Exact current head 2351cabdeeed5b024d86483a02baeecce14e35b2 passed the full repository suite: 1529 passed in 554.43s. Focused tool fallback suite: 94 passed. Review this exact SHA; the bounded classifier honors suppressed exception contexts and the public HTTP/SSE contracts remain secret-free.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Review current HEAD 2351cabdeeed5b024d86483a02baeecce14e35b2 only. A concurrent agent added the valid exception-chain boundary fix: suppressed raise from None contexts are no longer reclassified, and non-string tool names fail with the documented validation error. Fresh exact-head evidence: fallback suite 94 passed; compileall and diff check passed. The full suite is running on this exact SHA now. Please publish a formal same-head verdict; do not rely on the predecessor 6cc4466 evidence or push unrelated changes.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Fresh verification completed on exact current HEAD 2351cabdeeed5b024d86483a02baeecce14e35b2 after the concurrent exception-context repair:

  • Full suite: 1529 passed in 528.55s
  • Focused tool fallback suite: 94 passed
  • tool_fallback.py: 100% statement, branch, and docstring coverage
  • compileall and diff check: passed

The protected PR remains unmergeable until the same-head Hosted Checks and required independent approval are present. Please review this exact SHA only and publish the formal verdict.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Current-head review at f60adbc:

  • The structured tool-failure and external-admin resource-policy changes remain on the current head; the new CHANGELOG entry is additive and matches the security behavior.
  • I found no additional source defect in the current diff. Exact local evidence from the current stack remains subject to the hosted required Checks; no independent approval is recorded, so merge is not authorized.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact current head e1d4170e6e6a5f1c9878e3b16ca2c6b42ad3dd8a was rechecked. The focused fallback/security and affected HTTP contract suite passes (134 passed), and Ruff plus diff check pass. I repaired only stale test entry points, a duplicate dict literal, and an unused import; the product/security behavior is unchanged. Please review this exact SHA; protected main still requires independent approval and current hosted Checks.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Correction: the exact pushed head is e1d417070885d5c0d0f0e62bd7d2e07736e87f01. The prior comment contained an incorrect full-SHA expansion; use this correction as the only exact-head reference.

@seonghobae

Copy link
Copy Markdown
Contributor Author

@seonghobae

Copy link
Copy Markdown
Contributor Author

@devin review exact current HEAD e1d417070885d5c0d0f0e62bd7d2e07736e87f01 against main; focus on structured tool-failure classification, idempotent-only retries, fail-closed ambiguous outcomes, secret-free JSON/SSE error contracts, and circuit-breaker behavior.

@seonghobae

Copy link
Copy Markdown
Contributor Author

@opencode-agent Review exact current HEAD only. Verify structured tool-failure classification, idempotent-only retries, fail-closed ambiguous outcomes, secret-free HTTP/SSE errors, and circuit-breaker behavior. Publish a formal verdict from same-head evidence.

@seonghobae

Copy link
Copy Markdown
Contributor Author

@opencode-agent Review exact current HEAD e1d4170 only. Verify structured tool-failure classification, idempotent-only retries, fail-closed ambiguous outcomes, secret-free HTTP/SSE errors, and circuit-breaker behavior. Publish a formal verdict from same-head evidence.

coderabbitai[bot]

This comment was marked as resolved.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Applied the raw provider boundary follow-up at exact head cc806cdbd93b01dcb77ee96ba700ae59565ca2e2:

  • ProviderResponseError now exits TaskOrchestrator._invoke before tool-failure classification, retry, cross-agent failover, or circuit-breaker mutation.
  • Added regression coverage proving a malformed primary response does not call the backup agent and leaves circuit state empty.
  • Exact focused fallback/provider suite: 112 passed; compileall and git diff --check passed.

This keeps #771 safe when stacked with the provider boundary in #807. Please review this exact HEAD; merge remains gated by independent approval and protected Checks.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Correction to the preceding note: the exact pushed HEAD is cc806cdb809068b78388d843758086747a21750a. Use this full SHA as the only current-head reference; the abbreviated SHA in the preceding note was expanded incorrectly.

devin-ai-integration[bot]

This comment was marked as resolved.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head status refresh for PR #771\n\n- Live head: cc806cd\n- Base: e226e11\n- The branch advanced after the prior e258875 audit; all predecessor evidence is stale for merge decisions.\n- Current required workflows were newly queued at 2026-08-21T12:12:15Z; approvals remain 0.\n- Current-head review contains a valid provider 409 tool_execution_stopped boundary finding and a direct-run coverage finding. A local successor commit 276ed4f addresses both and passes focused 114, direct fallback 96, and full 1538 tests, plus compileall/actionlint/diff-check/Semgrep/pip-audit.\n- The normal push of that successor was rejected by active ruleset 18156473 because changes must be made through a pull request and the required workflows for the new head are not yet satisfied. No bypass, direct protected push, or stale evidence reuse was used.\n- Repository coverage remains 90% statement with 146 partial branches and docstring coverage 95.9%, below the 100% standard.\n- Decision: WAIT_AND_REMEDIATE.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Reviewed the current head cc806cdb809068b78388d843758086747a21750a against the provider transport boundary. The remaining issue was real: HTTP 409 tool_execution_stopped was correctly recognized as non-transient, but _send_with_retry() and _send_raw_with_retry() discarded it as a generic RuntimeError, and the streaming provider-open path did not preserve the public fail-closed contract. Stacked fix PR #811 (f0b0dd565f93d8f4aa90ca6ad67544c6b6b8051f) preserves the existing ToolFallbackStoppedError / 409 tool_execution_stopped contract and adds HTTP and SSE end-to-end regressions. Local focused validation: 115 passed, compileall, and diff check passed. PR #811 is open against this PR's branch and must merge before #771.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact remote-head evidence for cc806cdb809068b78388d843758086747a21750a:

  • Full suite: 1536 passed in 527.97s.
  • Focused provider/tool-fallback suite on the same remote HEAD: 112 passed.
  • Compileall and git diff --check passed.

A concurrent local follow-up exists as 276ed4f04a065735577ab81e065dad54250b90b3 (not present on the remote PR): it preserves the terminal provider 409 tool_execution_stopped HTTP contract and asserts the provider URL/body is absent. Its focused provider/tool-fallback suite is 112 passed; compileall and diff check pass. A normal fast-forward push was attempted and rejected by the active repository rule until required central workflows are satisfied, so this follow-up is not claimed as remote PR content.

The remote PR remains protected and unmerged pending current-head Checks and independent approval.

devin-ai-integration[bot]

This comment was marked as resolved.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head verification after the normal parent repair push:

  • Remote HEAD: 276ed4f04a065735577ab81e065dad54250b90b3
  • Base: e226e1197bdfc890c9d8e5b9b648c78857d7e465
  • The current tree preserves the provider raw-error boundary and immediately re-raises ProviderResponseError before tool-failure classification, retry, failover, or circuit mutation.
  • Focused provider/tool fallback coverage: 112 passed; direct fallback module execution, compileall, and diff check passed.
  • The parent exact tree had already passed the full repository suite at 1536 tests; the pushed successor adds the provider terminal-tool regression and was revalidated locally on the exact commit.

Fresh hosted Checks are pending and no qualifying independent formal approval is present. No merge, self-approval, bypass, or force push was performed.

@seonghobae

Copy link
Copy Markdown
Contributor Author

Exact-head follow-up: removed the unreachable last_error/ProviderResponseError tail from TaskOrchestrator._invoke after the immediate sanitized re-raise introduced by the provider-error boundary.

  • Current pushed head: b24b3a3c434069057a2853f03736d06245988e4a
  • Base: e226e1197bdfc890c9d8e5b9b648c78857d7e465
  • Focused provider reliability and tool-fallback tests: 117 passed
  • git diff --check: passed
  • No formal approval or terminal hosted Checks yet; no merge attempted.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Open in Devin Review

Comment thread contextual_orchestrator/orchestrator.py
@seonghobae
seonghobae enabled auto-merge (squash) August 21, 2026 16:40
@opencode-agent opencode-agent Bot added priority: medium Normal-priority or P2 work status: needs-review Open pull request requiring current-head review or checks type: bug Defect or incorrect behavior labels Aug 22, 2026
@seonghobae
seonghobae merged commit 84a40cb into main Aug 24, 2026
34 of 35 checks passed
@seonghobae
seonghobae deleted the fix/issue-567-tool-failure branch August 24, 2026 07:58
seonghobae added a commit that referenced this pull request Aug 25, 2026
* fix: extend provider error boundary to streaming and batch paths

- _stream_send: mid-stream failures surface one package-owned error; the
  terminal tool-stop SSE contract is preserved (CWE-209)
- batch_chat: upload/poll/download failures no longer leak raw urllib text
- model discovery: raw connection resets (OSError, non-URLError) map to the
  stable transport_error code
- ADR 0011: document the streaming/batch boundary extension

Supersedes the still-valid delta of #807 after #771 landed the core boundary.

* fix(orchestrator): tolerate empty or missing choices in SSE streams

A provider usage-only frame can emit choices: [] or omit choices entirely.
The previous  raised IndexError when the key
existed but the list was empty, and the broad provider error boundary would
then abort an otherwise valid stream. Normalise the choices list before
indexing and add a regression test covering both empty and missing choices.

Devin Review: contextual-orchestrator#830

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: medium Normal-priority or P2 work status: needs-review Open pull request requiring current-head review or checks type: bug Defect or incorrect behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: classify tool-runtime failures and fall back safely

1 participant