feat(discovery): record parallel tool-call capability and exclude single-tool models from orchestrator/free by devin-ai-integration[bot] · Pull Request #972 · ContextualWisdomLab/contextual-orchestrator

devin-ai-integration · 2026-08-31T09:57:37Z

Summary

orchestrator/free is the zero-cost, capability-blind pool used by the org's central review agents (OpenCode, Noema, Strix). A model in this pool must accept arbitrary chat requests, including requests with multiple tool calls. Issue #940 shows that NIM's meta/llama-3.2-11b-vision-instruct rejects multi-tool requests with This model only supports single tool-calls at once!. Previously DiscoveredModel carried no tool-call parallelism signal, so there was no honest way to keep such a model out of the general free pool.

This PR adds supports_parallel_tool_calls: bool | None = None to DiscoveredModel, wires it through discovery-time selectors and runtime ModelAgent tags, and rejects single-tool-call agents from _is_general_free_agent.

What changed

DiscoveredModel now has supports_parallel_tool_calls.
_parallel_tool_call_evidence() reads a positive signal from provider supported_parameters containing "parallel_tool_calls".
probe_discovered_model_tool_call_capability() performs an opt-in live POST /chat/completions probe with parallel_tool_calls: true and two tool definitions. It returns True on 200, False when the 400 body explicitly says single-tool-call only, and None on any ambiguous/network/auth error.
is_general_chat_candidate(..., supports_parallel_tool_calls=...) returns False when the evidence is False.
is_discovered_chat_candidate and is_routable_discovered_model now use the field.
general_free_serving_candidates excludes models whose evidence is False (via _requires_single_tool_call).
agent_from_discovered and provider_bootstrap.serving_tags_for_discovered emit tool_call:multi or tool_call:single tags.
TaskOrchestrator._is_general_chat_agent derives the value from those tags and passes it to is_general_chat_candidate.
TaskOrchestrator._is_general_free_agent now requires not self._agent_requires_single_tool_call(agent) in addition to the existing non-text-input exclusion.
ADR 0039 and docs/product-technical-gap-baseline.md are added/updated.

Follow-up

ContextualWisdomLab/.github's review sidecar still builds its own catalog and does not consume general_free_serving_candidates or preserve input:/tool_call: tags. A separate PR to the .github repo is needed before the NIM vision/single-tool models are fully excluded from CI orchestrator/free selection.

Fixes #940.

Link to Devin session: https://app.devin.ai/sessions/97593a84ee7649e5afa96899f5c4c475
Open in Devin Desktop: https://app.devin.ai/desktop/session/97593a84ee7649e5afa96899f5c4c475?variant=devin

…gle-tool models from orchestrator/free - Add supports_parallel_tool_calls to DiscoveredModel (True/False/None). - Populate from provider supported_parameters listing parallel_tool_calls. - Add probe_discovered_model_tool_call_capability for live 400 evidence. - Wire field through is_general_chat_candidate, is_discovered_chat_candidate, is_routable_discovered_model, general_free_serving_candidates, agent_from_discovered, serving_tags_for_discovered, _is_general_chat_agent, and _is_general_free_agent. - Emit tool_call:multi/tool_call:single tags and reject tool_call:single agents from the general free pool. - Add tests and ADR 0039; update product-technical-gap-baseline.md. Fixes #940. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-08-31T09:57:38Z

Original prompt from Automation

당신은 ContextualWisdomLab 조직의 자율 개발 루프를 수행합니다. 이 세션은 스케줄에 의해 무인으로 시작되며, 사람에게 중간 보고를 하지 않습니다. 실행 결과를 보고하지 말고 다음 루프 작업을 계속하십시오. 질문으로 멈추지 말고, 판단 가능한 것은 스스로 판단해 진행하십시오.

#``# 0. 대상 선택

저장소는 고정되어 있지 않습니다. 매 실행마다 ContextualWisdomLab 조직에서 소유자가 온전한 권리를 가진 저장소들을 살펴보고, 레버리지가 큰 순서대로 작업 대상을 스스로 선택하십시오.

저장소를 선택할 때에는 이름이 아니라 제품·책임·재사용 경계와 기존 문서·구현 구조, 소비 저장소까지 대조해 실제 PR 위치가 맞는지 확인하십시오.

참고 저장소:

@ContextualWisdomLab/.github

@ContextualWisdomLab/contextual-orchestrator

@ContextualWisdomLab/TEPP

@ContextualWisdomLab/fast-mlsirm

@ContextualWisdomLab/keyverse

@ContextualWisdomLab/RankWeave

@ContextualWisdomLab/ThreadWeave

@ContextualWisdomLab/disksage

@ContextualWisdomLab/wardnet

@ContextualWisdomLab/LineageWeave

naruon 저장소 및 조직 중앙 .github 와의 연계를 항상 고려하십시오.

#``# 1. 기본 루프 (중간 보고 없이 반복)
열린 PR마다: 리뷰 확인 → 수정 → Checks 재검증 → 병합 → 다음 개발.

대상 저장소의 열린 PR과 Issue를 모두 수집합니다.

각 PR의 리뷰 코멘트와 GitHub Checks 상태를 확인하고, 실패가 있으면 원인을 추적(Root Cause Analysis)해 근본 원인을 고칩니다.

not-merge-ready 상태의 PR은 직접 merge-ready 상태로 만듭니다.

Stacking 가능한 PR은 Stacking 하여 효율을 높입니다. Stacking 된 PR에서 중앙 OpenCode Agent가 리뷰를 하지 않으면 ContextualWisdomLab/.github 저장소를 수정합니다.

병합 가능한 PR은 병합합니다. 조직 내 유관 프로젝트의 PR이 있으면 함께 처리합니다(대개 원인이 엮여 있습니다).

PR과 Issue가 모두 소진되면, 구매자가 체감할 제품 Gap을 찾아 개발을 이어가고 새 PR을 만들어 같은 루프를 반복합니다.

소프트웨어를 완성하기 위한 Loop을 스스로 지속적으로 작성/제거하며 갱신하십시오. ADR과 Goal을 수시로 설정/변경/업데이트하고, Goal 수정이 불가능하면 Loop을 갱신하십시오.

이 예약 프롬프트 자체가 개선되어야 한다고 판단되면 Devin Automation을 업데이트해 프롬프트를 개선하십시오.

기준: 이 소프트웨어를 200억 달러에 판매할 수 있다는 자신이 있어야 합니다. 상용 소프트웨어 수준을 목표로 하십시오.

#``# 2. Gap 베이스라인 문서
ADR 및 리서치 자료, 현행 데이터를 근거로 기능 명세, PRD, TRD, UML 등을 도출하고 Gap을 찾으십시오. 현행 PR을 포함해 점검한 결과를 docs/product-technical-gap-baseline.md에 작성/갱신하고, 그 문서에 적힌 Gap을 개발하는 것도 루프에 포함합니다.

#``# 3. 아키텍처 원칙

조직 중앙 .github, naruon, 기타 저장소와 연결 가능한 구조. 따로 또 같이 쓰는 모듈러 MSA: 개별 소프트웨어로도 동작하고, 모듈로 반입해도 훌륭히 동작해야 합니다.

단일 소프트웨어가 Monolithic 하게 비대해지면 과감히 저장소를 분리하십시오.

소프트웨어 이름과 내부 호출자 이름이 크게 다르면(옛 이름 잔존 등) 교체하십시오.

코드 장애 ... (3993 chars truncated...)

devin-ai-integration · 2026-08-31T09:57:39Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

…er probe Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

…l-capability' into devin/17592943-parallel-tool-call-capability # Conflicts: # tests/test_auto_discovery_server.py

seonghobae · 2026-09-01T00:05:20Z

Cross-PR integration contract: routing identity is provider-neutral model_group only; do not add or preserve a provider-family abstraction. OpenRouter discovery must retain concrete free model IDs, while the aggregate openrouter/free router is not a serving candidate. OpenCode, Noema, and Strix must call contextual-orchestrator. Do not impose fixed wall-clock deadlines on inference, initial ping, readiness/health, provider discovery, or OpenRouter ZDR-list retrieval; use explicit cancellation and evidence-backed transport failure instead. Reconcile this PR with #971 and central .github #1508 before merge.

Resolves the merge conflict between this PR's parallel-tool-call capability work and main's independent evolution since this branch forked (stale main commit c6c3a0c). - orchestrator.py: combine HEAD's supports_parallel_tool_calls computation/threading in _is_general_chat_agent with main's independently-added "structured:blocked" guard. - model_discovery.py: combine HEAD's supported_parameters consensus aggregation and _parallel_tool_call_evidence call with main's max_output_tokens/context_window aggregation and top_provider lookup in _parse_openai_compatible. - __main__.py: restructure _auto_discover_runtime_agents to layer HEAD's capability-blocked tracking (discovery:blocked:capability tag with preserve-disabled semantics, mirroring main's existing spend/structured block-tag pattern) on top of main's independently landed configured-gateway structured-chat probing, embedding routability, and limits-changed tracking. The capability_blocked signal is now computed from a stripped (supports_parallel_tool_calls reset to None) DiscoveredModel so it stays a distinct block reason rather than silently folding into spend_routable. Removed HEAD's now-superseded _should_preserve_operator_disabled_state helper in favor of main's more general block_markers-based preserve-disabled computation. - tests: merged independently-added test functions in test_auto_discovery_server.py and test_model_discovery.py (no logical overlap, both sides' tests kept). - docs: renamed docs/planning/adrs/0039-parallel-tool-call-capability.md to 0042 (main independently claimed 0039 for 0039-request-scoped-configured-endpoint-routing.md since this branch forked); updated the matching gap-baseline.md cross-reference. Verified: PYTHONPATH=. python -m pytest tests -q (3342 passed, only the 5 known pre-existing failures unrelated to this change), 100% interrogate docstring coverage on touched files, git diff --check clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

seonghobae · 2026-09-01T22:57:15Z

Merged current main (4d143601) into this branch to bring it forward from its stale fork point (c6c3a0c9) and resolved the conflicts. Head is now a1103da4.

Why this wasn't redundant with main. Before resolving, I verified whether orchestrator/free-pool single-tool-call exclusion had already landed independently on main (it looked plausible, since model_discovery.py's discovery-time tool-call-evidence infrastructure — discovery_tool_call_tags, _parallel_tool_call_evidence, probe_discovered_model_tool_call_capability, etc. — is already present there). It turned out main already has TaskOrchestrator._agent_requires_single_tool_call wired into _is_general_free_agent (the free-pool eligibility check), but the broader _is_general_chat_agent free function — used for ordinary routing, synthesizer selection, and every non-free chat-selection path — did not yet exclude single-tool-call agents on main. So this PR's contribution to orchestrator.py is real and additive, not redundant.

Conflict resolution, file by file:

contextual_orchestrator/orchestrator.py (1 conflict) — combined this PR's supports_parallel_tool_calls computation/threading into _is_general_chat_agent with main's independently-added "structured:blocked" not in agent.tags guard. Both predicates now apply.
contextual_orchestrator/model_discovery.py (2 conflicts) — combined this PR's supported_parameters consensus field and _parallel_tool_call_evidence(...) call with main's independently-added max_output_tokens/context_window aggregation and top_provider lookup in _parse_openai_compatible. No overlap between the two additions.
contextual_orchestrator/__main__.py (6 conflicts, the substantial one) — main independently restructured _auto_discover_runtime_agents since this branch forked: configured-gateway structured-chat probing (_probe_configured_gateway_structured_chat), embedding-model routability, and limits-changed tracking as its own append path. I layered this PR's capability-blocked tracking on top of that structure rather than keeping this PR's older three-way branch: capability_blocked is now computed from a stripped DiscoveredModel (supports_parallel_tool_calls reset to None) before feeding is_routable_discovered_model/is_discovered_chat_candidate, so it stays a distinct, separately-tagged block reason (discovery:blocked:capability, with :preserve-disabled semantics mirroring main's existing spend:blocked/structured:blocked pattern) instead of silently collapsing into spend_routable. This PR's original _should_preserve_operator_disabled_state helper is superseded by main's more general block_markers-based preserve-disabled computation and was removed as dead code.
tests/test_auto_discovery_server.py, tests/test_model_discovery.py — all conflicts were independently-added, non-overlapping test functions from both sides; kept both.
docs/planning/adrs/0039-parallel-tool-call-capability.md → renamed to 0042. main independently claimed 0039 for 0039-request-scoped-configured-endpoint-routing.md since this branch forked (a same-number collision, not a redesign, per this repo's ADR-uniqueness convention); updated the matching docs/product-technical-gap-baseline.md cross-reference.

Verification: PYTHONPATH=. python -m pytest tests -q — 3342 passed, only the repo's 5 known pre-existing failures remain (test_orchestrated_responses_stream.py ×3 orchestrator/free regression, test_psychometric_routing.py fast_mlsirm import gap, test_spend_analytics.py::test_exact_output_without_prompt_usage_is_explicitly_unavailable), none of them related to this change. interrogate reports 100% docstring coverage on the touched files. git diff --check clean.

Generated by Claude Code

devin-ai-integration

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 3 new potential issues.

devin-ai-integration · 2026-09-01T22:59:46Z

+## Research grounding
+
+This decision is a capability-constrained routing safeguard, not a learned quality claim.
+It reuses the repository's existing vendored routing literature:
+
+- Chen, L., Zaharia, M., & Zou, J. (2023). *FrugalGPT: How to use large language models while reducing cost and improving performance*. arXiv. https://arxiv.org/abs/2305.05176
+- Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Rühle, V., Lakshmanan, L. V. S., & Awadallah, A. H. (2024). *Hybrid LLM: Cost-efficient and quality-aware query routing*. International Conference on Learning Representations. https://arxiv.org/abs/2404.14618
+- Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., & Stoica, I. (2024). *RouteLLM: Learning to route LLMs with preference data*. arXiv. https://arxiv.org/abs/2406.18665
+
+These papers justify preserving explicit capability evidence at the routing boundary.
+They do not justify inferring multi-tool support from model names or from a bare 200
+response, so this ADR keeps the field fail-closed on ambiguity.
+
+The cited PDFs are already vendored in `docs/papers/`; no additional restricted paper is
+copied in this run.


📝 Info: Research artifacts already exist

The cited routing papers already reside under docs/papers. The repository's research-grounding rule does not require duplicate PDFs for this feature.

Was this helpful? React with 👍 or 👎 to provide feedback.

…all tag loss, unbounded probe read) Two real findings from Devin's review of a1103da (the main-merge commit for this PR): 1. _refresh_discovered_tool_call_tags (contextual_orchestrator/__main__.py) conflated "a discovery marker for EITHER polarity has ever been seen" with "this specific visible tag is discovery-owned". Once discovery supplied evidence once, a later refresh would strip an operator's pre-existing tool_call:single/multi override too, even though it predated any discovery marker and belongs to the opposite polarity. Now only the visible tag paired with its own hidden discovery marker is treated as discovery-owned; an operator-authored tag of the other polarity survives evidence going from known back to unknown. 2. probe_discovered_model_tool_call_capability (model_discovery.py) read the entire provider response body with an unbounded response.read()/ exc.read(). Applied the same bounded-read-then-check pattern already used elsewhere in this module (MAX_DISCOVERY_RESPONSE_BYTES): an oversized body now returns None (ambiguous evidence) instead of buffering an unbounded amount of memory. Added regression tests for both: - test_auto_discovery_preserves_operator_tool_call_override_when_evidence_goes_stale - test_probe_discovered_model_tool_call_capability_rejects_oversized_response - test_probe_discovered_model_tool_call_capability_rejects_oversized_400_body The third finding (ADR 0042 already cites its research PDFs under docs/papers/) was informational, no action needed. Verified: targeted test files pass (194 passed), interrogate 100% on touched files, git diff --check clean; full suite rerun in progress. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

seonghobae · 2026-09-01T23:04:18Z

Addressed Devin's review of the merge commit (a1103da4). Head is now bd422bda.

Fixed — contextual_orchestrator/__main__.py:717-720, "Operator tool limits disappear". Real bug. _refresh_discovered_tool_call_tags used a single boolean (has_discovery_provenance, true whenever either discovery marker — single or multi — was present anywhere in the tag set) to decide whether to strip any plain tool_call:single/tool_call:multi tag. That meant an operator's pre-existing override of one polarity would get silently deleted the next time discovery evidence for the other polarity arrived and later went back to unknown — exactly the operator single -> discovery multi -> discovery unknown sequence you traced. Fixed by pairing each visible tag with its own hidden marker specifically (a discovery pass always writes them together, so a marker of one polarity never implies ownership of the other visible tag) and only stripping the visible tag that's actually paired with a marker present in the current tag set. Added test_auto_discovery_preserves_operator_tool_call_override_when_evidence_goes_stale covering exactly this sequence.

Fixed — contextual_orchestrator/model_discovery.py:253-258, "Capability probe accepts unbounded responses". Real, and this module already has an established fix for exactly this shape of problem: MAX_DISCOVERY_RESPONSE_BYTES (8 MiB) + the bounded-read-then-check pattern already used by _fetch_json, _fetch_json_same_host_https, and _fetch_configured_gateway_json. Applied the same pattern to both the success-path response.read() and the 400-error-path exc.read() in probe_discovered_model_tool_call_capability; an oversized body now returns None (ambiguous evidence, matching this function's existing fail-open-on-ambiguity design) instead of buffering an unbounded amount into memory. Added test_probe_discovered_model_tool_call_capability_rejects_oversized_response and ..._rejects_oversized_400_body.

No action — docs/planning/adrs/0042-parallel-tool-call-capability.md:99-113, "Research artifacts already exist". Correct and informational only; the cited papers are already vendored under docs/papers/, nothing to change.

Verification: targeted test files (test_model_discovery.py, test_auto_discovery_server.py) pass (194 passed), interrogate 100% on touched files, git diff --check clean. Full PYTHONPATH=. python -m pytest tests -q rerun in progress; will follow up if it surfaces anything.

Generated by Claude Code

seonghobae · 2026-09-01T23:07:23Z

Confirmed — the same-polarity case is real and distinct from what I just fixed. Reproduced directly:

model = DiscoveredModel(..., supports_parallel_tool_calls=False)  # discovery agrees: single
tags = ("discovered", "chat", "tool_call:single")  # operator already set this, no discovery marker yet

tags = _refresh_discovered_tool_call_tags(tags, model)
# -> ('discovered', 'chat', 'tool_call:single', 'tool_call:single', 'discovery:tool_call:single')
#    (duplicate 'tool_call:single' -- one from the operator, one just appended by discovery_tool_call_tags)

tags = _refresh_discovered_tool_call_tags(tags, replace(model, supports_parallel_tool_calls=None))
# -> ('discovered', 'chat')  -- BOTH copies of 'tool_call:single' are gone, including the operator's

Why I'm not pushing a fix for this one unilaterally. Unlike the different-polarity case, this isn't fixable by pairing markers with their own visible tag more carefully — the two tool_call:single occurrences are literally the same string, so once discovery agrees with an operator's pre-existing override even once, there is no remaining information in the flat tags: tuple[str, ...] that says which occurrence was the operator's and which was discovery's. That's a genuine data-model gap, not a bug in the refresh logic's control flow, and ModelAgent.patch_agent() already lets an operator set arbitrary tags directly (patch_agent(pool, id, {"tags": [...]})), so this isn't a theoretical scenario.

Three ways to actually close this, none of which I want to pick unilaterally:

Namespace operator overrides separately — e.g. a tool_call_override:single/tool_call_override:multi prefix that _refresh_discovered_tool_call_tags never touches, with _is_general_chat_agent/_agent_requires_single_tool_call-style consumers checking the override namespace first. Clean and permanent, but touches every consumer of tool_call:* semantics, and there's no way to retroactively distinguish already-persisted plain tags that were operator-set under the current scheme.
Make "unknown" sticky instead of clearing — once discovery or an operator has established tool_call:single/tool_call:multi, a later None (unknown) result no longer wipes it; only an explicit opposite-polarity discovery result would replace it. This incidentally fixes the same-polarity case too (nothing gets removed on unknown, regardless of origin), but changes what "unknown evidence" means for the pure-discovery case as well — worth checking against ADR 0042's "None means no evidence either way" framing before deciding it's still consistent.
Document it as a known limitation and leave the current (already-improved) behavior as best-effort — the residual gap only bites when an operator's override happens to agree with discovery at least once before discovery's evidence later degrades to unknown, which is narrower than the different-polarity case I already fixed.

Happy to implement whichever direction you'd rather have; I don't think this should be guessed at silently given it's a safety-relevant override (excluding a model from parallel tool calls).

Generated by Claude Code

Seonghobae pushed test_discovery_tool_call_ownership.py directly to this branch, specifying the desired same-polarity behavior as failing acceptance tests (the different-polarity case was already fixed in bd422bd; this closes the remaining gap Devin's re-review confirmed was distinct). The gap: when an operator's pre-existing tool_call:single/multi tag happens to match discovery's own value, the two are string-identical, so _refresh_discovered_tool_call_tags had no way to tell them apart once discovery's hidden marker got attached -- a later refresh would treat both occurrences as discovery-owned and strip the operator's tag along with discovery's when evidence went stale. Fix: discovery never claims ownership (by adding its hidden marker) of a visible tag that is already present without one. Since the two tags are indistinguishable strings, the only safe rule is "if it was already there unclaimed, it stays unclaimed" -- discovery's matching evidence is still effectively honored (the correct value is already showing), but the marker that would let a later refresh treat it as discovery's own (and thus remove it in the same-polarity case) is never added. Verified by exhaustive trace across every existing and new scenario (different-polarity known->known->unknown, same-polarity known-> known->unknown, fresh-discovery no-operator, discovery value flips, repeated identical discovery evidence) before implementing, then confirmed: tests/test_discovery_tool_call_ownership.py (the new acceptance tests) + test_auto_discovery_server.py + test_model_discovery.py all pass (196 passed), plus the broader chat_capability/provider_bootstrap/ discover_models_cli/provider_catalog_store/multimodal_model_group_http suites (147 passed). interrogate 100% on touched files. git diff --check clean. Full suite rerun in progress. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

seonghobae · 2026-09-01T23:19:31Z

Merged 62d59fc1 (the new tests/test_discovery_tool_call_ownership.py acceptance tests) and implemented _refresh_discovered_tool_call_tags to satisfy both of them. Head is now b542003f.

The fix: discovery never claims ownership (by attaching its hidden discovery:tool_call:* marker) of a visible tool_call:single/tool_call:multi tag that's already present without one. Since a same-polarity operator tag and discovery's own tag are string-identical, that's the only safe rule — once a plain tag is unclaimed, it stays unclaimed forever, so a later refresh can never mistake it for discovery's own and strip it when evidence goes stale. Discovery's matching evidence is still effectively honored (the correct value is already showing on the agent), it just never becomes the thing a future "evidence withdrawn" pass is allowed to remove.

I traced this against every scenario before implementing — both new same-polarity tests, the different-polarity test from the previous fix, the pure-discovery lifecycle (fresh agent, evidence flipping between values, repeated identical evidence — no duplicate-tag regression), all check out. Verified: tests/test_discovery_tool_call_ownership.py + test_auto_discovery_server.py + test_model_discovery.py (196 passed), plus test_chat_capability.py/test_cli_role_effort_catalog.py/test_multimodal_model_group_http.py/test_provider_bootstrap.py/test_discover_models_cli.py/test_provider_catalog_store.py (147 passed), interrogate 100% on touched files, git diff --check clean. Full suite rerun in progress; will follow up if it surfaces anything.

Generated by Claude Code

…el-tool-call-capability

…el-tool-call-capability Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

seonghobae · 2026-09-02T17:27:02Z

noema-review failure diagnosis + branch update

Actual cause of the noema-review failure (checked the job log, not assumed stale-head): this was a genuine Noema review-model citation failure, not merely staleness. The reviewer's adversarial probe cited contextual_orchestrator/__main__.py line=889 as an "exact changed-side line," but that line wasn't an exact match against the diff's changed lines (nearest matches were 892, 893, 881, 880, 872, ...). The bounded repair mechanism couldn't reconcile the citation within its 900s wall-clock deadline (NoemaRepairDeadlineExceeded), so the check failed with no publishable verdict. All other checks on this head (CodeQL, Semgrep, Trivy, OSV, Scorecard, Atheris, Hypothesis, full unit/contract suite, package smoke) were green.

What I did:

git fetch origin main && git merge origin/main --no-edit — merged current main (through commit 212ff437, including the rater-observation/evaluation-criterion-binding and admin-audit-refresh work) into this branch. The merge was clean, no conflicts — including docs/product-technical-gap-baseline.md, which only received additive insertions from both sides.
Set up a venv per requirements.lock + pip install --no-deps -e ., then ran:
- The tests directly touching this PR's feature (supports_parallel_tool_calls, _parallel_tool_call_evidence, probe_discovered_model_tool_call_capability, _is_general_free_agent, tool-call tag ownership) across tests/test_discover_models_cli.py, tests/test_discovery_tool_call_ownership.py, tests/test_auto_discovery_server.py, tests/test_chat_capability.py, tests/test_multimodal_model_group_http.py, tests/test_provider_catalog_store.py, tests/test_model_discovery.py, tests/test_provider_bootstrap.py, tests/test_chat_model_capability_isolation.py — 357 passed.
- The full suite (python -m pytest tests -q, excluding test_psychometric_routing.py which needs numpy on Python ≥3.12 and isn't available in this sandbox's 3.11 venv): 3390 passed, 2 skipped, 2 failed in ~14 min. Both failures are the known pre-existing baseline issues (not regressions from this merge): test_admin_contract.py::test_model_group_mutations_refresh_audit_events (missing json import on main, tracked separately in fix(admin): repair test_model_group_mutations_refresh_audit_events #1029) and test_spend_analytics.py::test_exact_output_without_prompt_usage_is_explicitly_unavailable (usage_source mismatch). No new/unexplained failures.
Confirmed the remote branch hadn't moved past the head I started from, then pushed the merge commit directly (no force-push): 424a3613..0208f2ea.

This should give the required checks a fresh head to review/re-run against.

Generated by Claude Code

PR #972 (feat(discovery): record parallel tool-call capability and exclude single-tool models from orchestrator/free) independently added docs/planning/adrs/0042-parallel-tool-call-capability.md, colliding with this PR's 0042-opencode-go-provider-discovery.md -- neither number exists on main yet. Per this repo's CLAUDE.md, "a same-number collision is a rename, not a redesign," and PR #972 was updated earlier than this PR, so this PR's ADR renumbers instead. 0130 is the next free number after checking docs/planning/adrs/ on current origin/main (highest: 0126) and every other open PR's added ADR files (0127 PR #1020, 0128 PR #1012, 0129 PR #1030, plus PR #972's untouched 0042). Renamed docs/planning/adrs/0042-opencode-go-provider-discovery.md to 0130-opencode-go-provider-discovery.md, updated its front-matter id, and updated the three in-repo prose references to "ADR 0042" for this ADR (contextual_orchestrator/model_discovery.py comment, tests/test_model_discovery.py docstring, docs/kv-credentials.md, CHANGELOG.d/opencode-go-provider-discovery.md). Left the coincidental "0042" substring in docs/planning/adrs/0004-pr-review-merge-loop.md (part of an unrelated git SHA) untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BV96rXhqoR3tYZ9AeAVur4

cwl-noema-review

Noema LLM review

PR adds supports_parallel_tool_calls to DiscoveredModel and propagates it through gateway metadata merge, OpenAI-compatible parsing, duplicate reconciliation, catalog persistence, bootstrap tags, and runtime routing to exclude single-tool models from general chat/free pools. Prior review threads show all reported bugs were addressed with tests. Fresh diff review found no concrete blocking regressions.

Reviewed changed lines

contextual_orchestrator/chat_capability.py:143 (RIGHT): Guard if supports_parallel_tool_calls is False: return False excludes models with explicit single-tool evidence from general chat candidates while preserving existing eligibility for None evidence.
contextual_orchestrator/orchestrator.py:740 (RIGHT): _is_general_chat_agent derives supports_parallel_tool_calls from persisted tags and fails closed for conflicting tool_call:single/tool_call:multi evidence.

Adversarial validation

contextual_orchestrator/chat_capability.py:143 (RIGHT) falsified: Adding if supports_parallel_tool_calls is False: return False could regress routing for models where evidence is None, e.g., an implicit None argument now blocks a previously eligible model. — Default is None; the early return is only triggered for explicit False. The existing test_unproven_tool_call_parallelism_keeps_existing_eligibility in tests/test_chat_capability.py asserts None and True both preserve True eligibility.
contextual_orchestrator/orchestrator.py:740 (RIGHT) falsified: _is_general_chat_agent derives supports_parallel_tool_calls from tags, but a malformed agent carrying both tool_call:single and tool_call:multi could be admitted to general chat despite contradictory evidence. — The if "tool_call:single" in agent.tags branch takes precedence over tool_call:multi, setting supports_parallel_tool_calls to False. tests/test_chat_capability.py::test_conflicting_tool_call_tags_fail_closed explicitly constructs this scenario and asserts False.
Residual risk: None identified.

Findings

No blocking findings.

Result: APPROVE
Head SHA: 0208f2ea182d8a0134d53eac0d7e3fa3a1cb6ada
Reviewer credential: noema-review-github-app-refresh
Actor: cwl-noema-review[bot]

…el-tool-call-capability-local

seonghobae · 2026-09-03T06:47:33Z

`Full unit and contract suite` failure diagnosis + branch update

Diagnosis confirmed: pre-39a4348e staleness, not a defect in this PR's diff. The failing check (NameError: name 'json' is not defined in tests/test_admin_contract.py::test_model_group_mutations_refresh_audit_events) is the same shared bug introduced on main by commit 212ff437 (this PR's merge base) and fixed on main by 39a4348e (#1035, missing import json/shutil/subprocess plus three JS-extraction boundary bugs in that test). This PR's branch (previously merged to main@212ff437 via the earlier 424a3613..0208f2ea update) predates 39a4348e, so it inherited the bug. Confirmed via git diff origin/main..HEAD -- tests/test_admin_contract.py before merging: the diff was exactly the inverse of 39a4348e's fix, and 39a4348e was not yet an ancestor of this branch's head. This PR's own diff (discovery.py/chat_capability.py/orchestrator.py tool-call-parallelism work) does not touch test_admin_contract.py — same pattern already independently confirmed and fixed on #976 and #984 in the preceding hour.

The separately-failing CodeQL analysis check is the known, already-tracked org-wide infra issue ("CodeQL analyses from advanced configurations cannot be processed when the default setup is enabled"); left untouched here, per the standing note that it's being fixed centrally in ContextualWisdomLab/.github.

What was done:

Confirmed no concurrent activity on devin/17592943-parallel-tool-call-capability: head commit (0208f2ea, merge from the earlier main-sync) timestamped 2026-09-02T17:26:42Z, well outside any active-edit window, and unmoved on a re-fetch immediately before push.
git fetch origin main && git merge --no-edit origin/main on the branch in a scratch worktree — clean merge, no conflicts (main had advanced to c594b682, which includes 39a4348e). Merge commit: 876f1679.
Verified tests/test_admin_contract.py alone: 3 passed (previously 1 failed with the NameError).
Verified this PR's own affected test files (test_auto_discovery_server.py, test_chat_capability.py, test_discover_models_cli.py, test_discovery_tool_call_ownership.py, test_model_discovery.py, test_provider_bootstrap.py, test_provider_catalog_store.py): 301 passed.
Full suite, uv run --locked --extra api --extra db --extra queue --group dev python -m pytest -q (CI's exact invocation): 3437 passed, 2 skipped in 869.78s (14:29) — zero failures, no regressions.
Re-checked for concurrent branch movement immediately before pushing (head still 0208f2ea), then pushed the merge commit non-force: 0208f2ea..876f1679.

This should give the required checks a fresh, unblocked head to re-run against.

Generated by Claude Code

seonghobae · 2026-09-05T08:54:42Z

ADR number collision: `0042` is claimed by two open PRs

Found by an org-wide sweep of ADR filenames across main and all 62 open PRs. CLAUDE.md requires: "Planning filenames use four digits and must be unique across current main and every open PR. A same-number collision is a rename, not a redesign."

The collision

PR	File (status `added`)	PR created
#911	`docs/planning/adrs/0042-time-windowed-routing-observations.md`	2026-08-28T20:58:42Z
#972 (this PR)	`docs/planning/adrs/0042-parallel-tool-call-capability.md`	2026-08-31T09:57:37Z

0042 does not exist on main, so both are genuinely new claims on the same number rather than edits to an existing ADR.

Suggested resolution — a rename, not a redesign. #911 claimed the number first, so the smaller change is for this PR to renumber. The next free number is 0043 (occupied: 0001–0011, 0015, 0019–0042, 0124–0129; the mid-series gaps 0012–0014 and 0016–0018 look deliberate, so 0043 is the safe next sequential choice).

Nothing in this PR's content is in question — only the filename and any in-document self-reference to the number.

Method / limitations, so this is checkable rather than taken on faith. ADR filenames were collected via pulls/{n}/files for every open PR and intersected by four-digit prefix; only status != "removed" entries counted. No open PR exceeds 100 changed files, so single-page pagination did not truncate the scan. main itself is clean — 38 ADRs, zero duplicate numbers.

seonghobae · 2026-09-05T12:36:44Z

중복 판정: #972 × #1028 — 둘 다 살아 있고, 승계되지 않았으며, 태그 방출기만 정리하면 됩니다

두 PR이 비테스트 소스 5개 파일을 공유하고 서로 9곳에서 충돌하기에 중복 여부를 트리로 확인했습니다.

먼저: 어느 쪽도 `main`에 승계되지 않았습니다

main에 이미 있는 것과 없는 것을 갈랐습니다.

main 에 존재:
  contextual_orchestrator/model_discovery.py:79  def discovery_tool_call_tags(model) -> tuple[str, ...]
  contextual_orchestrator/model_discovery.py:81  model.supports_parallel_tool_calls is True
  → c7774d3e (2026-09-01) "fix(discovery): reject echoed tool probe definitions" 로 유입.
    #972 가 아니라 별개 PR입니다.

main 에 부재:
  _requires_single_tool_call        0건   ← #972 의 고유 서술어
  supports_tool_calls               0건   ← #1028 의 고유 필드
  tool_call_tags_for_discovered     0건   ← #1028 의 신규 함수

따라서 #972 를 "이미 반영됨"으로 닫으면 안 됩니다. 유입된 것은 태그 방출기와 필드뿐이고, 자유 풀에서 단일 도구 모델을 배제하는 서술어 자체는 main에 없습니다. 이 PR의 핵심 delta가 그대로 남아 있습니다.

둘은 중복이 아니라 상보입니다 — 방향이 반대라서 그렇게 보일 뿐입니다

#972    supports_parallel_tool_calls (병렬 도구 호출 가능 여부)
        _requires_single_tool_call(...)  →  병렬 불가 모델을 자유 풀에서 배제

#1028   supports_tool_calls (도구 호출 자체의 가능 여부)
        _declares_text_input(...)        →  도구 호출 가능한 비텍스트 입력 모델을 배제에서 면제

두 필드는 서로 다른 능력입니다 — "도구 호출이 되는가"와 "도구 호출을 병렬로 할 수 있는가"는 별개이고, 둘 다 있는 것이 맞습니다. 배제와 면제도 서로 다른 축(병렬 능력 / 입력 양식)에 겁니다. 어느 쪽도 다른 쪽을 대체하지 않습니다.

실제로 정리가 필요한 지점은 하나입니다

main에 이미 discovery_tool_call_tags가 있는데 #1028이 tool_call_tags_for_discovered라는 두 번째 태그 방출기를 같은 모듈에 추가합니다. 방출하는 태그와 읽는 필드가 달라 기능 중복은 아니지만, 같은 모듈에 역할과 이름이 거의 같은 함수가 둘이 되는 상태입니다.

이 저장소에서 오늘 이미 겪은 함정과 같은 모양입니다 — 이름이 비슷한 두 심볼의 안전 성질이 다르면, 다음 사람이 구조만 보고 하나를 지우거나 잘못된 쪽을 호출합니다.

권고: #1028의 신규 함수를 main의 discovery_tool_call_tags에 흡수하고, 두 능력 축의 태그를 한 함수가 함께 방출하게 하십시오. 필드는 둘 다 유지합니다.

순서

#1028   mergeable_state = behind   (충돌 없음)
#972    mergeable_state = dirty    (main 과 충돌)

#1028 먼저, 그다음 #972 rebase를 권합니다. #1028이 깨끗하고, #972가 어차피 재작업이 필요하므로 태그 방출기 통합을 그때 함께 처리하면 왕복이 한 번 줍니다.

판정 요약

어느 쪽도 닫지 마십시오. 중복은 태그 방출기 한 곳뿐이고, 나머지는 각자의 고유 delta입니다. AGENTS.md의 "Verifying a 'superseded — closing' claim" 기준으로, 두 PR의 핵심 서술어가 모두 main에 부재함을 트리로 확인했습니다.

측정 기준: origin/main 대비 three-dot diff, 각 PR head는 refs/pull/<n>/head. 어느 PR도 닫거나 편집하지 않았습니다.

devin-ai-integration Bot requested a review from seonghobae as a code owner August 31, 2026 09:57