Skip to content

fix(delegation): runtime handles 202+queued; canvas surfaces delegation rows - #2126

Merged
hongmingwang-moleculeai merged 2 commits into
stagingfrom
fix/director-bypass-and-agent-comms
Apr 26, 2026
Merged

fix(delegation): runtime handles 202+queued; canvas surfaces delegation rows#2126
hongmingwang-moleculeai merged 2 commits into
stagingfrom
fix/director-bypass-and-agent-comms

Conversation

@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor

What this fixes

Two bugs that compounded into the "Director does the work itself" UX you saw today (workspace `8b0bf0b3`):

1. Runtime treated 202+queued as silent failure

`workspace/builtin_tools/delegation.py:_execute_delegation` only handled HTTP 200. When the peer's a2a-proxy returned HTTP 202 + `{queued: true}` (single-SDK-session bottleneck on the peer), the loop fell through. Two iterations later `if "error" in result` tried to access an unbound `result`, the goroutine ended quietly, the delegation stayed at FAILED with `error="None"`. The LLM checking status saw "failed" + the platform's "Delegation queued — target at capacity" log line in chat context, concluded the peer was permanently unavailable, and bypassed delegation to do the work itself.

DB evidence from the live workspace (6 delegation rows, all `status='queued'`):

```
delegation queued → UX Researcher "Delegating to f6f3a023..."
delegation queued → UX Researcher "Delegation queued — target at capacity"
delegation queued → Visual Designer "Delegating to a2e92524..."
delegation queued → Visual Designer "Delegation queued — target at capacity"
delegation queued → React Engineer "Delegating to fd1a1f96..."
delegation queued → React Engineer "Delegation queued — target at capacity"
```

2. Canvas Agent Comms tab dropped every delegation row

`AgentCommsPanel.tsx` filtered to `a2a_send | a2a_receive` only. `activity_type='delegation'` rows never reached `toCommMessage`. User saw "No agent-to-agent communications yet" while the DB had 6+ delegations.

How it's fixed

File Change
`workspace/builtin_tools/delegation.py` Add `DelegationStatus.QUEUED`. Explicit `if a2a_resp.status_code == 202` branch in `_execute_delegation` — recognizes `{queued: true}`, marks local state QUEUED, mirrors to platform, returns cleanly without retrying. `check_delegation_status` docstring extended with explicit per-status guidance: "queued → wait, peer is on prior task, do NOT bypass."
`canvas/src/components/tabs/chat/AgentCommsPanel.tsx` Include `delegation` in both the initial filter and the WS push filter. New `delegation` branch in `toCommMessage` — maps as outbound (always; platform proxies on our behalf), uses `summary` as primary text.

Tests

  • 3 new Python tests (`workspace/tests/test_delegation.py` → `TestA2AQueued`):
    • 202+queued marks status as QUEUED, no error field
    • 202+queued does NOT retry the A2A POST (counted by URL match — mock is shared across all AsyncClient calls)
    • bare 202 without `{queued:true}` still falls through to existing retry-then-FAILED path
  • 3 new TS tests (`AgentCommsPanel.test.ts`):
    • `delegate` row maps as outbound with summary text
    • `delegate_result` queued row preserves `status='queued'` (load-bearing for the LLM's wait-vs-bypass decision)
    • missing `target_id` returns `null` instead of rendering a ghost

All 20 `test_delegation.py` tests pass; all 8 `AgentCommsPanel.test.ts` tests pass.

What this PR does NOT solve

The underlying single-SDK-session bottleneck (peer can only handle one A2A request at a time). Tracked as task #102 — real architectural work. This PR makes the runtime handle the queueing correctly so the LLM doesn't bail, and makes delegations visible in Agent Comms so operators can see what's happening. Once #102 lands, the queueing will become rare anyway.

Test plan

  • `pytest workspace/tests/test_delegation.py` green (20/20)
  • `vitest run AgentCommsPanel.test.ts` green (8/8)
  • Manual: provision a director + 3 sub-agents, send director a multi-step task, observe Agent Comms panel populates with delegation rows including queued ones; observe director NOT bypassing when peer is queued (status surfaced as "queued" in chat, director acknowledges and waits)

🤖 Generated with Claude Code

…on rows

Two bugs that compounded into the "Director does the work itself" UX:

1. workspace/builtin_tools/delegation.py: _execute_delegation only
   handled HTTP 200 in the response branch. When the peer's a2a-proxy
   returned HTTP 202 + {queued: true} (single-SDK-session bottleneck
   on the peer), the loop fell through. Two iterations later the
   `if "error" in result` check tried to access an unbound `result`,
   the goroutine ended quietly, and the delegation stayed at FAILED
   with error="None". The LLM checking status saw "failed" + the
   platform's "Delegation queued — target at capacity" log line in
   chat context, concluded the peer was permanently unavailable, and
   bypassed delegation to do the work itself.

   Fix: explicit 202+queued branch. Adds DelegationStatus.QUEUED,
   marks the local delegation as QUEUED, mirrors to the platform,
   and returns cleanly without retrying. The retry loop is for
   transient transport errors — queueing is a real ack, not a failure
   to retry against (retrying would just re-queue the same task).

   check_delegation_status docstring extended with explicit per-status
   guidance: pending/in_progress → wait, queued → wait (peer busy on
   prior task, reply WILL arrive), completed → use result, failed →
   real error in error field; only fall back on failed, never queued.

2. canvas/src/components/tabs/chat/AgentCommsPanel.tsx: filter dropped
   every delegation row because it whitelisted only a2a_send /
   a2a_receive. activity_type='delegation' rows (written by the
   platform's /delegate handler with method='delegate' or
   'delegate_result') never reached toCommMessage. User saw "No
   agent-to-agent communications yet" while 6+ delegations existed
   in the DB.

   Fix: include "delegation" in the both the initial filter and the
   WS push filter, plus a delegation branch in toCommMessage that
   maps the row as outbound (always — platform proxies on our behalf)
   and uses summary as the primary text source.

Tests:
  - 3 new Python tests cover the 202+queued path: status becomes
    QUEUED not FAILED; no retry on queued (counted by URL match
    against the A2A target since the mock is shared across all
    AsyncClient calls); bare 202 without {queued:true} still
    falls through to the existing retry-then-FAILED path.
  - 3 new TS tests cover the delegation mapper: 'delegate' row
    maps as outbound to target with summary text; queued
    'delegate_result' preserves status='queued' (load-bearing for
    the LLM's wait-vs-bypass decision); missing target_id returns
    null instead of rendering a ghost.

Does NOT solve: the underlying single-SDK-session bottleneck that
causes peers to queue in the first place. Tracked as task #102
(parallel SDK sessions per workspace) — real architectural work.
This PR makes the runtime handle the queueing correctly so the LLM
doesn't bail out, and makes the delegations visible in Agent Comms
so operators can see what's happening.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor Author

Still skipping: failed E2E correlates with the active Cloudflare Minor Service Outage (status page indicator updated 22:08:40Z). CP #284's fast-fail surfaces CF API failures in ~16s instead of 900s — that's working as intended; the underlying CF outage is the blocker. Re-trigger when cloudflarestatus.com returns to operational.

…TION_* events

Critical follow-up to PR #2126's review. Two real bugs:

1. **Runtime QUEUED never resolved.** Platform's drain stitch updates
   the platform's delegate_result row when a queued delegation finally
   completes, but never pushes back to the runtime. The LLM polling
   check_delegation_status saw status="queued" forever — combined with
   the new docstring guidance ("queued → wait, peer will reply"), the
   model would wait indefinitely on a state that never resolves.
   Strictly worse than pre-PR behavior where it would have at least
   bypassed.

2. **Live updates dead code.** delegation.go writes activity rows by
   direct INSERT INTO activity_logs, bypassing the LogActivity helper
   that fires ACTIVITY_LOGGED. Adding "delegation" to the canvas's
   ACTIVITY_LOGGED filter (PR #2126 first cut) was inert — initial
   GET worked, live updates did not.

Fix:

(1) Runtime side, workspace/builtin_tools/delegation.py:
  - New `_refresh_queued_from_platform(task_id)` async helper that
    pulls /workspaces/<self>/delegations and finds the platform-side
    delegate_result row for our task_id.
  - check_delegation_status calls _refresh when local status is
    QUEUED, so the LLM's poll itself drives state convergence.
  - Best-effort: GET failure leaves local state untouched, next
    poll retries.
  - Docstring updated to reflect the actual behavior ("polls
    transparently — keep polling and you'll see the flip").
  - 4 new tests cover: QUEUED → completed via refresh; QUEUED →
    failed via refresh; refresh keeps QUEUED when platform hasn't
    resolved; refresh swallows network errors safely.

(2) Canvas side, AgentCommsPanel.tsx WS push handler:
  - Listens for DELEGATION_SENT / DELEGATION_STATUS / DELEGATION_COMPLETE
    / DELEGATION_FAILED in addition to ACTIVITY_LOGGED.
  - Each event's payload synthesized into an ActivityEntry shape
    so toCommMessage's existing delegation branch maps it. Status
    derived: STATUS uses payload.status, COMPLETE → "completed",
    FAILED → "failed", SENT → "pending".
  - The ACTIVITY_LOGGED branch keeps the "delegation" type accepted
    as a no-op-today / future-proof path: if delegation handlers
    are ever refactored to call LogActivity, this lights up
    automatically without another canvas change.

Doesn't change: the docstring guidance ("queued → wait, don't bypass")
is now actually load-bearing because the refresh path will deliver
the eventual outcome. Without the refresh, the guidance was a trap.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@HongmingWang-Rabbit

Copy link
Copy Markdown
Contributor Author

Code review (5-axis)

Correctness: ✓ Two-part fix lands cleanly:

  1. Runtime side (delegation.py) — explicit 202+queued branch returns early after marking DelegationStatus.QUEUED, calling _notify_completion(..., 'queued'), and _update_delegation_on_platform(..., 'queued', ...). The malformed-JSON guard (try: a2a_resp.json() except: queued_body = {}) plus the strict is True check on queued correctly falls through to the existing failure path on 202-without-queued-flag — no silent state confusion.

  2. Canvas side (AgentCommsPanel) — filter expanded + toCommMessage mapping for activity_type === 'delegation'. The !peerId guard returns null for malformed rows (matches the test contract).

Tests: ✓ Coverage is the right shape — TestA2AQueued.test_queued_marks_status_queued_not_failed AND test_queued_does_not_retry. The retry-count assertion is non-trivial (filtering A2A POSTs from platform-sync POSTs by URL match) but correct. Canvas tests cover delegate / delegate_result + the null-target_id defensive case.

Architecture:QUEUED enum addition is forward-compatible (existing clients see other values unchanged). Status-semantics docstring on check_delegation_status is the right place to teach the LLM what queued means — that's where the prompt-side guidance has to live.

Security: N/A — no auth/data-boundary changes; the queued-body is parsed but only .get('queued') is read, so a hostile peer can't smuggle anything via the rest of the JSON.

Performance: ✓ One additional branch + one body parse on 202 responses. Negligible.

FYI

  • The queued_body malformed-JSON path silently sets {} and falls through. Worth a one-liner log here too (mirrors the env-var pattern in fix(a2a-proxy): close 60s context-canceled gap on long silent runs #2128) — would help diagnose a future "peer is returning weird 202s" incident. Not blocking.
  • The LLM-prompt-side "don't bypass on queued" guidance lives in the docstring. Whether the runtime actually honors it depends on the agent's prompt construction — outside this PR's scope, but worth a follow-up to verify the LLM does the right thing in a real queued scenario before declaring incident-closed.

LGTM. Auto-merge already armed.

Merged via the queue into staging with commit fdf8b65 Apr 26, 2026
14 checks passed
@molecule-ai
molecule-ai Bot deleted the fix/director-bypass-and-agent-comms branch May 20, 2026 06:21
HongmingWang-Rabbit pushed a commit that referenced this pull request Jun 12, 2026
…alse (#2113)' (#2126) from fix/continue-on-error-triage-2113 into main
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant