fix(relay): bound native scope lifecycle ops so a wedged observability pipeline cannot block the agent

[victor-kyriazakos]

Symptom

With the NeMo Relay observability plugin active, every delegated-subagent batch wedges at child completion. Children finish their work (stream final text, log Turn ended: reason=text_response) but run_conversation() never returns; futures never resolve; the delegate heartbeat declares children stale and the batch is force-finalized at ~600s. Cron jobs using delegate_task hit the same wall, surfaced as a misleading "provider timeout".

Observed live on an enterprise staging fleet: four independent occurrences (2 cron, 2 gateway).

Evidence

Falsification probe (decisive): booting the same fleet instance with the observability plugin disabled — identical config otherwise — the same 2-subagent batch completes and delivers normally. Re-enabled: wedges every time. Toggled twice.

The blocking call is in the agent's finalization path:

finish_logical_calls(turn) -> end_turn(turn) -> lease.host.run_in_session(session, relay.scope.pop, ...)

Every call site already wraps these in except Exception (warn + continue), but an exception guard cannot help against a call that never returns.

On the mechanism — what we can and cannot claim

An earlier revision of this PR asserted that nemo_relay.scope.pop blocks because it synchronously waits for the scope to close through the export pipeline. Reading the NeMo Relay 0.7.2 source, that is not supported and I've retracted it. pop_scope calls pop_scope_inner(params, false) — it does not request a delivery receipt — and event publication goes onto an unbounded mpsc::channel, which cannot block the sender. SubscriberDelivery::wait() is reachable only through the #[doc(hidden)] pop_scope_with_subscriber_delivery, which nothing in Hermes calls.

So the wedge is not an unbounded export wait. The remaining blocking operations on that path are the scope-stack and global-context read locks, and NeMo Relay's own docs note a related hazard — SubscriberDelivery::wait carries an explicit "do not call this from a subscriber, event-sanitizer, guardrail, or intercept callback... waiting there creates a wait cycle." Our plugin hooks do run inside callbacks, which makes a lock/re-entrancy interaction the leading hypothesis. I have not localized it, and this PR does not claim to.

What we know with confidence: the plugin's scope finalization is the wedge site (falsification probe), and the agent must not be hostage to it.

What this PR actually is

A defensive bound at our integration boundary with a third-party native library — not a root-cause fix. run_in_session takes an optional timeout; when set, the native call runs on the existing shared DaemonThreadPoolExecutor (tools.daemon_pool — abandoned workers cannot block interpreter exit) and honors future.result(timeout=...). On breach, TimeoutError propagates into each site's existing exception handling: warn, retain the unclosed-prefix diagnostics, continue.

Bounded sites (the whole class, not just the one that bit us):

• ensure_session session-scope push
• begin_turn turn-scope push
• end_turn turn-scope pop
• _finish_logical_calls logical-LLM pops
• close_session session pop + subscribers.flush()

Properties:

• timeout=None (default) → byte-identical synchronous behavior for every other caller.
• Interpreter-shutdown paths fall back to the synchronous call, so the atexit flush still exports.
• Executor exhaustion is safe: Future.result(timeout=...) bounds even unstarted futures, so a wedge-poisoned executor degrades to fast timeouts rather than a new hang.
• Lock semantics unchanged — the context copy still happens on the calling thread under session.lock (wrapping run_in_session wholesale would move the lock off-thread and self-deadlock in close_session).

Worst case is one lost span. Observability must never block the product.

Thread-safety of moving native scope ops off the calling thread

This deserves explicit treatment, because NeMo Relay scope stacks are thread-local and strict-LIFO: pop_scope returns InvalidArgument("scope handle is not at the top of the stack") when the handle is not on top of the calling thread's stack. Running push/pop on a shared executor could therefore plausibly corrupt cross-session stack state.

It does not, for two reasons that are worth checking rather than assuming:

1. We submit via context.run(...), so the caller's ContextVars — including the scope-stack ContextVar — are copied onto the executor thread.
2. The Python binding re-syncs the native thread-local stack from that ContextVar on every call (_ensure_scope_stack() in python/nemo_relay/_context.py), which is exactly the mechanism the library provides for mirroring a ContextVar-based stack onto a worker thread.

Empirically, across a four-instance staging fleet running this build: zero not at the top of the stack and zero scope handle not found diagnostics, with spans arriving normally for every scope type. Had the stack been corrupted by the thread move, strict LIFO would have surfaced it loudly and immediately.

Known trade-off

A defensive timeout can mask a genuine bug rather than fix it. Two things make me comfortable shipping it anyway: the blocking call is across an FFI boundary into a library we don't own, where a bound is appropriate regardless of cause; and the failure it prevents is total (agent deadlock) while its cost is minimal (one span). Upstream investigation continues separately — this stops the bleeding, it doesn't close the question.

Tests

tests/agent/test_relay_runtime_bounded_scope_ops.py — fake relay whose pop/flush block on a never-set Event (a stand-in for any wedge cause, which is the right level of abstraction given the mechanism is open):

1. end_turn returns when the native pop wedges
2. close_session returns when the native pop wedges
3. close_session returns when subscribers.flush wedges
4. finish_logical_calls returns and retains the unclosed prefix for diagnostics
5. • 6. healthy-relay lifecycle: push/pop counts unchanged, pop completes synchronously before end_turn returns (guards the default lane against fire-and-forget regressions)

Verified RED→GREEN: tests 1–4 hang on unfixed code. Mutation check: reverting the fix turns them red again; restoring turns them green.

Regression run (tests/run_agent/test_run_agent.py, tests/plugins/test_nemo_relay_plugin.py, tests/hermes_cli/test_relay_shared_metrics_runtime.py + the new file): 296 passed, 1 failed — the single failure (TestAnthropicInterruptHandler::test_interruptible_anthropic_interrupt_never_closes_shared_client) is pre-existing on the branch point, verified by running it against unmodified relay_runtime.py from 03da1606b.

Live verification

Deployed across a four-instance staging fleet with the plugin re-enabled: delegation batches that previously wedged for 600s now complete in seconds across workload shapes (instant children and children that run tool turns).

Worth noting honestly: the timeout never actually fires in production. Moving the native call onto an executor thread appears to change the finalization topology enough to avoid the wedge entirely. That is itself evidence for the thread-affinity hypothesis above — and it means the bound is insurance rather than the active mechanism of the fix.

[victor-kyriazakos]

Status: refreshed onto current main (merge, no rewrite). The slice-6 failure was the same pre-existing busy-mode test fixed on main in f51aa6a, not this change. Relay suite green post-merge (177 passed). This bounds native scope lifecycle ops so a wedged observability pipeline cannot stall the agent loop — the fix behind the enterprise perf report we chased last week. Ready for review.

[victor-kyriazakos]

CI update: the slice-12 red was REAL and is fixed in ccc071e — the test file hung AFTER '6 passed in 4.32s' until the runner's 300s file-timeout SIGKILL. Two defects, same class this PR targets: (1) the interpreter-shutdown fallback lane ran the native call unbounded (a wedged pipeline could block process exit — now a bounded daemon exit-thread with identical timeout/abandon semantics); (2) the wedge tests left daemon workers parked on Event.wait() plus live sessions on the atexit hook, so exit replayed wedged pops until the file timeout. Autouse teardown now unwedges and drains. Canonical runner: 4.4s. Note: bare pytest was a false green for this class — it reports before the exit-path cost.