Skip to content

fix(agent-orchestrator): scope admin-stop suppression to fresh stamps - #22986

Merged
lalalune merged 1 commit into
elizaOS:developfrom
nothingxnowhere:fix/22981-admin-stop-marker-staleness
Aug 21, 2026
Merged

lalalune merged 1 commit into
elizaOS:developfrom
nothingxnowhere:fix/22981-admin-stop-marker-staleness

Conversation

@nothingxnowhere

Copy link
Copy Markdown
Contributor

Relates to

Closes #22981. Hardens the administrative-stop marker introduced in #22852 (flagged in my review there; the PR merged before the review landed).

Contribution provenance

  • AI assistance: yes
  • Model(s) used: anthropic/claude-fable-5
  • Agent tooling: Claude Code
  • Skill path: elizaOS/army@9259107132edeab02d9e47dbb7ce383721bada77:skills/contribute-to-eliza
  • Provenance status: self-reported

Sync with develop

  • Rebased onto origin/develop at f43d944af3 (zero conflicts); exact head 02aaba94d484cc96f944ffa0e81b32bbb372fa66.

Risks

Low. The change narrows when an existing suppression fires; it never widens it. Every path that previously synthesized still synthesizes. The only behavior change for markers stamped by current develop (timestamp-less) is that their stopped synthesizes — which is exactly the pre-#22852, never-silent behavior, and the only such sessions still alive to emit stopped are the failed-stop survivors this fix exists for.

Background

What does this PR do?

#22852's adminStopReason stamp is never cleared and the coordinator honors it unconditionally, so the stamp outlives the administrative action it was written for. The reachable bad timeline: stamp succeeds → acp.stopSession throws (the user_stop catch even marks the session stop_failed) → the session survives wearing the marker → its later genuine crash emits stopped → the fresh-metadata read still sees the marker → the crash is silently swallowed, forever. That violates the #11689 never-silent-terminal invariant that the suppression's own comment names as its regression line. The adjacent HANDOFF_PENDING_META_KEY already solves the analogous staleness ("a stale marker is ignored AND cleared"); the admin-stop marker had no staleness handling.

The fix scopes suppression to fresh stamps:

  • markSessionAdministrativelyStopped now stamps adminStopStampedAt (ISO-8601) alongside the reason; still best-effort, fail-open, never blocking the stop.
  • The coordinator honors the marker only within ADMIN_STOP_MARKER_TTL_MS (10 minutes — teardown stopped events follow the stop within seconds; the window is generous headroom for a wedged subprocess while bounding how long a failed stop can shadow a survivor). Duplicate teardown stopped events from one admin action (the one-shot runner pattern documented in the coordinator) all land inside the window and stay suppressed, without clearing the marker.
  • A stale stamp — or a timestamp-less pre-fix stamp — is cleared best-effort (clearStaleAdminStopMarker, mirroring clearStalePendingHandoffMarker: null-patch + enrichment-cache eviction + error-policy:J6) and the stop synthesizes, failing toward never-silent.
  • The module header's claim that "the mid-task forwarder drops marked sessions" is corrected — nothing reads the marker in the forwarder (also flagged in the fix(orchestrator): port administrative-stop marker so lifecycle stops stop narrating as failures #22852 review).

What kind of change is this?

Bug fix (non-breaking change which fixes an issue).

Documentation changes needed?

My changes do not require a documentation change. The marker module's header documents the freshness contract.

Testing

Where should a reviewer start?

plugins/plugin-agent-orchestrator/src/__tests__/admin-stop-suppression.test.ts — the stale-stamp and timestamp-less cases are the regression line; the harness drives the real SwarmCoordinatorService via bindToAcp (same harness #22852 shipped).

Detailed testing steps

  1. Check out exact head 02aaba94d484cc96f944ffa0e81b32bbb372fa66.
  2. bun x vitest run src/__tests__/admin-stop-suppression.test.ts src/__tests__/admin-stop-marker.test.ts from plugins/plugin-agent-orchestrator11/11: fresh stamp suppresses without claiming the dedupe slot; duplicate stopped events stay suppressed with the marker intact; a stale stamp synthesizes AND clears both keys; a timestamp-less stamp synthesizes; an unmarked stop synthesizes; the stamp writes reason + parseable instant; fail-open and method-absent tolerance hold; the freshness predicate pins the TTL boundary, missing, and unparseable timestamps.
  3. Counterfactual: revert only src/services/ to origin/develop and re-run — exactly the six new pins fail (the stale and timestamp-less stops are silently suppressed; the stamp writes only one key), while the five preserved-behavior cases still pass. Restore.
  4. Neighborhood: verify-retry-stop-race 9/9 and verify-retry-busy-session 6/6 with the change (26/26 across the four suites).
  5. tsc6 --noEmit -p tsconfig.json in the plugin: clean. Biome on the four changed files: clean. git diff --check: clean.

Evidence

  • Before screenshots: N/A — backend coordinator behavior with no UI surface.
  • After screenshots: N/A — backend coordinator behavior with no UI surface.
  • Walkthrough video: N/A — deterministic unit/regression proof below stands alone.
  • Backend logs:
Counterfactual + head suite runs (vitest)
# head 02aaba94d484cc96f944ffa0e81b32bbb372fa66
admin-stop-suppression.test.ts + admin-stop-marker.test.ts
 Test Files  2 passed (2)
      Tests  11 passed (11)

# src/services reverted to origin/develop (unfixed), same tests
 × stamps the reason and a parseable stamped-at instant together
 × honors a stamp inside the freshness window, including the boundary
 × treats a stamp past the window as stale
 × treats missing and unparseable timestamps as stale (pre-#22981 stamps)
 × a STALE stamp does not silence the survivor's genuine crash — it synthesizes and the marker is cleared
 × a timestamp-less (pre-#22981) stamp is stale — the stop synthesizes
      Tests  6 failed | 5 passed (11)

# head, full neighborhood
admin-stop-suppression + admin-stop-marker + verify-retry-stop-race + verify-retry-busy-session
 Test Files  4 passed (4)
      Tests  26 passed (26)
  • Frontend console/network logs: N/A — no frontend surface.
  • Live-model trajectory: N/A — no agent-behavior change; deterministic coordinator plumbing.
  • Domain artifacts:
Static gates at exact head (tsc6, Biome, whitespace)
$ tsc6 --noEmit -p tsconfig.json        # plugins/plugin-agent-orchestrator
(clean, exit 0)

$ biome check src/services/admin-stop-marker.ts \
    src/services/swarm-coordinator-service.ts \
    src/__tests__/admin-stop-marker.test.ts \
    src/__tests__/admin-stop-suppression.test.ts
Checked 4 files in 42ms. No fixes applied.

$ git diff --check
(clean)
- OCR review: N/A — no captured imagery.

A stamped stopSession that throws leaves the surviving session wearing
adminStopReason forever; the coordinator then silently swallows the
survivor's later genuine crash on every stopped event, violating the
never-silent-terminal invariant (elizaOS#11689) that the marker's own
suppression names as its regression line.

Stamp adminStopStampedAt alongside the reason and honor the marker only
within a ten-minute freshness window: duplicate teardown stopped events
from one administrative action stay suppressed, while a stale stamp - or
a timestamp-less pre-fix stamp - is cleared best-effort and the stop
synthesizes, mirroring the handoff-pending staleness contract.

Closes elizaOS#22981.

@MLuber-bot MLuber-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review — fix(agent-orchestrator): scope admin-stop suppression to fresh markers

The fix adds freshness gating to the administrative-stop suppression in SwarmCoordinatorService. Previously any session carrying adminStopReason in metadata would suppress synthesis forever — including sessions whose admin stop threw before tearing down, leaving a surviving agent wearing the marker indefinitely. A later genuine crash on that survivor would be silently dropped, violating the #11689 never-silent-terminal invariant.

The fix introduces ADMIN_STOP_STAMPED_AT_META_KEY (written atomically with the reason by markSessionAdministrativelyStopped) and isAdminStopMarkerCurrent(stampedAt, nowMs), which returns false for missing, unparseable, or TTL-expired timestamps. The 10-minute TTL is generous headroom for teardown latency while still bounding how long a failed-stop marker can shadow a survivor. Stale or timestamp-less markers are actively cleared via clearStaleAdminStopMarker so they don't re-suppress on the next stopped event.

Pre-#22981 stamps (no timestamp field) are treated as stale by isAdminStopMarkerCurrent returning false on undefined — a correct backward-compatible default.

Tests cover: fresh stamp suppresses; duplicate teardown events stay suppressed while fresh; stale stamp synthesizes and marker is cleared; timestamp-less legacy stamp synthesizes; unmarked stop synthesizes (existing regression line preserved).

@ss251 ss251 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact-head review at 02aaba94d484cc96f944ffa0e81b32bbb372fa66 vs origin/develop. Trusted control: gh pr diff 22986. Static review (docker review-sandbox hangs on docker info).

#22852's admin-stop marker suppressed every later stopped for a session that still wore adminStopReason. A stopSession that threw left a live session stamped forever, so a genuine later crash stayed silent — the #11689 never-silent-terminal line.

This head pairs the reason with adminStopStampedAt and honours suppression only while isAdminStopMarkerCurrent (10 min TTL, inclusive boundary). Missing / unparseable stamps (pre-#22981) are stale. Fresh duplicate teardown stopped events stay quiet and do not claim the synthesis dedupe slot. A stale stamp synthesizes and clearStaleAdminStopMarker nulls both keys (J6 if the clear fails). Unmarked stops still synthesize. Stamp still fail-opens if updateSessionMetadata throws.

Tests pin fresh / duplicate / stale / timestamp-less / unmarked. Not leftover-tax, not a timeout twin.

{"schemaVersion":"2","projectId":"eliza","artifactUrl":"https://github.com/elizaOS/eliza/pull/22986","headSha":"02aaba94d484cc96f944ffa0e81b32bbb372fa66","provider":"xai","model":"grok-4.6","client":"Grok Bot.app","runId":"","traceSha256":"","recommendation":"accept","reproduced":false,"securityRisk":"none","duplicateRisk":"none","splitRisk":"none","effortBand":"medium","complexity":"moderate","impact":"meaningful","reviewLoad":"standard","recommendedTier":"medium","recommendedThirds":6,"workUnitId":"wu_eliza_admin_stop_marker_freshness","confidenceBasisPoints":9000,"valueRationale":"Admin-stop suppression is freshness-scoped so a failed stop cannot silence a later genuine crash; duplicate teardown stopped events stay quiet inside the TTL.","usefulArtifacts":["admin-stop-marker.ts isAdminStopMarkerCurrent","swarm-coordinator-service.ts stale clear","admin-stop-suppression.test.ts"],"commands":["gh pr diff 22986 --repo elizaOS/eliza"],"evidenceUrls":["https://github.com/elizaOS/eliza/pull/22986"],"summary":"Approve 02aaba94: scope admin-stop suppression to a fresh stamped-at window so failed stops cannot mute a survivor crash."}

AI provider/model: xAI / grok-4.6
Client / agent tooling: Grok Bot.app
Contribution skill revision: SlopDotCash/slopdotcash@6000f78:skills/contribute-to-eliza
Attribution status: self-reported
— [grok-bot-ss251]

@lalalune

Copy link
Copy Markdown
Member

SHA-locked GREEN on 02aaba94d484cc96f944ffa0e81b32bbb372fa66. Two exact approvals confirm the freshness-scoped suppression: fresh duplicate teardown events stay suppressed, while stale, missing, or malformed stamps synthesize the genuine terminal event and clear both marker keys best-effort. Exact 26/26 neighborhood tests, typecheck, Biome, and diff evidence are recorded. GitHub is CLEAN/MERGEABLE and 165 intervening commits have zero touched-path overlap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(orchestrator): stale administrative-stop marker silences later genuine crashes after a failed stop

4 participants