Arm recovery as convergence: observe, assess, plan at arm grain; delete the leftover classifier

[gunbai-bot]

Replaces the native arm's leftover classifier with convergence: every launch, rollback and recovery observes every rank, assesses against a desired arm (std.goal_assessment), and decides (std.upsert_decision). Design: docs/plans/arm-convergence-recovery-design.md. Stacked on #13319.

Why. On 2026-10-05 (run 37254736596), Group A was left in a state that matched no enumerated leftover kind: three ranks had rolled back and were running, and the head was fence-only and stopped. Group A stayed down until a human intervened. The fence's 2 h horizon was a timer standing in for "is the writer dead".

What changes

• gunbc.spark.arm_convergence (new). Per rank, it reads:
  • the unit file hash vs the planned unit's (hashed on the host);
  • the unit's ActiveState;
  • the container's incarnation;
  • the committed-arm record.
    Each fact is read, absent, or unread, and unread is never treated as absent. The decision is Noop / Refuse / Apply. Apply carries the whole desired arm and nothing that could name a subset.
• native_serving_apply is one loop: observe → (a realized arm is read back first) Noop / Refuse / Apply → preflight → stop every rank and read each container gone → vacancy → apply head first → readback → commit (write the committed-arm record). On failure it returns to the last committed arm, through the same readback, and only if the re-render reproduces the recorded unit hashes. With no committed arm to return to, or a failed return, every rank is stopped.
• Deleted:
  • the leftover classifier, the horizon, adoption and marker recovery;
  • the fence, the incumbent backup and the dispositions;
  • the declared-incumbent bracket and the incumbent dispatch input;
  • the adoption-only admission.
• The arm's own units are the converge's subject, not occupants. Vacancy is judged after the stop. A foreign occupant still refuses.
• Serialization: there is no fence. fleet_converge_mode_writes_arm is a total per-mode answer, read against each mode's concurrency domain; Group A keeps srv1's key on purpose. Hand actuation is unserialized and is caught only as a deviation or an unknown.

Switching Group A between model families (GLM to V4.1, or back): what this now requires. The declared-incumbent bracket is gone, so a launch whose hosts carry ANOTHER arm's running units refuses at admission ("occupied by a serving unit that is not this arm's"). Stopping the other arm by hand is not the route: the stop must be a declared converge of that arm toward a desired state of VACANT (every rank stopped, its container read gone, its committed-arm record cleared), run through the same loop and the same concurrency group. That converge toward vacant is not in this PR; it is the named follow-up, and until it lands a cross-family switch on Group A refuses. Within one family (every GLM staircase step), switching is an ordinary converge, because the arm's own units are its subject.

Evidence. Remote claim_batch at 4da4570:

witness file	pass
arm_convergence	11/11
fleet_converge_arm_writer	3/3
native_serving_resumable_apply	17/17
native_serving_roce_transport	17/17
group_arm_launch	14/14
v41_group_a_launch	26/26

The converge itself has not run on hardware; its first execution is a Group A launch, held until the parent says go.

🤖 Generated with Claude Code

[gunbai-bot]

superseded: Group A decommissioned, arm path has no remaining consumer

— sent from cool-carp-342