Spark remote legs carry a liveness deadline; a leg that never began is typed dark-or-down undecided

[gunbai-bot]

Lessons A1 and A3 from the 2026-10-03/04 Spark Group A bring-up (node://adhoc-f0195bdb-ef2).

A1 — every fleet ssh leg carries a liveness bound, not only a connect bound.

• Fact: new variants ServerAliveInterval and ServerAliveCountMax in extdeps.ssh.client_options, cited from ssh_config(5).
  • The interval defaults to 0, which means never, so an established session whose path goes dark waits on TCP alone.
  • That is how fleet-converge run 37156689444 sat ~1h inside one leg.
• New row gunbc.fleet_known_hosts_anchor fleet_ssh_leg_deadline (connect 10 s, keepalive 15 s × 4).
  • fleet_ssh_leg_deadline_options is the one source of the deadline words for both fleet shapers (shape_fleet_ssh_exec and fleet_ssh_password_client_options). The inline ConnectTimeout=10 is deleted.
  • Neither shaper takes an options parameter, so no caller can shape a fleet leg without the deadline.
  • Every Spark remote leg reaches these shapers: gunbc.spark.model_snapshot_materialize spark_remote_run → gunbc.fleet_bootstrap_principal_session → the shapers.
  • sshd answers keepalives itself, so a silent remote command is not cut off. Only a path that returns nothing is.

A3 — "LAN leg dark" vs "host down", typed honestly, and no "did not run" for a leg that may have run.

• SparkRemoteRun.RemoteUnreachable now carries reach: SparkLegUnreachability, with two arms:
  • LegNotAttempted: nothing was sent. This is the only arm that projects to ArgvLegDidNotRun.
  • LegBeginUnobserved: ssh exited with its own 255 and the in-band began-line never ARRIVED.
• Why not "never began" (review of this PR): the marker and the command travel in one unacknowledged stream. The path can go dark after the shell wrote the marker, or after the command started, before the marker reaches the client. So the command MAY HAVE RUN.
  • gunbc.spark.host_effect_quiescence ArgvRun gains a third arm, ArgvLegMayHaveRun.
  • spark_argv_run_of_unreachable is the one projection. It sends LegBeginUnobserved to ArgvLegMayHaveRun, never to ArgvLegDidNotRun.
  • Every ArgvRun consumer handles the new arm: host_effect_quiescence and gunbc.compute.host_occupancy treat it as unread (new HostOperationResultUnreceived), and fleet_wireless_link refuses with "read back before retrying".
  • host_occupancy_admission now goes through the same projection.
• There is deliberately no HostDown arm. From one ssh leg the executor cannot tell a dark wlP9s9 from a host that is off, so the cause text says "the command MAY HAVE RUN; management LAN leg dark or host down, UNDECIDED".
  • The first settling readback is a probe of the host's fabric-rail address (192.168.110.x) from a same-group peer: rail answers → LAN dark, host up.
  • Uptime and the wlP9s9 journal come after, since they need a session (DESIGN §4d).
• ONE began-line protocol (review 75397): the marker, payload line, reader and stripper are lifted out of gunbc.spark.pair_serving_apply into new gunbc.spark.remote_leg_began. Both pair_serving_apply and spark_remote_run consume it, so there is one marker string and one parser (no stderr-sniffing, per gunbc.command_runner ssh_exit_255_conflation_dissolution). It is stripped before any consumer sees stdout.
• A session lost AFTER the began-line arrived stays RemoteRan with 255.
• Next rungs, named in the type's comment and the ledger, not built here:
  • (1) a two-phase protocol: the client receives an acknowledgement before it sends the effect command, after which a NeverBegan arm is supportable;
  • (2) an automatic rail-jump classifier with its own consumer.

Witnesses (claim_batch on BuildBuddy, all PASS):

• New test.claim.spark.spark_remote_leg_deadline_witness_test, 10 claims:
  • the exact deadline argv;
  • the route through the password session;
  • a red control: a zeroed keepalive is not a bound;
  • 255 with no began-line → LegBeginUnobserved;
  • the MAY-HAVE-RUN + UNDECIDED + rail-probe text;
  • LegBeginUnobserved projects to ArgvLegMayHaveRun, never ArgvLegDidNotRun (reverting the projection reds it);
  • LegNotAttempted projects to ArgvLegDidNotRun;
  • a session lost after the began-line stays a ran result;
  • the began-line never reaches a consumer.
• native_serving_resumable_apply_witness_test (22, the pair_serving_apply consumer of the shared protocol) passes after the lift. Re-run after the repair and passing: host_occupancy_admission_witness_test (15), fleet_wireless_link_witness_test (16), pair_serving_d0_release_witness_test (22). Earlier on this branch: ssh_client_options_witness_test (8), serving_load_probe_witness_test (40, includes the updated RemoteUnreachable construction), spark_bootstrap_password_session_witness_test (2).

Ledger:

• New gunbc.recurring_failure_mode a_dark_management_lan_leg_hangs_and_reads_as_a_down_host.
• an_unreached_effect_leg_reads_as_one_that_ran_and_refused records spark_remote_run's progress, and that the same may-have-run residue applies to its own Unreached reading.

Residue, stated:

• extdeps.bmc.openbmc_password_ssh_transport and extdeps.ssh.password_session CopyFile still carry ConnectTimeout only, as transport literals. They are the new row's next-rung trigger. They are not Spark legs.

🤖 Generated with Claude Code