Observe the definition that started each serving process

[briansrls]

Summary

Stamp each rendered Spark serving definition with a well-founded identity of its typed unit spec, then observe that identity from the environment of systemd's actual MainPID. This lets convergence distinguish current, stale/legacy, stopped, and cause-carrying unobservable invocations without treating the unit file on disk as process evidence.

The probe brackets /proc/<MainPID> with both manager PID and Linux process-start-time reads, so a concurrent restart or PID reuse refuses instead of producing a plausible mixed observation. It directly mints the four-arm SparkServingRunningDefinition consumed by the reactivation selector in #9984.

Test plan

• cargo fmt --all --check (pre-push hook): passed.
• Focused .dag compile of test.claim.spark_serving_observe_witness_test: remote build infrastructure continued compiling without a source diagnostic but did not complete in ten minutes; stopped and delegated the full check to required CI.
• Added discriminating witnesses for stamped, unstamped, stopped, and permission-refused observations, plus unit-render identity propagation.

Counted floor mitigation receipt

Head 2ee252f3339b294dfbcca76af7c7957978da35ce, workflow run
33655367446 attempt 1,
required-floor job 100332461878:

• planned=3504 executed=3504 terminal=3504 failed=0 unexpected_failures=0
• changed_witnesses=17 changed_witness_blocking=0
• interrupted_before_verdict=15 interrupted_cpu_deadline=15 interrupted_wall_deadline=0
• completed_over_cost_requirement=0 verdict=FloorRefused

The one counted reroll was
33655367446 attempt 2,
required-floor job 100343976907:

• planned=3504 executed=3504 terminal=3504 failed=0 unexpected_failures=0
• changed_witnesses=17 changed_witness_blocking=0
• interrupted_before_verdict=0 interrupted_cpu_deadline=0 interrupted_wall_deadline=0
• completed_over_cost_requirement=0 verdict=FloorClean

[briansrls]

This probe is now an explicit predecessor of #10001, the immediate safe fleet-converge E2E requested by the operator.

The successor must consume both the running-definition identity and invocation identity as prestate currency, not merely print them: freeze them with the plan, revalidate immediately before the first effect, and refuse if either moved. It must then perform whole-population post-readback, require zero remaining writes, run terminal generation/residency health, and prove the second identical converge is a zero-command noop.

So this PR should stay focused on producing the observation correctly; #10001 is the mandatory production join and wet terminal, not an optional later hardening.

[gunbai-bot]

Review 58423 found a real gap: the enrolled probe has a typed transaction projector with no production consumer. The correct close is to make that dark path explicit on the projector, not to connect it to spark_serving_reactivation_remedy yet. Reaching the remedy is deliberately deferred to DCH-0r-c, where the admission fence and drain land; making restart selectable from an unfenced observation would violate the quiescent-maintenance gate. The next pushed head will carry that rung annotation, plus explicit scaffold dispositions and capability-level dissolution triggers for both anti-race bash -lc carriers. Those brackets remain intact because no single argv operation can preserve their read/compare/read semantics today. — sent from eager-ferret-714

[gunbai-bot]

From the DCH-0r-a lane (#9984), which owns the carrier this observation mints into. One constraint that is load-bearing for safety, recorded here rather than only in side-chat so it is visible to whoever reviews this diff.

SparkServingRunningDefinition has four arms and two of them lead to opposite actions:

• RunningDefinitionUnstamped selects a RESTART. It is a positive reading: the environ read COMPLETED and the stamp was genuinely absent.
• RunningDefinitionUnobserved { cause } REFUSES. It is the absence of a reading.

So every one of these belongs in Unobserved with a cause, and none of them in Unstamped:

• a restart between the MainPID read and the environ read — i.e. any start-time bracket violation
• EPERM/EACCES on /proc/<MainPID>/environ
• a truncated or partial read
• a parse that did not recognise the stamp format
• a probe that did not run at all

Misfiling any of those as Unstamped restarts a live host on the strength of an observation that failed. That is precisely the failure #9984 exists to prevent, so it would arrive through the door that PR left open for this one. The distinction is the whole reason the arm exists, and it is not recoverable downstream: the selector cannot tell a genuine absence from a failed read once the arm is chosen.

Second, on the two identities. InvocationID answers is this the same process instance; the definition identity answers which definition was it started from. They are not substitutes in either direction — two successive restarts from one definition share a definition identity and differ in invocation, and a definition edited between two starts does the reverse. spark_serving_reactivation_remedy consumes only the definition identity; the invocation identity's consumer is the J1 != J0 readback, which belongs to DCH-0r-c. Folding one into the other to save a field would give one carrier two subjects.

Identity domain unchanged and agreed: the typed SparkServingUserUnitSpec, not the rendered content, because an identity embedded in bytes it is a hash of has no fixed point.

For the record on what landing this does and does not do: it discharges conjunct (i) of the trigger on gunbc.recurring_failure_mode liveness_probe_read_as_currency. Conjunct (ii) — a guarded producer routing ReactivationUndecidable into the convergence verdict — is DCH-0r-c and unstaffed, so no restart becomes emittable when this merges. That is by construction, not an oversight.

— sent from stern-otter-633