Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6cd030e
fix: surface stalled secondmate queues and wake handoffs
kunchenguid Aug 23, 2026
8135c9c
no-mistakes(review): Make handoff wakes retryable and stall alerts cr…
kunchenguid Aug 23, 2026
111aca4
no-mistakes(review): Prevent duplicate handoff wakes and cover remote…
kunchenguid Aug 23, 2026
e357cf1
no-mistakes(review): Serialize local handoffs and preserve pre-move w…
kunchenguid Aug 23, 2026
2f23f26
no-mistakes(review): Serialize teardown with handoffs and retain remo…
kunchenguid Aug 23, 2026
bf0ebab
no-mistakes(review): Reconcile correlated handoff wake delivery after…
kunchenguid Aug 23, 2026
24b3c15
no-mistakes(review): Keep failed wakes retryable and isolate stall re…
kunchenguid Aug 23, 2026
4779409
no-mistakes(review): Reset known-undelivered wake attempts for durabl…
kunchenguid Aug 23, 2026
698e6b9
no-mistakes(review): Refuse duplicate sends for unresolved delivery a…
kunchenguid Aug 23, 2026
1ec2ea2
no-mistakes(review): Atomically restore retryability after reconciled…
kunchenguid Aug 23, 2026
e1196e1
no-mistakes(review): Serialize delivery confirmation with reconciliation
kunchenguid Aug 23, 2026
32dcfdb
no-mistakes(document): Document routed wake and stall supervision
kunchenguid Aug 23, 2026
1c875a9
no-mistakes(lint): Fix ShellCheck expansion and subshell warnings
kunchenguid Aug 23, 2026
039392f
no-mistakes: apply CI fixes
kunchenguid Aug 23, 2026
2204c97
no-mistakes: apply CI fixes
kunchenguid Aug 23, 2026
dfb6660
no-mistakes(review): Retire stale wake state and defer pre-move wakes
kunchenguid Aug 23, 2026
7217c31
no-mistakes(review): Secure markers, bind batches, and preserve teard…
kunchenguid Aug 23, 2026
ab9a115
no-mistakes(review): Preserve unresolved prepared wakes across unrela…
kunchenguid Aug 23, 2026
f155ae8
no-mistakes(review): Preserve prepared wakes before unrelated moving …
kunchenguid Aug 23, 2026
4ae17f0
no-mistakes(document): Document prepared wake batch ownership
kunchenguid Aug 23, 2026
e53661c
no-mistakes: apply CI fixes
kunchenguid Aug 23, 2026
92a8dc0
no-mistakes: apply CI fixes
kunchenguid Aug 23, 2026
a0428b3
no-mistakes(review): Make local wake retirement recoverable
kunchenguid Aug 23, 2026
8727e97
no-mistakes(document): Clarify handoff recovery and teardown document…
kunchenguid Aug 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/skills/bootstrap-diagnostics/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ When any diagnostic needs captain attention, report the plain consequence and re
- `SECONDMATE_SYNC: secondmate <id>: skipped: <reason>` - secondmate convergence left a live home on its existing checkout because the home was dirty, diverged, unsafe, on the wrong branch, missing its placement-specific target commit, unreachable, or otherwise not fast-forwardable, or because inherited local-material propagation failed; bootstrap continued, but inspect the reason because the secondmate's tracked instructions, inherited settings, or shared captain preferences may be stale after a primary update.
- `SECONDMATE_LIVENESS: secondmate <id>: skipped: <reason>|respawn failed after <cause>: <reason>` - the session-start liveness sweep could not guarantee that the registered secondmate is running a real agent process.
Investigate the reason because that secondmate is not guaranteed live.
- `SECONDMATE_HANDOFF: secondmate <id>: pending delivery: <n> item(s)` - queued work has already left the main dispatchable backlog and remains safe in the named remote route's backlog-format outbox.
Preserve that outbox and rerun `bin/fm-backlog-handoff.sh --resume-pending` after same-host connectivity returns; never re-add or dispatch the items from the main backlog.
- `SECONDMATE_HANDOFF: secondmate <id>: pending delivery: <n> item(s)` - queued work has already left the main dispatchable backlog and remains safe in the named remote route's backlog-format outbox, pending backlog receipt or receiver-wake confirmation.
Preserve that outbox and rerun `bin/fm-backlog-handoff.sh --resume-pending` after the route or endpoint problem is resolved; never re-add or dispatch the items from the main backlog.
An unsafe-outbox variant requires path and file-type inspection before any retry.
- `NUDGE_SECONDMATES: secondmate <id>: send failed: <reason>` - secondmate convergence changed a running home's loaded instructions or inherited config, but the deterministic `fm-send.sh fm-<id>` re-read nudge failed.
Inspect the reason, keep the pending marker under `state/.secondmate-nudge-pending/` intact, and rerun session start after the endpoint or metadata issue is fixed so bootstrap can retry the exact same marked send on the same local or remote route.
Expand Down
4 changes: 3 additions & 1 deletion .agents/skills/secondmate-provisioning/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,9 @@ After seeding, run this handoff for the new secondmate's in-scope queued items.
For an existing or inherited domain, complete record intake first so no already-shipped plan row is handed off as open work.
For a local route, the helper resolves and validates the secondmate home from `data/secondmates.md`, then delegates the item move to `tasks-axi mv` (the single owner of the backlog format), which moves each named item - and a whole connected set, blocker plus dependents, atomically - from the main `data/backlog.md` into the secondmate home's `data/backlog.md`.
For a remote route, the same helper first moves the dependency-closed set atomically from the main backlog into `data/handoff/<id>.outbox.md`, then transfers that backlog-format outbox through `fm-on.sh` and lets the remote home's `fm-backlog-receive.sh` move every not-already-present key under the destination lock.
The outbox is the whole recovery record: its presence means delivery is unfinished, `--resume-pending` safely re-delivers it, and confirmed receipt removes it.
After a new local placement or a remote outbox receipt becomes durable, the helper sends one marked routed-work instruction through the receiving secondmate's recorded endpoint; missing or failed delivery makes the command fail loudly with the moved work intact, and the same handoff command retries known-undelivered wake intent without moving an already-present item again.
An unresolved delivery attempt is never blindly resent.
For a remote route, the outbox remains until both backlog receipt and receiver wake are confirmed; `--resume-pending` retries unfinished outboxes, while the script header owns its stable wake-correlation recovery state.
There is no two-phase handoff journal and no tasks-axi release beyond the already-required atomic `mv` capability.
Bootstrap retries pending outboxes when mutation is authorized and emits `SECONDMATE_HANDOFF:` for any that remain.
This delegated route remains required when `config/backlog-backend=manual`, which controls only routine firstmate backlog edits.
Expand Down
303 changes: 299 additions & 4 deletions bin/fm-backlog-handoff.sh

Large diffs are not rendered by default.

78 changes: 76 additions & 2 deletions bin/fm-pending-reply-lib.sh
Original file line number Diff line number Diff line change
Expand Up @@ -344,6 +344,18 @@ fm_pending_reply_prepare_delivery() { # <state-dir> <corr_id>
}

fm_pending_reply_confirm_delivery() { # <state-dir> <corr_id>
local state=$1 corr=$2 lock rc=0
local STATE FM_WAKE_QUEUE FM_WAKE_QUEUE_LOCK
STATE=$state
lock="$state/.pending-reply-$corr.lock"
. "$_FM_PENDING_REPLY_LIB_DIR/fm-wake-lib.sh"
fm_lock_acquire_wait "$lock" || return 1
_fm_pending_reply_confirm_delivery_locked "$@" || rc=$?
fm_lock_release "$lock"
return "$rc"
}

_fm_pending_reply_confirm_delivery_locked() { # <state-dir> <corr_id>
local state=$1 corr=$2 now marker
marker=$(fm_pending_reply_delivery_confirmation_path "$state" "$corr")
if ! fm_pending_reply_prepare_delivery "$state" "$corr"; then
Expand Down Expand Up @@ -372,7 +384,7 @@ fm_pending_reply_mark_delivery_unknown() { # <state-dir> <corr_id>
fm_pending_reply_set "$rec" phase delivery_unknown
}

fm_pending_reply_reconcile_delivery() { # <state-dir> <corr_id>
_fm_pending_reply_reconcile_delivery_locked() { # <state-dir> <corr_id>
local state=$1 corr=$2 rec delivered marker entry delivery_state value epoch
local grace now age phase
rec=$(fm_pending_reply_path "$state" "$corr")
Expand Down Expand Up @@ -412,6 +424,68 @@ fm_pending_reply_reconcile_delivery() { # <state-dir> <corr_id>
return 1
}

fm_pending_reply_reconcile_delivery() { # <state-dir> <corr_id>
local state=$1 corr=$2 lock rc=0
local STATE FM_WAKE_QUEUE FM_WAKE_QUEUE_LOCK
STATE=$state
lock="$state/.pending-reply-$corr.lock"
. "$_FM_PENDING_REPLY_LIB_DIR/fm-wake-lib.sh"
fm_lock_acquire_wait "$lock" || return 1
_fm_pending_reply_reconcile_delivery_locked "$@" || rc=$?
fm_lock_release "$lock"
return "$rc"
}

fm_pending_reply_delivery_attempt_unresolved() { # <state-dir> <corr_id>
local state=$1 corr=$2 rec delivered marker entry
rec=$(fm_pending_reply_path "$state" "$corr")
[ -f "$rec" ] && [ ! -L "$rec" ] || return 1
delivered=$(fm_pending_reply_get "$rec" delivered_epoch)
[ -z "$delivered" ] || return 1
marker=$(fm_pending_reply_delivery_confirmation_path "$state" "$corr")
[ -f "$marker" ] && [ ! -L "$marker" ] || return 1
entry=$(cat "$marker" 2>/dev/null || true)
case "$entry" in attempted=*) return 0 ;; esac
return 1
}

# A definitive backend rejection makes the existing correlation retryable again.
# Reconciliation may have aged the same attempted sidecar to delivery_unknown
# while the backend call was in flight, so both undelivered phases converge here
# under the per-correlation lock; a confirmed delivery can never be reset.
fm_pending_reply_reset_known_undelivered() { # <state-dir> <corr_id>
local state=$1 corr=$2 lock rc=0
local STATE FM_WAKE_QUEUE FM_WAKE_QUEUE_LOCK
STATE=$state
lock="$state/.pending-reply-$corr.lock"
. "$_FM_PENDING_REPLY_LIB_DIR/fm-wake-lib.sh"
fm_lock_acquire_wait "$lock" || return 1
_fm_pending_reply_reset_known_undelivered_locked "$@" || rc=$?
fm_lock_release "$lock"
return "$rc"
}

_fm_pending_reply_reset_known_undelivered_locked() { # <state-dir> <corr_id>
local state=$1 corr=$2 rec delivered phase marker entry
rec=$(fm_pending_reply_path "$state" "$corr")
[ -f "$rec" ] && [ ! -L "$rec" ] || return 1
delivered=$(fm_pending_reply_get "$rec" delivered_epoch)
[ -z "$delivered" ] || return 1
phase=$(fm_pending_reply_get "$rec" phase)
case "$phase" in awaiting_report|delivery_unknown) ;; *) return 1 ;; esac
marker=$(fm_pending_reply_delivery_confirmation_path "$state" "$corr")
[ -e "$marker" ] || [ -L "$marker" ] || {
[ "$phase" = awaiting_report ]
return $?
}
[ -f "$marker" ] && [ ! -L "$marker" ] || return 1
entry=$(cat "$marker" 2>/dev/null || true)
case "$entry" in attempted=*) ;; *) return 1 ;; esac
[ "$phase" = awaiting_report ] \
|| fm_pending_reply_set "$rec" phase awaiting_report || return 1
rm -f -- "$marker"
}

# Drop an undelivered expectation after a failed send so transport failure does
# not masquerade as a missed report later.
fm_pending_reply_discard_undelivered() { # <state-dir> <corr_id>
Expand Down Expand Up @@ -1049,7 +1123,7 @@ _fm_pending_reply_maybe_escalate_locked() { # <state-dir> <corr_id>
[ -f "$rec" ] || return 1
phase=$(fm_pending_reply_get "$rec" phase)
if [ "$phase" = delivery_unknown ]; then
fm_pending_reply_reconcile_delivery "$state" "$corr" || true
_fm_pending_reply_reconcile_delivery_locked "$state" "$corr" || true
phase=$(fm_pending_reply_get "$rec" phase)
[ "$phase" = delivery_unknown ] || return 0
fi
Expand Down
29 changes: 20 additions & 9 deletions bin/fm-send.sh
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,14 @@ MARK_FROM_FIRSTMATE=0
PENDING_REPLY_CORR=
PENDING_REPLY_CREATED=0
TARGET_TASK_ID=
fm_send_known_undelivered_cleanup() {
[ -n "$PENDING_REPLY_CORR" ] || return 0
if [ "$PENDING_REPLY_CREATED" = 1 ]; then
fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR"
else
fm_pending_reply_reset_known_undelivered "$STATE" "$PENDING_REPLY_CORR"
fi
}
if [ -n "$TARGET_SELECTOR" ] && [ -n "$TARGET_META" ] && [ "$(fm_meta_get "$TARGET_META" kind)" = secondmate ]; then
MARK_FROM_FIRSTMATE=1
TARGET_TASK_ID=$(fm_send_id_from_meta "$TARGET_META")
Expand Down Expand Up @@ -548,9 +556,14 @@ else
PENDING_REPLY_CREATED=1
fi
fm_pending_reply_embed_corr "$MESSAGE" "$PENDING_REPLY_CORR" MESSAGE
if [ "$PENDING_REPLY_CREATED" = 1 ] \
&& ! fm_pending_reply_prepare_delivery "$STATE" "$PENDING_REPLY_CORR"; then
fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true
if [ "$PENDING_REPLY_CREATED" != 1 ] \
&& fm_pending_reply_delivery_attempt_unresolved "$STATE" "$PENDING_REPLY_CORR"; then
echo "error: pending-reply delivery for $TARGET_TASK_ID is unresolved; refusing to resend correlation $PENDING_REPLY_CORR" >&2
exit 1
fi
if ! fm_pending_reply_prepare_delivery "$STATE" "$PENDING_REPLY_CORR"; then
[ "$PENDING_REPLY_CREATED" != 1 ] \
|| fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true
echo "error: failed to durably prepare pending-reply delivery for $TARGET_TASK_ID" >&2
exit 1
fi
Expand Down Expand Up @@ -608,19 +621,17 @@ else
echo "error: text delivery to remote secondmate $TARGET_REMOTE_ID is unknown; do not resend - same-host reconciliation is required" >&2
exit 1
fi
if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then
fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true
fi
fm_send_known_undelivered_cleanup || \
echo "error: known-undelivered pending-reply state could not be reset for $TARGET_TASK_ID" >&2
echo "error: text not sent to $T ($TARGET_BACKEND send failed; tried $RESOLUTION_TRIED)" >&2
exit 1
fi
case "$verdict" in
empty)
;;
send-failed)
if [ "$PENDING_REPLY_CREATED" = 1 ] && [ -n "$PENDING_REPLY_CORR" ]; then
fm_pending_reply_discard_undelivered "$STATE" "$PENDING_REPLY_CORR" || true
fi
fm_send_known_undelivered_cleanup || \
echo "error: known-undelivered pending-reply state could not be reset for $TARGET_TASK_ID" >&2
echo "error: text not sent to $T ($TARGET_BACKEND send failed; tried $RESOLUTION_TRIED)" >&2
exit 1
;;
Expand Down
Loading
Loading