Skip to content

fix: avoid redundant lint analysis while preserving source checks - #19

Merged
MrGTV-love merged 30 commits into
mainfrom
fm/fm-lint-memory-fix
Oct 6, 2026
Merged

MrGTV-love merged 30 commits into
mainfrom
fm/fm-lint-memory-fix

Conversation

@MrGTV-love

@MrGTV-love MrGTV-love commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

Intent

Host load investigation (2026-09-29, 10 cores, 32 GB, 12 of 13 GB swap in use; memory pressure is the main load driver). firstmate's own lint was the second-largest CPU user (about 0.9 of a core averaged over 2 minutes) and used 3.5 GB of memory across 3 shellcheck processes; one shellcheck run on tests/fm-pending-reply.test.sh reached 3.9 GB RSS. Observed shapes: shellcheck --norc --external-sources -- <file> under bin/fm-lint.sh --internal-worker, and shellcheck -x -S warning bin/fm-claude-launcher-lib.sh bin/fm-teamclaude-launch.sh bin/fm-spawn.sh bin/fm-control.sh test... run by workers.

The captain's words: "load continues to be something we navigate around. It would be helpful to learn the drivers of load and see if there is anything that we can deactivate that would increase resources." Then, on the proposed fork fix (lint only changed files and cap memory): "yes to lint memory fix". Standing preference: "I don't want any non-valuable friction. handicapping processes is almost never a good idea. I prefer fixing processes, not handicapping them".

What Changed

  • Select changed shell roots and their transitive sourcing callers for local lint, conservatively including unresolved imports. Restore source-aware analysis and cross-file diagnostics locally while keeping CI's complete, uncached checks.
  • Cache successful, input-stable ShellCheck results across worktrees using content fingerprints and serialize identical concurrent misses. Record per-root peak RSS even without execution bounds and recognize Perl memory-exhaustion errors.
  • Split status classification into focused record, decision, event, I/O, wake, and UTC libraries; narrow consumer imports and consolidate repeated pending-reply source statements. Update deployment lists, changed-owner test selection, regression coverage, and ownership/lint documentation.

Cold Measurements (before and after)

Method: one cold, uncached analysis per root in git-archived snapshots, run as /usr/bin/time -lp -o <time> bash bin/fm-lint.sh --telemetry <telemetry> <root>. Before is origin/main 12b3221. After is head 1a5a261. ShellCheck 0.11.0, default 2 jobs, full external-source context, cache off, no memory caps, no concurrency change. All four runs exited 0 and did real analysis.

Root Peak RSS before → after (bytes) User CPU (s) Wall (s)
tests/fm-pending-reply.test.sh 8156807168 → 2286043136 (-71.97%) 73.53 → 16.33 227.32 → 57.56
bin/fm-watch.sh 5673205760 → 5464358912 (-3.68%) 34.92 → 28.77 79.92 → 70.42

Limits of this evidence:

  • One paired sample per root on a busy shared host (ambient load1 24 to 49). This is not a fleet, P95, CI, or statistical claim.
  • The watcher root barely improves and still peaks near 5.46 GB (decimal GB, not GiB).
  • Warm-cache reuse and the cache protocol are not counted as a cold improvement.
  • Earlier same-run pairs with different times and earlier disconfirming measurements are retained in the run evidence and are not mixed into this table.
  • Raw data: cold-comparison.md and cold-measurements.json in the pipeline run evidence.

Test Exception (pre-existing remote-transport failures)

The Test step's verdict is no-go, not clean. The live remote-transport scenarios fail on the unmodified base as well as on this head. The captain-side supervisor ruled these failures pre-existing and separate from this lint change, and approved the Test step with a recorded exception reason. This change makes no transport edits and no deadline changes.

Evidence: five interleaved pairs, base 13f00a5 and head 1a5a261, same host, identical instrumentation, original deadlines, full command streams kept.

  • Base reproduces cross-home latency over 3 s (pairs 1, 5), a running-cancellation overrun of 7 s against 6 s (pair 2), a combined burst overrun over 12 s (pair 3), and a real c2 15 s guard expiry with exit 124 (pair 4).
  • Head shows the same failure classes (cross-home latency in pairs 2, 3, 4; burst overrun in pairs 1, 4). No failure class occurs only on head. Head pair 5 passes the full original sequence.
  • All five executions that reached the c1 command returned 0. Five stopped earlier.
  • Not explained: an earlier c1 exit 124 is not reproduced and its cause is unknown. Load is not proven to be the cause.
  • Ten samples support no statistical claim and do not show the transport is correct or reliable. A separate fix is owned outside this PR.
  • Raw data: transport-comparison/valid-pairs/ten-results.json and report.json in the pipeline run evidence.

Risk Assessment

⚠️ Medium: No material source defects were found across all 37 changed files, but dependency scanning and shared-cache correctness are nontrivial, and the documented measurements do not yet establish a general cold-memory or latency improvement.

Testing

The prior payload records an audit of the completed ten-run live transport comparison, including original staging/FIFO/cancellation prerequisites, full command streams, pre-cleanup job records, timing observations, and host load. Baseline reproduces the observed timing failure classes; all five reached c1 commands returned 0, and one head execution completed the full sequence. The recorded human decision treats reproduced baseline issues as non-blocking, but the supported live failures require a no-go verdict under this output contract. Prior analyzer, cold-memory, cache, and joint-SC2119 evidence was preserved.

  • Live validation: ❌ no-go - 5 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Dispatch a short job to another home while the first home's long job runs ❌ fail live Cross-home latency exceeded the unchanged 3-second limit in base pairs 1 and 5 and head pairs 2 and 3; ten-results.json contains every measurement.
Cancel a running caller and remove its job within the existing cancellation budget ❌ fail live Base pair 2 measured 7 seconds against the unchanged 6-second limit. Head pair 5 completed this assertion and the full sequence.
After queued, running, and orphan cancellation, execute c1 within its 15-second guard ✅ pass live c1 returned 0 in base pairs 3 and 4 and head pairs 1, 4, and 5, with retained burst-c1 markers. The other five executions failed earlier and did not reach c1.
Complete all three post-cancellation short commands within their individual and combined budgets ❌ fail live Base pair 3 exceeded the 12-second combined budget; base pair 4 hit an actual c2 guard expiry at 124. Head pairs 1 and 4 exceeded the combined budget; head pair 5 passed.
Complete the original transport sequence including FIFO, staging publication, cancellation, stdin, and litter reaping ✅ pass live pair-5-head/stdout.log shows the complete original sequence passing with test exit 0.
Evidence: Complete comparative attribution report
{
  "assigned_phase": "Test: focused transport attribution only",
  "base_sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
  "head_sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
  "execution_order": "base1,head1,base2,head2,base3,head3,base4,head4,base5,head5",
  "valid_execution_count": 10,
  "summary": "Pinned baseline reproduces cross-home latency and post-cancellation short-command timing failures; no head-only failure class was observed. No transport or permanent source/test edits were made.",
  "classification": {
    "cross_home_latency": "pre-existing: base pairs1/5 fail at4 seconds; head pairs2/3 fail at5/4 seconds against the unchanged3-second assertion",
    "post_cancellation_burst_latency": "pre-existing: base pair3 fails total18 seconds; head pairs1/4 fail total14/18 seconds against unchanged12-second assertion",
    "short_command_deadline": "pre-existing scenario failure reproduced at base pair4 c2=124 with c1=0; all five reached c1 invocations across both revisions return0",
    "original_c1_failure": "Not reproduced in this batch. Historical c1=124 streams/records were discarded in the original execution, so its exact stage/cause remains unavailable. The new baseline c2 timeout demonstrates the same post-cancellation short-command deadline failure class, not the exact historical c1 cause.",
    "branch_disposition": "Reproduced baseline timing failures are pre-existing and do not block this change under the recorded human decision; separate transport investigation remains with Main. This is not a declaration that all transport tests pass.",
    "head_only_defect_observed": false
  },
  "new_timeout_attribution": {
    "execution": "pair-4-base",
    "command": "c2",
    "timeout_status_allocation_exit": 0,
    "status_file": "fm-timeout-status.QNjrrb",
    "runner_pid": 42259,
    "runner_exit": 124,
    "command_completion_status": "empty",
    "guard_seconds": 15,
    "conclusion": "External command guard actually expired; not a status-file mktemp allocation failure.",
    "stage": "[INFERENCE] Evidence points to pre-publication/entrypoint validation or readiness work, not a retained queued/running/done/reap c2 job.",
    "process_observation": "Runner42259 -> wrapper42287 -> entrypoint42512 remains at elapsed13 seconds; descendant chain includes id. No retained argv identifies burst-c2; final fixture has no burst-c2 marker.",
    "limit": "Exact expiration instruction and causal contribution of ambient load are not established. This new c2 evidence cannot recover the historical c1 failure.",
    "evidence": [
      "pair-4-base/c2.stderr",
      "pair-4-base/timeout-allocations.tsv",
      "pair-4-base/reaped/80546-fm-timeout-status.QNjrrb",
      "pair-4-base/process-samples.log",
      "pair-4-base/reaped-paths.tsv",
      "pair-4-base/fixture-before-cleanup.tar"
    ]
  },
  "host": {
    "platform": "macOS-27.0.1-arm64-arm-64bit-Mach-O",
    "logical_cpus": 18,
    "one_minute_load_at_run_boundaries_min": 62.88330078125,
    "one_minute_load_at_run_boundaries_max": 91.82373046875,
    "load_policy": "Ambient load recorded every execution and approximately each second; no services/configuration/load controls, admission waits, or resource caps. Interleaving controls host identity, not equal load.",
    "instrumentation_limit": "Identical diagnostic tracing and capture shims were applied to both source snapshots; their unmeasured overhead can affect timing. These observations do not establish load as the root cause."
  },
  "rows": [
    {
      "pair": 1,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 60.208,
      "cross_home_seconds": 4,
      "c1_result": "not_reached",
      "failure": [
        "not ok - home B's job waited 4s behind home A's long job"
      ],
      "load_start": [
        79.3359375,
        58.5693359375,
        63.66162109375
      ],
      "load_end": [
        88.5634765625,
        64.69287109375,
        65.59326171875
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-1-base",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {},
      "burst_elapsed_seconds": "not_measured_before_failure",
      "not_reached_reason": "not ok - home B's job waited 4s behind home A's long job",
      "archive_integrity_verified": true
    },
    {
      "pair": 1,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 1,
      "wall_seconds": 92.934,
      "cross_home_seconds": 3,
      "c1_result": 0,
      "failure": [
        "not ok - the post-cancellation burst convoyed for 14s"
      ],
      "load_start": [
        88.5634765625,
        64.69287109375,
        65.59326171875
      ],
      "load_end": [
        70.87060546875,
        63.6669921875,
        64.990234375
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-1-head",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {
        "c1": 0,
        "c2": 0,
        "c3": 0
      },
      "burst_elapsed_seconds": 14,
      "not_reached_reason": null,
      "archive_integrity_verified": true
    },
    {
      "pair": 2,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 76.723,
      "cross_home_seconds": 2,
      "c1_result": "not_reached",
      "failure": [
        "not ok - running-job cancellation took 7s"
      ],
      "load_start": [
        70.87060546875,
        63.6669921875,
        64.990234375
      ],
      "load_end": [
        65.39208984375,
        63.77392578125,
        64.93310546875
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-2-base",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {},
      "burst_elapsed_seconds": "not_measured_before_failure",
      "not_reached_reason": "not ok - running-job cancellation took 7s",
      "archive_integrity_verified": true
    },
    {
      "pair": 2,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 1,
      "wall_seconds": 21.984,
      "cross_home_seconds": 5,
      "c1_result": "not_reached",
      "failure": [
        "not ok - home B's job waited 5s behind home A's long job"
      ],
      "load_start": [
        65.39208984375,
        63.77392578125,
        64.93310546875
      ],
      "load_end": [
        73.64501953125,
        66.013671875,
        65.7158203125
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-2-head",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {},
      "burst_elapsed_seconds": "not_measured_before_failure",
      "not_reached_reason": "not ok - home B's job waited 5s behind home A's long job",
      "archive_integrity_verified": true
    },
    {
      "pair": 3,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 102.228,
      "cross_home_seconds": 3,
      "c1_result": 0,
      "failure": [
        "not ok - the post-cancellation burst convoyed for 18s"
      ],
      "load_start": [
        73.64501953125,
        66.013671875,
        65.7158203125
      ],
      "load_end": [
        67.578125,
        66.85009765625,
        66.12646484375
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-3-base",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {
        "c1": 0,
        "c2": 0,
        "c3": 0
      },
      "burst_elapsed_seconds": 18,
      "not_reached_reason": null,
      "archive_integrity_verified": true
    },
    {
      "pair": 3,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 1,
      "wall_seconds": 14.478,
      "cross_home_seconds": 4,
      "c1_result": "not_reached",
      "failure": [
        "not ok - home B's job waited 4s behind home A's long job"
      ],
      "load_start": [
        67.578125,
        66.85009765625,
        66.12646484375
      ],
      "load_end": [
        66.45556640625,
        66.5908203125,
        66.04345703125
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-3-head",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {},
      "burst_elapsed_seconds": "not_measured_before_failure",
      "not_reached_reason": "not ok - home B's job waited 4s behind home A's long job",
      "archive_integrity_verified": true
    },
    {
      "pair": 4,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 160.226,
      "cross_home_seconds": 2,
      "c1_result": 0,
      "failure": [
        "not ok - post-cancellation burst command c2 failed with 124"
      ],
      "load_start": [
        66.45556640625,
        66.5908203125,
        66.04345703125
      ],
      "load_end": [
        91.82373046875,
        77.7236328125,
        70.6767578125
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-4-base",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {
        "c1": 0,
        "c2": 124
      },
      "burst_elapsed_seconds": "not_measured_before_failure",
      "not_reached_reason": null,
      "archive_integrity_verified": true
    },
    {
      "pair": 4,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 1,
      "wall_seconds": 108.022,
      "cross_home_seconds": 3,
      "c1_result": 0,
      "failure": [
        "not ok - the post-cancellation burst convoyed for 18s"
      ],
      "load_start": [
        91.82373046875,
        77.7236328125,
        70.6767578125
      ],
      "load_end": [
        62.88330078125,
        73.3603515625,
        69.93408203125
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-4-head",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {
        "c1": 0,
        "c2": 0,
        "c3": 0
      },
      "burst_elapsed_seconds": 18,
      "not_reached_reason": null,
      "archive_integrity_verified": true
    },
    {
      "pair": 5,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 18.434,
      "cross_home_seconds": 4,
      "c1_result": "not_reached",
      "failure": [
        "not ok - home B's job waited 4s behind home A's long job"
      ],
      "load_start": [
        62.88330078125,
        73.3603515625,
        69.93408203125
      ],
      "load_end": [
        64.314453125,
        73.17529296875,
        69.92822265625
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-5-base",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {},
      "burst_elapsed_seconds": "not_measured_before_failure",
      "not_reached_reason": "not ok - home B's job waited 4s behind home A's long job",
      "archive_integrity_verified": true
    },
    {
      "pair": 5,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 0,
      "wall_seconds": 105.54,
      "cross_home_seconds": 3,
      "c1_result": 0,
      "failure": [],
      "load_start": [
        64.314453125,
        73.17529296875,
        69.92822265625
      ],
      "load_end": [
        50.4716796875,
        66.3427734375,
        67.68603515625
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-5-head",
      "fixture_captured_before_cleanup": true,
      "burst_command_results": {
        "c1": 0,
        "c2": 0,
        "c3": 0
      },
      "burst_elapsed_seconds": 11,
      "not_reached_reason": null,
      "archive_integrity_verified": true
    }
  ],
  "evidence_integrity": {
    "archives": "All ten pre-cleanup fixture archives traversed and regular file lengths verified; all capture stderr empty.",
    "streams": "Full test stdout/stderr and separately captured callers/burst commands; job stdout/stderr/status preserved through pre-deletion shims and final archive.",
    "races": "Two duplicate concurrent deletion-copy attempts raced; the SAME original files survive through alternate captures. Raw errors and resolutions retained in capture-race-recovery.json.",
    "review_resolution": "capture-review-resolution.json",
    "contaminated_attempt": "Initial stock-Bash BASHPID diagnostic attempt cancelled and excluded. Its original raw evidence remains in the parent directory; ten reported rows are the corrected batch only."
  },
  "changes": [],
  "deadline_relaxations": false,
  "transport_code_changes": false,
  "other_pipeline_phases_run": false,
  "broad_tests_lint_or_static_analysis_run": false,
  "prior_evidence": "Historical analyzer/cold/cache/joint-SC2119 artifacts and original failed/disconfirming evidence left intact.",
  "cleanup": {
    "fixture_scoped_processes_remaining_after_batch": 0,
    "workspace_snapshot_removal": "Removed .transport-comparison; glob confirms snapshot and descendants absent. No permanent source/test edits or prepared dependencies removed."
  }
}
Evidence: Ten exact execution results with ambient load and evidence locations
{
  "base_sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
  "head_sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
  "execution_order": "base1,head1,base2,head2,base3,head3,base4,head4,base5,head5",
  "rows": [
    {
      "pair": 1,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 60.208,
      "cross_home_seconds": "4",
      "c1_result": "not_reached",
      "failure": [
        "not ok - home B's job waited 4s behind home A's long job"
      ],
      "load_start": [
        79.3359375,
        58.5693359375,
        63.66162109375
      ],
      "load_end": [
        88.5634765625,
        64.69287109375,
        65.59326171875
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-1-base",
      "fixture_captured_before_cleanup": true
    },
    {
      "pair": 1,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 1,
      "wall_seconds": 92.934,
      "cross_home_seconds": "3",
      "c1_result": "0",
      "failure": [
        "not ok - the post-cancellation burst convoyed for 14s"
      ],
      "load_start": [
        88.5634765625,
        64.69287109375,
        65.59326171875
      ],
      "load_end": [
        70.87060546875,
        63.6669921875,
        64.990234375
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-1-head",
      "fixture_captured_before_cleanup": true
    },
    {
      "pair": 2,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 76.723,
      "cross_home_seconds": "2",
      "c1_result": "not_reached",
      "failure": [
        "not ok - running-job cancellation took 7s"
      ],
      "load_start": [
        70.87060546875,
        63.6669921875,
        64.990234375
      ],
      "load_end": [
        65.39208984375,
        63.77392578125,
        64.93310546875
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-2-base",
      "fixture_captured_before_cleanup": true
    },
    {
      "pair": 2,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 1,
      "wall_seconds": 21.984,
      "cross_home_seconds": "5",
      "c1_result": "not_reached",
      "failure": [
        "not ok - home B's job waited 5s behind home A's long job"
      ],
      "load_start": [
        65.39208984375,
        63.77392578125,
        64.93310546875
      ],
      "load_end": [
        73.64501953125,
        66.013671875,
        65.7158203125
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-2-head",
      "fixture_captured_before_cleanup": true
    },
    {
      "pair": 3,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 102.228,
      "cross_home_seconds": "3",
      "c1_result": "0",
      "failure": [
        "not ok - the post-cancellation burst convoyed for 18s"
      ],
      "load_start": [
        73.64501953125,
        66.013671875,
        65.7158203125
      ],
      "load_end": [
        67.578125,
        66.85009765625,
        66.12646484375
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-3-base",
      "fixture_captured_before_cleanup": true
    },
    {
      "pair": 3,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 1,
      "wall_seconds": 14.478,
      "cross_home_seconds": "4",
      "c1_result": "not_reached",
      "failure": [
        "not ok - home B's job waited 4s behind home A's long job"
      ],
      "load_start": [
        67.578125,
        66.85009765625,
        66.12646484375
      ],
      "load_end": [
        66.45556640625,
        66.5908203125,
        66.04345703125
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-3-head",
      "fixture_captured_before_cleanup": true
    },
    {
      "pair": 4,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 160.226,
      "cross_home_seconds": "2",
      "c1_result": "0",
      "failure": [
        "not ok - post-cancellation burst command c2 failed with 124"
      ],
      "load_start": [
        66.45556640625,
        66.5908203125,
        66.04345703125
      ],
      "load_end": [
        91.82373046875,
        77.7236328125,
        70.6767578125
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-4-base",
      "fixture_captured_before_cleanup": true
    },
    {
      "pair": 4,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 1,
      "wall_seconds": 108.022,
      "cross_home_seconds": "3",
      "c1_result": "0",
      "failure": [
        "not ok - the post-cancellation burst convoyed for 18s"
      ],
      "load_start": [
        91.82373046875,
        77.7236328125,
        70.6767578125
      ],
      "load_end": [
        62.88330078125,
        73.3603515625,
        69.93408203125
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-4-head",
      "fixture_captured_before_cleanup": true
    },
    {
      "pair": 5,
      "revision": "base",
      "sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
      "exit_code": 1,
      "wall_seconds": 18.434,
      "cross_home_seconds": "4",
      "c1_result": "not_reached",
      "failure": [
        "not ok - home B's job waited 4s behind home A's long job"
      ],
      "load_start": [
        62.88330078125,
        73.3603515625,
        69.93408203125
      ],
      "load_end": [
        64.314453125,
        73.17529296875,
        69.92822265625
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-5-base",
      "fixture_captured_before_cleanup": true
    },
    {
      "pair": 5,
      "revision": "head",
      "sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
      "exit_code": 0,
      "wall_seconds": 105.54,
      "cross_home_seconds": "3",
      "c1_result": "0",
      "failure": [],
      "load_start": [
        64.314453125,
        73.17529296875,
        69.92822265625
      ],
      "load_end": [
        50.4716796875,
        66.3427734375,
        67.68603515625
      ],
      "timeout_mechanism": "timeout",
      "evidence_directory": "~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-5-head",
      "fixture_captured_before_cleanup": true
    }
  ]
}
Evidence: Independent audit of streams, persisted markers, archives, pinned hashes, and cleanup
{
  "base_sha": "13f00a597029189f5f6594c3952a47b41fb9a21b",
  "head_sha": "1a5a26139e65c4151a8bb2c55e59055854825639",
  "source": "Completed live batch already retained in this named test run; independently audited, not re-executed by this continuation.",
  "runtime_hashes_match_git_blobs": true,
  "transport_and_timeout_blobs_identical": true,
  "all_observed_results_match_raw_streams_and_persisted_markers": true,
  "all_fixture_archives_readable": true,
  "disposable_snapshot_absent": true,
  "rows": [
    {
      "pair": 1,
      "revision": "base",
      "exit_code": 1,
      "cross_home_seconds": "4",
      "c1_result": "not_reached",
      "burst_results": {},
      "failure": [
        "not ok - home B's job waited 4s behind home A's long job"
      ],
      "load_start": [
        79.3359375,
        58.5693359375,
        63.66162109375
      ],
      "load_end": [
        88.5634765625,
        64.69287109375,
        65.59326171875
      ],
      "archive_members": 238,
      "load_samples": 14
    },
    {
      "pair": 1,
      "revision": "head",
      "exit_code": 1,
      "cross_home_seconds": "3",
      "c1_result": "0",
      "burst_results": {
        "c1": "0",
        "c2": "0",
        "c3": "0"
      },
      "failure": [
        "not ok - the post-cancellation burst convoyed for 14s"
      ],
      "load_start": [
        88.5634765625,
        64.69287109375,
        65.59326171875
      ],
      "load_end": [
        70.87060546875,
        63.6669921875,
        64.990234375
      ],
      "archive_members": 230,
      "load_samples": 52
    },
    {
      "pair": 2,
      "revision": "base",
      "exit_code": 1,
      "cross_home_seconds": "2",
      "c1_result": "not_reached",
      "burst_results": {},
      "failure": [
        "not ok - running-job cancellation took 7s"
      ],
      "load_start": [
        70.87060546875,
        63.6669921875,
        64.990234375
      ],
      "load_end": [
        65.39208984375,
        63.77392578125,
        64.93310546875
      ],
      "archive_members": 205,
      "load_samples": 32
    },
    {
      "pair": 2,
      "revision": "head",
      "exit_code": 1,
      "cross_home_seconds": "5",
      "c1_result": "not_reached",
      "burst_results": {},
      "failure": [
        "not ok - home B's job waited 5s behind home A's long job"
      ],
      "load_start": [
        65.39208984375,
        63.77392578125,
        64.93310546875
      ],
      "load_end": [
        73.64501953125,
        66.013671875,
        65.7158203125
      ],
      "archive_members": 260,
      "load_samples": 6
    },
    {
      "pair": 3,
      "revision": "base",
      "exit_code": 1,
      "cross_home_seconds": "3",
      "c1_result": "0",
      "burst_results": {
        "c1": "0",
        "c2": "0",
        "c3": "0"
      },
      "failure": [
        "not ok - the post-cancellation burst convoyed for 18s"
      ],
      "load_start": [
        73.64501953125,
        66.013671875,
        65.7158203125
      ],
      "load_end": [
        67.578125,
        66.85009765625,
        66.12646484375
      ],
      "archive_members": 213,
      "load_samples": 39
    },
    {
      "pair": 3,
      "revision": "head",
      "exit_code": 1,
      "cross_home_seconds": "4",
      "c1_result": "not_reached",
      "burst_results": {},
      "failure": [
        "not ok - home B's job waited 4s behind home A's long job"
      ],
      "load_start": [
        67.578125,
        66.85009765625,
        66.12646484375
      ],
      "load_end": [
        66.45556640625,
        66.5908203125,
        66.04345703125
      ],
      "archive_members": 258,
      "load_samples": 6
    },
    {
      "pair": 4,
      "revision": "base",
      "exit_code": 1,
      "cross_home_seconds": "2",
      "c1_result": "0",
      "burst_results": {
        "c1": "0",
        "c2": "124"
      },
      "failure": [
        "not ok - post-cancellation burst command c2 failed with 124"
      ],
      "load_start": [
        66.45556640625,
        66.5908203125,
        66.04345703125
      ],
      "load_end": [
        91.82373046875,
        77.7236328125,
        70.6767578125
      ],
      "archive_members": 209,
      "load_samples": 46
    },
    {
      "pair": 4,
      "revision": "head",
      "exit_code": 1,
      "cross_home_seconds": "3",
      "c1_result": "0",
      "burst_results": {
        "c1": "0",
        "c2": "0",
        "c3": "0"
      },
      "failure": [
        "not ok - the post-cancellation burst convoyed for 18s"
      ],
      "load_start": [
        91.82373046875,
        77.7236328125,
        70.6767578125
      ],
      "load_end": [
        62.88330078125,
        73.3603515625,
        69.93408203125
      ],
      "archive_members": 230,
      "load_samples": 43
    },
    {
      "pair": 5,
      "revision": "base",
      "exit_code": 1,
      "cross_home_seconds": "4",
      "c1_result": "not_reached",
      "burst_results": {},
      "failure": [
        "not ok - home B's job waited 4s behind home A's long job"
      ],
      "load_start": [
        62.88330078125,
        73.3603515625,
        69.93408203125
      ],
      "load_end": [
        64.314453125,
        73.17529296875,
        69.92822265625
      ],
      "archive_members": 247,
      "load_samples": 8
    },
    {
      "pair": 5,
      "revision": "head",
      "exit_code": 0,
      "cross_home_seconds": "3",
      "c1_result": "0",
      "burst_results": {
        "c1": "0",
        "c2": "0",
        "c3": "0"
      },
      "failure": [],
      "load_start": [
        64.314453125,
        73.17529296875,
        69.92822265625
      ],
      "load_end": [
        50.4716796875,
        66.3427734375,
        67.68603515625
      ],
      "archive_members": 243,
      "load_samples": 68
    }
  ]
}
  • Evidence: Baseline post-cancellation c2 deadline-expiry trace (local file: ~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/transport-comparison/valid-pairs/pair-4-base/c2.stderr)
Evidence: Complete passing head sequence
ok - atomic sequence claims remain unique and reap only after expiry
ok - lanes run homes concurrently while each home stays FIFO
ok - same-home jobs staged in the same second execute in staging-sequence order
ok - same-home sequence order follows completed staging publication
ok - a caller killed mid-wait cancels its queued job before execution
ok - a caller killed mid-wait stops its running job's process group
ok - a signal-less caller disconnect cancels the abandoned job through the parent probe
ok - bounded reads after a cancellation meet their own budget with no convoy
ok - an open caller stdin no longer wedges a non-payload remote command
ok - a live explicit-stdin stage survives the staging-litter age bound
ok - --stdin still delivers a payload caller's bytes
ok - abandoned stage litter is reaped by age while live staging survives
ALL TESTS PASSED
Evidence: All ten results in execution order
Pair revision | test exit | cross-home seconds | c1 exit | outcome | 1-minute load start→end
1 base | 1 | 4 | not reached | cross-home >3s | 79.34→88.56
1 head | 1 | 3 | 0 | burst 14s >12s; c1/c2/c3 exit 0 | 88.56→70.87
2 base | 1 | 2 | not reached | running cancellation 7s >6s | 70.87→65.39
2 head | 1 | 5 | not reached | cross-home >3s | 65.39→73.65
3 base | 1 | 3 | 0 | burst 18s >12s; c1/c2/c3 exit 0 | 73.65→67.58
3 head | 1 | 4 | not reached | cross-home >3s | 67.58→66.46
4 base | 1 | 2 | 0 | c2 exit 124; c3 not reached | 66.46→91.82
4 head | 1 | 3 | 0 | burst 18s >12s; c1/c2/c3 exit 0 | 91.82→62.88
5 base | 1 | 4 | not reached | cross-home >3s | 62.88→64.31
5 head | 0 | 3 | 0 | complete sequence passes | 64.31→50.47
- Outcome: ⚠️ 2 issues (1 error, 1 warning) across 6 runs (2h32m26s)

Pipeline

Updates from git push no-mistakes

... (12 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)

⚠️ **Test** - 2 issues (1 error, 1 warning)
  • shellcheck --version confirmed the installed 0.11.0 pin; binary identity was retained.
  • python3 .live-validation/measure.py ran cold baseline origin/main and final-head bash bin/fm-lint.sh --telemetry &lt;path&gt; &lt;root&gt; analyses for tests/fm-pending-reply.test.sh and bin/fm-watch.sh, then real production pending-library cold/warm/source-change checks.
  • bin/fm-lint.sh --list-files in isolated real-git fixtures exercised changed/deleted direct and transitive imports, executable child-shell/stdin imports, inert text exclusion, and unresolved-import conservatism.
  • bin/fm-test-run.sh --list --changed --base HEAD exercised all six status/UTC owner mappings against the nine required families and exact additional consuming scripts.
  • python3 .live-validation/analyzer.py exercised real analyzer diagnostics, shared/concurrent cache reuse, source/binary invalidation, version refusal, and explicit/changed/cold/mutated-cache modes.
  • python3 .live-validation/joint-seams.py exercised graph-preserving baseline/final production pending, watcher, and resolve seams with seeded SC2119 defects and owner-only/hidden-source counterfactuals.
  • python3 .live-validation/ci-recheck.py exercised complete disposable canonical inventories: clean CI analysis and seeded CI rejection, both without local cache reuse.
  • python3 .live-validation/production-cache.py exercised seven actual nested-import roots and real analyzer execution after changes/deletions of the production stdin-imported test helpers.
  • bash bin/fm-test-run.sh tests/fm-lint-workflows.test.sh --json &lt;evidence&gt;/workflow-timing.json exercised workflow-product validation and malformed workflow rejection.
  • bash bin/fm-test-run.sh --jobs 1 --json &lt;evidence&gt;/runtime-consumers-timing.json tests/fm-classify-decision-key.test.sh tests/fm-classify-corr-token.test.sh tests/fm-pending-reply.test.sh tests/fm-remote-reply.test.sh tests/fm-remote-transport-lanes.test.sh tests/fm-gotmp.test.sh tests/fm-stat-shadowing.test.sh tests/fm-test-fixtures.test.sh.
  • Re-drove tests/fm-remote-reply.test.sh from a disposable copied checkout with a sibling remote home after diagnosing the original root/home overlap refusal.
  • Ran disposable runtime-import and direct report/drain CLI smoke scripts through bin/fm-test-run.sh, covering reload, dynamic locals, home switching, snapshots, correlation, and parent publication retries.
  • Ran one focused cancellation-to-burst diagnostic with unchanged deadlines and retained command, worker, job, and process evidence before teardown.
  • Removed all owned .live-validation fixtures and retained evidence; no permanent source, test, or documentation changes were made.

🔧 No changes applied.
2 issues (1 error, 1 warning) still open:

  • ⚠️ tests/fm-remote-transport-lanes.test.sh:363 - The original targeted runtime execution returned 124 for c1, the first short command after queued, running, and orphaned-job cancellation. Subsequent complete-sequence reproduction and instrumented execution passed, but neither explains the original failure. The failing fixture discarded command diagnostics and removed job/worker records, so the timed-out stage cannot now be attributed. The transport implementation is unchanged versus the supplied base; a changed-code regression versus intermittent infrastructure/timing failure remains unresolved. No demonstrated root cause supports a safe fix or deadline relaxation. Retained evidence: runtime-consumers.log:90–99 and transport-fix/report.json. A human decision or an attributed reproduction is required before treating this scenario as green.
  • 🚨 live validation verdict: no-go (13 of 13 scenarios were driven live against the product); failed: Cancel queued, running, and orphaned jobs and complete the next short command without a convoy timeout
  • Live validation: ❌ no-go - 13 of 13 scenarios driven live against the product
Scenario Result Live Evidence
Change or delete an imported shell file and select its direct and transitive lint callers ✅ pass live final-stdin-smoke.json and selection-proof/manifest.json
Track executable child-shell and stdin imports, including production helper payloads ✅ pass live final-stdin-smoke.json, selection-proof/manifest.json, production-cache.json
Exclude inert quoted programs and heredoc data while retaining unresolved-import safety ✅ pass live final-stdin-smoke.json and analyzer-scenarios.json
Reuse unchanged production pending-reply analysis and invalidate it after source changes ✅ pass live pending-cache-cold.log, pending-cache-warm.log, pending-cache-source-miss.log, analyzer-authorized/phase-result.json
Share successful analyses across isolated copies and invalidate changed analyzer inputs ✅ pass live analyzer-scenarios.json, shared-copy.log, concurrent-0.log, concurrent-1.log
Detect imported positional-argument defects in full joint consumer analysis ✅ pass live joint-seams.json and analyzer-authorized/phase-result.json
Run uncached canonical CI analysis and reject broken workflow input ✅ pass live canonical-ci.json, canonical-ci-clean.log, canonical-ci-seeded.log, workflow-observation.txt
Select the existing consuming tests when an extracted UTC or status owner changes ✅ pass live selection-proof/manifest.json
Reduce cold pending-reply analysis resources without changing concurrency or imposing caps ✅ pass live cold-comparison.md and cold-measurements.json
Measure cold watcher analysis resources under the same analyzer and resource settings ✅ pass live cold-comparison.md and cold-measurements.json
Preserve runtime initialization and open/resolve a correlated decision through report and drain ✅ pass live final-runtime-import-smoke.log and final-runtime-cli-smoke-corrected.log
Publish a remote reply through the real local worker and relay chain ✅ pass live runtime-remote-reply-separated.log and runtime-remote-reply-separated-timing.json
Cancel queued, running, and orphaned jobs and complete the next short command without a convoy timeout ❌ fail live runtime-consumers.log:90–99 and transport-fix/report.json
  • Fresh isolated real-product executions of bash bin/fm-lint.sh bin/stdin.sh, bin/caller.sh, and bin/inert.sh: cold/warm reuse, dependency-change misses, deletion misses, and --list-files direct/transitive selection.
  • Fresh disposable runtime execution of runtime-import-smoke.test.sh: initialization, reload, dynamic locals, home switching, timestamp output variables, frozen snapshots, and parent-publication retry.
  • Fresh disposable runtime execution of runtime-cli-smoke-corrected.test.sh: real fm-secondmate-report.sh and fm-wake-drain.sh open and resolve a correlated decision.
  • Retained earlier executions in this same run: production fm-lint.sh pending-root cold/warm/copy reuse, source and analyzer invalidation, unknown-import safety, and pinned-version rejection.
  • Retained earlier same-run source-aware ShellCheck 0.11.0 executions: seeded joint SC2119 findings in pending, watcher, and resolve graphs, with owner-only and hidden-source counterfactuals.
  • Retained earlier same-run executions of fm-lint.sh --list-files, fm-test-run.sh --list --changed --base HEAD, canonical CI fixture analysis, and workflow validation.
  • Retained same-run cold CLI measurements: /usr/bin/time -lp -o &lt;time&gt; bash bin/fm-lint.sh --telemetry &lt;telemetry&gt; tests/fm-pending-reply.test.sh and the corresponding bin/fm-watch.sh invocation on baseline and final snapshots.
  • Retained targeted runtime command: bash bin/fm-test-run.sh --jobs 1 --json &lt;timing&gt; tests/fm-classify-decision-key.test.sh tests/fm-classify-corr-token.test.sh tests/fm-pending-reply.test.sh tests/fm-remote-reply.test.sh tests/fm-remote-transport-lanes.test.sh tests/fm-gotmp.test.sh tests/fm-stat-shadowing.test.sh tests/fm-test-fixtures.test.sh.
  • Retained corrected remote-reply execution with separate checkout/home paths, complete transport reproduction, and instrumented complete-history transport execution; subsequent transport passes did not erase the original timeout.
  • Verified removal of this turn's .final-stdin-smoke-* and .final-runtime-smoke-* disposable fixtures.

🔧 No changes applied.
2 issues (1 error, 1 warning) still open:

  • ⚠️ tests/fm-remote-transport-lanes.test.sh:363 - The original targeted execution returned 124 for c1, the first short command after queued, running, and orphaned-job cancellation. Subsequent complete-sequence reproductions and instrumented execution passed, but did not explain the failure. The original fixture discarded command diagnostics and removed job/worker records, so the timed-out stage cannot be reconstructed. The transport implementation is unchanged versus the supplied base; a changed-code regression versus an intermittent timing/infrastructure failure remains unresolved. No demonstrated cause supports a safe code fix or deadline relaxation. Evidence: runtime-consumers.log:90–99, transport-fix/report.json, and transport-fix/focused-repair-result.json. An attributed reproduction or explicit human decision is required before treating this scenario as green.
  • 🚨 live validation verdict: no-go (13 of 13 scenarios were driven live against the product); failed: Cancel queued, running, and orphaned jobs and complete the next short command without a convoy timeout
  • Live validation: ❌ no-go - 13 of 13 scenarios driven live against the product
Scenario Result Live Evidence
Change or delete an imported helper and select its direct and transitive shell callers ✅ pass live selection-proof/manifest.json; final-test-phase-analyzer-transcript.log
Put import-like text in inert programs or heredoc data without selecting unrelated consumers ✅ pass live selection-proof/manifest.json; analyzer-scenarios.json; final-test-phase-analyzer-transcript.log
Lint deterministic nested source expressions and reuse unchanged successful analysis ✅ pass live production-cache.json; nested--cold.log; nested--warm.log
Lint the pending-reply production closure, reuse success, and invalidate it after an imported-source change ✅ pass live pending-cache-cold.log; pending-cache-warm.log; pending-cache-source-miss.log; pending-cache-source-warm.log
Reuse identical successful checks across isolated copies and concurrent callers without caching failures ✅ pass live analyzer-scenarios.json; concurrent-0.log; concurrent-1.log
Keep unresolved imports conservative and invalidate reuse when analyzer identity changes ✅ pass live analyzer-scenarios.json; final-test-phase-analyzer-transcript.log
Detect joint imported-owner and caller defects in explicit, changed, cold, and CI modes ✅ pass live analyzer-scenarios.json; joint-seams.json; canonical-ci.json; joint-seam-after-pending.log; joint-seam-after-watch.log; joint-seam-after-resolve.log
Run cold pending-reply and watcher lint with unchanged resources and measure before/after cost ✅ pass live cold-comparison.md; cold-measurements.json; before-.time.txt; after-.time.txt
Run workflow validation and reject malformed workflow input through the product interface ✅ pass live workflow-observation.txt; workflow-timing.json
Change each extracted status or UTC owner and select its established consuming tests ✅ pass live selection-proof/manifest.json
Initialize, reload, switch homes, and publish correlated status through the real runtime ✅ pass live final-test-phase-runtime-import.log; final-test-phase-runtime-cli.log; runtime-consumers.log
Capture and resolve remote replies locally and clean up copied-runtime task directories ✅ pass live runtime-remote-reply-separated.log; runtime-consumers.log
Cancel queued, running, and orphaned jobs and complete the next short command without a convoy timeout ❌ fail live runtime-consumers.log:90–99; transport-fix/report.json; transport-fix/focused-repair-result.json
  • shellcheck --version confirmed official ShellCheck 0.11.0 on PATH.
  • Fresh disposable-fixture execution of bash bin/fm-lint.sh tests/stdin.sh: cold analysis, unchanged cache reuse, source-change miss, and source-deletion miss.
  • Fresh disposable-fixture execution of bash bin/fm-lint.sh --list-files: changed and deleted executable-heredoc imports selected direct and transitive callers while excluding inert heredoc data.
  • Fresh disposable-fixture execution of bash bin/fm-lint.sh tests/inert.sh and bash bin/fm-lint.sh tests/unknown.sh: inert text retained its cache; genuinely unresolved executable imports never reused success.
  • /bin/bash ~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/runtime-import-smoke.test.sh exercised real initialization, reload, dynamic locals, home switching, timestamp parsing, frozen snapshots, and parent-channel publication retry.
  • /bin/bash ~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/runtime-cli-smoke-corrected.test.sh drove real fm-secondmate-report and fm-wake-drain CLIs through correlated decision open/close.
  • Retained exact-target, same-run cold comparisons executed /usr/bin/time -lp around bash bin/fm-lint.sh --telemetry &lt;artifact&gt; tests/fm-pending-reply.test.sh and bash bin/fm-lint.sh --telemetry &lt;artifact&gt; bin/fm-watch.sh on origin/main 12b322120dc015f8a526a1d00d323b31e453c2c4 and target 1a5a26139e65c4151a8bb2c55e59055854825639. Complete exact argv, raw timings, and telemetry are in cold-measurements.json; these expensive measurements were not repeated.
  • Retained same-run production analysis executed bash bin/fm-lint.sh bin/fm-pending-reply-lib.sh through cold, warm, imported-source mutation, and subsequent warm states using the actual production backend closure.
  • Retained same-run analyzer scenarios exercised explicit-root, changed-root, cold/miss, shared-copy, concurrent-cache, unresolved-import, analyzer-identity, and CI-mode behavior through fm-lint.sh; graph-preserving pending/watch/resolve probes verified joint SC2119 diagnostics and hidden-source/owner-only counterfactuals.
  • Retained same-run selection checks executed bin/fm-test-run.sh --list --changed --base HEAD for each of the six extracted owners and bin/fm-lint.sh --list-files for production child-shell and executable-stdin helper changes/deletions.
  • Retained same-run targeted execution: bash bin/fm-test-run.sh --jobs 1 --json ~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/runtime-consumers-timing.json tests/fm-classify-decision-key.test.sh tests/fm-classify-corr-token.test.sh tests/fm-pending-reply.test.sh tests/fm-remote-reply.test.sh tests/fm-remote-transport-lanes.test.sh tests/fm-gotmp.test.sh tests/fm-stat-shadowing.test.sh tests/fm-test-fixtures.test.sh.
  • Retained same-run workflow check: bash bin/fm-test-run.sh tests/fm-lint-workflows.test.sh --json ~/.no-mistakes/evidence/01M490FVKQ7G2KX8GST7M1P2CP/workflow-timing.json.
  • Retained corrected remote-reply execution used an isolated copied checkout with a sibling disposable home; complete transport reproductions and the instrumented full-history execution are recorded in transport-fix/report.json.
  • Removed both fresh worktree-local disposable fixtures after execution; retained product transcripts and evidence indexes in the designated evidence directory.

🔧 No changes applied.
2 issues (1 error, 1 warning) still open:

  • ⚠️ tests/fm-remote-transport-lanes.test.sh:363 - The original targeted execution returned 124 for c1 after queued, running, and orphaned-job cancellation. Its suppressed command streams and deleted job records prevent attributing that failure. Subsequent passing sequences did not explain it. The latest attribution attempts failed earlier at the cross-home latency assertion (6 seconds uninstrumented, 4 seconds instrumented), before reaching cancellation; the trace shows home B completed in its own lane but does not establish the delay's cause. The transport implementation is unchanged versus the supplied base. No demonstrated cause supports a safe code fix or deadline relaxation. An attributed reproduction or explicit human decision is required before treating this scenario as green. Evidence: runtime-consumers.log:90–99 and transport-fix/round-five-attribution/report.json.
  • 🚨 live validation verdict: no-go (14 of 14 scenarios were driven live against the product); failed: Cancel queued, running, and orphaned jobs and complete the next short command without a convoy timeout
  • Live validation: ❌ no-go - 14 of 14 scenarios driven live against the product
Scenario Result Live Evidence
Change or delete a sourced input and select its direct, transitive, and runtime sourcing callers ✅ pass live selection-proof/manifest.json, using the corrected fresh production-fixture results
Change an unrelated shell file without selecting pending consumers through inert jq, awk, prose, or heredoc text ✅ pass live selection-proof/production-unrelated-shell.json; final-test-phase-analyzer-recheck.json
Lint deterministic nested source expressions and reuse unchanged successful analysis ✅ pass live production-cache.json; nested--cold.log and nested--warm.log
Change executable bash -c imports and invalidate both caller selection and cached success ✅ pass live selection-proof/manifest.json; analyzer-scenarios.json child-source-change
Change or delete helpers imported by executable shell stdin without treating inert heredocs as imports ✅ pass live selection-proof/manifest.json; production-cache.json; final-test-phase-analyzer-recheck.json
Lint a genuinely unresolved executable import without reusing a potentially stale success ✅ pass live final-recheck-unknown-1.log; final-recheck-unknown-2.log; selection-proof/manifest.json
Change an extracted status or UTC owner and select its established consuming tests ✅ pass live selection-proof/manifest.json status-owner scenarios
Reuse successful analysis across isolated copies and concurrent callers while invalidating analyzer changes ✅ pass live analyzer-scenarios.json; shared-copy.log; concurrent-0.log; concurrent-1.log; binary-binary-change.log; version-refusal.log
Lint the production pending-reply backend closure, reuse success, and reanalyze after a source change ✅ pass live pending-cache-cold.log; pending-cache-warm.log; pending-cache-source-miss.log; pending-cache-source-warm.log
Detect joint imported-function and caller defects in explicit, changed-root, and full-CI analysis ✅ pass live round-six-analyzer-import-defect.log; round-six-analyzer-failed-retry.log; joint-seams.json; canonical-ci.json
Run cold pending and watcher analysis with unchanged resources and measure against same-host origin/main ✅ pass live cold-comparison.md; cold-measurements.json; environment.json
Reject malformed workflows through the normal lint entry point ✅ pass live workflow-observation.txt; workflow-timing.json
Preserve extracted-owner initialization, dynamic scope, home switching, snapshots, and correlated publication ✅ pass live round-six-runtime-import.log; round-six-runtime-cli.log; runtime-remote-reply-separated.log; runtime-consumers.log
Cancel queued, running, and orphaned jobs and complete the next short command without a convoy timeout ❌ fail live runtime-consumers.log:90–99; transport-fix/round-five-attribution/report.json
  • Fresh isolated production execution: /bin/bash bin/fm-lint.sh bin/lab-consumer.sh, exercising cold success, warm reuse, imported-source invalidation with joint SC2119, and repeated analysis of findings.
  • Fresh isolated runtime execution: /bin/bash &lt;evidence&gt;/runtime-import-smoke.test.sh, exercising real initialization, reload, dynamic locals, home switching, timestamp parsing, frozen snapshots, and parent-publication retry.
  • Fresh isolated CLI execution: /bin/bash &lt;evidence&gt;/runtime-cli-smoke-corrected.test.sh, driving real fm-secondmate-report.sh and fm-wake-drain.sh through correlated decision opening and resolution.
  • Retained same-run product selection checks: bin/fm-lint.sh --list-files and bin/fm-test-run.sh --list --changed --base HEAD against disposable Git repositories with changed and deleted dependencies.
  • Retained same-run official ShellCheck 0.11.0 checks through fm-lint.sh: production pending-root cache reuse/invalidation, deterministic nested imports, executable child-shell imports, executable-stdin imports, unresolved-import safety, cross-copy/concurrent reuse, binary invalidation, and version refusal.
  • Retained same-run explicit, changed-root, and disposable canonical-CI seeded SC2119 checks; production pending/watch/resolve import-seam checks with hidden-source and owner-only diagnostic controls.
  • Retained same-run cold comparison: /usr/bin/time -lp -o &lt;time&gt; bash bin/fm-lint.sh --telemetry &lt;telemetry&gt; &lt;root&gt; for tests/fm-pending-reply.test.sh and bin/fm-watch.sh, comparing origin/main 12b322120dc015f8a526a1d00d323b31e453c2c4 with target 1a5a26139e65c4151a8bb2c55e59055854825639.
  • Retained targeted runner execution for classify-decision-key, classify-corr-token, pending-reply, remote-reply, remote-transport-lanes, gotmp, stat-shadowing, and test-fixtures; remote-reply setup corrected using a separate disposable checkout.
  • Retained workflow consumer execution: bash bin/fm-test-run.sh tests/fm-lint-workflows.test.sh --json &lt;evidence&gt;/workflow-timing.json.
  • Inspected original cancellation failure and latest timestamped attribution evidence; compared transport files with the supplied base without rerunning the reported failures.
  • Removed both fresh disposable fixtures; verified their worktree path patterns were absent.

🔧 No changes applied.
2 issues (1 error, 1 warning) still open:

  • ⚠️ tests/fm-remote-transport-lanes.test.sh - The pinned baseline reproduces cross-home latency failures, post-cancellation burst overruns, and a post-cancellation c2=124 deadline expiry. The new timeout was an actual 15-second guard expiry, not a status-file mktemp failure. No head-only failure class was observed. The historical c1=124 was not reproduced and its exact cause remains unavailable because the original streams and records were discarded. Per the recorded human decision, these baseline failures belong to Main's separate transport investigation and do not block this change.
  • 🚨 live validation verdict: no-go (5 of 5 scenarios were driven live against the product); failed: Dispatch a short job to another home while the first home's long job runs, Cancel a running caller and remove its job within the existing cancellation budget, Complete all three post-cancellation short commands within their individual and combined budgets
  • Live validation: ❌ no-go - 5 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Dispatch a short job to another home while the first home's long job runs ❌ fail live Cross-home latency exceeded the unchanged 3-second limit in base pairs 1 and 5 and head pairs 2 and 3; ten-results.json contains every measurement.
Cancel a running caller and remove its job within the existing cancellation budget ❌ fail live Base pair 2 measured 7 seconds against the unchanged 6-second limit. Head pair 5 completed this assertion and the full sequence.
After queued, running, and orphan cancellation, execute c1 within its 15-second guard ✅ pass live c1 returned 0 in base pairs 3 and 4 and head pairs 1, 4, and 5, with retained burst-c1 markers. The other five executions failed earlier and did not reach c1.
Complete all three post-cancellation short commands within their individual and combined budgets ❌ fail live Base pair 3 exceeded the 12-second combined budget; base pair 4 hit an actual c2 guard expiry at 124. Head pairs 1 and 4 exceeded the combined budget; head pair 5 passed.
Complete the original transport sequence including FIFO, staging publication, cancellation, stdin, and litter reaping ✅ pass live pair-5-head/stdout.log shows the complete original sequence passing with test exit 0.
  • Audited the completed five interleaved pairs of /bin/bash -x tests/fm-remote-transport-lanes.test.sh against workspace-local snapshots of the exact supplied base and head, preserving the original sequence, assertions, and deadlines.
  • Checked every execution's recorded cross-home latency and c1 result against raw command streams, burst exit records, and persisted c1 markers; early failures were explicitly recorded as c1 not reached.
  • Read every regular file in all ten pre-cleanup fixture archives and checked capture diagnostics and ambient-load samples.
  • Checked pair-4-base/c2.stderr, timeout-allocations.tsv, and retained timeout status records: allocation exited 0, the external runner exited 124, and command completion status remained empty.
  • Compared retained transport, timeout, and harness SHA-256 hashes against git show &lt;pinned-sha&gt;:&lt;file&gt;; those files are identical between base and head.
  • Confirmed the disposable .transport-comparison snapshot directory is absent. No repeat cold benchmarks, broad suite, linters, static analysis, or other pipeline phases were run.
✅ **Document** - passed

✅ No issues found.

🔧 **Lint** - 1 issue found → auto-fixed ✅
  • ⚠️ linter found issues (exit code 1)

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Push** - passed

✅ No issues found.

…e failure is from this PR. 1. Behavior portable parallel 1: tests/fm-lint.test.sh test_pinned_shellcheck_memory_limit failed with "the memory-bound pinned root was not named". This PR caused it. The per-root analysis command now runs `perl bin/fm-lint-cache.pl check ...` inside the per-root `ulimit -v`. Under the test's 8 MiB cap, Perl 5.38 (Ubuntu 24.04) prints "Out of memory!" and exits 1. fm_lint_classify_root mapped rc 1 to "findings", so the root was not named and not recorded as an abnormal end. I reproduced this in an ubuntu:24.04 container. Perl 5.40 and 5.42 crash with SIGSEGV instead, which is already classified as a named signal. Fix: bin/fm-lint.sh adds `^Out of memory!$` (a whole stderr line) to the existing memory-evidence regex, and the classifier comment is updated. Regression: tests/fm-lint.test.sh gains an `oom-perl` stub case in test_memory_evidence_outranks_findings_and_signal_reasons. It prints "Out of memory!" and exits 1, and it must classify as memory. It runs portably in unbounded mode, plus bounded mode on Linux. Verified: with the old fm-lint.sh the new case fails ("oom-perl was classified 'findings', expected memory"). With the fix, the full tests/fm-lint.test.sh passes on macOS (bounded tests skipped). It also passes on Linux (ubuntu:24.04, pinned ShellCheck 0.11.0 + actionlint): 44 ok, including the pinned memory-envelope test and modes 0 1. `bin/fm-lint.sh bin/fm-lint.sh tests/fm-lint.test.sh` passes. 2. Behavior portable serial 2 (tests/fm-calm-pi-extension.test.sh, "calm mode was not off by default") and 3. Behavior portable serial 4 (tests/fm-pi-branch-extension.test.sh, "Calm-off ToolExecutionComponent rendering differs from Pi stock"): not caused by this PR. This PR touches only lint files. The same two Pi-extension tests fail today on unrelated branches: CI runs 36928154699 (fm/fm-jev-mem-guard-macos), 36926345705 (fm/fm-jev-never-send-sections) and 36887810827 (fm/fm-teardown-leak-kill-identity). That points to external Pi drift. No code change for these
…ause it. No branch change; this needs a main-first repair. The failing assertion is tests/fm-dispatch-resolve.test.sh expect_withheld line 267, assert_equals '1' "$(grep -c . <<<"$err")", in the "a directory at the list path" case. In CI it got 2 lines. The test does not print the raw stderr, so the extra line's text is not in the log. Facts: (1) the PR does not change bin/fm-dispatch-resolve.sh, its test, tests/lib.sh, the workflow, or anything the behavior shard runs. Those files are the same as base a4d559e and main. (2) The same test, shard order and runner image (ubuntu24 20260927.320.1) passed on this PR's previous head (run 36933934075). It also passed on main a4d559e and on 4 other branches today. (3) Counterfactual: 3 full test runs plus 40 runs under 10x load passed in ubuntu:24.04 with bash 5.2.21. 2,400 aarch64 runs and 1,200 x86_64 runs of that exact case each printed exactly one line. The cause is an intermittent extra stderr line in main's test/tool path. Main-first prerequisite: on main, delete the incidental line-count assertion in expect_withheld. Keep the off-reason, "nothing sent", exit 0, empty stdout, no-curl, no-quota and no-value-printed checks. ci-1, ci-2 and ci-4 stay unapproved, with their own owners. cold-source-aware-memory-still-multigib: fixed one real cause of extra expansion. bin/fm-pending-reply-lib.sh had a `# shellcheck source=bin/fm-marker-lib.sh` directive on the _FM_PENDING_REPLY_LIB_DIR assignment (from main kunchenguid#5753, "double duty"). I confirmed with a small probe that ShellCheck then binds every later "undirected" `. "$_FM_PENDING_REPLY_LIB_DIR/..."` source to fm-marker-lib.sh. So each of 8 sites re-read fm-marker-lib.sh plus fm-operational-input.sh. An strace of ShellCheck 0.11.0 shows fm-marker-lib.sh opened 9x for bin/fm-watch.sh and 20x for tests/fm-pending-reply.test.sh. Fix: remove the bind directive. Add `# shellcheck source=/dev/null` to the 8 sites that should not be followed. Only comment lines change. Directed sites, cross-file analysis, selection, cache, CI parity, flags and caps are unchanged. Controlled A/B, same container, pinned ShellCheck 0.11.0 (linux aarch64), --norc --external-sources, all 17 roots whose closure has this library. Output and exit code are identical (clean). Peak RSS in KiB: watch 5587052->4852980 (-13.1%); pending-reply.test 5713888->4498528 (-21.3%); pending-reply-lib 1997132->1508624 (-24.5%); backlog-handoff -14.5%; send -14.5%; procevent-remote-reply -18.5%; remote-secondmate-control -24.1%; secondmate-report -24.6%; secondmate-restart -17.6%; remote-reply.test -16.2%; send-remote-delivery.test -18.1%; stat-shadowing.test -4.9%; 3 small roots ~0%. teardown and launch-prompt-signals: the base side was OOM-killed at 7.6 GiB; the fix side completed clean at 7.28 GB and 7.14 GB. Seeded-defect check: an undefined variable is still reported the same way, and variables from marker/operational-input and parent-channel still resolve across files. tests/fm-pending-reply.test.sh passes (40 ok). I added no new test: a memory regression cannot be checked deterministically in CI, and a source-text check is not allowed. Remaining limit, not solved: the joint source-aware analysis still needs multi-GiB per root (fm-watch is about 4.9 GB on Linux). Other repeats are real directed sites (classify-lib through wake/parent-channel/afk-contract, timeout-lib). Removing them would lose cross-file context for other callers. Going further needs Main to choose an architecture
…e failure is from this PR. 1. Behavior portable parallel 1: tests/fm-lint.test.sh test_pinned_shellcheck_memory_limit failed with "the memory-bound pinned root was not named". This PR caused it. The per-root analysis command now runs `perl bin/fm-lint-cache.pl check ...` inside the per-root `ulimit -v`. Under the test's 8 MiB cap, Perl 5.38 (Ubuntu 24.04) prints "Out of memory!" and exits 1. fm_lint_classify_root mapped rc 1 to "findings", so the root was not named and not recorded as an abnormal end. I reproduced this in an ubuntu:24.04 container. Perl 5.40 and 5.42 crash with SIGSEGV instead, which is already classified as a named signal. Fix: bin/fm-lint.sh adds `^Out of memory!$` (a whole stderr line) to the existing memory-evidence regex, and the classifier comment is updated. Regression: tests/fm-lint.test.sh gains an `oom-perl` stub case in test_memory_evidence_outranks_findings_and_signal_reasons. It prints "Out of memory!" and exits 1, and it must classify as memory. It runs portably in unbounded mode, plus bounded mode on Linux. Verified: with the old fm-lint.sh the new case fails ("oom-perl was classified 'findings', expected memory"). With the fix, the full tests/fm-lint.test.sh passes on macOS (bounded tests skipped). It also passes on Linux (ubuntu:24.04, pinned ShellCheck 0.11.0 + actionlint): 44 ok, including the pinned memory-envelope test and modes 0 1. `bin/fm-lint.sh bin/fm-lint.sh tests/fm-lint.test.sh` passes. 2. Behavior portable serial 2 (tests/fm-calm-pi-extension.test.sh, "calm mode was not off by default") and 3. Behavior portable serial 4 (tests/fm-pi-branch-extension.test.sh, "Calm-off ToolExecutionComponent rendering differs from Pi stock"): not caused by this PR. This PR touches only lint files. The same two Pi-extension tests fail today on unrelated branches: CI runs 36928154699 (fm/fm-jev-mem-guard-macos), 36926345705 (fm/fm-jev-never-send-sections) and 36887810827 (fm/fm-teardown-leak-kill-identity). That points to external Pi drift. No code change for these
…ause it. No branch change; this needs a main-first repair. The failing assertion is tests/fm-dispatch-resolve.test.sh expect_withheld line 267, assert_equals '1' "$(grep -c . <<<"$err")", in the "a directory at the list path" case. In CI it got 2 lines. The test does not print the raw stderr, so the extra line's text is not in the log. Facts: (1) the PR does not change bin/fm-dispatch-resolve.sh, its test, tests/lib.sh, the workflow, or anything the behavior shard runs. Those files are the same as base a4d559e and main. (2) The same test, shard order and runner image (ubuntu24 20260927.320.1) passed on this PR's previous head (run 36933934075). It also passed on main a4d559e and on 4 other branches today. (3) Counterfactual: 3 full test runs plus 40 runs under 10x load passed in ubuntu:24.04 with bash 5.2.21. 2,400 aarch64 runs and 1,200 x86_64 runs of that exact case each printed exactly one line. The cause is an intermittent extra stderr line in main's test/tool path. Main-first prerequisite: on main, delete the incidental line-count assertion in expect_withheld. Keep the off-reason, "nothing sent", exit 0, empty stdout, no-curl, no-quota and no-value-printed checks. ci-1, ci-2 and ci-4 stay unapproved, with their own owners. cold-source-aware-memory-still-multigib: fixed one real cause of extra expansion. bin/fm-pending-reply-lib.sh had a `# shellcheck source=bin/fm-marker-lib.sh` directive on the _FM_PENDING_REPLY_LIB_DIR assignment (from main kunchenguid#5753, "double duty"). I confirmed with a small probe that ShellCheck then binds every later "undirected" `. "$_FM_PENDING_REPLY_LIB_DIR/..."` source to fm-marker-lib.sh. So each of 8 sites re-read fm-marker-lib.sh plus fm-operational-input.sh. An strace of ShellCheck 0.11.0 shows fm-marker-lib.sh opened 9x for bin/fm-watch.sh and 20x for tests/fm-pending-reply.test.sh. Fix: remove the bind directive. Add `# shellcheck source=/dev/null` to the 8 sites that should not be followed. Only comment lines change. Directed sites, cross-file analysis, selection, cache, CI parity, flags and caps are unchanged. Controlled A/B, same container, pinned ShellCheck 0.11.0 (linux aarch64), --norc --external-sources, all 17 roots whose closure has this library. Output and exit code are identical (clean). Peak RSS in KiB: watch 5587052->4852980 (-13.1%); pending-reply.test 5713888->4498528 (-21.3%); pending-reply-lib 1997132->1508624 (-24.5%); backlog-handoff -14.5%; send -14.5%; procevent-remote-reply -18.5%; remote-secondmate-control -24.1%; secondmate-report -24.6%; secondmate-restart -17.6%; remote-reply.test -16.2%; send-remote-delivery.test -18.1%; stat-shadowing.test -4.9%; 3 small roots ~0%. teardown and launch-prompt-signals: the base side was OOM-killed at 7.6 GiB; the fix side completed clean at 7.28 GB and 7.14 GB. Seeded-defect check: an undefined variable is still reported the same way, and variables from marker/operational-input and parent-channel still resolve across files. tests/fm-pending-reply.test.sh passes (40 ok). I added no new test: a memory regression cannot be checked deterministically in CI, and a source-text check is not allowed. Remaining limit, not solved: the joint source-aware analysis still needs multi-GiB per root (fm-watch is about 4.9 GB on Linux). Other repeats are real directed sites (classify-lib through wake/parent-channel/afk-contract, timeout-lib). Removing them would lose cross-file context for other callers. Going further needs Main to choose an architecture
@MrGTV-love MrGTV-love changed the title fix(bin): select lint roots by source closure and reuse identical ShellCheck results fix: avoid redundant lint analysis while preserving source checks Oct 6, 2026
@MrGTV-love
MrGTV-love merged commit 527dbaf into main Oct 6, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant