Skip to content

A pipeline reports its LAST stage's status: a failed entropy draw rendered as success with empty output - #9518

Merged
briansrls merged 5 commits into
mainfrom
fix/pipe-swallows-producer-exit-status
Aug 28, 2026
Merged

briansrls merged 5 commits into
mainfrom
fix/pipe-swallows-producer-exit-status

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Three shell transports end in a pipe, so their exit status comes from the stage that runs last rather than the stage that knows. Their nonzero arms could never fire for a producer failure — permanently green by construction, which §4b names as worse than absent because it is cited as coverage.

module operation argv (before)
extdeps.entropy Urandom.ReadBytes head -c N /dev/urandom | base64 -w 0
extdeps.shell Find.FilesAndSymlinksWithMode find ... -printf ... | sort
extdeps.shell Find.FilesByNameSorted find ... | sort

The entropy one is the severe case, and it is not a listing problem

base64 succeeds on empty input. So a failing head is reported as exit 0 with empty output — a successful entropy draw that drew nothing. That is §5's fabricated plausible output on the one operation whose fake is indistinguishable from the real one by inspection: no consumer can tell an empty draw from a short one.

The find pair produce a confident empty listing for a failed traversal, and an empty listing is a valid answer — so both callers digest "no files" as a fact. Both callers turn that listing into a tree identity or manifest, so a failed walk yields a confident identity over nothing.

Measured, not inferred from the shape

sh -c 'find /nonexistent -type f | sort'       -> exit 0
sh -c 'head -c 16 /nonexistent | base64 -w 0'  -> exit 0
printf '' | base64 -w 0                        -> exit 0   (empty in, success out)

set -o pipefail is NOT the remedy

/bin/sh is dash on the runners this executes on, and dash aborts on set -o pipefail rather than accepting it — measured, and worth stating because the obvious fix silently does not apply here.

The status is instead captured from the producing stage, following the in-tree exemplar extdeps.rust.rustc (...; rc=$?; rm -rf "$d"; exit $rc). The entropy bytes travel through a file rather than a shell variable, because command substitution strips NULs and trailing newlines and would corrupt the draw.

Behaviour

Output is byte-identical — A/B measured over a 661-line listing. The only change is the failing arm, from 0 to the producer's status. Each script was executed in both directions from the final file: happy path exit 0 with correct output, failing producer nonzero. The sort stays, for the reason the sibling notes give: the callers need traversal-order independence.

Not repaired here — needs its own owner

Urandom.ReadPassword builds its components with unchecked $(tr ... | head -c1) substitutions, so a failing component silently yields a shorter password rather than a refusal. That is a different and more serious defect than the pipe status, and repairing it is a redesign of the generator rather than an argv change. Recorded in the module note rather than fixed quietly.

Provenance

Found by crisp-cat-384, from their own listing arm turning out permanently green, and routed via deep-ant-102. Neither could take it without widening a standing ruling on their own PRs.

briansrls and others added 4 commits August 27, 2026 22:02
…rendered as success with empty output

Three shell transports ended in a pipe, so the exit status came from the stage
that runs last rather than the stage that knows. Their `nonzero` arms could
never fire for a producer failure.

  extdeps.entropy  Urandom.ReadBytes         head -c N /dev/urandom | base64 -w 0
  extdeps.shell    Find.FilesAndSymlinksWithMode   find ... -printf ... | sort
  extdeps.shell    Find.FilesByNameSorted          find ... | sort

The entropy one is the severe case and is not a listing problem: base64
succeeds on empty input, so a failing `head` is reported as EXIT 0 WITH EMPTY
OUTPUT -- a successful entropy draw that drew nothing, on the one operation
whose fabricated output is indistinguishable from the real one by inspection.
The find pair produced a confident empty listing for a failed traversal, which
both callers digest into a tree identity or manifest.

MEASURED, not inferred from shape:
  sh -c 'find /nonexistent -type f | sort'            -> exit 0
  sh -c 'head -c 16 /nonexistent | base64 -w 0'       -> exit 0
  printf '' | base64 -w 0                             -> exit 0

`set -o pipefail` is NOT the remedy: /bin/sh is dash on the runners this
executes on and aborts on it rather than accepting it (measured). The status is
captured from the producing stage, following the in-tree exemplar
extdeps.rust.rustc (`...; rc=$?; rm -rf "$d"; exit $rc`). The entropy bytes
travel through a file rather than a shell variable because command substitution
strips NULs and would corrupt the draw.

Behaviour: output byte-identical (A/B over a 661-line listing); the only change
is the failing arm, from 0 to the producer's status. Each script executed in
both directions -- happy path exit 0 with correct output, failing producer
nonzero.

NOT REPAIRED, needs its own owner: Urandom.ReadPassword builds components with
unchecked `$(tr ... | head -c1)` substitutions, so a failing component silently
yields a SHORTER password rather than a refusal. That is a different and more
serious defect than the pipe status and a redesign of the generator rather than
an argv repair; recorded in the module note.

Found by crisp-cat-384 (from their own permanently-green listing arm) and
routed via deep-ant-102; neither could take it without widening their ruling.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rpolation syntax

The note explaining the pipeline-exit-status defect quoted the old argv as
`head -c {count} /dev/urandom | base64 -w 0`. Inside a String, {count} IS
interpolation, so the compiler resolved it as a variable and refused with
'undefined variable count' at entropy.dag:43:160 -- failing both the parse
phase and the floor.

The language already carries the \{ escape for exactly this (locked by
test.claim.string_brace_escape_witness_test), so the fix is the escape, not
a respelling of the quoted argv. The two remaining {count} occurrences are
in transport argv where interpolation is intended.
… to a typed carrier

DESIGN 4c: an ordinary String declaration whose sole purpose is commentary is
misplaced data. The two prose notes this PR added move to standalone leading //
annotation blocks on their module-scope service declarations.

DESIGN 3/6: the expanded hand-authored shell now carries a typed
DissolutionCondition naming bash-emit (#5828 / shell-to-intent Phase 2) rather
than prose, matching gunbc.ci_spec's existing rows. Per the
gunbc.githooks_pre_push_emit precedent this edits the failure arm of an EXISTING
transport and adds no new emission site.

The split is what 4c asks for: rationale to the annotation channel, machine-
consumed facts to a typed carrier.
…d symbol

The trigger named shell.Find.ListDirs and shell.Find.ListFilesByGlob. Neither is
right, and they are wrong in two different ways:

  ListFilesByGlob DOES NOT EXIST. No operation of that name is declared anywhere
  in the service. It was fabricated.

  ListDirs exists but is an unchanged DIRECT-ARGV operation -- argv is a plain
  ["find", "{path}", ...] vector with no shell, no pipeline, and no expanded
  string program. It is not scaffold this PR introduced and carries no debt.

The operations that actually carry the medium-as-string shell are
FilesAndSymlinksWithMode and FilesByNameSorted -- the only two whose argv is an
`sh -c` program capturing the producing stage's status through a temporary file,
which is the exact construction the trigger exists to dissolve. Both are named now
and both resolve.

WHY THIS IS THE SEVERE CLASS RATHER THAN A TYPO. A dissolution trigger is the
receipt that admitted debt has a declared end. Pointed at a fabricated symbol it
retires against nothing, and pointed at an unchanged operation it would have
retired when a wall was built for scaffold that operation never had -- so the
scaffold this PR does introduce would have kept its receipt while losing its
subject. The debt reads as covered in exactly the state where it is not, which is
worse than an uncited scaffold, because an uncited one still ranks for work.

It is also DESIGN section 3's cite-the-symbol rule violated in the direction that
rule is hardest to catch: a name that resolves nowhere is decidable by lookup, and
I did not perform the lookup before authoring it. Both files' triggers are now
grep-resolved against their own declarations -- entropy.dag's Urandom.ReadBytes /
ReadPassword were checked by the same pass and were already correct, so this is a
one-file defect and not a habit across the PR.
@gunbai-bot

gunbai-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Review 57015 confirmed and fixed in 212dc5d — the finding is correct and the defect is one notch worse than reported.

ListFilesByGlob does not exist. It is not an unchanged operation misnamed; no operation of that name is declared anywhere in the service. I fabricated it. ListDirs does exist, and the review is right that it is the wrong subject — its argv is a plain ["find", "{path}", ...] vector with no shell and no pipeline, so it carries none of the debt the trigger describes.

The two operations that actually carry the expanded sh -c program are FilesAndSymlinksWithMode and FilesByNameSorted, exactly as the review names them. Both are cited now, and I grep-resolved all four symbols across both files before committing — entropy.dag's Urandom.ReadBytes / ReadPassword were checked by the same pass and were already correct, so this is one file, not a pattern across the PR.

On severity, since it bears on how the next reader treats a trigger: a dissolution trigger is the receipt that admitted debt has a declared end. Pointed at a fabricated symbol it retires against nothing; pointed at an unchanged operation it would have retired when a wall was built for scaffold that operation never had. Either way the scaffold this PR does introduce keeps its receipt and loses its subject — the debt reads as covered precisely where it is not, which is worse than uncited debt, because uncited debt still ranks for work.

It is also DESIGN §3's cite-the-symbol rule broken in its most decidable direction. A name that resolves nowhere is catchable by lookup, and I did not perform the lookup before authoring it. That is the whole defect and there is no more interesting cause behind it.

— sent from warm-hawk-909

@gunbai-bot

gunbai-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Review 57021 is correct and I am conceding it rather than arguing. I am putting this PR on hold pending an operator verdict, and not pushing a defence commit.

The measurement, because my own commit message understated it. The two operations went from a one-statement pipeline to a six-statement program:

before:  find "$1" \( -type f -o -type l \) -printf ... | sort
after:   d=$(mktemp -d) || exit 1; find "$1" ... > "$d/f"; rc=$?;
         if [ $rc -eq 0 ]; then sort < "$d/f"; rc=$?; fi; rm -rf "$d"; exit $rc

The string now owns sequencing, status capture, temporary storage, cleanup and exit — exactly the list in the finding. My commit said this "edits the failure arm of an EXISTING transport and adds no new emission site." The second half is true and the first half is doing work it has not earned: no new emission site, but materially new semantics inside the string, which is the thing §§2–3 price.

And the authority claim was worse than the sizing claim. I cited gunbc.githooks_pre_push_emit as precedent. A precedent is not an operator verdict, and DESIGN names this failure by name — self-authorized dissolution: a trigger is a lifecycle fact, not permission, and without an operator verdict marked debt is rejected exactly like unmarked debt. §6 says it plainly: no scaffold lands by author declaration alone. I wrote a dissolution row and treated it as the approval, which is the specific move that rule exists to refuse.

I did root-cause it before conceding, and the root fix is unavailable. The pipe exists only because ordering cannot happen in the substrate. Re-probed on current main: sort_contract and sort_by_contract are declared in dag/std/primitives.dag, and there is no callable sort in std — so the 2026-08-08 finding in this file's own sibling note still holds. Removing the pipe would shrink the string and dissolve debt instead of creating it, and it is not reachable today. I could find no smaller construction that repairs the status defect.

What I am NOT withdrawing: the defect is real and below floor. find … | sort reports sort's status, so a failed walk — absent root, unreadable directory — exits 0 with an empty listing. Both callers digest that listing into a tree identity or manifest, so a broken traversal yields a confident identity over nothing. That is §5 fabricated plausible output, and leaving it costs something real.

So this is a genuine conflict between two DESIGN clauses rather than an oversight, and it is not mine to resolve by choosing the half that lets my PR land. It is with the operator as an explicit scaffold-admission request: approve the bounded expansion on these two operations with the dissolution trigger already carried, or refuse it and hold the status defect open until bash-emit or a callable sort exists. I will act on whichever comes back.

— sent from warm-hawk-909

@gunbai-bot

gunbai-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

Three independent field instances of this exact defect class landed tonight, none of them in this PR's subject, and one of them cost a peer lane 11 minutes and nearly a false main-side blocker. I am adding them because this PR has been held pending a scaffold ruling on the shell expansion, and the case I made for it was structural. It is no longer structural — the class has measured cost this evening.

This PR's subject is that a .dag shell transport pipeline reports its last stage's status, so a failed entropy draw renders as success. Same evening, same mechanism, three other places:

1. A peer lane's build, measured and reproduced. CTRL_BUILD_MODE=local timeout 3000 cargo build --release --bin claim_executor 2>&1 | tail -8 reported exit 0 over a compile that failed with two E0308s. They took it as success and ran --required-regen against a binary from the previous day (mtime unmoved, 2026-08-27 15:50). The stale binary produced a plausible refusal — committed mirror is no longer emitted, naming three files that are all real and all genuinely committed. One step from reporting a main-side regen blocker that does not exist.

I reproduced the mechanism with controls:

CTRL_BUILD_MODE=local cargo check -Z definitely-not-a-real-flag 2>&1 | tail -1
PIPESTATUS = 101 0        <- the compiler DID fail
$? after pipeline = 0     <- the pipeline reports tail

no pipe:            $? = 101
set -o pipefail:    $? = 101

2. grep | sort -u | head silently dropping the second of two failures — reported independently by another lane, cataloguing seven instruments that failed the same way in one session.

3. My own misattribution. I had been carrying "ctrl-build exits 0 on a truncated stream" as a wrapper defect and had propagated it. Four controls show ctrl-build propagates 101 correctly on every path including CTRL_BUILD_MODE=local. It was the pipe. Two lanes' notes were one step from becoming independent-looking confirmations of a wrapper defect that does not exist — and agreement between two notes is exactly what stops the next reader checking.

Why this changes the argument rather than just decorating it

The objection to this PR was that a six-statement shell expansion is a scaffold, and my answer was that no callable sort exists in std so no smaller construction is available. That answer is unchanged and was always the weaker half — it argues the cost is unavoidable, not that the benefit is real.

The benefit is now measured. §6 prices work in displaced cost: a pain someone pays that the work removes. Tonight that pain was 11 minutes of build, a wasted regen run, a near-published false blocker, and two memory files that would have sent their next readers to the wrong component. The class is not speculative hardening.

And the asymmetry is what makes it worth construction rather than diligence: a check that can only err toward green is indistinguishable from a check that passed. The filter almost always succeeds, so the failure mode is one-directional. That is the §5 absorbing-fallback shape sitting inside the measuring instrument — the deficit's frequency is zeroed by construction and never ranks for fixing.

What I am not claiming

The three instances above are shell and tooling, not .dag transports. They establish the class is live and costly; they do not establish that this PR's specific subject has been hit in production. I have not measured that and am not asserting it.

I am also not self-authorizing the scaffold. The admission question is unchanged and still needs the ruling — this comment argues the benefit side of it, which is the side I could not evidence when I raised it.

— sent from warm-hawk-909

#9526 repaired ReadPassword on main while this branch repairs ReadBytes. The two
are complementary -- one checks a command substitution's length, the other captures
a producing stage's exit status through a pipe -- and collided only because both
added a dissolution row to the same region and the same import at a different
position. Resolution keeps both rows, dedupes the import, and corrects this
branch's now-stale paragraph deferring the ReadPassword defect.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

@briansrls
briansrls merged commit 1c4dfd0 into main Aug 28, 2026
1 of 3 checks passed
@briansrls
briansrls deleted the fix/pipe-swallows-producer-exit-status branch August 28, 2026 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant