Repository navigation
Conversation
…nded Attempt slice4-census-47eac00e. Every earlier receipt recorded cgroup placement, namespaces and capabilities as NOT OBSERVED for the same reason: the run was looked at only after it ended. A post-mortem cannot read a capability set from a process that is gone, and cannot tell a cleaned-up cgroup from one that never existed. CONFIRMED BY IDENTITY: the process launched as the jailer and the process running Firecracker were pid 1649, parented to init. That is the in-place exec the surface model was corrected to describe, established by measurement rather than by reading the jailer's source. PRIVILEGE IS FAR STRONGER THAN THE PREVIOUS RECEIPT COULD CLAIM. Zero effective, permitted, inheritable and ambient capabilities; no_new_privs set; seccomp in filter mode. The bounding set is undropped and that is recorded, since it is the one axis a reader would otherwise assume from the others. NAMESPACES BY INODE, and the shared answers are evidence too. Only the mount namespace differs, which is what an invocation without --new-pid-ns and without --netns should produce -- so the reading establishes both that the pivot happened and that no unrequested namespace was quietly introduced. THE FINDING: /proc/1649/cgroup read 0::/ while the guest was alive. The VMM ran in the ROOT CGROUP. Despite --cgroup-version 2 the attempt was placed in no cgroup of its own, so nothing bounded its cpu or memory -- the jailer's containment on this run was chroot, mount namespace and privilege drop only. The previous attempt saw an absent cgroup directory afterwards and honestly recorded NotObserved; the same absence is produced by successful cleanup and by placement never happening, and those differ by whether the machine had resource containment at all. The likely mechanism is stated as unverified -- the flag selects a version while limits request a cgroup -- and the discriminating run is owed. THE WATCHER WAS SEEN: pid 1651, busybox, uid 0, parented to init. That is why the fourth receipt's cleanup claim had to be narrowed, and why the surface now declares AttemptDeadlineWatcher. AND THE ROSTER DOES NOT YET MATCH THE MACHINE. Two interactive busybox shells were running that no process class names -- deliberate, this image's console loop, and undeclared, which is exactly the state the closed-world invariant forbids. They must be declared or removed before any run claims an exact-population match. The snapshot vocabulary keeps raw observation apart from classification: an earlier shape let an observation carry role_claimed, which invites the producer to label a process with the role it was looking for, so a firecracker running as root would be recorded as the attempt role and compared against the declaration it was meant to be tested against. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR
The observation split was made in runner_runtime_snapshot without removing what it superseded, so two observation types stood for one concept -- and the weaker one was still writable. ObservedProcess carried role_claimed, which invites the producer to label a process with the role it was LOOKING FOR: a firecracker running as root would be recorded as the attempt-execution role and then compared against the very declaration it was meant to be tested against. A role must be derived from executable identity, invocation, attempt and phase, never asserted by the thing doing the looking. It had no consumers, which is exactly why it survived a review pass -- nothing broke, and a fork with no consumers is still a fork with a fail-open arm someone can reach for. RawProcessObservation carries only what /proc said; this leaves one route to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR
The live census had found the VMM in the root cgroup -- a dedicated guest kernel with no host resource boundary at all. This closes it. The supervisor owns placement rather than asking the jailer for it. --cgroup-version 2 selects an implementation; it does not request a cgroup, which is why the earlier attempt moved into nothing. The supervisor now creates a leaf, writes the limits, and puts the launching identity into it BEFORE exec -- and since the jailer execs in place, the identity that ends up running Firecracker is already inside the bound. No window exists in which the VMM runs unbounded. Established from both directions while the guest was alive: /proc/1694/cgroup named 0::/gunbc/attempt-slice5b-cgroup-77e659df, and that cgroup's procs listed 1694. memory.current read 88 MiB against a 1 GiB max -- the kernel charging the attempt's pages to the attempt's cgroup, which is what separates a limit that applies from one merely written to a file. The deadline watcher is inside the same bound, which is correct rather than incidental since it is attempt-scoped. At retirement: procs empty, THEN removed, read in that order. An empty cgroup never removed is a leak; a removed cgroup that still held a process could not have been removed at all. THE BIDIRECTIONAL CHECK EARNED ITS KEEP ON FIRST USE. The first attempt wrote `( echo $$ > cgroup.procs; exec ... )`, and $$ in a POSIX subshell is the INVOKING shell's pid -- so it wrote 1, moved init into the attempt cgroup, and left the VMM in root. A forward-only read would have reported "not placed" and nothing more; the reverse read listed pid 1 in the leaf, which is not a plausible member and named the bug at once. It also left init in that cgroup so retirement could neither empty nor remove it -- cleared by the next boot, but a leak on a long-lived host. The fix is sh -c, whose new process has its own $$, preserved through exec into timeout, jailer and firecracker. STILL OPEN AND NOW VISIBLE: cpu.max reads back empty on a leaf whose cgroup.controllers includes cpu, while memory.max and pids.max apply. So it is neither a delegation failure nor a missing controller, and no cause is claimed. It surfaced only because the controller state is now printed -- the previous run wrote cpu.max, read nothing, and said nothing. Enforcement is not claimed. Nothing has tried to exceed memory.max, so the bound is observed to ACCOUNT rather than observed to STOP. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR
…carried The guest mounts its squashfs root read-only, runs /sbin/init, and brings systemd 255.4 to system mode on arm64. The host reaches it over an attempt-scoped point-to-point TAP: three of three pings, ARP resolved to the exact configured guest MAC, on a /30 with no room for a third party. The tap is absent after retirement, and cgroup placement stayed bidirectional across a run that now includes networking. A CLAIM CARRIED SINCE THE KVM PROBE IS WITHDRAWN. Earlier receipts listed guest userspace as unobserved, which was accurate -- and two went further, saying the guest was "killed at the deadline while still early in boot", and one said it "did NOT complete its boot". None of that was measured. It was inferred from rc=137 plus a head -25 of the guest log, and the tail was never read. The full log shows the guest reaching systemd, so rc=137 was the deadline firing on a HEALTHY guest rather than a stalled one -- different facts about the same exit code. The correction is narrow: those receipts are not rewritten to claim success their attempts did not observe. What is removed is the unmeasured clause about what the guest was doing when it was killed. THREE DEFECTS, ALL SELF-INFLICTED. Two boots failed with "Boot option loading failed" because a 302 MiB image was copied OVER the exact file the controller was serving over NFS. The on-disk digest matched what was built, so the file was intact and the controller's cached view was not; a fresh name fixed it immediately. This lane already had the rule -- machine_intake_boot_image_fetch publishes by atomic rename precisely so a reader never observes a path mid-write -- and a convenience cp in a build script went around it. Content-addressed naming would have made it structurally impossible rather than merely avoidable. $TAP and $GUEST_MAC were defined BELOW the heredoc that writes vm.json, so both expanded empty and Firecracker refused a blank guest_mac by name and column. Loud and exact, so it cost one run. The tap name was cut to the 15-character interface limit and ended in a dash -- functional, and indistinguishable from corruption in a diagnostic. It is now composed short rather than truncated. Recorded as still open: no guest service listens on anything, since ICMP is answered by the guest kernel; the guest has an unfiltered route to the host, which is not the default-deny its isolation profile requires; and the JIT-registered runner is the next thing the guest has to carry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR
The goal was guest egress -- a runner cannot register with GitHub over a /30 to its host -- by bridging the attempt tap and the host NIC onto the LAN. THE GUEST DID NOT OBTAIN EGRESS. This is its own receipt rather than a footnote on a later success, so a reader asking "does the guest have egress" finds no, and so the findings below are attributed to the run that produced them. THE NIC CARRIES A LINK, observed for the first time. igb eth0 negotiated 1000 Mbps full duplex on the port whose MAC the firmware used for PXE. runner_host_hardware_observation listed "that either interface carries traffic" among the things a PCI identity cannot establish, and every boot since had shown only binding. This is layer 1 and 2 on the fleet-facing port -- not egress, and the half of it hardware owns. THE CPU BOUND HAS A CAUSE NOW, where the previous receipt said none was claimed. Writing cpu.max fails with a located error rather than silently doing nothing: EPERM creating the interface file, in a leaf whose ROOT subtree_control reads "cpu memory pids" -- so the controller is delegated and the file still cannot be created, while memory.max and pids.max in that same leaf apply. That is a narrowing, not a diagnosis: an errno at a path instead of an empty read. AND THE BRIDGE WAS CORRECT ALL ALONG. An earlier run reported an empty bridge because `brctl show br0` takes no argument in busybox, so passing one printed nothing and nothing read as no members. Reading /sys/class/net/br0/brif shows both interfaces enslaved. The recurring lesson: an instrument that cannot express the question returns an answer that looks like a measurement. WHAT BLOCKED EGRESS, at the grain observed. busybox udhcpc applies nothing without a -s script and exited 0 with no address, which is indistinguishable from a working client. With a script supplied and present in the image, udhcpc STILL never invoked it -- the script logs unconditionally on entry and its log is empty -- while still exiting 0 under -n, the flag that should make a lease-less run exit non-zero. Those do not reconcile and no explanation is offered. Nothing in the run distinguishes "the segment offered no lease" from "a lease was obtained and discarded", which is exactly what the lease log was added to settle and failed to. The policy debt is stated rather than absorbed: bridging grants unfiltered segment access, which DefaultDenyEgress forbids. It is admissible only for the qualification profile with one pinned image and is a prerequisite for any real job. It was chosen because this image has no netfilter userspace at all and CONFIG_MODULES=n drops anything defconfig marks modular -- a capability gap, not a preference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR
LINK NEGOTIATION IS NOT TRAFFIC. The receipt read the igb link-up line as closing the hardware lane's open item "that either interface carries traffic". It does not: it establishes the driver bound and the PHY negotiated 1000 Mbps full duplex, and says nothing about a frame being carried. Closing that needs packet counters moving or a capture on this boot. Carrier is not a data plane, and the item stays open. THE brctl EXPLANATION WAS INVENTED TO FIT THE SYMPTOM AND IS WITHDRAWN. The receipt claimed `brctl show br0` takes no argument in busybox; busybox 1.36.1 documents `show [BRIDGE]...` and iterates what it is given, so the invocation was valid. Why that rendering came back empty while sysfs showed both members is unresolved. Replacing a bad measurement with a confident wrong cause is worse than leaving it open, because the correction reads as understanding and stops anyone looking. cpu.max IS DIAGNOSED RATHER THAN NARROWED, from the built config: CONFIG_CGROUP_SCHED=y, CONFIG_FAIR_GROUP_SCHED=y, and # CONFIG_CFS_BANDWIDTH is not set. Bandwidth provisioning is what creates cpu.max on a non-root cgroup and defaults off, so the file does not exist, the redirect tried to CREATE a file inside cgroupfs, and the kernel answered EPERM. The controller appearing in cgroup.controllers is consistent: weight control is present, bandwidth control is not. The repair is a kernel option -- and the acceptance is NOT "cpu.max reads back", since a present interface is not an enforced boundary. That owes a controlled load and cpu.stat throttling counters moving. The DHCP reading is also narrowed twice. "busybox applies nothing without -s" is too strong: it carries a compiled default script path and -s replaces it. And an empty callback log establishes only that the script body never reached its first statement -- not that udhcpc never attempted the callback, since udhcpc does not propagate a callback's result, so an exec failing before the log looks identical. The candidate causes are enumerated and none is excluded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR
Contributor
Author
|
The three receipts and three artifact logs unique to this branch have been cherry-picked onto current main in #10818, because Once #10818 lands, this PR's remaining content is on main and it can be closed. Not closing it yet -- #10818 has not merged, and closing the source before the destination lands would leave the citation dangling with nothing tracking it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A sibling slice, stacked on #10677, owning only the runtime-census subject: snapshot vocabulary and one receipt from a census taken while a guest was alive.
Every earlier receipt in this lane recorded cgroup placement, namespaces and capabilities as not observed, always for the same reason — the run was looked at only after it ended. A post-mortem cannot read a capability set from a process that is gone, and cannot tell a cleaned-up cgroup from one that never existed.
The finding: nothing was ever bounded
/proc/1649/cgroupread0::/while the guest was running. The VMM ran in the root cgroup. Despite--cgroup-version 2the attempt was placed in no cgroup of its own, so nothing bounded its CPU or memory — the jailer's containment on this run was chroot, mount namespace and privilege drop only.The previous attempt saw an absent cgroup directory afterwards and honestly recorded
NotObserved. The same absence is produced by successful cleanup and by placement never happening, and those differ by whether the machine had resource containment at all.Likely mechanism, stated as unverified: the flag selects a version while
--cgroup key=valuelimits request a cgroup. The discriminating run is owed.Confirmed by identity
The process launched as the jailer and the process running Firecracker were pid 1649, parented to init — the in-place exec the surface model was corrected to describe, established by measurement rather than by reading the jailer's source.
Privilege, all seven axes
CapEff/CapPrm/CapInh/CapAmbCapBnd000001ffffffffff— not droppedNoNewPrivsSeccompFar stronger than the previous receipt's "nonzero uid". The undropped bounding set is recorded because it's the one axis a reader would otherwise assume from the others.
Namespaces by inode — and the shared answers are evidence too
Only
mntdiffers;pid,net,user,ipc,uts,cgroupare all shared. That's exactly what an invocation without--new-pid-nsand without--netnsshould produce, so the reading establishes both that the pivot happened and that no unrequested namespace was quietly introduced. A booleannamespacedcould have said neither.The watcher, and a roster that doesn't match the machine
The deadline watcher was seen: pid 1651, busybox, uid 0. That's why the fourth receipt's cleanup claim had to be narrowed — a post-run census seeing busybox processes cannot tell console helpers from a surviving watcher by name.
And the census reports its own uncomfortable result: two interactive busybox shells were running that no process class names. Deliberate (this image's console loop) and undeclared — exactly the state the closed-world invariant forbids. They must be declared or removed before any run claims an exact-population match.
Vocabulary shape
Raw observation is kept apart from classification. An earlier shape let an observation carry
role_claimed, which invites the producer to label a process with the role it was looking for — so a Firecracker running as root would be recorded as the attempt-execution role and compared against the declaration it was supposed to be tested against.The snapshot carries its own completeness:
/procis not atomic, so a snapshot that raced the population it was enumerating, or that could not read a member, refuses rather than reporting agreement it did not observe.Open
cgroup.procsmembership read from the cgroup side (placement is bidirectional or it isn't established)switch_root--cgrouplimits produces placement — the run this finding owes🤖 Generated with Claude Code
https://claude.ai/code/session_01D7PLjw5FMYdZ2GWn3FeMpR