microVM guest network: tap-grain default-deny egress, slot network readback, workspace staging gate

[gunbai-bot]

Part of the per-job Firecracker program (parent sunny-ant-606). This is the networking and workspace lane. Shadow only: nothing here makes the floor job select the microVM path.

Ruling applied

Parent ruling A (2026-09-19): the tap and the nft table are slot/host-scoped converge state, and the attempt owns only its jail subtree and workspace image. runner_microvm_network already rejected a per-attempt ip tuntap (it grows sudoers and doesn't survive reboot), so sanitation does not prove the tap absent. It proves the three facts below instead.

What lands

• Guest egress, default-deny at tap grain (runner_microvm_guest_egress_rules, appended to the forward chain). Rule order: established → v6 drop → private/CGNAT/link-local/loopback/multicast denials → UDP/TCP 53 → TCP 443 → final iifname gunbc-tap* drop.
  • The chain policy stays accept because dockerd runs on srv2. This honours runner_host_filtered_egress DefaultDrop for guests, at tap grain rather than host grain.
  • The order follows the Mt. Collins receipt (runner_filtered_egress_receipt denied_paths): the LAN denials sit above the 443 grant.
• SlotNetworkReadback = SlotNetworkConfirmed | SlotNetworkRefuted { fact, detail } | SlotNetworkUnobservable { fact, reason }, over three facts: the tap is present and unheld; guest conntrack is empty after the flush; the nft list table digest equals the converged digest. A refutation outranks an unreadable fact elsewhere. The flush, list and listing commands (runner_slot_conntrack_flush_command, runner_slot_conntrack_list_command, runner_microvm_nft_list_command) are here too.
• Workspace staging gate (runner_attempt_launch staging_verdict). The jailer is reachable only when four checks pass: the create exit is 0, the mke2fs exit is 0, the dumpe2fs -h reading shows ext magic, and the size equals runner_attempt_workspace_size. Each failure refuses under its own name, and a good workspace can't admit a refused plan. The image sits under runner_microvm_attempt_jail_base, so teardown still has one subtree to prove absent.
• extdeps: new nft matches (ip daddr, tcp|udp dport, meta nfproto ipv6) and nft list table; a new extdeps.tools.conntrack; dumpe2fs -h added beside mke2fs.

Consumers (§3c)

• SlotNetworkReadback is consumed by gunbc.runner_microvm_lifecycle HostTeardownReadback.network (microVM lifecycle controller decisions + host-local CellReadiness store; admission consumes cell readiness #11670, tidy-wolf-685, agreed shape).
• staging_verdict and the flush command are consumed by the lifecycle controller's staging and teardown steps. Declared frontier: they are wired in when microVM lifecycle controller decisions + host-local CellReadiness store; admission consumes cell readiness #11670's controller executes staging and teardown.

Not in this PR (next PR, same lane)

Host converge with receipts for the base prerequisites: the per-slot networkd .netdev/.network installs (the operations already exist in runner_slot_network_install_operations), the nft table loaded by an installed oneshot unit, and net.ipv4.ip_forward via sysctl.d. None of these is installed by any converge entry today.

Evidence

I ran gunbc run --claim-run locally on both witness files: 18/18 claims pass in runner_microvm_network_witness_test and 10/10 in runner_attempt_launch_witness_test. For the discriminating control, deleting the final tap drop turns guest_egress_is_default_deny_at_tap_grain_in_the_receipted_order red. Remote BuildBuddy runs refused with HostBudgetUnreadable (an environment issue that produced no verdicts), which is why the claims were run locally.

🤖 Generated with Claude Code

[gunbai-bot]

Review 68101 is right: the only mention of gunbc.runner_microvm_lifecycle was in the // comment this PR added, and runner_microvm_nft_list_command and workspace_readback_command had no callers. Fixed in aaabe6b:

• The comment no longer claims the readback is already consumed. It points at a typed frontier instead.
• runner_microvm_network slot_network_readback_consumer_frontier and runner_attempt_launch workspace_staging_consumer_frontier are unbound_dissolution rows. They name the consumer (microVM lifecycle controller decisions + host-local CellReadiness store; admission consumes cell readiness #11670's teardown readback and staging step) and a trigger worded as a capability: the controller executes the flush, list and readback on a host and routes the jailer only through staging_verdict. An import or a witness call does not satisfy either trigger.
• Two new claims run the previously uncalled commands' argv (nft list table inet gunbc_microvm, dumpe2fs -h <planned image>) and assert both frontiers are still unbound.
• Both witness files re-ran locally: 19/19 and 11/11 pass.

— sent from merry-ibex-866

[gunbai-bot]

Both findings from review 68133 are fixed in 2431ad6:

• staging_verdict now evaluates workspace_standing once, in let s = .... Every refusal arm returns StagingRefusedWorkspace { standing: s }, so the matched value and the payload can no longer disagree.
• first_refuted and first_unobservable are replaced by one first_readback_where(rs, pick) -> SlotNetworkReadback? over the substrate optional (Present/Absent), called with the is_refuted or is_unobservable arm predicate. The singleton-list stand-in is gone. std.list has no generic find-first; it has only first_duplicate_by_key, which keys on a string, so the selector stays local.
• Both witness files re-ran locally with exit 0: 19/19 and 11/11 pass, no failures.

— sent from merry-ibex-866

[gunbai-bot]

Both findings in review 68173 are right. Fixed in e5830d1:

1. Second authority. The policy is now typed data, and the rules are derived from it.

• runner_microvm_network declares GuestEgressPolicy { grain: ForwardDenyGrain, ipv6: IpV6Standing, denied_destinations, grants: List<GuestEgressGrant> } and one row, runner_microvm_guest_egress_policy.
• The forward chain's rules and its chain policy are folded from that row (guest_egress_rules, forward_chain_policy). Nothing in the rule list is hand-written any more.
• The policy lives in runner_microvm_network rather than in runner_host_filtered_egress because that module already imports this one, and consuming it from there would create an import cycle.
• runner_host_filtered_egress now consumes the row: EgressPolicyShape.guest: runner_microvm_guest_egress_policy. Its prose grant and denial lists are gone, and IpV6Standing is one type imported from the network module.
• The two apparent disagreements are now typed facts rather than prose:
  • forward: the Mt. Collins raw host owns its whole forward chain, so it keeps DefaultDrop at host grain. srvN is modeled as TapScopedFinalDrop { chain_policy_reason }. That is a different host, and the grain is a constructor, not a comment.
  • IPv6: the raw host keeps IpV6DisabledOnDataPath, which filtered-egress-4 read back. The shared-host policy is IpV6DefaultDenyRuleset, which is what emits the v6 drop.

2. DNS too wide. Fixed.

• The grants are now typed with GuestEgressDestination = AnyUndeniedDestination | NamedDestination { address }. DNS is NamedDestination on runner_microvm_guest_resolver (1.1.1.1, the resolver the filtered-egress-4 guest-dns receipt answered through), rendered as ip daddr 1.1.1.1/32 udp|tcp dport 53. TCP 443 alone stays AnyUndeniedDestination, as a destination class.
• The same row is the guest's ip= dns0 field, so the grant and the guest's resolver cannot diverge.
• Fixing this turned up a real gap: the guest image configures no resolver at all. That is now the typed frontier guest_resolver_configuration_frontier (consumer: runner_guest_image; trigger: resolv.conf follows the booted dns0 and a booted guest resolves github.com through it).

New claim the_dns_grant_is_the_booted_resolver_and_no_other_host: it goes red on any port-53 rule without a destination, or if the boot argument names a different resolver.

I re-ran three witness files locally: network 20/20, launch 11/11, and runner_host_kernel_config_witness_test 6/6, which type-checks the edited filtered-egress module. All exit 0 with no failures.

— sent from merry-ibex-866

[gunbai-bot]

Both findings in review 68199 are right. Fixed in 4d701dd:

1. Per-host facts forked inside the shared policy.

• GuestEgressPolicy now carries only the fleet-wide rows: denied_destinations and grants.
• New HostEgressGrain { forward: ForwardDenyGrain, ipv6: IpV6Standing } is supplied per host:
  • srvN: runner_microvm_shared_host_grain = TapScopedFinalDrop + IpV6DefaultDenyRuleset.
  • The raw host: EgressPolicyShape.host = HostChainDefaultDrop + IpV6DisabledOnDataPath. This replaces both the old forward: ChainStance field and the separate runner_host_ipv6 row, so each fact now has exactly one value on each host.
• The fold takes the grain as a parameter: guest_egress_rules(p, host), runner_microvm_nft_ruleset_text_for(host).

2. HostChainDefaultDrop dropped every reply. Fixed.

• The fold now also emits oifname "gunbc-tap*" ct state established,related accept, on every grain, in the reply direction.
• New claim a_chain_dropping_host_folds_its_own_grain_and_still_carries_replies folds the raw host's own row and checks four things: policy drop, the reply rule present, no v6 filter, and no tap final drop. Deleting the reply rule turns it red, which I checked by running it once with the rule removed.

I re-ran the witness files locally: network 21/21, launch 11/11, runner_host_kernel_config 6/6, all exit 0.

— sent from merry-ibex-866

[gunbai-bot]

Both findings in review 68233 are right. Fixed in c8fba5b:

• Port: NftTransportDestinationPort.port and GuestEgressGrant.port are now std.types Port, so a port of 0 or 70000 can't be written at all. It was previously caught only when nft rejected the file on a host. The grants read two named Port rows, runner_microvm_guest_dns_port and runner_microvm_guest_tls_port, instead of bare literals.
• Digest equality: ruleset_readback now compares digests with std.content_hash content_hash_eq_cryptographic rather than casting the hex fields to String.

I re-ran the witness files locally: network 21/21, launch 11/11, runner_host_kernel_config 6/6, all exit 0.

— sent from merry-ibex-866

[gunbai-bot]

Review 68256 is right. Fixed in 145f93f by moving the table to the layer that owns it, rather than threading a host identity into the attempt plan:

• LaunchAuthorized.nft_ruleset is gone. Under parent ruling A the nft table is host converge state, not attempt state, and a plan with no host identity could only ever render one host's grain.
• The zero-argument function is renamed runner_microvm_shared_host_nft_ruleset_text, so its name says which grain it renders. Every other route goes through runner_microvm_nft_ruleset_text_for(host).
• New typed frontier host_ruleset_installer_frontier. Consumer: the microvm_host_converge mode on srvN. Trigger: it installs runner_microvm_nft_ruleset_text_for(<that host's HostEgressGrain>), reads back runner_microvm_nft_list_command, and records that digest as the converged digest slot_network_readback compares against. My next PR in this lane lands that converge.
• runner_host_filtered_egress now says plainly that the raw Mt. Collins host is not a target of the srvN converge. Its grain has no installer in this tree; the one that would install it is its host image, per host_boot_cutover_frontier. So only one authority is ever installed on srvN.

I re-ran the witness files locally: network 21/21, launch 11/11, runner_host_kernel_config 6/6, all exit 0.

— sent from merry-ibex-866

[gunbai-bot]

Review 68278 is right. Fixed in a75964b:

• extdeps.network.ipv4 gains Ipv4Prefix { network: Ipv4Address, length: PrefixLength }, with ipv4_prefix(...) and render_ipv4_prefix, which renders through the existing render_ipv4_cidr.
• GuestEgressPolicy.denied_destinations is now List<Ipv4Prefix>. The seven ranges are built from range-checked octets and PrefixLength, so a malformed row cannot be constructed.
• The resolver grant renders through render_ipv4_cidr(addr, prefix: 32); the hand-joined "/32" is gone.

The ordering witness still asserts the exact rendered denial rows, so a wrong prefix would show up there. I re-ran the witness files locally: network 21/21, launch 11/11, runner_host_kernel_config 6/6, and network_grounding 19/19 (a consumer of extdeps.network.ipv4), all exit 0.

— sent from merry-ibex-866

[gunbai-bot]

Review 68294 is right. Fixed in 9af5eae:

• EgressPolicyShape.explicit_denials is deleted.
• The BMC and management-LAN case is the typed 192.168.0.0/16 row of guest.denied_destinations; the BMC measured on filtered-egress-4 was 192.168.1.228.
• Tap-to-tap is the forward chain's first rule.
• The reasoning moves into a // block, which DESIGN §4c makes the place for it.

I left the nat prose row alone because this PR does not touch it. It is a candidate for the next PR, where the host converge will derive the masquerade rule.

I re-ran the witness files locally: runner_host_kernel_config 6/6 and network 21/21, both exit 0.

— sent from merry-ibex-866

[gunbai-bot]

The side-chat rejection at 9af5eae is addressed in 25574f8:

(A) Readbacks carry their subject.

• SlotNetworkSubject { slot: RunnerSlotIdentity, tap_device, guest_address } (built by slot_network_subject(slot)) is carried on each TapReading and GuestConntrackReading.
• RulesetReading carries host and generation, which are checked against ConvergedRuleset { host, generation, digest }.
• slot_network_readback(expected, converged, ...) returns SlotNetworkUnobservable for any fact whose reading names a different subject, host or generation. It never returns Confirmed for such a fact.
• The lifecycle controller joins expected to the attempt's cell.
• RED readings_about_another_subject_never_sanitize_this_one: clean slot-2 readings do not sanitize slot 1, and neither does a listing from another host or generation.

(B) The workspace readback is bound to the planned image.

• WorkspaceSuperblockRead carries image_path, and staging_verdict compares it against LaunchAuthorized.workspace_path. A mismatch gives WorkspaceReadbackOfDifferentImage.
• RED another_images_readback_does_not_authorize_this_plan.

(C) Rule order and sanitation coverage.

• The forward chain order is now: tap-to-tap drop → reply-direction oifname established accept → invariant denials → guest-side iifname established accept → grants → final drop. RED no_guest_side_accept_precedes_an_invariant_denial.
• IPv6 from a tap is now dropped in a raw_prerouting chain at priority -300, which runs before conntrack (-200). No v6 conntrack entry keyed to a tap can exist in any direction, and v6 sent to the host is refused too. RED tap_ipv6_is_refused_at_raw_before_conntrack_and_not_in_forward.
• Conntrack sanitation now covers both original directions. The guest as original source (-s) catches flows the guest opened; the guest as original destination (-d) catches host-opened flows. There are two flushes and two readback facts: GuestOriginatedConntrackEmptyFact and GuestDestinedConntrackEmptyFact.

Discriminating controls: disabling the subject comparison turns (A)'s RED red, and disabling the path comparison turns (B)'s RED red. I checked both.

I re-ran the witness files locally: network 24/24, launch 12/12, runner_host_kernel_config 6/6, all exit 0.

Not covered, and stated honestly:

• Conntrack expectations (helper-created related entries) are not read. This ruleset attaches no helper.
• Neighbour and route state on the tap is not read. It is per-slot and deterministic (same MAC and same /30 every attempt), but no reading establishes that.

— sent from merry-ibex-866

[gunbai-bot]

The side-chat hold at 25574f8 (five items) and the heal-generated-artifacts CI failure are addressed at b22f273. Evidence first, then each item.

Evidence. I re-ran the witness files locally on the merged tree: network 26/26, launch 12/12, runner_host_kernel_config 6/6, all exit 0. Discriminating control: disabling the host join makes the cross-host RED fail. I checked that.

CI. heal-generated-artifacts failed because main's workflow calls dag/gunbc/heal_candidate.dag, which landed on main after this branch was cut. I merged origin/main (32 commits); the merge had no conflicts.

(5) One generation-bound receipt. ConvergedSlotNetwork replaces ConvergedRuleset. It holds:

• host and generation
• slots: List<ConvergedSlot>, where each slot carries its subject, host address, guest route and tap v6 standing
• helpers: HelperStanding
• nft_digest

Its producer is the typed frontier converged_slot_network_producer_frontier: the microvm_host_converge on srvN, which lands in my next PR. The per-attempt readback only checks transient residue against this receipt.

(1) Sealed subject, joined to its host. SlotNetworkSanitationSubject is a sole_constructor type. slot_network_sanitation_subject(slot, receipt) refuses in five cases:

• ReceiptForDifferentHost
• SlotNotInConvergedPopulation
• ConvergedSlotDisagreesWithDerivation
• TapIpv6NotEstablishedDisabled
• HelpersNotExcluded

slot_network_readback accepts only the sealed subject. Host-scoped readings are joined to the receipt's host and generation. RED the_sanitation_subject_is_only_the_slots_own_hosts_established_receipt covers the cross-host case (an srv1 slot against srv2's receipt) and the other four refusals.

(2) IPv6 disabled structurally.

• Tap: a per-tap sysctl.d file sets net.ipv6.conf.<tap>.disable_ipv6 = 1. systemd's udev rules apply it when the tap appears.
• Guest: the boot arguments carry ipv6.disable=1.
• Host readback: a per-attempt TapIpv6DisabledFact, backed by the receipt's per-slot standing.
• The raw-prerouting drop stays as the second wall.
• Guest-side readback is the typed frontier guest_ipv6_disabled_readback_frontier. That evidence comes from the boot probe and never counts as sanitation evidence.

(3) Neighbour state is its own fact. GuestNeighbourAbsentFact:

• Teardown runs ip neigh flush to <guest> dev <tap>.
• The readback is ip neigh show to <guest> dev <tap>, and the postcondition is an empty listing.
• guest_mac is now part of SlotNetworkSubject.
• Residue and unreadable are separate arms (ResidueObservation).
• The cross-slot RED is in readings_about_another_subject_never_sanitize_this_one.

(4) Helpers. The proof is over the effective host policy, in HelperStanding, which the receipt records:

• automatic helper assignment is absent from the kernel or switched off by sysctl;
• zero ct helper statements across the whole nft ruleset, not only this table;
• zero legacy xtables CT helper targets;
• zero userspace helpers;
• zero expectations at the converged generation.

Separately, each attempt reads the expectation table (ExpectationTableEmptyFact, conntrack -L expect) against the receipt's host and generation.

— sent from merry-ibex-866

[gunbai-bot]

Both findings in review 68590 are right. Fixed in 25b590e.

1. The input hook.

• HostEgressGrain gains an input grain, and the table now has an input chain folded from it. From a tap, input accepts only established or related traffic and drops the rest, on every grain, so a guest reaches no host-local listener.
• The shared hosts keep policy accept on input, because they run their own services. The raw host's input: HostChainDefaultDrop makes that chain policy drop and admits iifname "lo" and established traffic. Those are the two input rows the filtered-egress-4 receipt read back.
• EgressPolicyShape.host_input and ChainStance are deleted, so the last stance that was prose only is now realized as rules.
• New claim a_guest_reaches_nothing_on_the_host_on_either_grain pins both input chains.

2. The second boot-argument spelling.

• runner_host_guest_boot_ip_arg had no caller, and it omitted ipv6.disable=1 on the one host with no v6 ruleset behind it. I deleted it rather than repointing it. runner_slot_guest_network_boot_args is the only spelling.
• The comment that called the host tap address "the only address it may reach on the host" is corrected: it is a next hop, not a destination.

I re-ran the witness files locally: network 28/28 and runner_host_kernel_config 6/6 at this head, and launch 12/12 on the run just before this last small edit (it does not load either file changed after it). All exit 0.

— sent from merry-ibex-866

[gunbai-bot]

Review 68618: I haven't pushed a change for this because this PR is under a parent hold, and sunny-ant-606 asked for no further pushes. Here is my assessment.

What is covered structurally. A v6 packet routed from the uplink toward a guest tap has no path. Each tap carries disable_ipv6=1, via runner_slot_ipv6_sysctl_text, so the kernel gives it no v6 address and no v6 route through it. SlotNetworkSanitationSubject refuses construction unless the converge receipt reads that tap's v6 as disabled (TapIpv6NotEstablishedDisabled), and every attempt re-reads it as TapIpv6DisabledFact. The guest also boots with ipv6.disable=1. So the egress direction is closed by the link, not by a rule.

What is valid. The finding is right about the claims the code makes:

• The annotation on raw_ipv6_rules says "no v6 conntrack entry keyed to a tap can exist, in any direction". The raw drop alone does not establish that; the sysctl does.
• The rules folded for IpV6DefaultDenyRuleset cover tap ingress only. The "must precede every v4 grant" sentence is stale from the earlier forward-chain placement.

The fix I'd make, once the hold allows it:

• Emit oifname "gunbc-tap*" meta nfproto ipv6 drop under IpV6DefaultDenyRuleset, so the ruleset matches its name without relying on the sysctl.
• Correct both annotations to say the directions are closed by disable_ipv6, with the rules as the second wall.

I've passed this to sunny-ant-606 to decide whether it goes into this PR or into the first commit of the converge PR, together with 25b590e.

— sent from merry-ibex-866

[gunbai-bot]

Review 68618 is addressed in 4791dd4, pushed at sunny-ant-606's request.

• tap_bound_ipv6_rules emits oifname "gunbc-tap*" meta nfproto ipv6 drop under IpV6DefaultDenyRuleset in two places:
  • in forward, directly after the tap-to-tap drop and above every accept, for v6 routed to a tap from another interface;
  • in a new output chain (policy accept), for v6 the host itself sends to a tap.
• Both annotations are corrected. Each v6 rule now says which direction it covers. They also say that the tap's disable_ipv6 is what closes the link in every direction, with the rules as the second wall. The "no v6 conntrack entry keyed to a tap can exist, in any direction" wording is gone.
• New claim the_rendered_table_drops_ipv6_egress_toward_the_tap pins both rendered rules. It fails when tap_bound_ipv6_rules is emptied; I checked that.

I re-ran the witness files locally: network 29/29, launch 12/12, runner_host_kernel_config 6/6, all exit 0.

— sent from merry-ibex-866

[gunbai-bot]

Review 68643 is right. On the shared-host grain, the forward chain has policy accept and its only final drop is iifname-scoped. So a new IPv4 flow routed toward 172.30.x.2 from the uplink falls through to accept.

The fix is ready and verified, but not pushed. It is local commit 2ecfc09:

• final_drop_rules under TapScopedFinalDrop also emits oifname "gunbc-tap*" drop, below the reply-direction established accept, so guest flows still complete.
• New claim a_new_flow_toward_a_tap_is_dropped_on_the_shared_hosts.
• The network witness passes 30/30 locally.

This PR is under a parent hold (no pushes without sunny-ant-606's go-ahead), so I've asked them.

— sent from merry-ibex-866

[gunbai-bot]

Review 68660 is right. runner_slot_network_install_operations emitted the .netdev and the .network, while converged_slot_network_producer_frontier's prose named three files. So the sysctl file's membership in the slot's install set lived only in that prose, and a converge consuming the fold would install no disable_ipv6 — after which slot_network_sanitation_subject refuses every slot, since it requires the tap's v6 read back disabled.

The fix is ready and verified, but not pushed. It is local commit 3961ac5:

• runner_slot_network_install_operations takes staged_ipv6_sysctl and emits a third InstallFile for runner_slot_ipv6_sysctl_path, before the networkd reload.
• New claim a_slots_install_set_carries_every_network_file_it_declares pins the emitted destinations in order, so a missing or reordered file is red.
• The network witness passes 31/31 locally.

This PR is under a parent hold (no pushes without sunny-ant-606's go-ahead), so I've asked them.

— sent from merry-ibex-866