feat(kubernetes): add proxy-pod topology (in-pod process supervisor, out-of-pod proxy)

[russellb]

Summary

Adds the Kubernetes proxy-pod supervisor topology. The process supervisor
stays in the agent pod and launches the workload as it does in combined;
only the network proxy moves out, into a paired, per-sandbox supervisor
Deployment. The sole thing that crosses the pod boundary is the workload's
egress, which is fenced by two per-sandbox Kubernetes NetworkPolicy objects
rather than by pod-local nftables.

Because the supervisor still runs beside the workload, proxy-pod keeps the
full supervisor contract — SSH, exec, connect, upload/download, port
forwarding, openshell logs, and Landlock/seccomp filesystem+process policy —
while producing the least-privileged sandbox pod any OpenShell topology
produces: runAsNonRoot, all Linux capabilities dropped, no privilege
escalation, no added caps at any layer, no privileged init container, no shared
process namespace. That is what lets it run on OpenShift under a stock,
Red Hat-shipped non-root SCC with no bespoke security grant.

Two mechanisms make this safe with the supervisor out of one process' reach:

• Scoped credential. The agent pod holds a per-sandbox, process-kind
  gateway JWT. It can drive its own sandbox's control plane (relays, policy
  fetch, log push, token refresh) but is denied provider-secret and inference
  RPCs at the gateway authorization chokepoint. Provider credentials and
  inference routing live only with the out-of-pod proxy.
• NetworkPolicy fence. An agent-egress policy lets the workload reach only
  the proxy (plus cluster DNS); a supervisor-ingress policy lets only the
  paired workload reach the proxy port. All other egress is dropped by the CNI.

This supersedes the earlier network-only proxy-pod design (supervisor entirely
out of the pod), which lost SSH/exec/sync/filesystem policy and required a
workload-command override. Keeping the process supervisor in-pod recovers those
features; see the design-pivot note below.

This continues @TaylorMutch's original proxy-pod work in #2077 (and the
earlier #2016), rebased onto current main. It is the sibling of the
cni-sidecar topology in #2606.

Design pivot: in-pod process supervisor

The original revision of this PR moved the entire supervisor out of the
sandbox pod. That made SSH/exec/upload/sync structurally impossible (every
relay targets something inside the sandbox, and the SSH server needs the
workload's PID/mount/network namespaces), dropped filesystem/process policy, and
required running the image through a containers.agent.command override.

The current design moves only the network proxy out. The process supervisor
— the piece that launches the workload, applies Landlock/seccomp, and serves
relays — stays in the agent pod and owns its own scoped gateway session. Nothing
about the workload's namespaces has to cross a pod boundary, so the feature loss
is recovered. The privilege win is preserved because network enforcement (the
only part that needed NET_ADMIN/nftables) is what left the pod.

Sandbox-pod privileges: none added

To retain full functionality the agent pod needs zero added privileges:

Setting	Value
runAsNonRoot	true
allowPrivilegeEscalation	false
capabilities	drop: [ALL] (none added)
Privileged / privileged init container	no
Added caps (e.g. NET_ADMIN)	none
shareProcessNamespace / host namespaces / hostPath	none
Init containers (supervisor sideload, CA install, wait-for-proxy, workspace)	all non-root

Filesystem/process policy works without privilege because Landlock is
unprivileged and seccomp applies under no_new_privs (set by
allowPrivilegeEscalation:false); relays bind an abstract unix socket (no caps);
network/L7 policy is enforced out-of-pod, so the agent needs no NET_ADMIN.
proxy-pod maps to ProcessEnforcementMode::NetworkOnly, which skips only the
root/capability-requiring setup while still enforcing the child sandbox.

Topology tradeoffs (combined / sidecar / cni-sidecar / proxy-pod)

In every topology the network supervisor (the OpenShell proxy) is the egress
policy engine. The topologies differ in where the proxy runs, how the
workload is confined to it, and — for proxy-pod — the credential scope in the
sandbox pod.

Dimension	combined (default)	sidecar	cni-sidecar	proxy-pod
Process supervisor location	agent pod	agent pod	agent pod	agent pod
Egress policy engine (proxy) location	agent pod	agent pod	agent pod	separate pod
Confines workload to the proxy via	in-pod nftables	in-pod nftables	node CNI rules	Kubernetes NetworkPolicy
Network + L7 policy	yes	yes	yes	yes
Filesystem / process / binary policy	yes	partial (Landlock)	partial (Landlock)	yes (Landlock/seccomp)
SSH / exec / connect / upload / sync	yes	yes	yes	yes
Port forwarding	yes	yes	yes	yes
Workload stdout/stderr in openshell logs	yes	yes	yes	yes
Sandbox-pod gateway credential	full	full	full	scoped (process-kind: no provider/inference)
Process attribution on net events	full (binary + PID)	full	full	none (proxy is in another pod)
Added caps in sandbox pod	yes	SYS_PTRACE, DAC_READ_SEARCH¹	SYS_PTRACE, DAC_READ_SEARCH¹	none
Shared PID namespace	no	yes	yes	no
Privileged init container	no	yes	no	no
Node-level privileged DaemonSet	no	no	yes	no
Pods per sandbox	1	1	1	2
Kernel isolation under Kata (workload vs supervisor)	shared VM	shared VM	shared VM	separate VMs
Requires NetworkPolicy enforcement	no	no	no	yes
OpenShift SCC	privileged	custom	custom + privileged CNI	stock non-root

¹ On the in-pod network sidecar container (--mode=network), added when
binary-aware network policy is enabled — the capabilities (plus the shared PID
namespace) let the in-pod network supervisor read /proc/<pid> of workload
processes to attribute connections to a binary/PID. That is exactly the feature
proxy-pod gives up: its proxy runs in a separate pod with no view of the
workload's processes, so it needs neither capability. The agent container drops
all capabilities in every topology.

When to use which. combined stays the default. sidecar/cni-sidecar
keep the full contract while lowering pod privilege, at the cost of a per-pod
privileged init container or a node-level CNI DaemonSet. proxy-pod keeps the
full contract and the lowest pod privilege — the only topology admitted by a
stock OpenShift non-root SCC — at the cost of a second pod per sandbox, a
dependency on CNI NetworkPolicy enforcement, and a scoped sandbox-pod
credential. It also raises the isolation ceiling under a VM RuntimeClass:
because the workload and proxy are in separate pods, Kata places them in separate
VMs with separate kernels, so a kernel compromise in the workload VM does not by
itself reach the proxy or its provider credentials.

The remaining structural gap versus the in-pod topologies is process
attribution on network events: the proxy runs in a different pod and cannot map
a connection back to the originating PID/binary, so those events render without
process identity.

RFC

rfc/proxy-pod-topology-DRAFT.md (included in this PR, unnumbered pending a
maintainer-assigned number) documents the in-pod-process-supervisor design:
motivation, per-sandbox resources, the credential-scope and NetworkPolicy model,
OpenShift enablement (cluster DNS peers/port, stock non-root SCC), readiness
without an in-sandbox session, the retained feature set, remaining tradeoffs,
risks, alternatives, and open questions. The superseded network-only design is
retained in the RFC for context.

Related Issue

Continues @TaylorMutch's proxy-pod PR #2077, which references #1827, #981,
#899, and #1305. Maintainers assign the RFC number from the originating issue
before it leaves draft.

Changes

Topology:

• Add the proxy-pod supervisor topology: the agent pod runs the process
  supervisor (--mode=process) which launches the workload and serves relays;
  a per-sandbox supervisor Deployment runs the network proxy (--mode=network),
  fronted by a headless Service, a generated proxy CA Secret, and an
  agent-egress / supervisor-ingress NetworkPolicy pair.
• The agent pod is fully non-root (drop ALL caps, no privilege escalation) with a
  non-root supervisor-binary sideload init container; the SSH relay uses a
  netns-scoped abstract socket so no writable /run path is needed.
• The Deployment, Service, Secret, and supervisor-ingress policy are
  owner-referenced to the Sandbox CR and GC'd with it. The agent-egress fence is
  deliberately not owner-referenced (Kubernetes GC does not order sibling
  deletion, so a GC-owned fence could be removed alongside the workload pod and
  let a SIGTERM-ignoring workload regain direct egress); the gateway manages it
  directly, deleting it only after the workload pod is gone and reaping any fence
  orphaned by a gateway crash.
• Companion names are keyed on the immutable sandbox UUID, stable across
  sandbox-name reuse. Nested proxy_pod.proxy_uid / proxy_pod.affinity config
  and Helm values.

Scoped credential (new):

• SandboxCallerKind {Full, Process} claim in the sandbox JWT. The agent pod is
  minted a Process-kind token (based on its openshell.ai/sandbox-role=agent
  pod label); Process callers are denied GetSandboxProviderEnvironment,
  ExchangeProviderSubjectToken, and GetInferenceBundle at the multiplex
  authorization chokepoint. Token refresh preserves the caller kind.

OpenShift enablement:

• Configurable cluster DNS peers (proxy_pod.dns_peers) and gateway peers
  (proxy_pod.gateway_peers, so the in-pod supervisor's own gateway session is
  allowed through the egress fence), with per-peer namespace/pod selectors and
  port. Empty lists are rejected; an empty peer renders no rule rather than
  allow-all.
• Gated stock non-root SCC grant (sandboxServiceAccount.openshift.nonrootSCC,
  default off) — a ClusterRole/Binding only, no custom SCC object.

Readiness & lifecycle:

• proxy-pod reports SupervisorSessionModel::Required (relays are available).
• wait-for-proxy agent-pod init container (via a wait-for-tcp supervisor
  subcommand) so pod readiness transitively means egress works.
• The supervisor Deployment's live availability folds into sandbox readiness
  (tri-state, so a transient GET error never fails open to Ready); a shared-mode
  Deployment watch pushes status within seconds, with get/list and the periodic
  reconcile as backstops.
• Periodic companion reconciliation corrects supervisor replica drift and reaps
  orphaned egress fences (re-confirming the Sandbox CR is gone immediately before
  deleting a fence).
• stop/start scale the supervisor Deployment to zero and back.

Correctness fixes (rebase + cluster testing):

• Reject corporate upstream-proxy credential Secrets in proxy-pod (would land
  in the workload pod).
• Reject containers.agent.command/args overrides in every topology (the
  supervisor is the entrypoint).
• Non-root supervisor sideload init container + abstract SSH socket (the two
  OpenShift nonroot-v2 admission/startup fixes found in cluster validation).

Docs: RFC, docs/kubernetes/topology.mdx, docs/kubernetes/openshift.mdx,
docs/reference/sandbox-compute-drivers.mdx, docs/reference/gateway-config.mdx,
Helm README, and the debug-openshell-cluster / helm-dev-environment skills.

Testing

• cargo test -p openshell-driver-kubernetes (251) and -p openshell-server --lib (1432).
• mise run helm:test — 122 pass (proxy-pod DNS/gateway-peer, SCC-grant, RBAC cases).
• mise run pre-commit — clean.
• CI: the proxy_pod e2e suite (rewritten for the in-pod design) asserts a
  plain create reaches Ready and that exec and upload round-trip through the
  in-pod supervisor.
• Live OpenShift / OVN-Kubernetes (policy-enforcing): end-to-end validated —
  sandbox reaches Ready with the process supervisor running non-root in the
  agent pod (all four init containers non-root, admitted by a stock non-root SCC);
  exec, exec --tty (PTY → /dev/pts/0), interactive connect,
  upload/download, and ssh -L port forwarding all work; openshell logs
  streams workload/supervisor output and shows Landlock policy being applied;
  the agent pod's Process-kind token is denied provider RPCs (gateway returns
  PERMISSION_DENIED) while the controller's Full token succeeds; workload
  egress is fenced (direct egress blocked, proxied egress policy-evaluated:
  allow → 200, deny → 403 at CONNECT) with the generated CA trusted; the in-pod
  supervisor reaches the gateway via the gateway_peers egress rule.
• Kata Containers (VM RuntimeClass): re-validated the full flow with
  runtimeClassName: kata on both pods. Each pod runs in its own Kata VM
  (confirmed via the hypervisor CPU flag and VM-bounded memory), so the
  workload and network proxy get separate VMs/kernels. Ready, exec/--tty,
  upload/download, the cross-VM egress fence (direct blocked, proxied L7 403),
  the scoped process-kind credential, in-guest Landlock policy, and openshell logs all work; no topology-specific issues under Kata.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Original authorship from feat(kubernetes): add proxy-pod supervisor topology #2077 preserved on rebased commits

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[russellb]

I'm still running local review, but I think this is ready to look at and try out

[russellb]

Major update: proxy-pod now keeps the full supervisor feature set

This PR has been redesigned. The earlier revision moved the entire
supervisor out of the sandbox pod, which structurally lost SSH, exec,
upload/sync, filesystem/process policy, and openshell logs, and required a
workload-command override.

The new design moves only the network proxy into the separate per-sandbox
pod. The process supervisor stays in the agent pod, launches the workload,
and serves relays over its own scoped gateway session — so those features come
back, at the same low pod privilege.

Retained (validated on a live OpenShift / OVN-Kubernetes cluster):

• SSH, exec, exec --tty (PTY → /dev/pts/0), interactive connect
• upload / download, and ssh -L port forwarding
• openshell logs streaming, with Landlock/seccomp filesystem+process policy applied

Still least-privileged. The agent pod adds no privileges to keep this
functionality: runAsNonRoot, drop: [ALL], no privilege escalation, no added
caps, no privileged init container, no shared PID namespace — it admits under a
stock OpenShift non-root SCC. Network enforcement (the only part needing
NET_ADMIN/nftables) is what left the pod, replaced by a NetworkPolicy egress
fence.

Safe with the proxy out of reach. The agent pod holds a per-sandbox
process-kind credential that can drive its own control plane but is denied
provider-secret and inference RPCs (verified: the gateway returns
PERMISSION_DENIED); provider credentials live only with the out-of-pod proxy.
Workload egress is fenced to that proxy (direct egress blocked; proxied egress
policy-evaluated — allow → 200, deny → 403).

The one remaining structural gap vs. the in-pod topologies is process
attribution on network events (the proxy is in a different pod). The PR
description and RFC have the full tradeoff table and details.

[grs]

The proxy-pod mode appears to leave the provider SPIFFE Workload API socket mounted into the agent pod. The base pod template adds SPIFFE_WORKLOAD_API_VOLUME_NAME and OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET when provider_spiffe_enabled is true. In sidecar mode both are removed from the agent container. The proxy-pod transform does not do the same cleanup. The workload API is only needed in the supervisor proxy pod.

[russellb]

Correct — the proxy-pod transform was missing the SPIFFE cleanup that sidecar does. Fixed in b3b3c75.

The proxy-pod agent transform now strips all three from the agent pod: the OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET env, the container volume mount, and — since no container in the agent pod uses it — the csi.spiffe.io volume from the pod spec entirely (so the agent pod never receives an SVID). The supervisor (proxy) pod keeps the mount and volume, since it's the only side that mints provider credentials. A unit test asserts both halves (agent stripped, supervisor retained).

This reinforces the scoped-credential model: the agent pod's process-kind token is already denied provider RPCs, and now it also has no Workload API socket to obtain an SVID from.

[2000krysztof]

Suggestion: topology-aware SCC guidance for OpenShift
This section instructs all OpenShift users to grant privileged SCC to the sandbox service account. With the proxy-pod topology, the enforcement boundary moves outside the pod, sandbox pods run with drop: ALL capabilities and need zero elevated privileges.
Could this guide be updated to show the topology-specific path?

• proxy-pod: nonroot-v2 is sufficient (built-in, no security exception required). The chart already supports sandboxServiceAccount.openshift.nonrootSCC=true which handles this automatically — no manual oc adm policy step needed.
• combined / sidecar: still requires privileged SCC as today.
  As written, users deploying proxy-pod on OpenShift will follow these instructions and grant privileged unnecessarily, which undermines the security benefit this topology provides.

OpenShell/deploy/helm/openshell/README.md

Lines 49 to 50 in 06f0aa6

	# Sandboxes are deployed into the openshell namespace and use the openshell-sandbox service account
	oc adm policy add-scc-to-user privileged -z openshell-sandbox -n openshell

[russellb]

Validated under Kata Containers (VM-based RuntimeClass)

Ran the proxy-pod topology on OpenShift with runtimeClassName: kata on both pods (OVN-Kubernetes, policy-enforcing). It works, and the separate-pods → separate-VMs isolation the design claims holds up.

Confirmed real VM isolation: the agent pod reports the hypervisor CPU flag and sees only its VM-bounded memory (~1.9 GB vs the host's ~131 GB). Each pod is its own Kata VM, so the workload and the network proxy run in separate VMs with separate kernels.

Full feature set works in-guest:

Check	Result
Both pods Ready under kata RuntimeClass	✅
exec / exec --tty (PTY)	✅ non-root, /dev/pts/0
upload / download round-trip	✅
Egress fence (cross-VM)	✅ direct blocked; proxied reaches proxy VM, L7 policy enforced (403)
Scoped process-kind credential	✅ provider RPCs denied
Landlock filesystem policy	✅ applied in the Kata guest (abi:v5, 9 rules)
Log push (openshell logs)	✅

No topology-specific issues surfaced under Kata.

[russellb]

@2000krysztof Correct — the chart README's quick-start granted privileged unconditionally, which is wrong for proxy-pod. Fixed in 8d70aa8.

The README OpenShift section is now split by topology:

• combined/sidecar: keep the privileged grant (in-pod enforcement).
• proxy-pod: skip the oc adm policy step entirely and bind the built-in nonroot-v2 via sandboxServiceAccount.openshift.nonrootSCC=true (requires the default workspaceMode=shared).

Edited the README.md.gotmpl source and regenerated README.md. The published docs page (docs/kubernetes/openshift.mdx) already carried the topology-aware guidance, so this brings the chart README in line with it.

[jhjaggars]

OpenShift/OVN validation found that proxy-pod agent Pods receive HTTP_PROXY/HTTPS_PROXY, but commands launched through openshell sandbox exec do not. The process supervisor clears the child environment and reconstructs proxy variables from OPENSHELL_PROXY_URL, which the proxy-pod transform did not set.

A controlled test confirmed that adding only OPENSHELL_PROXY_URL makes implicit exec egress work while preserving L7 denials and the direct-bypass fence. I opened a stacked fix with regression coverage: russellb#1

[russellb]

Thanks, @jhjaggars! I pulled your change in.

[cgwalters]

I pretty strongly think we should do this for the podman driver too; podman very intentionally supports Kubernetes concepts like pods, so we can move the proxy out to a separate container in the same pod there too.