CNTRLPLANE-3329: Enable fuse-overlayfs in ARC runner pods via user namespaces - #8571
CNTRLPLANE-3329: Enable fuse-overlayfs in ARC runner pods via user namespaces#8571celebdor wants to merge 1 commit into
Conversation
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
Skipping CI for Draft Pull Request. |
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
openshift-ci-robot
added
the
jira/valid-reference
|
@celebdor: This pull request references CNTRLPLANE-3329 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR enables unprivileged FUSE for the GitHub Actions runner by adding an OpenShift SecurityContextConstraints resource (restricted-v3-fuse) that permits CAP_SYS_ADMIN and UID/GID ranges 0–65534 under pod-scoped user namespaces, and by updating the runner Helm values to annotate the pod for CRI‑O device access (/dev/fuse), set spec.hostUsers:false, and configure pod/container securityContexts (runAsUser: 0; container drops all capabilities and adds SYS_ADMIN). Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 12✅ Passed checks (12 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
openshift-ci
Bot
added
area/ci-tooling
|
Scheduling tests matching the |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bryan-cox, celebdor The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
openshift-ci
Bot
removed
the
do-not-merge/work-in-progress
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@hack/github-actions-runner/values.yaml`:
- Around line 19-21: The securityContext currently only adds SYS_ADMIN which
widens capability surface; update the capabilities block in the relevant
resource (securityContext) to explicitly drop all capabilities and then re-add
only SYS_ADMIN (i.e., add a capabilities.drop: ["ALL"] alongside
capabilities.add: ["SYS_ADMIN"]) so only that single capability is granted.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 77e6e385-4146-4d6a-a460-57c1b131090e
📒 Files selected for processing (2)
hack/github-actions-runner/restricted-v3-fuse-scc.yamlhack/github-actions-runner/values.yaml
|
/verified by @celebdor |
openshift-ci-robot
added
the
verified
|
@bryan-cox: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/override ci/prow/e2e-aws |
|
@celebdor: Overrode contexts on behalf of celebdor: ci/prow/e2e-aks, ci/prow/e2e-aws, ci/prow/e2e-aws-upgrade-hypershift-operator, ci/prow/e2e-azure-self-managed, ci/prow/e2e-kubevirt-aws-ovn-reduced, ci/prow/e2e-v2-aws, ci/prow/e2e-v2-gke, ci/prow/images, ci/prow/okd-scos-images, ci/prow/security, ci/prow/verify-deps DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
The fuse-overlayfs binary was added to the runner image in the previous
PR but /dev/fuse was not exposed and the SCC blocked the mount syscall.
Enable unprivileged FUSE mounts in ARC runner pods by:
- Adding the CRI-O annotation to expose /dev/fuse
- Running pods in user namespaces (hostUsers: false) so uid 0 maps
to an unprivileged host identity
- Requesting CAP_SYS_ADMIN explicitly on the container (needed for
mount(2) inside the user namespace)
- Introducing restricted-v3-fuse SCC: identical to restricted-v3
except it allows SYS_ADMIN in allowedCapabilities, drops only
MKNOD and NET_RAW (instead of ALL), and permits uid 0 inside the
user namespace via userNamespaceLevel: RequirePodLevel
The runner SA must be granted the SCC before deploying:
oc adm policy add-scc-to-user restricted-v3-fuse \
-z arc-runner-set-gha-rs-no-permission -n arc-runners
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3bebac7 to
7f51242
Compare
openshift-ci-robot
removed
the
verified
|
/lgtm |
|
@celebdor: This pull request references CNTRLPLANE-3329 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/override ci/prow/e2e-aws |
|
Scheduling tests matching the |
|
@celebdor: Overrode contexts on behalf of celebdor: ci/prow/e2e-aks, ci/prow/e2e-aws, ci/prow/e2e-aws-upgrade-hypershift-operator, ci/prow/e2e-azure-self-managed, ci/prow/e2e-kubevirt-aws-ovn-reduced, ci/prow/e2e-v2-aws, ci/prow/e2e-v2-gke, ci/prow/images, ci/prow/okd-scos-images, ci/prow/security, ci/prow/verify-deps DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Test Resultse2e-aws
e2e-aks
Failed TestsTotal failed tests: 4
|
AI Test Failure AnalysisJob: Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6 |
Test Failure Analysis CompleteJob Information
Test Failure AnalysisErrorSummaryThis is a pre-existing infrastructure flake unrelated to PR #8571. The PR only modifies GitHub Actions runner configuration files ( Root CauseThe AKS management cluster was under severe resource pressure while running 9 concurrent hosted clusters, leading to two distinct but related failures: TestCreateClusterHABreakGlassCredentials — The HA kube-controller-manager deployment requires 2 replicas. Pod TestCreateCluster — This cluster's version rollout was delayed (remained Underlying infrastructure cause: The management cluster (3× Standard_D4lds_v5 + 6× Standard_D14_v2 via NAP) hit limits documented in the build log: Why this is NOT caused by PR #8571: The PR modifies only Recommendations
Evidence
|
|
@celebdor: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Closing in favor of GOCACHEPROG approachThis PR attempted to enable NFS idmapped mount support was added upstream in kernel 6.3 and has not been backported to RHEL 9. Alternative: Use Go 1.24+'s |
4 participants
Summary
Enables the fuse-overlayfs mount in ARC runner pods so the EFS-backed Go build cache can be used as a zero-copy overlay instead of a full
cp -aat each job start. PR #8568 added fuse-overlayfs to the runner image and the warm-go-cache action, but the mount was failing because/dev/fusewas not exposed and the SCC blocked themount(2)syscall.Changes:
values.yaml: Adds the CRI-O annotation to expose/dev/fuse, enables pod-level user namespaces (hostUsers: false), runs as uid 0 inside the user namespace (maps to unprivileged host uid), and explicitly requestsCAP_SYS_ADMINon the container for the mount syscallrestricted-v3-fuse-scc.yaml: Custom SCC based onrestricted-v3with minimal changes — allowsSYS_ADMINinallowedCapabilities, drops onlyMKNOD/NET_RAWinstead ofALL, permits uid 0, and requiresuserNamespaceLevel: RequirePodLevelso these privileges only exist inside the user namespaceWhy user namespaces?
fuse-overlayfsneeds eitherfusermount3(setuid, blocked byallowPrivilegeEscalation: false) orCAP_SYS_ADMINformount(2). Running in a user namespace (hostUsers: false) means uid 0 andCAP_SYS_ADMINinside the pod map to an unprivileged host identity — no actual privilege escalation on the host.Pre-deploy step
The runner service account must be granted the SCC before helm upgrade:
Test plan
contrib/ci/gha-cache-timing.sh🤖 Generated with Claude Code
Summary by CodeRabbit