Skip to content

OSAC-1568: Add local Vault infrastructure and e2e workflow - #72

Merged
eliorerz merged 1 commit into
osac-project:mainfrom
eliorerz:OSAC-1568-vault-infra
Jun 18, 2026
Merged

eliorerz merged 1 commit into
osac-project:mainfrom
eliorerz:OSAC-1568-vault-infra

Conversation

@eliorerz

@eliorerz eliorerz commented Jun 18, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add Vault server infrastructure (config, Podman Quadlet, systemd units, setup/unseal/backup/health-check scripts)
  • Vault runs as a localhost-only Podman container on the self-hosted runner, managed by systemd via Quadlet
  • JWT auth backed by GitHub OIDC with bound claims (repository_owner: osac-project, environment: e2e-test) and bound audiences
  • Add reusable e2e workflow using hashicorp/vault-action@v4 for secret retrieval via OIDC JWT
  • Add caller workflow triggered on pull_request and workflow_dispatch

Files

File Purpose
vault/config/vault.hcl Vault server config (file storage, 0.0.0.0:8200 in container, TLS disabled, disable_mlock)
vault/quadlet/vault.container Podman Quadlet unit (vault:2.0, localhost-only publish, SELinux :Z volumes, auto-unseal via ExecStartPost)
vault/scripts/vault-prerequisites.sh Installs podman, jq, vault CLI (RHEL/CentOS/Fedora); root guard; conditional dnf/yum support
vault/scripts/vault-setup.sh 10-phase one-time setup: dirs, config, systemd, init (3/2 Shamir), unseal, KV v2, JWT auth, policy+role, linger
vault/scripts/vault-unseal.sh Auto-unseal using stored keys; handles uninitialized/already-unsealed states gracefully
vault/scripts/vault-backup.sh Tar backup of data dir with 30-day retention
vault/scripts/vault-health-check.sh 10-point check (reachable, unsealed, initialized, version, JWT, role, policy, KV, systemd, timer)
vault/vault-unseal.service Systemd oneshot: fallback manual unseal
vault/vault-backup.service Systemd oneshot: backup execution
vault/vault-backup.timer Daily 03:00 backup with 10min random delay
.github/workflows/e2e-vault-secrets.yml Reusable e2e workflow: OIDC JWT auth to Vault, secret retrieval, podman test execution
.github/workflows/e2e-test.yml Caller workflow: triggers e2e on pull_request and workflow_dispatch

Deployment

# On the runner machine:
sudo bash vault/scripts/vault-prerequisites.sh   # one-time: install deps
bash vault/scripts/vault-setup.sh                 # one-time: deploy + configure
bash vault/scripts/vault-health-check.sh          # verify

Test plan

  • Full setup from scratch on CI runner -- all 10 phases pass
  • Idempotent re-run -- detects existing config, skips init
  • Health check -- 10/10 pass
  • Secret lifecycle -- write, read, delete at secret/osac/e2e/test
  • Backup -- systemctl --user start vault-backup.service creates timestamped tarball
  • Vault role tightened with environment + audience bound claims
  • Secrets populated (pull-secret, aap-license)
  • Reboot persistence -- verify auto-start and auto-unseal after machine reboot
  • JWT auth smoke test -- real GitHub Actions workflow with id-token: write

Summary by CodeRabbit

Release Notes

  • New Features
    • Added a managed Vault deployment using a Podman container with systemd integration, automatic startup unsealing, and a manual unseal option.
    • Introduced OSAC Vault configuration, prerequisites installation, and one-time setup to initialize KV and JWT/OIDC authentication for CI.
    • Enabled daily encrypted Vault backups with a retention policy and added a 10-point health check.
    • Added GitHub Actions E2E workflows that obtain Vault secrets using GitHub OIDC authentication.

@openshift-ci-robot

openshift-ci-robot commented Jun 18, 2026 •

Copy link
Copy Markdown

@eliorerz: This pull request references OSAC-1568 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

  • Add Vault server infrastructure (config, Podman Quadlet, systemd units, setup/unseal/backup/health-check scripts)
  • Vault runs as a localhost-only Podman container on the self-hosted runner, managed by systemd via Quadlet
  • JWT auth backed by GitHub OIDC — e2e workflows authenticate with their GitHub identity, no stored secrets needed
  • Tested end-to-end on the CI runner: all 10 health checks pass, secret write/read/delete lifecycle verified, backup and idempotent re-run confirmed

Files

File Purpose
vault/config/vault.hcl Vault server config (file storage, 0.0.0.0:8200 in container, TLS disabled, disable_mlock)
vault/quadlet/vault.container Podman Quadlet unit (vault:2.0, localhost-only publish, SELinux :Z volumes, auto-unseal via ExecStartPost)
vault/scripts/vault-prerequisites.sh Installs podman, jq, vault CLI (RHEL/CentOS/Fedora)
vault/scripts/vault-setup.sh 10-phase one-time setup: dirs, config, systemd, init (3/2 Shamir), unseal, KV v2, JWT auth, policy+role, linger
vault/scripts/vault-unseal.sh Auto-unseal using stored keys; handles uninitialized/already-unsealed states gracefully
vault/scripts/vault-backup.sh Tar backup of data dir with 30-day retention
vault/scripts/vault-health-check.sh 10-point check (reachable, unsealed, initialized, version, JWT, role, policy, KV, systemd, timer)
vault/vault-unseal.service Systemd oneshot: fallback manual unseal
vault/vault-backup.service Systemd oneshot: backup execution
vault/vault-backup.timer Daily 03:00 backup with 10min random delay

Deployment

# On the runner machine (as root or the runner user):
sudo bash vault/scripts/vault-prerequisites.sh   # one-time: install deps
bash vault/scripts/vault-setup.sh                 # one-time: deploy + configure
bash vault/scripts/vault-health-check.sh          # verify

Test plan

  • Full setup from scratch on CI runner — all 10 phases pass
  • Idempotent re-run — detects existing config, skips init
  • Health check — 10/10 pass
  • Secret lifecycle — write, read, delete at secret/osac/e2e/test
  • Backup — systemctl --user start vault-backup.service creates timestamped tarball
  • Reboot persistence — verify auto-start and auto-unseal after machine reboot
  • JWT auth smoke test — real GitHub Actions workflow with id-token: write (follow-on)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jun 18, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@eliorerz, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 41 minutes and 28 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: d11a20ec-7b5f-4752-ae71-6ad95e0cdef8

📥 Commits

Reviewing files that changed from the base of the PR and between 74086e1 and e577b14.

📒 Files selected for processing (12)
  • .github/workflows/e2e-test.yml
  • .github/workflows/e2e-vault-secrets.yml
  • vault/config/vault.hcl
  • vault/quadlet/vault.container
  • vault/scripts/vault-backup.sh
  • vault/scripts/vault-health-check.sh
  • vault/scripts/vault-prerequisites.sh
  • vault/scripts/vault-setup.sh
  • vault/scripts/vault-unseal.sh
  • vault/vault-backup.service
  • vault/vault-backup.timer
  • vault/vault-unseal.service

Walkthrough

Adds a complete HashiCorp Vault deployment for OSAC CI runners: Vault HCL configuration and Podman Quadlet container unit for containerized server, host prerequisite installer script, 10-phase orchestration setup script covering initialization through policy/role creation, unseal script with systemd fallback service, backup script with daily timer, and health-check validation script. Also adds GitHub Actions E2E test dispatcher workflow and reusable workflow that authenticate to Vault via GitHub OIDC JWT and retrieve secrets for test execution.

Changes

Vault Deployment Infrastructure

Layer / File(s) Summary
Vault config and Podman Quadlet container unit
vault/config/vault.hcl, vault/quadlet/vault.container
vault.hcl sets file-based storage at /vault/data, TCP listener on 0.0.0.0:8200 with TLS disabled, mlock disabled, and UI enabled. vault.container is the Podman Quadlet systemd unit that runs hashicorp/vault:2.0 with localhost-only port binding, SELinux-relabeled volume mounts for config/data/logs, rootless environment variables, registry auto-update, restart/timeout settings, and an ExecStartPost hook invoking vault-unseal.sh for automatic post-startup unsealing.
Host prerequisite installer script
vault/scripts/vault-prerequisites.sh
Detects and uses dnf or yum package manager, enforces root/sudo execution, installs podman, jq, and appropriate repo-management tools, conditionally adds HashiCorp RPM repository, installs vault CLI, verifies versions, and directs user to run vault-setup.sh next.
Unseal script and systemd fallback service
vault/scripts/vault-unseal.sh, vault/vault-unseal.service
vault-unseal.sh polls for Vault reachability (30 s timeout), exits gracefully for uninitialized or already-unsealed instances, reads base64-decoded keys from ~/.vault-server/.unseal-keys, applies up to threshold 2 unseal operations, and exits 1 if Vault remains sealed. vault-unseal.service wraps it as a oneshot systemd unit ordered after vault.service for manual re-unseal capability.
One-time setup orchestrator
vault/scripts/vault-setup.sh
Orchestrates 10 sequential phases: (1) creates ~/.vault-server directory structure with Vault container UID/GID ownership on data/ and logs/; (2) copies vault.hcl and helper scripts with proper permissions; (3) installs Quadlet and systemd units, reloads daemon, starts vault.service, and polls API reachability; (4) initializes Vault (3 shares, threshold 2) if not already initialized, writing 0600-mode init JSON; (5) extracts and saves unseal keys to .unseal-keys file; (6) invokes vault-unseal.sh to unseal; (7) enables KV v2 at secret/ and establishes VAULT_TOKEN; (8) conditionally enables JWT auth mount and configures GitHub OIDC discovery URL and bound issuer; (9) writes osac-e2e policy and JWT role with bound claims/audience and TTL/max TTL; (10) enables loginctl linger, enables and starts backup timer, and prints completion banner.
Backup script, systemd service, and daily timer
vault/scripts/vault-backup.sh, vault/vault-backup.service, vault/vault-backup.timer
vault-backup.sh creates timestamped tar.gz archives of ~/.vault-server/data with 0600 permissions and prunes archives older than 30 days. vault-backup.service is a oneshot systemd unit executing the script. vault-backup.timer schedules daily execution at 03:00 UTC with 600-second randomized delay and Persistent=true for missed-execution handling.
Health check validation script
vault/scripts/vault-health-check.sh
Performs 10 ordered checks: API reachability, sealed status, initialization status, version presence, JWT auth mount, osac-e2e role and policy existence, KV v2 engine at secret/, and systemctl status verification for vault.service and vault-backup.timer. Loads VAULT_TOKEN from init JSON or warns if unset, prints numbered [PASS]/[FAIL] results, and exits 1 if any check fails.

E2E Test CI Workflow Integration

Layer / File(s) Summary
E2E test dispatcher workflow
.github/workflows/e2e-test.yml
Defines a public CI workflow triggered on pull_request to main and manual workflow_dispatch, accepts optional test-suite (default vmaas) and test-filter (default empty) inputs, and delegates execution to the reusable workflow e2e-vault-secrets.yml.
Reusable E2E workflow with Vault OIDC authentication
.github/workflows/e2e-vault-secrets.yml
Reusable workflow that authenticates to Vault via hashicorp/vault-action@v4 using GitHub OIDC JWT, retrieves pull-secret and aap-license Vault secrets, writes them to temporary runner files with restrictive permissions and base64 decode, launches the test container image via podman with mounted kubeconfig and secret files while passing OSAC environment variables (namespace, VM template, pull secret path), runs pytest with JUnit XML output and verbose logging, and cleans up temporary secret files in an always() step. Exposes result output (PASSED/FAILED) based on test step outcome.

Sequence Diagram(s)

sequenceDiagram
  participant op as Operator
  participant setup as vault-setup.sh
  participant sd as systemd --user
  participant api as Vault API<br/>(127.0.0.1:8200)
  participant gh as GitHub OIDC

  op->>setup: bash vault-setup.sh
  setup->>setup: Phase 1-2: mkdir ~/.vault-server, copy config + scripts
  setup->>sd: Phase 3: install Quadlet + units, daemon-reload, start vault.service
  sd-->>api: vault.service launching Podman container
  setup->>api: poll vault status until reachable
  setup->>api: Phase 4: vault operator init (shares=3, threshold=2)
  api-->>setup: init JSON (root_token + unseal_keys_b64)
  setup->>setup: Phase 5: extract .unseal-keys (chmod 0600)
  setup->>api: Phase 6: vault-unseal.sh (apply keys)
  setup->>api: Phase 7: enable KV v2 at secret/
  setup->>gh: Phase 8: JWT auth mount + GitHub OIDC discovery URL
  setup->>api: Phase 9: osac-e2e policy + JWT role (bound claims)
  setup->>sd: Phase 10: loginctl linger + backup timer enable/start
  setup-->>op: completion banner + next steps
Loading
sequenceDiagram
  participant gh as GitHub (OIDC)
  participant workflow as e2e-vault-secrets.yml
  participant vault as Vault API
  participant runner as osac-ci runner
  participant container as Test container<br/>(podman)

  gh->>workflow: trigger (pull_request or workflow_dispatch)
  workflow->>gh: request OIDC JWT token
  gh-->>workflow: GitHub OIDC JWT
  workflow->>vault: authenticate via hashicorp/vault-action@v4 + JWT
  vault-->>workflow: Vault token (JWT role osac-e2e)
  workflow->>vault: retrieve pull-secret and aap-license
  vault-->>workflow: secret values
  workflow->>runner: write secrets to /tmp/*.txt (chmod 600)
  runner->>container: podman run with --volume mounts for secrets
  container->>container: pytest with env vars, kubeconfig, secret files
  container-->>runner: test results (JUnit XML, verbose logs)
  runner->>runner: always(): remove /tmp secret files
  workflow-->>gh: result output (PASSED or FAILED)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Suggested reviewers

  • larsks
  • danmanor

Poem

🔐 From init to unseal, ten phases complete,
Vault stands ready on 127.0.0.1:8200 street,
GitHub OIDC tokens grant secret access with grace,
E2E tests run fast in a containerized space,
Backups at midnight—the keys are secure! 🛡️


Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (1 error, 2 warnings)

Check name Status Explanation Resolution
No-Injection-Vectors ❌ Error Shell injection vulnerability in .github/workflows/e2e-vault-secrets.yml: unquoted TEST_FILTER and TEST_SUITE user inputs used in pytest command construction (lines 83-88, 103) allow arbitrary comm... Quote variables in shell commands: FILTER_ARGS="-k '${TEST_FILTER}'" and TEST_ARGS="tests/'${TEST_SUITE}'/" to prevent shell injection attacks from untrusted workflow inputs.
Docstring Coverage ⚠️ Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Ai-Attribution ⚠️ Warning AI tools were used (evident from "AI-generated summary" in summary section), but commit 74086e1 lacks required Assisted-by/Generated-by attribution trailers in the commit message. Add Assisted-by or Generated-by trailer(s) to the commit message to properly attribute AI tool usage (e.g., 'Assisted-by: Claude noreply@anthropic.com').
✅ Passed checks (8 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding local Vault infrastructure and an e2e workflow. It is specific, concise, and directly reflects the changeset's primary objectives.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
No-Hardcoded-Secrets ✅ Passed No hardcoded secrets found. Vault config contains no credentials; sensitive data (tokens, unseal keys) stored in chmod 0600 files; workflows use hashicorp/vault-action@v4 with OIDC JWT auth and run...
No-Weak-Crypto ✅ Passed PR contains no weak crypto algorithms (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom implementations, or non-constant-time secret comparisons. All cryptography delegated to HashiCorp Vault.
Container-Privileges ✅ Passed Container manifests contain no privileged: true, hostPID/Network/IPC, SYS_ADMIN caps, or allowPrivilegeEscalation flags. Vault container runs rootless with localhost-only binding and disabled mlock.
No-Sensitive-Data-In-Logs ✅ Passed All sensitive operations properly suppress output or redirect to files. Unseal keys/tokens never logged to stdout; init output redirected to file only; secrets written to disk with 0600 permissions...
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 9

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@vault/scripts/vault-backup.sh`:
- Around line 27-29: The pruned backup count in the find command is inaccurate
because the current order of `-print -delete` causes files to be printed before
deletion is attempted, meaning files that fail to delete are still counted. Fix
this by reordering the find predicates so that `-delete` comes before `-print`,
which ensures only successfully deleted files are counted in the pruned variable
and logged in the subsequent echo statement.

In `@vault/scripts/vault-health-check.sh`:
- Around line 45-51: The authenticated vault commands in checks 6 and 7 (vault
read auth/jwt/role/osac-e2e and vault policy read osac-e2e) will fail without a
VAULT_TOKEN set in the script. Source the root token from the vault
initialization file ~/.vault-server/.vault-init.json at the beginning of the
script (similar to how vault-setup.sh does it) to extract and set VAULT_TOKEN as
an environment variable. This will allow the script to run independently without
requiring the caller to pre-export the token.

In `@vault/scripts/vault-prerequisites.sh`:
- Around line 1-4: The vault-prerequisites.sh script documents that it should be
run as root but lacks a programmatic guard to enforce this requirement. Add an
EUID check immediately after the "set -euo pipefail" line to verify that the
effective user ID is 0 (root). If the check fails, the script should print a
clear error message and exit with a non-zero status code. This prevents delayed,
confusing failures during package installation attempts and provides immediate
feedback to users running the script without sufficient privileges.
- Around line 21-33: The script unconditionally installs yum-utils in the
initial package installation step, but yum-config-manager used later in the
HashiCorp repository configuration requires different packages depending on the
package manager: yum-utils for YUM-based systems and dnf-plugins-core for
DNF-based systems. Modify the package installation logic at the beginning of the
script to conditionally install the appropriate package manager utilities based
on the value of PKG_MGR variable, ensuring that the correct plugin package is
installed to support yum-config-manager (or its DNF equivalent) before the
HashiCorp repository configuration attempt.

In `@vault/scripts/vault-setup.sh`:
- Around line 137-149: The VAULT_TOKEN is sourced from the .vault-init.json file
in the conditional block (checking the VAULT_HOME path), but this file is
deleted later in the script around line 215. This causes subsequent reruns of
the script to fail authentication during phases 7-9 operations since there is no
automatic token source available. Either move the deletion of the
.vault-init.json file to the very end of the script after all Vault operations
complete, or add a validation check before phases 7-9 that ensures VAULT_TOKEN
is set and provides clear error messaging if it is not available, guiding the
user to provide authentication credentials.
- Around line 182-190: The JWT role for osac-e2e in the vault write
auth/jwt/role/osac-e2e command currently only restricts access based on
repository_owner, allowing any repository in the org to request tokens with the
osac-e2e policy. Enhance the bound_claims object to include additional
restrictive claims such as explicit repository name, ref, and workflow
constraints to limit token issuance to only the specific repositories and
workflows that actually need the osac-e2e policy, following the principle of
least privilege.
- Around line 87-92: The for loop that retries checking the Vault status field
'type' has no failure handling—if all 30 iterations complete without the
condition being met, the script silently continues to the next part. After the
for loop ends (after the done statement), add an explicit check that verifies
the vault_status_field 'type' is still valid, and if it is empty or null, echo
an error message and exit with a non-zero status code to immediately surface the
root cause of Vault being unreachable rather than allowing the setup to continue
with a failed Vault connection.

In `@vault/scripts/vault-unseal.sh`:
- Line 63: The hardcoded threshold value of 2 in vault-unseal.sh duplicates the
vault configuration defined in vault-setup.sh phase 4, which creates a
maintenance risk if these values diverge. Extract the threshold value into a
shared configuration file that both vault-unseal.sh and vault-setup.sh can
source, or if that's not feasible, add a clear comment above the threshold
assignment that explicitly references the corresponding threshold definition in
vault-setup.sh phase 4 to document the dependency and keep them synchronized.
- Around line 56-59: The script exits with code 0 when the UNSEAL_KEYS_FILE is
not found, which signals success even though Vault remains sealed and unsealing
failed. This masks critical failures when the keys file is accidentally deleted
or missing. Change the exit code from 0 to 1 in the conditional block that
checks for the missing unseal keys file (the if statement checking [[ ! -f
"${UNSEAL_KEYS_FILE}" ]]), so the script properly signals failure when required
keys are missing and cannot proceed with unsealing.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 9c1ca0b3-1f16-4596-8183-7c30d17d670b

📥 Commits

Reviewing files that changed from the base of the PR and between 5c2b986 and e8c1a3c.

📒 Files selected for processing (10)
  • vault/config/vault.hcl
  • vault/quadlet/vault.container
  • vault/scripts/vault-backup.sh
  • vault/scripts/vault-health-check.sh
  • vault/scripts/vault-prerequisites.sh
  • vault/scripts/vault-setup.sh
  • vault/scripts/vault-unseal.sh
  • vault/vault-backup.service
  • vault/vault-backup.timer
  • vault/vault-unseal.service

Comment thread vault/scripts/vault-backup.sh
Comment thread vault/scripts/vault-health-check.sh
Comment thread vault/scripts/vault-prerequisites.sh
Comment thread vault/scripts/vault-prerequisites.sh Outdated
Comment thread vault/scripts/vault-setup.sh
Comment thread vault/scripts/vault-setup.sh
Comment on lines +182 to +190
vault write auth/jwt/role/osac-e2e - <<'ROLE'
{
"role_type": "jwt",
"bound_claims": {"repository_owner": "osac-project"},
"user_claim": "repository",
"token_policies": ["osac-e2e"],
"token_ttl": "60m",
"token_max_ttl": "120m"
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Tighten JWT role claims to least privilege.

The role is only bound to repository_owner, so any repository in that org can request a Vault token with osac-e2e policy. Restrict to explicit repository/ref/workflow claims.

Proposed fix
 {
   "role_type": "jwt",
-  "bound_claims": {"repository_owner": "osac-project"},
+  "bound_claims_type": "glob",
+  "bound_claims": {
+    "repository": "osac-project/osac-repo-name",
+    "ref": "refs/heads/main"
+  },
   "user_claim": "repository",
   "token_policies": ["osac-e2e"],
   "token_ttl": "60m",
   "token_max_ttl": "120m"
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@vault/scripts/vault-setup.sh` around lines 182 - 190, The JWT role for
osac-e2e in the vault write auth/jwt/role/osac-e2e command currently only
restricts access based on repository_owner, allowing any repository in the org
to request tokens with the osac-e2e policy. Enhance the bound_claims object to
include additional restrictive claims such as explicit repository name, ref, and
workflow constraints to limit token issuance to only the specific repositories
and workflows that actually need the osac-e2e policy, following the principle of
least privilege.

Comment thread vault/scripts/vault-unseal.sh Outdated
Comment thread vault/scripts/vault-unseal.sh
@eliorerz
eliorerz force-pushed the OSAC-1568-vault-infra branch from 6d663ea to ade5688 Compare June 18, 2026 16:54
@eliorerz eliorerz changed the title OSAC-1568: Add local Vault infrastructure for CI e2e test secrets OSAC-1568: Add local Vault infrastructure and e2e workflow Jun 18, 2026
@eliorerz
eliorerz force-pushed the OSAC-1568-vault-infra branch from ade5688 to 6a091bf Compare June 18, 2026 17:11

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

♻️ Duplicate comments (1)
vault/scripts/vault-setup.sh (1)

194-207: 🧹 Nitpick | 🔵 Trivial | 💤 Low value

Consider tightening JWT role claims to explicit repository and ref patterns.

The role binds to repository_owner and environment, which restricts access to workflows in the osac-project org running in the e2e-test environment. However, this still allows any repository in the org to obtain tokens. For least-privilege security, consider adding explicit repository and ref claim restrictions.

🔒 Example with tighter claim restrictions
   "bound_claims": {
     "repository_owner": "osac-project",
+    "repository": "osac-project/osac-test-infra",
+    "ref": "refs/heads/main"
   },

Or use glob patterns if multiple repos/refs need access:

+  "bound_claims_type": "glob",
   "bound_claims": {
-    "repository_owner": "osac-project",
-    "environment": "e2e-test"
+    "repository": "osac-project/*",
+    "ref": "refs/heads/main"
   },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@vault/scripts/vault-setup.sh` around lines 194 - 207, The JWT role
configuration for osac-e2e in the vault write command currently has too
permissive bound_claims that only restrict by repository_owner and environment,
allowing any repository in the org to obtain tokens. Add explicit repository and
ref claim restrictions to the bound_claims object within the osac-e2e role
configuration to enforce least-privilege access, either by specifying exact
repository names and git refs or by using glob patterns if multiple repos or
refs need access.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/e2e-vault-secrets.yml:
- Around line 69-94: The "Run E2E tests" step directly expands workflow inputs
(inputs.test-suite, inputs.test-filter, inputs.namespace, inputs.vm-template)
within the shell script context, which creates a command injection
vulnerability. Move these workflow inputs to an env block at the step level,
then reference them as shell environment variables (like $TEST_SUITE_INPUT,
$TEST_FILTER_INPUT, $NAMESPACE_INPUT, $VM_TEMPLATE_INPUT) instead of using the
GitHub Actions template syntax directly in the run script. This ensures proper
variable handling and prevents shell metacharacter injection.
- Around line 61-67: The shell commands directly interpolate Vault secrets using
template expansion syntax which is vulnerable to shell injection if secrets
contain special characters like single quotes. To fix this, pass the secrets as
environment variables to the step instead of directly expanding them inline,
then reference those environment variables in the shell commands. This prevents
the shell from interpreting any special characters within the secret values.
Apply this approach to both the PULL_SECRET and AAP_LICENSE secret outputs in
the "Write secrets to files" step.
- Line 50: In the `.github/workflows/e2e-vault-secrets.yml` file, replace the
`hashicorp/vault-action@v4` reference with the pinned commit SHA
`hashicorp/vault-action@892a26828f195e65540a40b4768ae4571f51ebfc` to ensure the
workflow uses an immutable commit SHA instead of a mutable tag, which prevents
potential supply-chain attacks from tag modifications.

In `@vault/quadlet/vault.container`:
- Around line 33-34: The AutoUpdate=registry directive in the vault.container
file causes automatic updates to new minor/patch versions within the 2.0.x line,
introducing instability for CI runners. Replace the floating version reference
in the Image specification with an exact Vault version tag (for example, using
2.0.2 instead of 2.0) and either remove the AutoUpdate=registry line or change
it to a more restrictive auto-update policy that maintains CI stability by
requiring deliberate version updates after testing.

In `@vault/scripts/vault-prerequisites.sh`:
- Around line 43-47: The code at lines 43-47 is redundantly checking for package
managers using command -v checks for dnf-3 and dnf, but the package manager was
already detected and stored in the PKG_MGR variable at lines 14-21. Remove the
conditional logic that checks for dnf-3 and dnf commands, and instead directly
use the PKG_MGR variable to decide whether to use dnf config-manager or
yum-config-manager for adding the repository. This leverages the existing
package manager detection and eliminates the check for the non-standard dnf-3
command.

---

Duplicate comments:
In `@vault/scripts/vault-setup.sh`:
- Around line 194-207: The JWT role configuration for osac-e2e in the vault
write command currently has too permissive bound_claims that only restrict by
repository_owner and environment, allowing any repository in the org to obtain
tokens. Add explicit repository and ref claim restrictions to the bound_claims
object within the osac-e2e role configuration to enforce least-privilege access,
either by specifying exact repository names and git refs or by using glob
patterns if multiple repos or refs need access.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: c2cf55d5-4fb5-4311-87b5-d7f2489189f4

📥 Commits

Reviewing files that changed from the base of the PR and between e8c1a3c and 6a091bf.

📒 Files selected for processing (12)
  • .github/workflows/e2e-test.yml
  • .github/workflows/e2e-vault-secrets.yml
  • vault/config/vault.hcl
  • vault/quadlet/vault.container
  • vault/scripts/vault-backup.sh
  • vault/scripts/vault-health-check.sh
  • vault/scripts/vault-prerequisites.sh
  • vault/scripts/vault-setup.sh
  • vault/scripts/vault-unseal.sh
  • vault/vault-backup.service
  • vault/vault-backup.timer
  • vault/vault-unseal.service

Comment thread .github/workflows/e2e-vault-secrets.yml Outdated
Comment thread .github/workflows/e2e-vault-secrets.yml
Comment thread .github/workflows/e2e-vault-secrets.yml
Comment thread vault/quadlet/vault.container Outdated
Comment thread vault/scripts/vault-prerequisites.sh Outdated
@eliorerz
eliorerz force-pushed the OSAC-1568-vault-infra branch from 6a091bf to 74086e1 Compare June 18, 2026 19:45
@omer-vishlitzky

Copy link
Copy Markdown
Contributor

/lgtm
/approve

@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eliorerz, omer-vishlitzky

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [eliorerz,omer-vishlitzky]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (2)
.github/workflows/e2e-vault-secrets.yml (1)

81-103: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Quote pytest args via an array; current expansion is unsafe and can misparse filters.

On Line 102, pytest ${TEST_ARGS} ${FILTER_ARGS} performs word splitting on input-derived values. This breaks valid filters like foo and bar and allows unintended extra pytest flags to be injected via test-filter. Build pytest args as an array and expand with "${PYTEST_ARGS[@]}".

Suggested fix
       - name: Run E2E tests
         id: test
         env:
           TEST_SUITE: ${{ inputs.test-suite }}
           TEST_FILTER: ${{ inputs.test-filter }}
           OSAC_NAMESPACE: ${{ inputs.namespace }}
           OSAC_VM_TEMPLATE: ${{ inputs.vm-template }}
           TEST_IMAGE: ${{ inputs.test-image }}
         run: |
-          TEST_ARGS="tests/"
+          TEST_PATH="tests/"
           if [ -n "${TEST_SUITE}" ]; then
-            TEST_ARGS="tests/${TEST_SUITE}/"
+            TEST_PATH="tests/${TEST_SUITE}/"
           fi
 
-          FILTER_ARGS=""
+          PYTEST_ARGS=("${TEST_PATH}")
           if [ -n "${TEST_FILTER}" ]; then
-            FILTER_ARGS="-k ${TEST_FILTER}"
+            PYTEST_ARGS+=("-k" "${TEST_FILTER}")
           fi
 
           podman run --rm \
             --network host \
             -v "${HOME}/.kube/config:/root/.kube/config:ro,Z" \
             -v "${RUNNER_TEMP}/pull-secret.json:/root/pull-secret:ro,Z" \
             -v "${RUNNER_TEMP}/aap-license.zip:/root/aap-license.zip:ro,Z" \
             -e KUBECONFIG=/root/.kube/config \
             -e OSAC_VM_KUBECONFIG=/root/.kube/config \
             -e OSAC_NAMESPACE="${OSAC_NAMESPACE}" \
             -e OSAC_VM_TEMPLATE="${OSAC_VM_TEMPLATE}" \
             -e OSAC_PULL_SECRET_PATH=/root/pull-secret \
             "${TEST_IMAGE}" \
-            pytest ${TEST_ARGS} ${FILTER_ARGS} \
+            pytest "${PYTEST_ARGS[@]}" \
               --junitxml=/tmp/junit.xml -v

As per coding guidelines, "Flag injection vectors: SQL string concatenation, shell=True with user input, eval/exec on untrusted data, ... os.system with variables, and dangerouslySetInnerHTML with user data", this input-derived shell argument expansion should not be unquoted.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/e2e-vault-secrets.yml around lines 81 - 103, The pytest
command on line 102 uses unquoted variable expansion with TEST_ARGS and
FILTER_ARGS which is unsafe and can cause word splitting and command injection.
Convert the pytest arguments to a bash array named PYTEST_ARGS, initialize it
with the TEST_ARGS value, conditionally add FILTER_ARGS to the array if
TEST_FILTER is set, and then invoke pytest with proper array expansion using
"${PYTEST_ARGS[@]}" instead of the current unquoted ${TEST_ARGS} ${FILTER_ARGS}
pattern to ensure safe argument handling.

Sources: Coding guidelines, Linters/SAST tools

vault/scripts/vault-setup.sh (1)

194-206: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

JWT role binding is still broader than needed for least privilege.

Lines 197-200 allow any repo under osac-project with matching environment claim to request this policy. Narrow claims to the exact repository (and optionally workflow) that should read osac-e2e secrets.

🔒 Proposed tightening
 {
   "role_type": "jwt",
   "bound_claims": {
-    "repository_owner": "osac-project",
+    "repository": "osac-project/osac-test-infra",
     "environment": "e2e-test"
   },
   "bound_audiences": ["https://github.com/osac-project"],
   "user_claim": "repository",
   "token_policies": ["osac-e2e"],
   "token_ttl": "60m",
   "token_max_ttl": "120m"
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@vault/scripts/vault-setup.sh` around lines 194 - 206, The bound_claims in the
JWT role binding for osac-e2e is too permissive and violates the principle of
least privilege. Currently, it only checks repository_owner and environment,
allowing any repository under osac-project with the matching environment claim
to request access. Add a more specific constraint to the bound_claims by
including the exact repository name (and optionally the workflow name) that
should be permitted to authenticate with this role, so that only the intended
repository can request the osac-e2e policy.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@vault/scripts/vault-setup.sh`:
- Around line 49-53: The chown command that sets ownership on the VAULT_HOME
data and logs directories requires root privileges but the script runs in
user-session mode with set -e enabled, causing it to terminate the script if
executed by a non-root user. Make the ownership adjustment rootless-safe by
either appending || true to the chown line to allow it to fail gracefully
without aborting the script, or by conditionally executing it only when the user
has root privileges using a check like if [[ "$EUID" -eq 0 ]]; then before the
chown command.

In `@vault/vault-backup.timer`:
- Around line 7-9: The OnCalendar schedule in the [Timer] section is using the
host's local timezone instead of explicitly specifying UTC, which causes the
backup job to run at different local times depending on the runner's timezone
configuration. Add Timezone=UTC as a new line in the [Timer] section near the
existing OnCalendar and RandomizedDelaySec directives to ensure the 03:00:00
schedule runs consistently at UTC across all environments.

---

Duplicate comments:
In @.github/workflows/e2e-vault-secrets.yml:
- Around line 81-103: The pytest command on line 102 uses unquoted variable
expansion with TEST_ARGS and FILTER_ARGS which is unsafe and can cause word
splitting and command injection. Convert the pytest arguments to a bash array
named PYTEST_ARGS, initialize it with the TEST_ARGS value, conditionally add
FILTER_ARGS to the array if TEST_FILTER is set, and then invoke pytest with
proper array expansion using "${PYTEST_ARGS[@]}" instead of the current unquoted
${TEST_ARGS} ${FILTER_ARGS} pattern to ensure safe argument handling.

In `@vault/scripts/vault-setup.sh`:
- Around line 194-206: The bound_claims in the JWT role binding for osac-e2e is
too permissive and violates the principle of least privilege. Currently, it only
checks repository_owner and environment, allowing any repository under
osac-project with the matching environment claim to request access. Add a more
specific constraint to the bound_claims by including the exact repository name
(and optionally the workflow name) that should be permitted to authenticate with
this role, so that only the intended repository can request the osac-e2e policy.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: osac-project/coderabbit/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 41402ffd-8d91-47fc-85e6-dc50f947f66f

📥 Commits

Reviewing files that changed from the base of the PR and between 6a091bf and 74086e1.

📒 Files selected for processing (12)
  • .github/workflows/e2e-test.yml
  • .github/workflows/e2e-vault-secrets.yml
  • vault/config/vault.hcl
  • vault/quadlet/vault.container
  • vault/scripts/vault-backup.sh
  • vault/scripts/vault-health-check.sh
  • vault/scripts/vault-prerequisites.sh
  • vault/scripts/vault-setup.sh
  • vault/scripts/vault-unseal.sh
  • vault/vault-backup.service
  • vault/vault-backup.timer
  • vault/vault-unseal.service

Comment thread vault/scripts/vault-setup.sh Outdated
Comment thread vault/vault-backup.timer
Deploy a team-owned HashiCorp Vault instance for CI e2e test secrets:
- Vault server config (file storage, localhost-only, no TLS)
- Podman Quadlet unit file for systemd-managed container
- Setup, unseal, backup, health-check, and prerequisites scripts
- Systemd services for auto-unseal and daily backups
- JWT auth configured for GitHub OIDC with bound claims
  (repository_owner + environment) and bound audiences

Add reusable GitHub Actions e2e workflow:
- vault-action@v4 retrieves secrets via OIDC JWT auth
- Caller workflow triggers on pull_request and workflow_dispatch
- Runs on self-hosted osac-ci runners with e2e-test environment
@eliorerz
eliorerz force-pushed the OSAC-1568-vault-infra branch from 74086e1 to e577b14 Compare June 18, 2026 20:04
@openshift-ci openshift-ci Bot removed the lgtm label Jun 18, 2026
@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown

New changes are detected. LGTM label has been removed.

@eliorerz eliorerz added the lgtm label Jun 18, 2026
@eliorerz

Copy link
Copy Markdown
Contributor Author

/override ci/prow/e2e-vmaas

@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown

@eliorerz: Overrode contexts on behalf of eliorerz: ci/prow/e2e-vmaas

Details

In response to this:

/override ci/prow/e2e-vmaas

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@eliorerz
eliorerz merged commit 3950e95 into osac-project:main Jun 18, 2026
4 of 5 checks passed
@eliorerz
eliorerz deleted the OSAC-1568-vault-infra branch June 18, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants