ci(#5733): pin podman to 4.x to avoid crun incompatibility

[waynesun09]

Summary

GitHub Actions runner images have begun pre-installing podman 5.8.4, which requires crun >= 1.15. The runner's system crun stays at 1.14.1, so sandbox creation fails deterministically with crun: unknown version specified: OCI runtime error, killing agent runs before the agent starts.

Related Issue

Closes #5733

Changes

• Add .github/scripts/install-podman.sh (matching the existing install-openshell.sh convention): writes an apt preferences file pinning podman to the 4.* series with Pin-Priority: 1001 (authorizes downgrading an already-installed newer version), runs apt-get install -y --allow-downgrades podman, then asserts the installed version is actually 4.x — fails loudly with a clear error if the pin didn't take, instead of silently leaving 5.x in place and resurfacing later as a confusing crun error.
• action.yml and .github/workflows/functional-tests.yml: replace the duplicated inline "Install Podman" step with a call to the shared script.
• apt-get install -y podman alone does not downgrade an already-installed newer version, which is why the plain install left podman 5.8.4 in place — this is why both the pin's Pin-Priority and --allow-downgrades are needed together.

Testing

• make lint passes (staged changes, including shellcheck on the new script, the GitHub Actions workflow linter, and pinact)
• Tests added/updated for new or modified logic — not applicable, no Go code changed
• Manually validated via workflow_dispatch on this branch (gh workflow run functional-tests.yml --ref fix-5733-pin-podman-version): confirmed the new script executes (this PR's pull_request_target-triggered check does not exercise branch changes, since GitHub resolves that trigger's workflow YAML from main) and that podman resolves to 4.9.3+ds1-1ubuntu0.2 and passes the version assertion. Did not hit a runner with podman 5.x pre-installed during validation (that variant is still a partial rollout per Pin podman version in action.yml to prevent runner image upgrade breakage #5733), so the downgrade path itself is not yet empirically exercised — the version assertion is the safety net for that case.
• make e2e-test / full dogfood install of action.yml against a test org not run (requires live pool orgs + mint OIDC per docs/guides/dev/testing-workflows.md); reviewed as low-risk since action.yml's step is now the identical shared script already validated above

Checklist

• PR title follows Conventional Commits (ci(#5733), not fix(ci) — CI-only fix is ci, per the forbidden type/scope table in COMMITS.md; issue number used as scope per convention)
• Commits are signed off (DCO)
• I wrote this contribution myself and can explain all changes in it

[qodo-code-review]

PR Summary by Qodo

Pin Podman to 4.x in CI to avoid crun incompatibility on GitHub runners

🐞 Bug fix ⚙️ Configuration changes 🕐 10-20 Minutes

AI Description

• Pin podman to the 4.* series on GitHub-hosted runners to avoid crun runtime failures.
• Allow apt downgrades so preinstalled Podman 5.x gets replaced deterministically.
• Apply the same fix in both the reusable action and functional-test workflow.

Diagram

graph TD
  A["GitHub Actions runner"] --> B["Write apt pin: podman 4.*"] --> C["apt-get update"] --> D["Install podman (allow downgrade)"] --> E["Podman uses system crun"] --> F["Sandbox creation succeeds"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Upgrade/install newer crun (>= 1.15) instead of pinning Podman

• ➕ Keeps CI on the latest Podman major version
• ➕ Avoids relying on apt pinning behavior across runner image changes
• ➖ More invasive: introduces a new system component install path (and potential repo/source build complexity)
• ➖ Higher risk of breaking rootless/container setup on runners

2. Pin the runner image / use a containerized job environment

• ➕ Stabilizes the entire toolchain (podman/crun) against upstream runner drift
• ➕ Reproducible builds for CI
• ➖ May increase maintenance (image updates, security patch cadence)
• ➖ Can be harder to integrate with existing workflow assumptions (privileges, sockets, caching)

3. Install a specific Podman package version explicitly (exact 4.x version)

• ➕ Fully deterministic resolution (no wildcard semantics)
• ➕ Easier to audit and reproduce
• ➖ Requires periodic version bumps
• ➖ Can fail when the exact version disappears from apt repos/mirrors

Recommendation: The current apt-preferences pin to 4.* plus --allow-downgrades is the lowest-risk, fastest mitigation for runner drift and matches the documented failure mode (Podman 5.x requiring newer crun). Consider moving to an explicitly pinned Podman 4.x version (or a pinned runner image) later if runner variability continues.

Files changed (2) +12 / -2

Other (2) +12 / -2

functional-tests.ymlPin Podman to 4.x (with downgrade) in functional test workflow +6/-1

Pin Podman to 4.x (with downgrade) in functional test workflow

• Adds an apt preferences file to pin 'podman' to the 4.* series with a high priority to permit downgrades. Updates the install command to use '--allow-downgrades' so preinstalled Podman 5.x is replaced, preventing sandbox creation failures due to old 'crun'.

.github/workflows/functional-tests.yml

action.ymlPin Podman to 4.x (with downgrade) in the GitHub Action runtime setup +6/-1

Pin Podman to 4.x (with downgrade) in the GitHub Action runtime setup

• Mirrors the workflow fix inside the action by writing '/etc/apt/preferences.d/podman-pin' and installing Podman with '--allow-downgrades'. Ensures consumers of the action get a Podman/crun-compatible setup even when runners ship Podman 5.x by default.

action.yml

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 6:43 PM UTC · Completed 6:56 PM UTC
Commit: e8b5818 · View workflow run →

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

1. No preferences dir creation ✓ Resolved 🐞 Bug ☼ Reliability

Description

The Podman pin is written to /etc/apt/preferences.d/podman-pin without ensuring
/etc/apt/preferences.d exists; on minimal/custom runner images this will fail and stop the job
before Podman is installed/downgraded. This is especially relevant for the reusable composite
action, which may be run outside standard GitHub-hosted Ubuntu images.

Code

action.yml[265]

+        printf 'Package: podman\nPin: version 4.*\nPin-Priority: 1001\n' | sudo tee /etc/apt/preferences.d/podman-pin >/dev/null

Relevance

●●● Strong

Creating required dirs before writing system config is a straightforward robustness fix; similar
hardening changes were accepted.

PR-#1688

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Both modified install steps write directly into /etc/apt/preferences.d/... with sudo tee and do
not create the parent directory first, so the step relies on the directory being present in the
execution environment.

action.yml[258-268]
.github/workflows/functional-tests.yml[186-195]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The Podman install steps write an apt preferences file to `/etc/apt/preferences.d/podman-pin` but do not ensure the parent directory exists, which can cause `tee` to fail on some environments.

### Issue Context
This affects both the reusable composite action and the functional-tests workflow. While GitHub-hosted Ubuntu runners typically have `/etc/apt/preferences.d`, adding an explicit directory create is a safe no-op when it already exists and prevents brittle failures on minimal/custom images.

### Fix Focus Areas
- action.yml[258-268]
- .github/workflows/functional-tests.yml[186-195]

### Suggested change
Add a directory creation line before `tee`, e.g.:
```bash
sudo install -d /etc/apt/preferences.d
# or: sudo mkdir -p /etc/apt/preferences.d
printf 'Package: podman\nPin: version 4.*\nPin-Priority: 1001\n' | sudo tee /etc/apt/preferences.d/podman-pin >/dev/null
```

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. No downgrade verification log ✓ Resolved 🐞 Bug ◔ Observability

Description

The functional-tests workflow installs/downgrades Podman but does not log or assert the resulting
Podman version, making it harder to detect and debug cases where the pin/downgrade did not take
effect. This can delay diagnosis if runner images change again and the job later fails during
sandbox creation.

Code

.github/workflows/functional-tests.yml[195]

+          sudo apt-get install -y --allow-downgrades podman

Relevance

●●● Strong

Team often accepts small CI/workflow hardening for debuggability; adding podman --version is
low-risk.

PR-#1682

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The functional-tests workflow’s Podman install step ends immediately after the apt install, while
the composite action explicitly prints podman --version, demonstrating the diagnostic gap is only
in the workflow.

.github/workflows/functional-tests.yml[186-196]
action.yml[258-269]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
After installing Podman in `.github/workflows/functional-tests.yml`, the workflow does not print or validate the installed Podman version.

### Issue Context
The composite action already runs `podman --version` after installation, but the functional-tests workflow does not. Adding a version log (and optionally a simple assertion that major version is 4) makes it immediately obvious whether the apt pin/downgrade worked.

### Fix Focus Areas
- .github/workflows/functional-tests.yml[186-196]

### Suggested change
Add:
```bash
podman --version
# Optional enforcement:
podman --version | grep -qE '^podman version 4\.' || { echo "::error::Expected podman 4.x"; exit 1; }
```
right after the `apt-get install` line.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

_{To customize comments, go to the Qodo configuration screen, or learn more in the docs.}

[fullsend-ai-review]

Review

Findings

Medium

• [protected-path] .github/scripts/install-podman.sh, .github/workflows/functional-tests.yml — This PR adds and modifies files under the .github/ protected path. The changes are well-justified by linked issue Pin podman version in action.yml to prevent runner image upgrade breakage #5733 (pinning podman to 4.x to fix a deterministic crun incompatibility on newer runner images), but human approval is always required for protected-path changes regardless of context.

Low

• [injection] .github/scripts/install-podman.sh:31 — The ::error:: workflow command and the final echo (line 35) interpolate ${installed_version} (from podman --version) without sanitizing for GHA workflow command sequences. The value originates from a system-installed binary and is not directly attacker-controlled, making this a defense-in-depth observation rather than a practical vulnerability.

Previous run

Review

Findings

Medium

• [protected-path] .github/workflows/functional-tests.yml — This PR modifies a file under the .github/ protected path. The change is well-justified by linked issue Pin podman version in action.yml to prevent runner image upgrade breakage #5733 (pinning podman to 4.x to fix a deterministic crun incompatibility on newer runner images), but human approval is always required for protected-path changes regardless of context.

Low

• [architectural-precedent] action.yml:262 — Introduces apt version pinning (Pin-Priority: 1001 + --allow-downgrades) as a new CI dependency management pattern. The inline comments explain the rationale and link the tracking issue, mitigating the documentation gap. Consider filing a follow-up issue to remove the pin once podman 5.x/crun compatibility is resolved in runner images.

• [issue-reference-convention] .github/workflows/functional-tests.yml:192 — Issue reference uses full repo path fullsend-ai/fullsend#5733 but the established codebase convention is short-form #5733 for same-repo references.

• [issue-reference-convention] action.yml:264 — Same convention: change See fullsend-ai/fullsend#5733. to See #5733.

---

Labels: PR modifies CI workflow files (.github/) and podman sandbox infrastructure

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:02 PM UTC · Ended 7:07 PM UTC
Commit: f44dcd0 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ❌ Failure · Started 7:09 PM UTC · Completed 7:10 PM UTC
Commit: c5156f6 · View workflow run →

[fullsend-ai-review]

🤖 Review · ⚠️ Cancelled · Started 7:13 PM UTC · Ended 7:20 PM UTC
Commit: 39bd970 · View workflow run →

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 7:22 PM UTC · Completed 7:34 PM UTC
Commit: 1ebacee · View workflow run →

[fullsend-ai-review]

See the review comment for full details.

[fullsend-ai-review]

[low] injection

The ::error:: workflow command and the final echo (line 35) interpolate ${installed_version} (from podman --version) without sanitizing for GHA workflow command sequences. The value originates from a system-installed binary and is not directly attacker-controlled, making this a defense-in-depth observation rather than a practical vulnerability.

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 7:46 PM UTC · Completed 7:58 PM UTC
Commit: 1ebacee · View workflow run →

[fullsend-ai-retro]

Retro: PR #5738 — pin podman to 4.x to avoid crun incompatibility

Outcome: Human-authored PR merged in ~1 hour. Code agent failed; triage and review agents worked. No novel proposals — all improvement opportunities are tracked by existing open issues.

Timeline

1. 16:40 UTC — Issue #5733 opened (podman 5.x / crun incompatibility breaking sandbox creation).
2. 16:41–16:45 UTC — Triage agent (run 30471864372) succeeded. Correctly classified as Critical and recommended podman version pinning — aligned with the eventual fix.
3. 17:39 UTC — /fs-code triggered by ralphbean.
4. 17:39–17:40 UTC — Code agent (run 30476323037) failed within seconds. make go-build hit a transient Go module proxy HTTP/2 stream error (golang.org/x/text@v0.37.0: stream error: INTERNAL_ERROR). The agent never started. The finalize step also failed (fullsend: command not found) because the binary wasn't built, so no status comment was posted on the issue.
5. 18:41 UTC — waynesun09 manually created PR #5738.
6. 18:41–18:56 UTC — First review (run 30481095420) succeeded. Both qodo-code-review and fullsend-ai-review posted findings. ifireball approved at 18:45 (before either bot finished).
7. 19:07–19:10 UTC — Third review (run 30483017729) failed — runner got podman 5.8.4, hitting the exact bug the PR was fixing. Two other review runs were cancelled (superseded by force pushes).
8. 19:20–19:34 UTC — Final review (run 30483976041) succeeded on updated code.
9. 19:34 UTC — Added to merge queue. 19:44 UTC — Merged.

What went well

• Triage accuracy: The triage agent correctly identified the root cause and recommended the approach the author ultimately implemented.
• Complementary bot coverage: qodo caught a genuine reliability bug (missing mkdir -p /etc/apt/preferences.d before tee). fullsend-ai-review caught convention issues (short-form issue references) and surfaced the protected-path policy gate. Together they provided comprehensive coverage.
• Good re-review behavior: The final fullsend-ai-review run correctly dropped the two findings the author had addressed (architectural-precedent, issue-reference-convention) and retained only the still-applicable ones (protected-path, injection defense-in-depth).
• Well-qualified findings: The review agent's injection finding correctly noted the input originates from a system binary and is not attacker-controlled — honest uncertainty rather than false alarm.

Evidence for existing issues

• #5606 and #934 (auto-retry on pre-agent infra failures / fast failures): Code agent run 30476323037 failed at make go-build due to a transient Go module proxy HTTP/2 error. The agent exited with code 1 within seconds, with no output. Auto-retry at either the dispatch or build-step level would have likely succeeded.
• #3635 and #4323 (post failure status comment when agent fails before starting): The finalize step's fullsend reconcile-status errored with fullsend: command not found because the binary wasn't built. Issue Pin podman version in action.yml to prevent runner image upgrade breakage #5733 received no feedback about the code agent failure, requiring the human to manually inspect the workflow run.
• agents#131 and agents#298 (shell script pitfall checks / side-effect preconditions): The review agent missed the mkdir -p bug that qodo caught — writing to /etc/apt/preferences.d/podman-pin without ensuring the directory exists is a standard shell scripting pitfall and a side-effect precondition violation.