Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,7 @@ script-test:
$(call run-timed,bash internal/scaffold/fullsend-repo/scripts/pre-fetch-prior-review-test.sh)
$(call run-timed,bash internal/scaffold/fullsend-repo/.github/scripts/setup-agent-env-test.sh)
$(call run-timed,bash hack/gitlab-runner-vm/executor/prepare_validation_test.sh)
$(call run-timed,bash hack/gitlab-runner-vm/executor/gateway_test.sh)
$(call run-timed,bash hack/gitlab-runner-vm/lib_test.sh)
$(call run-timed,bash internal/runtime/kill_stray_processes_test.sh)
$(call run-timed,python3 skills/topissues/scripts/topissues_test.py)
Expand Down
14 changes: 9 additions & 5 deletions docs/guides/dev/cli-internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -412,11 +412,14 @@ Vendoring commit messages use title + body (upload and stale delete). `github st
│ └──────┬───────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ ImportProfile() │ Import openshell provider profiles │
│ │ ImportProfileVerified() │ Import openshell provider profiles │
│ │ │ (from resolved openshell.profiles; │
│ │ │ on GitLab, a fullsend-gitlab-forge │
│ │ │ profile is auto-generated from the │
│ │ │ forge host URL — see #6615) │
│ │ │ drops the os.TempDir() content cache │
│ │ │ and confirms the gateway lists each │
│ │ │ profile — see #7218. On GitLab, a │
│ │ │ fullsend-gitlab-forge profile is │
│ │ │ auto-generated from the forge host URL │
│ │ │ — see #6615) │
│ └──────┬───────────┘ │
│ ▼ │
│ ┌──────────────────┐ │
Expand Down Expand Up @@ -581,7 +584,8 @@ details, see [Agent runtimes](../../runtimes.md).
|-----------|------------|---------|
| `EnsureAvailable()` | Check `openshell` binary | Verify runtime available |
| `CheckGateway()` | `openshell gateway ...` | Start inference gateway |
| `ImportProfile()` | `openshell provider profile import ...` | Import openshell provider profile |
| `ImportProfile()` | `openshell provider profile import ...` | Import openshell provider profile (hash-cached) |
| `ImportProfileVerified()` | Forget cache → import → `list-profiles` | Same, then confirm the gateway lists it (#7218) |
| `EnsureProvider()` | `openshell provider ...` | Register model provider (bare-key form) |
| `Create()` | `openshell sandbox create --image ...` | Spin up container |
| `Exec()` | `openshell sandbox exec ...` | Run command in sandbox |
Expand Down
23 changes: 17 additions & 6 deletions hack/gitlab-runner-vm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,21 @@ fullsend agent jobs.
Each runner VM runs:
- **gitlab-runner** (custom executor) — receives CI jobs from GitLab
- **Podman** (rootless) — creates per-job containers
- **OpenShell gateway** — provides sandbox compute for fullsend agents
- **OpenShell gateway** — started per job in `prepare.sh`, torn down in
`cleanup.sh`. The VM does **not** keep a long-lived `systemd --user`
gateway: that accumulated a stale profile registry, a baked-in OpenShell
version, and leaked sandboxes (#7218). Each job recreates the gateway
with an empty registry, matching GitHub Actions (fresh, version-matched
install, throw the runner away). The ~6 GB image cache stays warm; only
the CLI (~39 MB) and a version-skewed supervisor (~30 MB) are fetched
when the job's pin differs from the host.

Job containers use `--network=host` to reach the gateway. An OCI
`createRuntime` hook injects the host CA trust bundle into every container
(Debian and RHEL-family layouts) so jobs can verify internal TLS endpoints.
Gateway mTLS credentials are mounted read-only from the runner user's
OpenShell config.
OpenShell config. `prepare.sh` reaps leftover `openshell-*` / `openshell.managed`
containers from an abruptly-killed prior job before starting the new gateway.

This is a deployment variant of the container isolation model described in
ADR-0036. It uses a Podman custom executor instead of Docker/Kubernetes
Expand Down Expand Up @@ -117,7 +125,7 @@ GCP_PROJECT=my-gcp-project ./delete-gcp-vm.sh --list
| `RUNNER_IMAGE` | yes (create) | — | Image pre-pulled as a warm cache; jobs must still set `image:` in `.gitlab-ci.yml` |
| `RUNNER_TAG` | no | `fullsend-gitlab-runner` | Runner tag for job matching |
| `RUNNER_ACCESS_LEVEL` | no | `not_protected` | `ref_protected` restricts the runner to protected branches and tags, so merge-request pipelines on unprotected source refs never match and sit `pending`. In project mode, `not_protected` means any job on any branch of the project can run; in group mode, any tag-matched job on any branch of any project invited into the group tree runs on this VM (see Security below) |
| `OPENSHELL_VERSION` | no | from `.github/scripts/openshell-version.sh` | OpenShell version (Renovate-tracked) |
| `OPENSHELL_VERSION` | no | from `.github/scripts/openshell-version.sh` | OpenShell version used at **VM provision** (Renovate-tracked). Per-job `prepare.sh` re-reads the job image's `openshell --version` and upgrades the host CLI + supervisor when they differ, so a stale VM pin cannot stick. |
| `GITLAB_RUNNER_VERSION` | no | `19.2.1` | gitlab-runner version |
| `REGISTRATION_TOKEN` | setup only | — | GitLab runner registration token |

Expand Down Expand Up @@ -154,9 +162,10 @@ GCP_PROJECT=my-gcp-project ./delete-gcp-vm.sh --list
- `setup.sh` — standalone VM configuration (called by create-openshift-vm.sh / create-gcp-vm.sh)
- `gitlab-runner-version.sh` — central pin for the gitlab-runner version
- `vm.yaml` — KubeVirt VirtualMachine template (OpenShift only)
- `executor/prepare.sh` — custom executor prepare stage
- `executor/prepare.sh` — custom executor prepare stage (reaps leftover OpenShell containers, starts a per-job gateway matched to the job image's OpenShell version)
- `executor/run.sh` — custom executor run stage
- `executor/cleanup.sh` — custom executor cleanup stage
- `executor/cleanup.sh` — custom executor cleanup stage (stops the gateway, wipes `~/.local/state/openshell/{gateway,tls}`, reaps sandboxes)
- `executor/gateway.sh` — shared per-job gateway helpers sourced by prepare/cleanup

## Security notes

Expand All @@ -176,7 +185,9 @@ GCP_PROJECT=my-gcp-project ./delete-gcp-vm.sh --list
- Job containers share the host network namespace (`--network=host`) to reach
the OpenShell gateway. The gateway binds to `0.0.0.0` (required for the
Podman compute driver — sandbox containers register via
`host.containers.internal`). mTLS protects the endpoint.
`host.containers.internal`). mTLS protects the endpoint. The gateway process
itself is per-job: `prepare.sh` starts it against a wiped store, `cleanup.sh`
stops it. A leftover from a killed job is reaped at the next prepare.
- Job containers receive read-only access to the runner's gateway mTLS
credentials (`~/.config/openshell`). This is required for the fullsend
agent inside job containers to authenticate to the gateway. Credential
Expand Down
6 changes: 3 additions & 3 deletions hack/gitlab-runner-vm/create-gcp-vm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -284,7 +284,7 @@ for tool in gcloud python3 curl timeout sha256sum; do
fi
done
for _f in setup.sh create-gcp-vm.sh gitlab-runner-version.sh \
executor/job_id.sh executor/prepare.sh executor/run.sh executor/cleanup.sh; do
executor/job_id.sh executor/prepare.sh executor/run.sh executor/cleanup.sh executor/gateway.sh; do
if [ ! -f "${SCRIPT_DIR}/${_f}" ]; then
echo "ERROR: required file not found: ${SCRIPT_DIR}/${_f}" >&2
_missing=1
Expand Down Expand Up @@ -503,7 +503,7 @@ cp "${SCRIPT_DIR}/setup.sh" "${_stage_dir}/"
cp "${SCRIPT_DIR}/create-gcp-vm.sh" "${_stage_dir}/"
cp "${SCRIPT_DIR}/gitlab-runner-version.sh" "${_stage_dir}/"
mkdir -p "${_stage_dir}/executor"
for file in job_id.sh prepare.sh run.sh cleanup.sh; do
for file in job_id.sh prepare.sh run.sh cleanup.sh gateway.sh; do
cp "${SCRIPT_DIR}/executor/${file}" "${_stage_dir}/executor/"
done
mkdir -p "${_stage_dir}/.github/scripts"
Expand Down Expand Up @@ -538,7 +538,7 @@ echo "==> Verifying copied files"
verify_copied_files() {
{
(cd "${SCRIPT_DIR}" && sha256sum setup.sh create-gcp-vm.sh gitlab-runner-version.sh \
executor/job_id.sh executor/prepare.sh executor/run.sh executor/cleanup.sh)
executor/job_id.sh executor/prepare.sh executor/run.sh executor/cleanup.sh executor/gateway.sh)
(cd "${REPO_ROOT}/.github/scripts" \
&& sha256sum install-openshell.sh openshell-version.sh \
| sed 's| | .github/scripts/|')
Expand Down
6 changes: 3 additions & 3 deletions hack/gitlab-runner-vm/create-openshift-vm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ for tool in oc virtctl python3 curl timeout sha256sum; do
fi
done
for _f in setup.sh create-openshift-vm.sh vm.yaml gitlab-runner-version.sh \
executor/job_id.sh executor/prepare.sh executor/run.sh executor/cleanup.sh; do
executor/job_id.sh executor/prepare.sh executor/run.sh executor/cleanup.sh executor/gateway.sh; do
if [ ! -f "${SCRIPT_DIR}/${_f}" ]; then
echo "ERROR: required file not found: ${SCRIPT_DIR}/${_f}" >&2
_missing=1
Expand Down Expand Up @@ -402,7 +402,7 @@ for file in setup.sh create-openshift-vm.sh vm.yaml gitlab-runner-version.sh; do
-c "cat > ~/gitlab-runner-vm/${file}" < "${SCRIPT_DIR}/${file}"
done

for file in job_id.sh prepare.sh run.sh cleanup.sh; do
for file in job_id.sh prepare.sh run.sh cleanup.sh gateway.sh; do
virtctl -n "${NAMESPACE}" ssh "${VM_USER}"@vm/"${vm_name}" \
-t "-o StrictHostKeyChecking=no" -t "-o UserKnownHostsFile=/dev/null" \
-c "cat > ~/gitlab-runner-vm/executor/${file}" < "${SCRIPT_DIR}/executor/${file}"
Expand All @@ -425,7 +425,7 @@ virtctl -n "${NAMESPACE}" ssh "${VM_USER}"@vm/"${vm_name}" \
echo "==> Verifying copied files"
{
(cd "${SCRIPT_DIR}" && sha256sum setup.sh create-openshift-vm.sh vm.yaml gitlab-runner-version.sh \
executor/job_id.sh executor/prepare.sh executor/run.sh executor/cleanup.sh)
executor/job_id.sh executor/prepare.sh executor/run.sh executor/cleanup.sh executor/gateway.sh)
(cd "${REPO_ROOT}/.github/scripts" \
&& sha256sum install-openshell.sh openshell-version.sh \
| sed 's| | .github/scripts/|')
Expand Down
48 changes: 29 additions & 19 deletions hack/gitlab-runner-vm/executor/cleanup.sh
Original file line number Diff line number Diff line change
@@ -1,32 +1,42 @@
#!/usr/bin/env bash
# GitLab Runner custom executor — cleanup stage.
# Stops and removes the job container. Always succeeds.
# Stops and removes the job container and the per-job OpenShell gateway.
# Always succeeds.
# -e intentionally omitted — cleanup must not abort on individual failures.
set -uo pipefail

# shellcheck source=job_id.sh
source "$(dirname "${BASH_SOURCE[0]}")/job_id.sh"
JOB_ID=$(resolve_job_id) || {
echo "WARN: could not read job id from JOB_RESPONSE_FILE — nothing to clean up"
exit 0
}
# shellcheck source=gateway.sh
source "$(dirname "${BASH_SOURCE[0]}")/gateway.sh"

JOB_ID=$(resolve_job_id) || JOB_ID=""
STATE_DIR="${HOME}/.local/state/gitlab-runner"
STATE_FILE="${STATE_DIR}/container-${JOB_ID}"

if [ -f "${STATE_FILE}" ]; then
CONTAINER_NAME=$(cat "${STATE_FILE}")
if [[ "${CONTAINER_NAME}" =~ ^runner-[0-9]+$ ]]; then
echo "Cleaning up container: ${CONTAINER_NAME}"
podman stop --time 10 "${CONTAINER_NAME}" 2>/dev/null || true
podman rm -f "${CONTAINER_NAME}" 2>/dev/null || true
else
# Skip only the podman calls — the staging copy of the gateway mTLS
# material below must still be removed.
echo "WARN: state file holds an unexpected container name (${CONTAINER_NAME}) — not touching podman"
if [ -n "${JOB_ID}" ]; then
STATE_FILE="${STATE_DIR}/container-${JOB_ID}"

if [ -f "${STATE_FILE}" ]; then
CONTAINER_NAME=$(cat "${STATE_FILE}")
if [[ "${CONTAINER_NAME}" =~ ^runner-[0-9]+$ ]]; then
echo "Cleaning up container: ${CONTAINER_NAME}"
podman stop --time 10 "${CONTAINER_NAME}" 2>/dev/null || true
podman rm -f "${CONTAINER_NAME}" 2>/dev/null || true
else
# Skip only the podman calls — the staging copy of the gateway mTLS
# material below must still be removed.
echo "WARN: state file holds an unexpected container name (${CONTAINER_NAME}) — not touching podman"
fi
rm -f "${STATE_FILE}"
fi
rm -f "${STATE_FILE}"

OPENSHELL_STAGING="${STATE_DIR}/openshell-${JOB_ID}"
rm -rf "${OPENSHELL_STAGING}" 2>/dev/null || true
else
echo "WARN: could not read job id from JOB_RESPONSE_FILE — skipping job container cleanup"
fi

OPENSHELL_STAGING="${STATE_DIR}/openshell-${JOB_ID}"
rm -rf "${OPENSHELL_STAGING}" 2>/dev/null || true
# Always tear the per-job gateway down (killing any sandboxes it still holds),
# even when the job id is missing. A leftover is also reaped by the next
# job's prepare.sh.
teardown_openshell_gateway || true
Loading
Loading