Skip to content
163 changes: 160 additions & 3 deletions .github/workflows/deploy-demo-box.yml
Original file line number Diff line number Diff line change
Expand Up @@ -211,25 +211,182 @@ jobs:
# when this job is skipped. actions:read exists for one reason: the
# agent-engine step below downloads the CI-built Apptainer image when the
# box has none, since that image is linux/amd64 only and cannot be built
# on the dev machine.
# on the dev machine. contents:read is for the Pull latest main step
# below: it authenticates the box's own HTTPS remote with this job's own
# ephemeral github.token instead of a credential stored on the box.
permissions:
actions: read
contents: read
Comment thread
sakibsadmanshajib marked this conversation as resolved.
timeout-minutes: 15
steps:
- name: Pull latest main
# Root cause of the 2026-08-13 outage (every deploy since failing
# `fatal: could not read Username for 'https://github.com': No such
# device or address`, exit 128): the box's persistent clone at
# /home/sakib/hive authenticates its HTTPS remote from some
# credential external to this workflow (a stored PAT or a git
# credential helper on the box; this runner has no shell access to
# inspect which). That credential worked through the 2026-08-12
# 20:04 UTC run (confirmed from that run's own log: `git fetch`
# printed a real ref listing, not an error) and was gone by the next
# deploy 17 hours later, with the workflow file byte-identical
# across that gap (`git log -- .github/workflows/deploy-demo-box.yml`
# between those two commits is empty). That is an expired or rotated
# credential, not a missing config line: nothing in this repo
# changed to cause it.
#
# Fix: stop depending on any credential that lives on the box at
# all. `github.token` is this job's own ephemeral, auto-rotated
# GITHUB_TOKEN (scoped read-only by `contents: read` above, minted
# fresh per run), the same mechanism actions/checkout uses
# internally. It is supplied as per-invocation `GIT_CONFIG_*`
# environment variables rather than `.git/config`, so there is
# nothing here to expire or rotate again. Two things a first pass
# of this comment got wrong, corrected after security review on
# this PR: the interpolated value IS written to disk, in the
# runner's own materialized step script under $RUNNER_TEMP (this
# runner is not ephemeral, see the box-reboot note below, so that
# file is not on a fresh tmpfs); and GitHub only auto-masks the
# literal token, not the base64 Basic-auth blob derived from it, so
# that blob is masked explicitly below instead. The token itself
# now reaches the script only via the `GH_TOKEN` env var, never
# interpolated into the script text, matching the pattern the
# agent-engine step below already uses.
#
# actions/checkout itself was considered instead of fixing the pull
# in place, letting the runner's own already-authenticated checkout
# do the fetch. Rejected: every step below assumes /home/sakib/hive is
# a long-lived directory carrying the box's own untracked .env
# (every secret this stack runs on) and Docker build cache.
# actions/checkout's default `clean: true` runs `git clean -ffdx`,
# which would delete that untracked .env on first use, a bigger
# and riskier change than fixing the credential the persistent
# clone already uses.
#
# GIT_TERMINAL_PROMPT=0 makes a still-bad credential fail in one
# git-native line instead of the opaque "could not read Username /
# No such device or address" this outage actually produced (that
# message is itself just what a denied non-interactive prompt looks
# like on a runner with no tty).
env:
GIT_TERMINAL_PROMPT: "0"
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
cd /home/sakib/hive

# A leftover .git/index.lock from a prior run killed mid checkout
# (the timeout-minutes below, a manual cancel, or a box reboot
# this file's own header says the runner does not survive) would
# fail this line with a lock message instead of the real problem.
# Age-gated, not unconditional: this clone is the box's shared
# checkout, other writers (git gc --auto, a human on the box) can
# legitimately hold this lock, and the workflow-level concurrency
# group only serializes this workflow's own runs against each
# other. 15 minutes matches this job's own timeout-minutes below,
# so a lock younger than that is still someone's live operation.
find .git -maxdepth 1 -name index.lock -mmin +15 -delete
git checkout main
git fetch origin
git pull --ff-only origin main

# The token below authenticates ANY public github.com repo, not
# just this one, so it would fetch happily from a repointed
# origin. Check first, or a repointed remote pulls the wrong
# content while reporting the same green this whole fix exists to
# make honest.
# Redacted unconditionally, not just when a mismatch is found:
# origin can embed a credential (https://user:token@github.com/...)
# and this value reaches both the error branch below and the
# diagnostic dump after it, so redacting once here keeps every
# later use safe by construction instead of relying on each print
# site to remember.
expected_url="https://github.com/${{ github.repository }}"
actual_url=$(git remote get-url origin)
# Greedy up to the LAST @, not the first: a credential whose
# secret half itself contains an unencoded @ would otherwise
# leave part of it printed after a first-@ match.
redacted_url=$(printf '%s' "$actual_url" | sed -E 's#//.*@#//REDACTED@#')
case "$actual_url" in
"$expected_url"|"$expected_url.git") ;;
*)
echo "::error::box remote 'origin' is $redacted_url, expected $expected_url(.git). Refusing to pull: this job's GITHUB_TOKEN would authenticate a fetch from any public github.com repo, so a repointed remote would silently deploy the wrong source. Fix with: git -C /home/sakib/hive remote set-url origin $expected_url.git" >&2
exit 1
;;
esac

# Secret-free, and answers exactly what this PR could not verify
# without box access: whether a credential path older than this
# fix could still shadow or rescue it. --name-only reports THAT a
# helper is configured, never its value: a helper can be an
# inline shell snippet with a PAT embedded in it, and that value
# is exactly the on-box credential this whole PR suspects.
git config --show-origin --name-only --get-regexp '^credential\.helper$' \
&& echo "credential.helper is configured (value withheld)" \
|| echo "no credential helper configured"
echo "origin: $redacted_url"
[ -f "$HOME/.git-credentials" ] && echo "a stored credential file exists on the box" || true
[ -f "$HOME/.netrc" ] && echo "a .netrc file exists on the box" || true

if [ -z "${GH_TOKEN:-}" ]; then
echo "::error::GH_TOKEN is empty. This job's permissions block (needs contents: read) or the runner's token minting is the problem, not the box." >&2
exit 1
fi

# GIT_CONFIG_* env vars, not `git -c` on argv: argv is world
# readable via /proc/<pid>/cmdline on a stock box with no
# hidepid, env is readable only by the same uid or root. Slot 1
# empties credential.helper for this invocation so a stored PAT
# or netrc on the box cannot shadow or rescue the header in slot
# 0: a failure below is unambiguously this token's failure.
b64=$(printf 'x-access-token:%s' "$GH_TOKEN" | base64 -w0)
echo "::add-mask::$b64"
export GIT_CONFIG_COUNT=2
export GIT_CONFIG_KEY_0="http.https://github.com/.extraheader"
export GIT_CONFIG_VALUE_0="AUTHORIZATION: basic $b64"
export GIT_CONFIG_KEY_1="credential.helper"
export GIT_CONFIG_VALUE_1=""

if ! git fetch origin main; then
echo "::error::git fetch origin main failed. See the git error above for the actual cause. If it names the username/credential, check the deploy job's permissions block (needs contents: read)." >&2
exit 1
fi
if ! git pull --ff-only origin main; then
echo "::error::git pull --ff-only origin main failed after a successful fetch, so this is a fast-forward divergence (a stray local commit on the box, or main force-pushed), not a credential problem. See the git error above; git -C /home/sakib/hive log --oneline -5 origin/main on the box will show what diverged." >&2
exit 1
Comment thread
sakibsadmanshajib marked this conversation as resolved.
fi

# pull --ff-only moves the ref and writes the worktree as two
# separate steps, not one atomic operation. A run killed in
# between (this job's own timeout, a cancel, or a box reboot)
# leaves HEAD advanced over a half-written tree; the next run's
# fetch/pull then reports "Already up to date" over that same
# half-written tree and goes green. This catches either case:
# this run's own interruption never reaches this line anyway, so
# only a genuinely dirty tree (this run's or a leftover one) fails
# it.
if [ -n "$(git status --porcelain -uno)" ]; then
echo "::error::working tree at /home/sakib/hive has tracked-file changes against HEAD after the pull, consistent with a previous deploy interrupted mid checkout. Recover with: git -C /home/sakib/hive reset --hard origin/main (safe: reset --hard does not remove untracked files, so the box's .env is not touched)." >&2
exit 1
fi

# --ff-only means this line only runs after a real advance or a
# no-op "Already up to date.", never after a silent partial pull,
# but nothing upstream of this could tell the two apart without
# reading raw git output. Record the outcome explicitly instead.
echo "deployed commit: $(git rev-parse HEAD)"
Comment thread
sakibsadmanshajib marked this conversation as resolved.

- name: Install and restart the agent-engine launch daemon
# Issue #780. Cowork tasks need an Apptainer sandbox per task, and
# control-plane's own container cannot exec Apptainer. The launcher
# therefore runs on the box as this unprivileged user and
# control-plane talks to it over a Unix socket; see the script header
# for why the container is deliberately NOT given that privilege.
# GH_TOKEN below is job-scoped, so the contents:read this job's
# permissions block grants for the Pull latest main step above also
# reaches this GH_TOKEN, which previously only carried the default
# actions:read scope. Correct and unavoidable at job-level
# permissions granularity; noted since it was not called out when
# contents:read was added. See issue #902 for the separate,
# pre-existing question of who can trigger this job at all.
working-directory: /home/sakib/hive
env:
GH_TOKEN: ${{ github.token }}
Expand Down
Loading