Skip to content

chore(deploy): add psycopg3 cutover + rollback script #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jphein merged 1 commit into from
May 26, 2026
Merged

chore(deploy): add psycopg3 cutover + rollback script #213

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 142 additions & 0 deletions scripts/deploy-psycopg3-cutover.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
#!/usr/bin/env bash
# deploy-psycopg3-cutover.sh — cut the KG extractor worker pools over to the
# psycopg3 + AsyncConnectionPool branch and roll back on regression.
#
# Hosts:
# katana — local (this machine). Worker runs in tmux session "kg-2080".
# familiar — remote. Worker runs in tmux session "kg-p102" + checkout at
# ~/kg-extract-deploy.
#
# Health check: ~/.local/bin/kg-backfill-per-pool.sh returns JSON with
# total_rate, katana_rate, famili_rate, *_flight, total_done, eta_h.
#
# Rollback trigger: post-cutover total_rate < BASELINE * REGRESSION_FACTOR
# for ROLLBACK_CONFIRM consecutive samples.
#
# Usage:
# scripts/deploy-psycopg3-cutover.sh <new-sha>
# DRY_RUN=1 scripts/deploy-psycopg3-cutover.sh <new-sha> # plan only
# ROLLBACK_TO=<old-sha> scripts/deploy-psycopg3-cutover.sh <new-sha>
#
# Env (defaults):
# KATANA_DIR=$HOME/Projects/kg-extract-katana
# FAMILIAR_DIR=\$HOME/kg-extract-deploy (expanded on familiar)
# BASELINE_SAMPLES=3
# POST_SAMPLES=5
# SAMPLE_INTERVAL=20 # seconds between samples
# REGRESSION_FACTOR=0.85 # rollback if rate drops below 85% of baseline
# ROLLBACK_CONFIRM=2 # consecutive bad samples before rollback
# DRY_RUN=0
# ROLLBACK_TO= # if unset, captured from each host pre-pull

set -euo pipefail

NEW_SHA="${1:-}"
[ -n "$NEW_SHA" ] || { echo "usage: $0 <new-sha>" >&2; exit 2; }

KATANA_DIR="${KATANA_DIR:-$HOME/Projects/kg-extract-katana}"
# Quoted single-tick on purpose: this string is sent to familiar over ssh and
# expanded by the remote shell (so $HOME = familiar's $HOME, not katana's).
# shellcheck disable=SC2016
FAMILIAR_DIR_RAW='${FAMILIAR_DIR:-$HOME/kg-extract-deploy}'
BASELINE_SAMPLES="${BASELINE_SAMPLES:-3}"
POST_SAMPLES="${POST_SAMPLES:-5}"
SAMPLE_INTERVAL="${SAMPLE_INTERVAL:-20}"
REGRESSION_FACTOR="${REGRESSION_FACTOR:-0.85}"
ROLLBACK_CONFIRM="${ROLLBACK_CONFIRM:-2}"
DRY_RUN="${DRY_RUN:-0}"
PER_POOL="$HOME/.local/bin/kg-backfill-per-pool.sh"

step() { printf '\n\033[1m▸ %s\033[0m\n' "$1"; }
ok() { printf ' \033[32m✓\033[0m %s\n' "$1"; }
warn() { printf ' \033[33m!\033[0m %s\n' "$1" >&2; }
fail() { printf ' \033[31m✗\033[0m %s\n' "$1" >&2; exit 1; }
run() { if [ "$DRY_RUN" = "1" ]; then printf ' [dry] %s\n' "$*"; else eval "$*"; fi; }

# total_rate of the latest sample, or 0 on parse failure
sample_total_rate() {
"$PER_POOL" 2>/dev/null | python3 -c 'import sys,json
try: print(json.load(sys.stdin).get("total_rate", 0.0))
except Exception: print(0.0)' 2>/dev/null || echo 0.0
}

# Mean of N samples taken SAMPLE_INTERVAL apart. Progress goes to stderr so
# the final mean (stdout) can be captured cleanly via $(mean_rate ...).
mean_rate() {
local n="$1" sum=0 i rate
for ((i=1; i<=n; i++)); do
rate=$(sample_total_rate)
printf ' sample %d/%d: %s/min\n' "$i" "$n" "$rate" >&2
sum=$(python3 -c "print($sum + $rate)")

@gemini-code-assist gemini-code-assist Bot May 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Interpolating shell variables directly into the inline Python script string can lead to syntax errors if the variables are empty or contain unexpected characters. It is safer to pass them as command-line arguments to Python and access them via sys.argv.

Suggested change
sum=$(python3 -c "print($sum + $rate)")
sum=$(python3 -c "import sys; print(float(sys.argv[1]) + float(sys.argv[2]))" "$sum" "$rate")

[ "$i" -lt "$n" ] && sleep "$SAMPLE_INTERVAL"
done
python3 -c "print(round($sum / $n, 1))"

@gemini-code-assist gemini-code-assist Bot May 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Pass $sum and $n as arguments to Python to avoid syntax errors from direct shell variable interpolation.

Suggested change
python3 -c "print(round($sum / $n, 1))"
python3 -c "import sys; print(round(float(sys.argv[1]) / float(sys.argv[2]), 1))" "$sum" "$n"

}

[ -x "$PER_POOL" ] || fail "$PER_POOL not executable — install/symlink it first"

step "1/6 capture pre-cutover SHAs (for rollback)"
KATANA_OLD=$(git -C "$KATANA_DIR" rev-parse HEAD)
ok "katana @ $KATANA_OLD"
# shellcheck disable=SC2029 # FAMILIAR_DIR_RAW intentionally expands remote-side
FAMILIAR_OLD=$(ssh familiar "git -C $FAMILIAR_DIR_RAW rev-parse HEAD")
ok "familiar @ $FAMILIAR_OLD"
ROLLBACK_KATANA="${ROLLBACK_TO:-$KATANA_OLD}"
ROLLBACK_FAMILIAR="${ROLLBACK_TO:-$FAMILIAR_OLD}"

step "2/6 baseline throughput ($BASELINE_SAMPLES samples × ${SAMPLE_INTERVAL}s)"
BASELINE=$(mean_rate "$BASELINE_SAMPLES")
ok "baseline total_rate = ${BASELINE}/min"
THRESHOLD=$(python3 -c "print(round($BASELINE * $REGRESSION_FACTOR, 1))")

@gemini-code-assist gemini-code-assist Bot May 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Pass $BASELINE and $REGRESSION_FACTOR as arguments to Python to prevent syntax errors if the variables are empty or malformed.

Suggested change
THRESHOLD=$(python3 -c "print(round($BASELINE * $REGRESSION_FACTOR, 1))")
THRESHOLD=$(python3 -c "import sys; print(round(float(sys.argv[1]) * float(sys.argv[2]), 1))" "$BASELINE" "$REGRESSION_FACTOR")

ok "rollback threshold = ${THRESHOLD}/min"

step "3/6 pull $NEW_SHA on both hosts + reinstall deps"
run "git -C '${KATANA_DIR}' fetch origin && git -C '${KATANA_DIR}' checkout '$NEW_SHA'"
run "git -C '${KATANA_DIR}' submodule update --init --recursive 2>/dev/null || true"
run "'${KATANA_DIR}/.venv/bin/pip' install -q -e '${KATANA_DIR}[kg-extract,postgres]'"
# shellcheck disable=SC2029 # remote-side expansion intentional
run "ssh familiar 'cd ${FAMILIAR_DIR_RAW} && git fetch origin && git checkout ${NEW_SHA} && .venv/bin/pip install -q -e .[kg-extract,postgres]'"
ok "code + deps on $NEW_SHA"

step "4/6 restart worker sessions"
# katana: tmux kg-2080 — Ctrl-C the running python, re-issue the launch line
# from its scrollback (preserves env). Sender of last command is in pane 0.
run "tmux send-keys -t kg-2080 C-c"
run "sleep 3"
run "tmux send-keys -t kg-2080 Up Enter"
Comment on lines +104 to +106

@gemini-code-assist gemini-code-assist Bot May 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Relying on tmux send-keys ... Up Enter to restart the worker is highly fragile. If any other command was run in that tmux pane (e.g., during manual debugging or log viewing), Up Enter will execute that command instead of the worker. Consider explicitly running the startup command or utilizing a systemd service (like the one defined in deploy/systemd/kg-extract.env.example) for reliable process management.

ok "katana worker restarted"
run "ssh familiar 'tmux send-keys -t kg-p102 C-c && sleep 3 && tmux send-keys -t kg-p102 Up Enter'"
ok "familiar worker restarted"

step "5/6 post-cutover verification ($POST_SAMPLES samples × ${SAMPLE_INTERVAL}s)"
BAD=0
for ((s=1; s<=POST_SAMPLES; s++)); do
rate=$(sample_total_rate)
printf ' sample %d/%d: %s/min (threshold %s)\n' "$s" "$POST_SAMPLES" "$rate" "$THRESHOLD"
if python3 -c "import sys; sys.exit(0 if $rate < $THRESHOLD else 1)"; then

@gemini-code-assist gemini-code-assist Bot May 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Directly interpolating $rate and $THRESHOLD into the Python command string is prone to syntax errors if either variable is empty or malformed. Passing them as arguments is much more robust.

Suggested change
if python3 -c "import sys; sys.exit(0 if $rate < $THRESHOLD else 1)"; then
if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) < float(sys.argv[2]) else 1)" "$rate" "$THRESHOLD"; then

BAD=$((BAD + 1))
warn "below threshold ($BAD/$ROLLBACK_CONFIRM consecutive)"
if [ "$BAD" -ge "$ROLLBACK_CONFIRM" ]; then
warn "regression confirmed — initiating rollback"
break
fi
else
BAD=0
fi
[ "$s" -lt "$POST_SAMPLES" ] && sleep "$SAMPLE_INTERVAL"
done

if [ "$BAD" -ge "$ROLLBACK_CONFIRM" ]; then
step "6/6 ROLLBACK to $ROLLBACK_KATANA / $ROLLBACK_FAMILIAR"
run "git -C '${KATANA_DIR}' reset --hard '${ROLLBACK_KATANA}'"
run "'${KATANA_DIR}/.venv/bin/pip' install -q -e '${KATANA_DIR}[kg-extract,postgres]'"
run "tmux send-keys -t kg-2080 C-c && sleep 3 && tmux send-keys -t kg-2080 Up Enter"
# shellcheck disable=SC2029 # remote-side expansion intentional
run "ssh familiar 'cd ${FAMILIAR_DIR_RAW} && git reset --hard ${ROLLBACK_FAMILIAR} && .venv/bin/pip install -q -e .[kg-extract,postgres] && tmux send-keys -t kg-p102 C-c && sleep 3 && tmux send-keys -t kg-p102 Up Enter'"
fail "rolled back to pre-cutover SHAs — investigate before retrying"
fi

step "6/6 cutover holds"
FINAL=$(sample_total_rate)
ok "post-cutover total_rate = ${FINAL}/min (baseline ${BASELINE}/min, threshold ${THRESHOLD}/min)"
ok "psycopg3 cutover landed; rollback SHAs were katana=$ROLLBACK_KATANA familiar=$ROLLBACK_FAMILIAR"