chore(deploy): add psycopg3 cutover + rollback script - #213
Conversation
scripts/deploy-psycopg3-cutover.sh stages the KG extractor cutover from
psycopg2 to Morpheus's psycopg3 + AsyncConnectionPool branch:
1. Capture pre-cutover SHAs on katana + familiar (rollback anchors)
2. Sample N baseline rates via ~/.local/bin/kg-backfill-per-pool.sh
3. Pull NEW_SHA + reinstall extras on both hosts
4. Restart worker tmux sessions (kg-2080 on katana, kg-p102 on familiar)
5. Sample N post-cutover rates; rollback if rate < BASELINE * 0.85 for
ROLLBACK_CONFIRM consecutive samples
6. Either announce success or git-reset-hard + reinstall + restart
Defaults: 3 baseline samples + 5 post samples, 20 s apart, 0.85 threshold,
2 consecutive bad samples to trip rollback. All overridable via env.
DRY_RUN=1 prints every state-changing command instead of executing it.
Smoke-tested both happy path and forced-rollback path against the live
queue (50.1/min baseline observed). bash -n + shellcheck clean.
141 lines, single bash script — no new deps.
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a robust deployment and rollback orchestration script designed to manage the transition of KG worker pools to the new psycopg3 implementation. By automating the code deployment, dependency installation, and service restarts across multiple hosts, the script minimizes manual intervention and provides a safety net to automatically revert changes if performance regressions are detected. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a deployment and rollback script (scripts/deploy-psycopg3-cutover.sh) designed to manage the cutover of worker pools to psycopg3. The script monitors throughput metrics and automatically rolls back to pre-cutover SHAs if performance drops below a specified threshold. The review feedback highlights several instances of direct shell variable interpolation inside inline Python scripts, which can lead to syntax errors, and suggests passing these variables as command-line arguments instead. Additionally, the reviewer points out that relying on tmux send-keys ... Up Enter to restart worker processes is fragile and recommends using explicit startup commands or systemd services.
| for ((i=1; i<=n; i++)); do | ||
| rate=$(sample_total_rate) | ||
| printf ' sample %d/%d: %s/min\n' "$i" "$n" "$rate" >&2 | ||
| sum=$(python3 -c "print($sum + $rate)") |
There was a problem hiding this comment.
Interpolating shell variables directly into the inline Python script string can lead to syntax errors if the variables are empty or contain unexpected characters. It is safer to pass them as command-line arguments to Python and access them via sys.argv.
| sum=$(python3 -c "print($sum + $rate)") | |
| sum=$(python3 -c "import sys; print(float(sys.argv[1]) + float(sys.argv[2]))" "$sum" "$rate") |
| sum=$(python3 -c "print($sum + $rate)") | ||
| [ "$i" -lt "$n" ] && sleep "$SAMPLE_INTERVAL" | ||
| done | ||
| python3 -c "print(round($sum / $n, 1))" |
There was a problem hiding this comment.
Pass $sum and $n as arguments to Python to avoid syntax errors from direct shell variable interpolation.
| python3 -c "print(round($sum / $n, 1))" | |
| python3 -c "import sys; print(round(float(sys.argv[1]) / float(sys.argv[2]), 1))" "$sum" "$n" |
| step "2/6 baseline throughput ($BASELINE_SAMPLES samples × ${SAMPLE_INTERVAL}s)" | ||
| BASELINE=$(mean_rate "$BASELINE_SAMPLES") | ||
| ok "baseline total_rate = ${BASELINE}/min" | ||
| THRESHOLD=$(python3 -c "print(round($BASELINE * $REGRESSION_FACTOR, 1))") |
There was a problem hiding this comment.
Pass $BASELINE and $REGRESSION_FACTOR as arguments to Python to prevent syntax errors if the variables are empty or malformed.
| THRESHOLD=$(python3 -c "print(round($BASELINE * $REGRESSION_FACTOR, 1))") | |
| THRESHOLD=$(python3 -c "import sys; print(round(float(sys.argv[1]) * float(sys.argv[2]), 1))" "$BASELINE" "$REGRESSION_FACTOR") |
| run "tmux send-keys -t kg-2080 C-c" | ||
| run "sleep 3" | ||
| run "tmux send-keys -t kg-2080 Up Enter" |
There was a problem hiding this comment.
Relying on tmux send-keys ... Up Enter to restart the worker is highly fragile. If any other command was run in that tmux pane (e.g., during manual debugging or log viewing), Up Enter will execute that command instead of the worker. Consider explicitly running the startup command or utilizing a systemd service (like the one defined in deploy/systemd/kg-extract.env.example) for reliable process management.
| for ((s=1; s<=POST_SAMPLES; s++)); do | ||
| rate=$(sample_total_rate) | ||
| printf ' sample %d/%d: %s/min (threshold %s)\n' "$s" "$POST_SAMPLES" "$rate" "$THRESHOLD" | ||
| if python3 -c "import sys; sys.exit(0 if $rate < $THRESHOLD else 1)"; then |
There was a problem hiding this comment.
Directly interpolating $rate and $THRESHOLD into the Python command string is prone to syntax errors if either variable is empty or malformed. Passing them as arguments is much more robust.
| if python3 -c "import sys; sys.exit(0 if $rate < $THRESHOLD else 1)"; then | |
| if python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) < float(sys.argv[2]) else 1)" "$rate" "$THRESHOLD"; then |
1 participant
Summary
scripts/deploy-psycopg3-cutover.sh— orchestrates cutting both KG worker pools (katanakg-2080, familiarkg-p102) over to the psycopg3 + AsyncConnectionPool branch landed in perf(kg-extract): migrate KG triple worker to psycopg3 AsyncConnectionPool #208.~/.local/bin/kg-backfill-per-pool.sh(per-pool JSON:total_rate,katana_rate,famili_rate,eta_h).total_ratedrops below 85% of baseline for 2 consecutive samples.DRY_RUN=1mode plans without acting;ROLLBACK_TO=<sha>allows explicit rollback target instead of auto-captured pre-pull HEAD.Why
Tonight's deploy needs a repeatable, observable cutover with a safety net — manual SSH + restart sequence is error-prone given the dual-host setup. This script is the cleaner standalone follow-up to #208 (which migrated the worker code).
Test plan
DRY_RUN=1 scripts/deploy-psycopg3-cutover.sh <current-sha>→ confirms plan output without executingscripts/deploy-psycopg3-cutover.sh <new-sha>→ both pools pick up new code, throughput stable🤖 Generated with Claude Code