refactor(server): Python restart control module — deploy stamp, image/SIF promotion, drain, rollback

.github/workflows/server-test.yml

@@ -138,7 +138,7 @@ jobs:
           python-version: "3.12"
 
       - name: Install Test Dependency
-        run: python -m pip install requests
+        run: python -m pip install requests PyYAML Werkzeug
 
       - name: Verify Docker Daemon
         run: docker info

CHANGELOG.md

@@ -47,6 +47,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - runner database/env config consolidated into `config/runners/<family>.yaml`.
   - ESMFold v1 checkpoint removed (2022 openfold kernels no longer compile); ESM-2/1v/IF1 remain.
   - `ENABLED_TASKRUNNERS` filters by runtime family.
+  - deployment control: `restart.sh` is a thin wrapper over the Python control module (`server/run/revocompute_ctl/`); restart walks run as named steps with a deploy stamp (commit, digests, changed families, SIF sha256s, config backup), `:next`→`:latest`→`:previous` promotion with changed-only churn and post-deploy prune, staged `<sif>.next` builds, `--dry-run`, `--drain=<minutes>` (maintenance sentinel + 503 submission gate), and `--rollback`; deployment-state file mutations run inside a throwaway container as the runner identity.
   - server docs/README reconciled with the executor/runtime model (runbook distinguishes build/dev/prod/prepared modes).
 - Package manager:
   - CI: source-aware checks — Bare Tests/Pylint run only for desktop changes, Server Tests only for server changes.
@@ -57,6 +58,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Server:
   - Result polling/viewers: terminal task states reload once without overlapping polls; pending result pages keep polling; Mol* teardown completes before iframe removal, stale teardown continuations cannot replace newer previews, and preview loaders remain visible.
   - AlphaFold stages: drain the stderr translator before wrapper exit and preserve both process statuses so final stage markers cannot be lost.
+  - SLURM stages: `srun -u`, unbuffered AlphaFold stderr, and a Python translator stream wrapper phases live, so `run_stage` records intermediates before job exit.
   - SLURM status: move allocated tasks from queued to running as soon as the wrapper publishes the real job ID, without waiting for scientific stage output.
   - Compute recovery: restarts preserve unstarted queued work, cancel only deployment-owned SLURM allocations, fully finalize orphans, and poll reconnected Docker jobs without blocking worker readiness.
   - RFdiffusion runner: redirect Hydra bookkeeping and generated schedule caches to the per-job writable `/tmp` filesystem under read-only container execution; lock task defaults to the pinned upstream inference configuration.

CLAUDE.md

@@ -61,7 +61,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
      profiles configured opposite to project conventions): don't fix, don't
      debate, don't treat them as blocking. No branch protection gates on them.
      Their findings get handled periodically in dedicated batch-fix PRs.
-  3. **Server PRs** (`server/` — REvoCompute): deploy to the live SLURM server with `REVODESIGN_SERVER_ENV=/repo/REvoDesign/server/.env.production.v7-slurm bash server/run/restart.sh restart --use-proxy` (absolute env path, exactly ONE restart running at a time). Before restarting, sync `server/config/task_types.yaml` to the production `CONFIG_DIR` copy (back it up, copy it over, re-apply the two machine lines `job_executor: slurm` / `container_runtime: apptainer`). New task types or runtime families must be appended to `ENABLED_TASKRUNNERS` in the deployment env file before activation — the server only enables registered-and-listed task types. When runner images changed: delete the old SIF under `/mnt/data/srv/revodesign/server-slurm/images/` and add `--build-sif` to the restart — no `.sif.partial` versioning, one SIF per family. Disk-full recovery: `docker buildx prune`, `APPTAINER_CACHEDIR=/home/yinying/.apptainer/ apptainer cache clean --type all`, remove obsolete SIFs. Then live-verify the affected pages on the canonical direct edge `https://revocompute.yaoyy.moe` in incognito (cache-free); `https://revocompute-relay.yaoyy.moe` is the optional relay edge and is not a deployment blocker. When behavior changed, submit a living test with a real data file from `tests/data` through the API using the group test account (ask the user for credentials if none are in session or memory): login `POST /compute/api/auth/login` with `{"username": …, "password": …}` → Bearer token; submit `POST /compute/api/post` (multipart `file` + `task_type` + `params[name]`/`workspace`); monitor the local SLURM job (`squeue`) and read results from the API — status `GET /compute/api/running/<md5>`, manifest `GET /compute/api/results/<md5>`, logs `GET /compute/api/results/<md5>/artifacts/<path>`. Verify the served static files contain the change AND the page behaves as designed, never just one of the two.
+  3. **Server PRs** (`server/` — REvoCompute): deploy to the live SLURM server with `REVODESIGN_SERVER_ENV=/repo/REvoDesign/server/.env.production.v7-slurm bash server/run/restart.sh restart --use-proxy --drain=15` (absolute env path, exactly ONE restart running at a time). Before restarting, sync `server/config/task_types.yaml` to the production `CONFIG_DIR` copy (back it up, copy it over, re-apply the two machine lines `job_executor: slurm` / `container_runtime: apptainer`). New task types or runtime families must be appended to `ENABLED_TASKRUNNERS` in the deployment env file before activation — the server only enables registered-and-listed task types. When runner images changed, add `--build-sif` to the restart — SIFs stage as `<sif>.next` and promote in place after down (the old SIF is saved as `<sif>.previous`); unchanged families are skipped automatically, no manual SIF deletion, no `.sif.partial` versioning. After the restart, verify `${CONFIG_DIR}/.deploy-stamp` (commit, digests, changed families, backup path); on failure, recover with `restart --rollback` (stamp-verified previous image/SIF set, config restored on registry drift). Disk-full recovery: `docker buildx prune`, `APPTAINER_CACHEDIR=/home/yinying/.apptainer/ apptainer cache clean --type all`, remove obsolete SIFs. Then live-verify the affected pages on the canonical direct edge `https://revocompute.yaoyy.moe` in incognito (cache-free); `https://revocompute-relay.yaoyy.moe` is the optional relay edge and is not a deployment blocker. When behavior changed, submit a living test with a real data file from `tests/data` through the API using the group test account (ask the user for credentials if none are in session or memory): login `POST /compute/api/auth/login` with `{"username": …, "password": …}` → Bearer token; submit `POST /compute/api/post` (multipart `file` + `task_type` + `params[name]`/`workspace`); monitor the local SLURM job (`squeue`) and read results from the API — status `GET /compute/api/running/<md5>`, manifest `GET /compute/api/results/<md5>`, logs `GET /compute/api/results/<md5>/artifacts/<path>`. Verify the served static files contain the change AND the page behaves as designed, never just one of the two.
   4. **Main program PRs** (PyMOL plugin): CI and review comments only — no server deploy. Run the relevant gates (`make kw-test PYTEST_KW='<keyword>'`); cross-Qt checks exist in `REvoDesignTestFlight` (PyQt5) and `REvoDesignTestFlightQt6`.
   5. When CI is green and every comment is fixed or debated, push a final empty marker commit `chore: Done fixing — <what was live/CI verified>` as the branch head (the plain `Done fixing` prefix is an intentional exception to the conventional-commit rule, matching the squash-merge habit); the user squash-merges from there.
 - **Standalone bootstrapper encoding**: Keep `src/REvoDesign/tools/package_manager.py` ASCII-only because it is published as `REvoDesign_PyMOL.py` and may be saved through locale-aware Windows tools. A UTF-8-to-GBK transcode turns characters such as `→` into bytes beginning with `0xA1`, which Python 3 rejects while parsing the file as UTF-8. Preserve both the GBK-compilation regression test and the `check-standalone-source-ascii` pre-commit guard; non-ASCII text remains acceptable in normal packaged modules.

server/OPERATIONS_AND_TASK_ADAPTER_GUIDE.md

@@ -147,8 +147,10 @@ runtime_families:
     slurm_image: /absolute/versioned/path/example_20260811.sif
 ```
 
-Every production `slurm_image` must be absolute and versioned. Never overwrite
-a working SIF in place. Runner YAML files must not contain `runner`,
+Every production `slurm_image` must be absolute and versioned. The control
+module never overwrites a working SIF in place: restarts stage rebuilds as
+`<sif>.next` and promote them after `down`, saving the previous file as
+`<sif>.previous` (see §8). Runner YAML files must not contain `runner`,
 `job_executor`, `container_runtime`, or `slurm_image`.
 
 One active runner file exists per runtime family:
@@ -261,15 +263,21 @@ Docker runner images need rebuilding only to produce a replacement SIF or test
 Docker execution.
 
 The build loop creates one image per runtime family and then the server image.
-Before rebuilding a production tag, preserve its current image ID with a
-timestamped rollback tag:
-
-```bash
-stamp=$(date -u +%Y%m%dT%H%M%SZ)
-docker image inspect revodesign-revocompute-runner-example:latest --format '{{.Id}}'
-docker tag revodesign-revocompute-runner-example:latest \
-  "revodesign-rollback/example-${stamp}:latest"
-```
+The deployment tag scheme is `next` → `latest` → `previous`, managed
+automatically:
+
+- `build` tags every runner family as `<image>:next` and leaves the running
+  deployment untouched. A candidate can be validated as `:next` (or a
+  hand-built `:candidate`) without changing `latest`.
+- A `restart` captures the pre-down digests, stops the stack, then promotes:
+  changed families advance `latest` → `previous` → `:next` → `latest`;
+  unchanged families see zero churn (the redundant `:next` tag is dropped).
+  `previous` always survives the post-deploy prune, so the last deployment
+  is one `restart --rollback` away.
+- `--mode=prod` pulls `latest` directly; the pre-pull image id becomes
+  `previous`.
+- `--mode=prepared` promotes nothing — it activates images that a prior
+  `build` left at `:next` (or a pull left at `latest`).
 
 For focused development, build a candidate tag first and validate it without
 changing `latest`:
@@ -317,20 +325,27 @@ docker run --rm --network none \
 
 ## 8. Rebuild SIFs from current images
 
-SIFs are rebuilt by deleting the old file for the family and running the
-restart with `--build-sif`; it rebuilds every family SIF from the current
-Docker images and then activates the stack:
+SIFs rebuild through staged `.next` files — the running SIF is never touched
+in place and no manual delete is needed:
 
 ```bash
-rm -f "/absolute/image-dir/example_v1.sif"
 REVODESIGN_SERVER_ENV="${REVODESIGN_SERVER_ENV}" \
   bash server/run/restart.sh restart --use-proxy --build-sif
 ```
 
-No `.sif.partial` staging or per-family versioning — one SIF per family at
-the registry path. `--build-sif` is incompatible with `--mode=prepared`.
-For a focused single-family iteration, the manual build from the exact
-registry `definition` remains available:
+`restart --build-sif` stages `<sif>.next` for every family whose SIF is
+missing or **older than the family's docker image** — image updates that
+were deployed without a SIF rebuild (in any earlier restart) are caught
+automatically. Limit a catch-up build to one family with
+`--enabled-runners=<name>` when the full set would be too costly. After
+`down`, promotion moves the staged file into place with `os.replace`,
+saving the current SIF as `<sif>.previous` for `restart --rollback`.
+Staging is atomic (built to `<sif>.next.build`, renamed on success), so a
+killed build can never leave a corrupt `.next`. One SIF per family at the
+registry path; no versioned `.sif.partial` files. `--build-sif` is
+incompatible with `--mode=prepared`. For a focused single-family
+iteration, the manual build from the exact registry `definition` remains
+available:
 
 ```bash
 apptainer build --fakeroot "/absolute/image-dir/example_v1.sif" \
@@ -367,16 +382,30 @@ apptainer run --cleanenv --containall \
 For a GPU task add `--nv` and prove SLURM allocated a GPU. CPU tasks must not
 receive `--nv`.
 
-## 9. Back up and promote external configuration
+## 9. Deploy stamp and external configuration backup
 
-Back up configuration outside the active config directory, preserving modes:
+Every prepared/prod `restart` automates the backup and writes a deploy stamp:
 
-```bash
-stamp=$(date -u +%Y%m%dT%H%M%SZ)
-backup_root="${SERVER_DIR}/backups/config-${stamp}"
-mkdir -p "${backup_root}"
-cp -a "${CONFIG_DIR}/." "${backup_root}/"
-```
+- **Config backup** — before `down`, `${CONFIG_DIR}` is copied (as the
+  runner identity, inside a throwaway container) to
+  `${SERVER_DIR}/backups/config-<timestamp>`. Older backups are never
+  deleted. For a manual, standalone backup the same result is:
+
+  ```bash
+  stamp=$(date -u +%Y%m%dT%H%M%SZ)
+  backup_root="${SERVER_DIR}/backups/config-${stamp}"
+  mkdir -p "${backup_root}"
+  cp -a "${CONFIG_DIR}/." "${backup_root}/"
+  ```
+
+- **Deploy stamp** — after a successful `up`, `${CONFIG_DIR}/.deploy-stamp`
+  records the commit sha and dirty flag, mode, per-step timings,
+  changed/unchanged families, `latest`/`previous`/`next` digests, SIF
+  sha256s for changed families, the registry sha256, and the config-backup
+  path. `restart --rollback` consumes it: it verifies the `previous` image
+  tags and SIFs exist, restores the config backup when the registry sha256
+  has drifted, retags `previous` → `latest` for the changed set, and sweeps
+  down/up to readiness — never touching tasks, results, or the user database.
 
 Do not delete older backups. Move obsolete runner files to a timestamped
 directory outside `${CONFIG_DIR}/runners`; one active YAML must remain per
@@ -420,13 +449,20 @@ must likewise use operator-provisioned shared weight mounts rather than
 The safe activation sequence is:
 
 ```text
-validate registry/runners/images/SIFs/Compose
+plan: --dry-run prints the walk + per-family change predictions
+                    |
+drain (optional): --drain=N blocks submissions, waits for SLURM jobs
+                    |
+            validate registry/runners/images/SIFs/Compose
                     |
              any failure? ---- yes ---> keep current stack running
                     |
                     no
                     v
-                  down
+       config backup + digest baseline + down (sweep kills stragglers)
+                    |
+                    v
+        promote: :next -> :latest, latest -> previous, SIF .next in place
                     |
                     v
           up --no-build (no pull)
@@ -436,24 +472,47 @@ validate registry/runners/images/SIFs/Compose
               |           |
             pass        failure
               |           |
-            smoke      restore config,
-                       tags and old SIF paths
+       prune + deploy    restart --rollback (stamp-verified previous set,
+       stamp written      config restored on registry drift)
+              |
+            smoke
 ```
 
-Activate only after rollback tags, old SIFs, config backups, free-space checks,
-and smoke inputs are ready:
+Dry-run first — it reads only and predicts exactly which families will
+change:
 
 ```bash
 REVODESIGN_SERVER_ENV="${REVODESIGN_SERVER_ENV}" \
-  bash server/run/restart.sh restart --mode=prepared
+  bash server/run/restart.sh restart --mode=prepared --dry-run
 ```
 
+Activate only after the dry-run prediction matches the prepared artifacts:
+
+```bash
+REVODESIGN_SERVER_ENV="${REVODESIGN_SERVER_ENV}" \
+  bash server/run/restart.sh restart --mode=prepared --drain=15
+```
+
+`--drain=15` blocks new submissions through the web-visible
+`${SERVER_DIR}/.maintenance` sentinel (the API answers 503 "submissions are
+paused") and waits up to 15 minutes for in-flight SLURM jobs to finish; the
+pre-stop sweep cancels the remainder. The sentinel is removed after a
+successful restart and on failure.
+
 Prepared mode performs all artifact/config/Compose checks before `down`, then
 starts with existing images and no build or pull. Verify Compose services,
 nginx routing, login, task schema, worker/maintenance/Redis health, and fresh
-logs. If readiness fails, do not loop restarts: restore the config backup,
-restore rollback image tags and old SIF paths, start the previous deployment,
-and verify the gateway.
+logs. If readiness fails, do not loop restarts — the deployment is one
+command away from the previous state:
+
+```bash
+REVODESIGN_SERVER_ENV="${REVODESIGN_SERVER_ENV}" \
+  bash server/run/restart.sh restart --rollback
+```
+
+`--rollback` refuses when no deploy stamp exists, or when any stamped
+`previous` image/SIF is missing, naming the stamped commit. It never touches
+tasks, results, or the user database.
 
 `--mode=prod` is for genuinely published, pullable images. Do not use it for
 local-only runtime tags because it performs pulls after stopping the stack.

server/README.md

@@ -84,9 +84,23 @@ therefore rebuild all runtime families.
 Image builds install Python packages with `uv pip install` (bootstrapped from
 its PyPI wheel) and keep PyPI in the index pool via `--extra-index-url`
 alongside the pytorch wheelhouses; `run.sh` edits invalidate only the final
-COPY layers. NOTE: `restart --build-sif` skips SIFs that already exist — after
-changing any runner `run.sh`, delete the corresponding `*.sif` files under the
-deployment images dir so the SIFs rebuild from the updated images.
+COPY layers. Runner images build to `:next` and are promoted to `:latest`
+after the stack is down; `:previous` always survives for rollback.
+`restart --build-sif` stages changed SIFs as `<sif>.next` and promotes them
+in place after `down` — unchanged families are skipped automatically, no
+manual SIF deletion needed.
+
+Deploy safety flags (all `restart`-only):
+
+| Flag | Effect |
+|------|--------|
+| `--dry-run` | Prints the planned step walk and per-family changed/unchanged predictions; executes and writes nothing |
+| `--drain=<minutes>` | Blocks new submissions through the `SERVER_DIR/.maintenance` sentinel (API answers 503) and waits for in-flight SLURM jobs; the sweep cancels the remainder at timeout |
+| `--rollback` | Restores the previous image/SIF set from the deploy stamp, restoring the config backup when the registry drifted; refuses when the stamped previous set is missing |
+
+A successful prepared/prod restart writes `${CONFIG_DIR}/.deploy-stamp` with
+the commit, digests, changed families, SIF sha256s, registry sha256, and the
+config-backup path.
 
 ## Overview
 

server/docker/runners/alphafold/Dockerfile

@@ -57,7 +57,7 @@ RUN groupadd -f -r ${RUNNER_GROUP} -g ${RUNNER_GID} 2>/dev/null || true && \
     (id -u ${RUNNER_USERNAME} >/dev/null 2>&1 && usermod -u ${RUNNER_UID} -g ${RUNNER_GID} ${RUNNER_USERNAME}) || \
     useradd -o -r -g ${RUNNER_GROUP} -u ${RUNNER_UID} ${RUNNER_USERNAME}
 RUN mkdir -p /app/revocompute && chown -R ${RUNNER_UID}:${RUNNER_GID} /app/revocompute /opt/alphafold
-COPY --chown=${RUNNER_UID}:${RUNNER_GID} ./docker/runners/alphafold/run.sh ./docker/runners/alphafold/alphafold.stages ./docker/runners/common/task_context.sh ./docker/runners/common/task_context.py ./docker/runners/common/stage_translate.awk /app/revocompute/
+COPY --chown=${RUNNER_UID}:${RUNNER_GID} ./docker/runners/alphafold/run.sh ./docker/runners/alphafold/alphafold.stages ./docker/runners/common/task_context.sh ./docker/runners/common/task_context.py ./docker/runners/common/stage_translate.py /app/revocompute/
 
 WORKDIR /app/revocompute/
 ENV RUNNER_USERNAME=${RUNNER_USERNAME} \