Skip to content
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
2a9fd69
fix(server): result preview — preserve the warm Mol* frame across sur…
YaoYinYing Aug 17, 2026
ec3af91
chore: Done fixing — warm Mol* frame preserved across surface clears;…
YaoYinYing Aug 17, 2026
c0105df
fix(server): warm preview polish — review fixes, structure cache, loa…
YaoYinYing Aug 17, 2026
6a6773b
fix(server): no loading flash on warm/cached structure swaps
YaoYinYing Aug 17, 2026
a344e7e
feat(server): staged first-load experience for the viewer shell
YaoYinYing Aug 17, 2026
713075e
feat(server): shared stderr->stage translator with AlphaFold2 phase m…
YaoYinYing Aug 17, 2026
18d33ec
fix(server): result page polls task status while pending
YaoYinYing Aug 17, 2026
5456170
fix(server): reliable SLURM job-id capture and orphan recovery for qu…
YaoYinYing Aug 18, 2026
e2e3bdd
fix(server): register the orphan-recovery worker_ready handler directly
YaoYinYing Aug 18, 2026
0e2ddea
fix(server): recovery fails tasks whose SLURM job left both sacct and…
YaoYinYing Aug 18, 2026
2d20545
fix(server): import subprocess for the recovery live-queue probe
YaoYinYing Aug 18, 2026
c130555
fix(server): alphafold stages are the four real phases, no wrapper ma…
YaoYinYing Aug 18, 2026
d79b356
fix(server): delegate job cancellation to the worker; implement recov…
YaoYinYing Aug 18, 2026
c535af1
fix(server): implement _recover_slurm_task — attach-and-poll after re…
YaoYinYing Aug 18, 2026
dc54e58
fix(server): dashboard polls non-terminal task statuses
YaoYinYing Aug 18, 2026
0fdbe28
fix(server): recovery flips queued orphans to running when the job is…
YaoYinYing Aug 18, 2026
69dfe0c
fix(server): liveness stage at wrapper start so queued flips to runni…
YaoYinYing Aug 18, 2026
3209d8f
refactor(server): pre-stop SLURM sweep replaces the restart-recovery …
YaoYinYing Aug 18, 2026
96a3399
fix(server): run the pre-stop sweep inside the worker container
YaoYinYing Aug 18, 2026
9e20560
fix(server): log every stage callback for the run_stage debugging pass
YaoYinYing Aug 18, 2026
dd06938
fix(server): address PR review findings
YaoYinYing Aug 18, 2026
dbd30cf
docs: correct REvoCompute edge topology [skip ci]
YaoYinYing Aug 18, 2026
18c25e5
fix(server): close restart recovery gaps
YaoYinYing Aug 18, 2026
9c9e8c4
fix(server): keep py2dmol controls visible
YaoYinYing Aug 18, 2026
9520b42
fix(server): scope restart recovery safely
YaoYinYing Aug 18, 2026
236d6d1
chore: Done fixing — CI, production, and living task verified
YaoYinYing Aug 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed
- Server:
- SLURM status: move allocated tasks from queued to running as soon as the wrapper publishes the real job ID, without waiting for scientific stage output.
- RFdiffusion runner: redirect Hydra bookkeeping and generated schedule caches to the per-job writable `/tmp` filesystem under read-only container execution; lock task defaults to the pinned upstream inference configuration.
- Mol* input workbenches: sequence-strip and canvas selections now report canonical residues; cached viewer-shell handshakes no longer race listener setup.
- task-type intro audit: intros/categories corrected against the runner docs (Pro-Prime = OGT prediction → `fitness`; ThermoMPNN-D = ΔΔG prediction; PLACER = all-atom ligand modeling; GREMLIN = conservation + couplings; HyperMPNN/LASErMPNN/OpenDDE reworded).
Expand Down
5 changes: 3 additions & 2 deletions docs/dev-guide/server.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,9 @@ failures, and create `task_finished` only after real success.
Docker and SLURM runners share this contract. SLURM starts a job in a valid
existing working directory, stores the actual SLURM job ID, polls queue/account
state, and maps cancellation to `scancel`. A task changes from queued to running
when SLURM reports execution or a stage marker arrives; it is not inferred only
from local `srun` process state.
when the allocation wrapper publishes its numeric job ID; the first declared
stage is emitted once as a liveness signal, and later stage markers advance
progress. Execution is not inferred only from local `srun` process state.

GPU task types require user GPU permission and configured GPU resources.
Apptainer receives `--nv` only for GPU tasks. CPU task families must not inherit
Expand Down
17 changes: 10 additions & 7 deletions server/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1170,9 +1170,10 @@ job is enqueued.

### 13.7 slurm_job_id Persistence

The SLURM job ID (`srun-<pid>`) is stored in the `slurm_job_id` column of
the tasks table. When a user cancels a running SLURM task, the web process
calls `scancel` with this ID before terminating the `srun` process.
The allocation wrapper writes `REVODESIGN_JOB_ID=<numeric-id>` as its first
stdout line. The worker stores that real SLURM ID in the `slurm_job_id` column
of the tasks table so cancellation and restart recovery can address the
allocation directly.

### 13.8 Live Output

Expand All @@ -1184,12 +1185,14 @@ Docker runner's live progress behaviour.
### 13.9 Task States

SLURM tasks use an additional `queued` status:

- **`queued`** — `srun` is waiting for a SLURM allocation (resource contention)
- **`running`** — first `REVODESIGN_STAGE:` marker received, job is executing
- **`running`** — the allocation wrapper has started on a compute node

This transition happens via the stage callback: when the first
`REVODESIGN_STAGE:` line appears on stdout, the status moves from `queued`
to `running`.
When the worker captures the wrapper's numeric job-ID line, it invokes the
stage callback with the first declared stage as a liveness signal. Later
`REVODESIGN_STAGE:` lines advance progress normally; a repeated first marker
is deduplicated.

## 14. Troubleshooting

Expand Down
5 changes: 4 additions & 1 deletion server/config/task_types.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -750,7 +750,10 @@ task_types:
input_label: "Protein sequence (FASTA)"
input_workspace: *fasta_workspace
stage_markers:
alphafold: "AlphaFold2 structure prediction"
msa_searching: "MSA searching"
featuring: "Building features"
modeling: "Folding models"
relaxing: "Amber relaxation"
params:
- name: "model_preset"
type: "str"
Expand Down
2 changes: 1 addition & 1 deletion server/docker/runners/alphafold/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ RUN groupadd -f -r ${RUNNER_GROUP} -g ${RUNNER_GID} 2>/dev/null || true && \
(id -u ${RUNNER_USERNAME} >/dev/null 2>&1 && usermod -u ${RUNNER_UID} -g ${RUNNER_GID} ${RUNNER_USERNAME}) || \
useradd -o -r -g ${RUNNER_GROUP} -u ${RUNNER_UID} ${RUNNER_USERNAME}
RUN mkdir -p /app/revocompute && chown -R ${RUNNER_UID}:${RUNNER_GID} /app/revocompute /opt/alphafold
COPY --chown=${RUNNER_UID}:${RUNNER_GID} ./docker/runners/alphafold/run.sh ./docker/runners/common/task_context.sh ./docker/runners/common/task_context.py /app/revocompute/
COPY --chown=${RUNNER_UID}:${RUNNER_GID} ./docker/runners/alphafold/run.sh ./docker/runners/alphafold/alphafold.stages ./docker/runners/common/task_context.sh ./docker/runners/common/task_context.py ./docker/runners/common/stage_translate.awk /app/revocompute/

WORKDIR /app/revocompute/
ENV RUNNER_USERNAME=${RUNNER_USERNAME} \
Expand Down
5 changes: 5 additions & 0 deletions server/docker/runners/alphafold/alphafold.stages
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# AlphaFold2 stderr phase patterns (absl logging) -> stage markers.
msa_searching:Started Jackhmmer|Started HHblits|Started HHsearch
featuring:Predicting [A-Za-z0-9]
modeling:Running model model_
relaxing:[Rr]elax|[Aa]mber
7 changes: 5 additions & 2 deletions server/docker/runners/alphafold/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,12 @@ else
)
fi

echo "REVODESIGN_STAGE:alphafold"
cd "${ALPHAFOLD_PATH:-/opt/alphafold}"
python3 run_alphafold.py "${af_args[@]}"
# AlphaFold logs its phases to stderr (absl logging); the shared translator
# rewrites the stable ones into the stdout stage protocol while passing the
# original lines through to the stderr log unchanged.
python3 run_alphafold.py "${af_args[@]}" 2> >(awk -f /app/revocompute/stage_translate.awk \
-v PATTERNS=/app/revocompute/alphafold.stages >&1)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

[[ -n "$(ls "${output_dir}"/*/ranked_0.pdb 2>/dev/null || true)" ]] || {
echo "AlphaFold produced no ranked_0.pdb" >&2; exit 1; }
Expand Down
33 changes: 33 additions & 0 deletions server/docker/runners/common/stage_translate.awk
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# stage_translate.awk — shared stderr->stage translator for runner wrappers.
#
# Rewrites tool stderr lines into the existing stdout stage protocol:
# every input line is passed through to stderr unchanged, and lines matching
# a family's pattern file emit "REVODESIGN_STAGE:<marker>" on stdout.
#
# Usage: tool ... 2> >(awk -f /app/revocompute/stage_translate.awk \
# -v PATTERNS=/app/revocompute/<family>.stages >&1)
#
# Pattern file format: "marker:regex" per line; # comments and blank lines
# ignored. Regexes are awk extended regexes (no ":" in patterns).

BEGIN {
while ((getline line < PATTERNS) > 0) {
sub(/^[ \t]*/, "", line)
if (line == "" || line ~ /^#/) continue
colon = index(line, ":")
if (colon == 0) continue
n += 1
markers[n] = substr(line, 1, colon - 1)
regexes[n] = substr(line, colon + 1)
}
close(PATTERNS)
}

{
print > "/dev/stderr"
for (i = 1; i <= n; i += 1)
if ($0 ~ regexes[i]) {
print "REVODESIGN_STAGE:" markers[i]
fflush()
}
}
55 changes: 41 additions & 14 deletions server/revocompute/job/runners/slurm_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,11 @@ class SlurmJob(Job):
"""A compute job submitted via SLURM + Apptainer.

``submit()`` launches ``srun`` via ``subprocess.Popen`` and returns
the real SLURM job id (captured from srun's stderr banner), falling
back to a pid-based id if the banner never arrives. ``poll()`` waits
for the process to exit and returns ``COMPLETED`` or ``FAILED`` based
on the exit code.
the real SLURM job id, captured from the allocation wrapper's first
stdout line or an ``srun`` stderr banner, and temporarily falls back to
a pid-based id if neither arrives during submission. ``poll()`` waits for
the process to exit and returns ``COMPLETED`` or ``FAILED`` based on the
exit code.
"""

def __init__(
Expand Down Expand Up @@ -114,9 +115,9 @@ def submit(self) -> str:
self._stdout_thread.start()
self._stderr_thread.start()

# srun prints "srun: job NNNN queued and waiting for resources" on
# stderr right after submitting — grab the real job id so scancel
# works. Fall back to the pid-based id if the banner never arrives.
# Capture the real id from the allocation wrapper's first stdout line
# or srun's stderr banner. Fall back to the pid-based id if neither
# arrives during submission.
self._job_id_event.wait(timeout=5.0)
if self._slurm_job_id:
self._job_id = self._slurm_job_id
Expand Down Expand Up @@ -239,7 +240,15 @@ def _build_wrapper_script(self) -> str:
return path

def _render_wrapper(self) -> str:
lines = ["#!/bin/bash", "set -euo pipefail", ""]
lines = [
"#!/bin/bash",
"set -euo pipefail",
"",
# The allocation carries the authoritative job id; publish it on
# stdout first so the runner never depends on srun's stderr
# banner (which SLURM 19.05 does not always print in time).
'echo "REVODESIGN_JOB_ID=${SLURM_JOB_ID}"',
]
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
self._render_input_staging(lines)
lines.append("")
self._render_apptainer_invocation(lines)
Expand Down Expand Up @@ -319,16 +328,34 @@ def _read_stdout(self) -> None:
stream = self._process.stdout
last_stage: str | None = None
markers = self.tt.stage_markers

def emit_stage(stage: str) -> None:
nonlocal last_stage
if stage == last_stage:
return
last_stage = stage
try:
self.stage_callback(stage)
except Exception as exc: # surface, never mask status updates
logging.error("Stage callback failed for task %s: %s", self.task_id, exc)

for line in iter(stream.readline, ""):
self._stdout_lines.append(line)
if line.startswith("REVODESIGN_JOB_ID=") and self._slurm_job_id is None:
candidate = line.split("=", 1)[1].strip()
if candidate.isdigit():
self._slurm_job_id = candidate
self._job_id_event.set()
if markers and self.stage_callback:
# The allocation is live before the scientific tool
# prints its first marker. Emit the first declared
# stage as a liveness signal so queued tasks become
# running as soon as the wrapper starts.
emit_stage(next(iter(markers)))
if markers and self.stage_callback:
stage = extract_stage_from_log_line(line, markers)
if stage and stage != last_stage:
last_stage = stage
try:
self.stage_callback(stage)
except Exception:
pass
if stage:
emit_stage(stage)
stream.close()

def _read_stderr(self) -> None:
Expand Down
31 changes: 8 additions & 23 deletions server/revocompute/routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@
_task_zip_path,
_virtual_upload_path,
build_results_archive,
cancel_compute_resources,
format_times,
format_walltime,
run_compute_task,
Expand Down Expand Up @@ -1146,29 +1147,13 @@ def cancel_task(md5sum):
400,
)

slurm_job_id = task.get("slurm_job_id")
if slurm_job_id:
scancel = shutil.which("scancel")
if not scancel:
logging.warning("scancel not found; cannot cancel SLURM job %s", slurm_job_id)
else:
try:
subprocess.run([scancel, str(slurm_job_id)], timeout=10, check=True)
logging.info("Cancelled SLURM job %s for task %s", slurm_job_id, md5sum)
except Exception as exc: # pylint: disable=broad-except
logging.warning("Failed to scancel SLURM job %s: %s", slurm_job_id, exc)

container_id = task.get("container_id")
if container_id:
docker = shutil.which("docker")
if not docker:
logging.warning("docker not found; cannot stop container %s", container_id)
else:
try:
subprocess.run([docker, "stop", str(container_id)], timeout=15, check=True)
logging.info("Stopped Docker container %s for task %s", container_id, md5sum)
except Exception as exc: # pylint: disable=broad-except
logging.warning("Failed to stop container %s: %s", container_id, exc)
# SLURM tooling and the Docker socket live only in the worker container;
# the web process delegates the kill there. The task record flips to
# cancelled immediately either way.
cancel_compute_resources.delay(
slurm_job_id=str(task["slurm_job_id"]) if task.get("slurm_job_id") else None,
container_id=str(task["container_id"]) if task.get("container_id") else None,
)

celery_id = task.get("celery_task_id")
if celery_id:
Expand Down
12 changes: 12 additions & 0 deletions server/revocompute/static/css/task-results.css
Original file line number Diff line number Diff line change
Expand Up @@ -76,3 +76,15 @@
.main-results { grid-template-columns: 1fr; }
.structure-color-bar { flex: 1 1 calc(100% - 2.4rem); margin-left: 0; padding-top: 0.4rem; padding-left: 0; border-top: 1px solid var(--line); border-left: 0; }
}

/* Structure preview loading state — quiet pulse, reduced-motion aware. */
.preview-loading { display: flex; flex-direction: column; align-items: center; gap: 0.6rem; padding: 3rem 1rem; }
.preview-loading-bars { display: flex; gap: 0.4rem; }
.preview-loading-bars span { width: 0.45rem; height: 1.6rem; border-radius: 999px; background: color-mix(in srgb, var(--accent) 35%, var(--paper)); animation: preview-loading-pulse 1.1s ease-in-out infinite; }
.preview-loading-bars span:nth-child(2) { animation-delay: 0.15s; }
.preview-loading-bars span:nth-child(3) { animation-delay: 0.3s; }
.preview-loading-label { margin: 0; color: var(--muted); font-size: 0.82rem; }
@keyframes preview-loading-pulse { 0%, 100% { transform: scaleY(0.55); opacity: 0.45; } 50% { transform: scaleY(1); opacity: 1; } }
@media (prefers-reduced-motion: reduce) {
.preview-loading-bars span { animation: none; opacity: 0.7; }
}
24 changes: 24 additions & 0 deletions server/revocompute/static/css/viewer-shell.css
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ html, body {
}

#shellState {
position: relative;
display: flex;
flex-direction: column;
gap: 0.85rem;
Expand All @@ -44,6 +45,29 @@ html, body {
text-align: center;
}

/* Ambient breathing glow behind the ring — the first-load signature. */
#shellState[data-state="loading"]::after {
position: absolute;
inset: 50% auto auto 50%;
width: 14rem;
height: 14rem;
border-radius: 50%;
background: radial-gradient(circle, color-mix(in srgb, var(--shell-accent) 16%, transparent) 0%, transparent 68%);
content: "";
transform: translate(-50%, -50%);
animation: shell-breathe 2.6s ease-in-out infinite;
pointer-events: none;
}

@keyframes shell-breathe {
0%, 100% { opacity: 0.55; transform: translate(-50%, -50%) scale(0.92); }
50% { opacity: 1; transform: translate(-50%, -50%) scale(1.06); }
}

@media (prefers-reduced-motion: reduce) {
#shellState[data-state="loading"]::after { animation: none; opacity: 0.7; }
}

#shellState::before {
width: 2.4rem;
height: 2.4rem;
Expand Down
28 changes: 27 additions & 1 deletion server/revocompute/static/js/dashboard.js
Original file line number Diff line number Diff line change
Expand Up @@ -207,7 +207,7 @@
'</div>' +
'</div>' +
'<div class="task-status-tools">' +
'<span class="status-pill ' + meta.css + ' ' + traceClass + '"' + traceAttr + '>' + escapeHtml(meta.label) + tracePopover + '</span>' +
'<span class="status-pill ' + meta.css + ' ' + traceClass + '"' + traceAttr + ' data-md5="' + escapeHtml(task.md5) + '" data-task-status="' + escapeHtml(task.status) + '">' + escapeHtml(meta.label) + tracePopover + '</span>' +
errorHelp +
'</div>' +
'</header>' +
Expand Down Expand Up @@ -255,6 +255,32 @@
});
}

// Status pills are rendered once at load; poll the non-terminal ones so a
// queued/running badge updates itself, and reload when a task finishes so
// the results buttons and traces appear without a manual refresh.
async function pollStatuses() {
var pills = document.querySelectorAll(".status-pill[data-task-status]");
for (var index = 0; index < pills.length; index += 1) {
var pill = pills[index];
var status = pill.dataset.taskStatus;
if (status === "finished" || status === "failed" || status === "cancelled" || status === "deleted") continue;
try {
var response = await A.authFetch("/compute/api/running/" + encodeURIComponent(pill.dataset.md5));
var payload = await response.json().catch(function () { return {}; });
if (!response.ok || !payload.status || payload.status === status) continue;
var meta = getStatusMeta(payload.status);
pill.textContent = meta.label;
pill.className = "status-pill " + meta.css;
pill.dataset.taskStatus = payload.status;
if (payload.status === "finished" || payload.status === "failed" || payload.status === "cancelled" || payload.status === "deleted") {
window.location.reload();
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
return;
}
} catch (error) { /* transient — retry next tick */ }
}
}
setInterval(pollStatuses, 15000);

function setActiveFilter(nextFilter) {
state.filter = nextFilter;
var chips = document.querySelectorAll("#statusFilters .chip");
Expand Down
Loading
Loading