Multi-file install: a single part error deletes the whole install tmpdir, including completed parts and resumable partials

[lstein]

Summary

When any single part of a multi-file install errors, _download_error_callback deletes the entire install tmpdir (model_install_default.py:1482-1489 → _safe_rmtree(install_job._install_tmpdir)) — including parts that finished completely and partials holding gigabytes of resumable progress. One transient failure (a 5xx on one file, a rename race) throws away everything the resume machinery exists to protect.

Found during the adversarial review of #9432; the rmtree behavior predates that PR, but #9432 added a new way to trip it (the sidecar rename race below).

Mechanism

Worker thread: any exception other than DownloadJobCancelledException in _do_download marks the part ERROR (download_default.py:327-330). For a part belonging to an install, _download_error_callback then pops the install job, sets it errored, cancels the multifile job, and rmtrees the whole _install_tmpdir.

Two concrete triggers:

1. Transient server error on one file. A single HTTP 500 on file k of an n-file install (raised at download_default.py:456-459) deletes the completed files 1..k-1 and all partial progress. The next attempt starts the entire install from zero.
2. Sidecar rename race (new surface from fix(download): finalize completed range resumes #9432). The 416-promotion path stats the sidecar (download_default.py:352) and renames it (download_default.py:446) a full network round-trip apart, with no existence guard. If the sidecar vanishes in between — user-triggered restart_file() runs clear_partials=True (model_install_default.py:665, :1314) against an in-flight request; manual deletion; AV/indexer interference on Windows — the rename raises FileNotFoundError → part ERROR → whole tmpdir deleted.

Reproduction (test sketch, deterministic)

Trigger 1 needs only two mounts — one good file, one 500 — through the install service, then assert the tmpdir is gone despite file 1 having completed.

Trigger 2 can be made deterministic at the download-queue level with an adapter that deletes the sidecar during the request, simulating the race:

class SidecarDeletingAdapter(TestAdapter):
    def __init__(self, *args, sidecar: Path, **kwargs):
        super().__init__(*args, **kwargs)
        self._sidecar = sidecar

    def send(self, request, **kwargs):
        self._sidecar.unlink()          # the race: sidecar vanishes mid-round-trip
        return super().send(request, **kwargs)

def test_sidecar_vanishing_during_416_roundtrip(tmp_path: Path) -> None:
    source = AnyHttpUrl("https://test.com/race.safetensors")
    content = b"complete"
    destination = tmp_path / "race.safetensors"
    sidecar = destination.with_name(destination.name + ".downloading")
    sidecar.write_bytes(content)

    session = TestSession()
    session.mount(
        str(source),
        SidecarDeletingAdapter(
            b"", status=416, headers={"Content-Range": f"bytes */{len(content)}"}, sidecar=sidecar
        ),
    )
    queue = DownloadQueueService(requests_session=session)
    queue.start()
    try:
        job = queue.download(source=source, dest=destination)
        queue.join()
    finally:
        queue.stop()

    # Today: FileNotFoundError escapes the rename -> job ERRORs; in an install context
    # _download_error_callback then rmtrees the entire tmpdir.
    assert job.status == DownloadJobStatus.ERROR

Suggested fix

• In _download_error_callback, preserve the tmpdir when any part has resumable progress (a .downloading file on disk) or has already completed: write the install marker with an errored/paused status instead of rmtree, so the existing restore/restart_failed machinery can pick it up. Only rmtree when nothing on disk is worth keeping.
• Guard the 416 promotion rename: if the sidecar is missing at rename time, fall through to the pause/restart path instead of letting FileNotFoundError escalate to a part ERROR.