fix(vllm): validate persistent container configuration

DEPENDENCIES.md

@@ -199,7 +199,9 @@ xr-ai-vllm  (utils/xr-ai-vllm/)
     `--stop` flow.  Besides `serve` / `stop_persistent_servers`, exposes the
     shared wrapper helpers `resolve_model_cache`, `load_config`, `setup_hf_env`,
     and `gpu_compute_major` (all stdlib-only; pyyaml is imported function-locally
-    inside `load_config` so the `--stop` path stays dependency-free).
+    inside `load_config` so the `--stop` path stays dependency-free). Docker
+    containers carry a deterministic launch fingerprint; containers created
+    with stale model, image, GPU, or vLLM arguments are replaced before reuse.
 
 xr-ai-vad  (utils/xr-ai-vad/)
     └── numpy >=1.24

docs/ai-services.md

@@ -233,15 +233,31 @@ The persistent vLLM-backed servers (`vlm_server`, `llama_nemotron_llm_server`,
 **survive stack restarts by design**. Each persistent wrapper script checks its
 health endpoint before spawning vLLM:
 
-- **Already running** → touch the ready file immediately, then idle. Stack is
-  ready in seconds; no model reload.
-- **Not running** → spawn vLLM normally, wait for `/health`, touch ready file.
+- **Already running with a matching launch fingerprint** → touch the ready
+  file immediately, then idle. Stack is ready in seconds; no model reload.
+- **Matching container still starting** → attach to its lifecycle and keep
+  waiting for `/health` instead of issuing a conflicting second `docker run`.
+- **Already running with changed or legacy configuration** → stop, remove, and
+  recreate the repository-owned container from the current YAML.
+- **Healthy endpoint without the expected running container** → fail without
+  stopping the unowned listener.
+- **Stopped Docker container with a matching launch fingerprint** → restart it,
+  wait for `/health`, then touch the ready file.
+- **Stopped Docker container with changed or legacy configuration** → remove
+  and recreate it from the current YAML before waiting for `/health`.
+- **Not running** → spawn vLLM normally, wait for `/health`, then touch the
+  ready file.
 
 In pip mode, vLLM is spawned with `start_new_session=True` so the launcher's
-`killpg()` does not reach it on shutdown. In docker mode, the container is
-launched detached (`docker run -d --name xr-ai-vllm-<service>`) so it
-similarly outlives the wrapper. Either way the wrapper exits cleanly and
-vLLM keeps running.
+`killpg()` does not reach it on shutdown. In docker mode, Docker owns the
+container while the foreground `docker run` client uses its own session.
+Either way vLLM keeps running after the orchestrator exits.
+
+Docker containers carry a fingerprint of their image, GPU assignment, model
+cache, environment, bootstrap packages, complete vLLM command, and a versioned
+launcher-controlled Docker contract. This prevents a failed container created
+by one sample profile—or by older launcher behavior—from being restarted later
+with stale memory limits, entrypoint, setup commands, or model arguments.
 
 **Stopping the persisted servers** — run from the sample directory:
 

docs/changelog.md

@@ -111,6 +111,18 @@ direct role mappings, and existing flat entries. Legacy flat constructors and
 read-only attribute aliases keep current callers compatible. Render profiles
 remain unchanged pending their owning refactor.
 
+### 2026-07-31 — Validate persistent vLLM containers before reuse
+
+Docker-hosted vLLM containers carry a fingerprint of their complete launch
+configuration, including a version for launcher-controlled Docker behavior.
+A healthy or stopped container is reused only when the expected named container
+owns the endpoint and its label matches the active image, GPU assignment,
+environment, bootstrap packages, and vLLM arguments. Stale or unlabeled
+repository-owned containers are recreated; healthy unowned listeners are
+rejected without being stopped. This keeps persistent model weights fast to
+reuse without allowing an older sample profile to silently override the current
+configuration.
+
 ### 2026-07-30 — Simple VLM adopts the native voice runtime
 
 `simple-vlm-example` is the first sample migrated from direct

docs/source/components/ai-services.md

@@ -224,15 +224,31 @@ The persistent vLLM-backed servers (`vlm_server`, `llama_nemotron_llm_server`,
 **survive stack restarts by design**. Each persistent wrapper script checks its
 health endpoint before spawning vLLM:
 
-- **Already running** → touch the ready file immediately, then idle. Stack is
-  ready in seconds; no model reload.
-- **Not running** → spawn vLLM normally, wait for `/health`, touch ready file.
+- **Already running with a matching launch fingerprint** → touch the ready
+  file immediately, then idle. Stack is ready in seconds; no model reload.
+- **Matching container still starting** → attach to its lifecycle and keep
+  waiting for `/health` instead of issuing a conflicting second `docker run`.
+- **Already running with changed or legacy configuration** → stop, remove, and
+  recreate the repository-owned container from the current YAML.
+- **Healthy endpoint without the expected running container** → fail without
+  stopping the unowned listener.
+- **Stopped Docker container with matching launch fingerprint** → restart it,
+  wait for `/health`, then touch the ready file.
+- **Stopped Docker container with changed or legacy configuration** → remove
+  and recreate it from the current YAML before waiting for `/health`.
+- **Not running** → spawn vLLM normally, wait for `/health`, then touch the
+  ready file.
 
 In pip mode, vLLM is spawned with `start_new_session=True` so the launcher's
-`killpg()` does not reach it on shutdown. In docker mode, the container is
-launched detached (`docker run -d --name xr-ai-vllm-<service>`) so it
-similarly outlives the wrapper. Either way the wrapper exits cleanly and
-vLLM keeps running.
+`killpg()` does not reach it on shutdown. In docker mode, Docker owns the
+container while the foreground `docker run` client uses its own session.
+Either way vLLM keeps running after the orchestrator exits.
+
+Docker containers carry a fingerprint of their image, GPU assignment, model
+cache, environment, bootstrap packages, complete vLLM command, and a versioned
+launcher-controlled Docker contract. This prevents a failed container created
+by one sample profile—or by older launcher behavior—from being restarted later
+with stale memory limits, entrypoint, setup commands, or model arguments.
 
 **Stopping the persisted servers** — run from the sample directory:
 

tests/test_vllm_docker.py

@@ -2,6 +2,7 @@
 # SPDX-License-Identifier: Apache-2.0
 
 """Unit tests for xr_ai_vllm._docker pure helpers."""
+
 from __future__ import annotations
 
 import json
@@ -10,12 +11,16 @@
 from pathlib import Path
 from unittest.mock import MagicMock, patch
 
+import pytest
 from xr_ai_vllm._docker import (
+    _CONFIG_LABEL,
     _already_logged_in,
+    _launch_fingerprint,
     _LogStreamer,
     _registry_for,
     build_run_argv,
     container_exists,
+    container_label,
     container_running,
     pid_on_port,
     run,
@@ -99,9 +104,37 @@ def test_container_name_present(self, tmp_path):
 
     def test_port_label_set(self, tmp_path):
         argv = build_run_argv(**self._base_kwargs(tmp_path))
-        assert "--label" in argv
-        idx = argv.index("--label")
-        assert argv[idx + 1] == "xr-ai-vllm.port=8100"
+        labels = [argv[index + 1] for index, value in enumerate(argv) if value == "--label"]
+        assert "xr-ai-vllm.port=8100" in labels
+        assert any(label.startswith(f"{_CONFIG_LABEL}=") for label in labels)
+
+    def test_configuration_fingerprint_changes_with_vllm_arguments(self, tmp_path):
+        kwargs = self._base_kwargs(tmp_path)
+        first = build_run_argv(**kwargs)
+        kwargs["vllm_argv"] = [*kwargs["vllm_argv"], "--gpu-memory-utilization", "0.78"]
+        second = build_run_argv(**kwargs)
+
+        def fingerprint(argv):
+            labels = [argv[index + 1] for index, value in enumerate(argv) if value == "--label"]
+            return next(label for label in labels if label.startswith(f"{_CONFIG_LABEL}="))
+
+        assert fingerprint(first) != fingerprint(second)
+
+    def test_configuration_fingerprint_changes_with_contract_version(
+        self,
+        tmp_path,
+        monkeypatch,
+    ):
+        kwargs = self._base_kwargs(tmp_path)
+        first = build_run_argv(**kwargs)
+        monkeypatch.setattr("xr_ai_vllm._docker._LAUNCH_CONTRACT_VERSION", 2)
+        second = build_run_argv(**kwargs)
+
+        def fingerprint(argv):
+            labels = [argv[index + 1] for index, value in enumerate(argv) if value == "--label"]
+            return next(label for label in labels if label.startswith(f"{_CONFIG_LABEL}="))
+
+        assert fingerprint(first) != fingerprint(second)
 
     def test_network_host(self, tmp_path):
         argv = build_run_argv(**self._base_kwargs(tmp_path))
@@ -326,6 +359,13 @@ def test_container_running_false_when_docker_missing(self):
         ):
             assert not container_running("some-name")
 
+    def test_container_label_returns_inspected_value(self):
+        with patch(
+            "xr_ai_vllm._docker.subprocess.check_output",
+            return_value="abc123\n",
+        ):
+            assert container_label("some-name", _CONFIG_LABEL) == "abc123"
+
     def test_pid_on_port_returns_none_when_tools_missing(self):
         with patch(
             "xr_ai_vllm._docker.subprocess.check_output",
@@ -334,18 +374,107 @@ def test_pid_on_port_returns_none_when_tools_missing(self):
             assert pid_on_port(8100) is None
 
 
+def _run_kwargs(tmp_path):
+    return dict(
+        image="vllm/vllm-openai:v0.20.0",
+        container_name="xr-ai-vllm-test",
+        log_prefix="test",
+        vllm_argv=["vllm", "serve", "model", "--gpu-memory-utilization", "0.78"],
+        host="0.0.0.0",
+        port=8107,
+        model_cache=tmp_path / "models",
+        hf_token=None,
+        cuda_visible_devices="1",
+        extra_env=None,
+        extra_pip=None,
+        ready_file=None,
+    )
+
+
+def _expected_fingerprint(kwargs):
+    return _launch_fingerprint(
+        image=kwargs["image"],
+        port=kwargs["port"],
+        model_cache=kwargs["model_cache"],
+        cuda_visible_devices=kwargs["cuda_visible_devices"],
+        extra_env=kwargs["extra_env"],
+        extra_pip=kwargs["extra_pip"],
+        vllm_argv=kwargs["vllm_argv"],
+    )
+
+
 class TestRun:
-    def test_stopped_container_is_removed_and_relaunched(self, tmp_path):
+    def test_healthy_unowned_listener_is_rejected(self, tmp_path):
+        kwargs = _run_kwargs(tmp_path)
+        with (
+            patch("xr_ai_vllm._docker._docker_available", return_value=True),
+            patch("xr_ai_vllm._docker._lifecycle.health_ok", return_value=True),
+            patch("xr_ai_vllm._docker.container_exists", return_value=False),
+            patch("xr_ai_vllm._docker.stop_container") as stop,
+            patch("xr_ai_vllm._docker.remove_container") as remove,
+            patch("xr_ai_vllm._docker.subprocess.Popen") as popen,
+            patch("xr_ai_vllm._docker.signal.getsignal", return_value=None),
+            patch("xr_ai_vllm._docker.signal.signal"),
+            pytest.raises(SystemExit, match="1"),
+        ):
+            run(**kwargs)
+
+        stop.assert_not_called()
+        remove.assert_not_called()
+        popen.assert_not_called()
+
+    def test_healthy_stale_container_is_recreated(self, tmp_path):
+        kwargs = _run_kwargs(tmp_path)
+        state = {"exists": True}
+        process = MagicMock()
+        process.poll.return_value = None
         argv = ["docker", "run", "fresh-container"]
+
+        def remove(_name):
+            state["exists"] = False
+            return True
+
+        with (
+            patch("xr_ai_vllm._docker._docker_available", return_value=True),
+            patch("xr_ai_vllm._docker._lifecycle.health_ok", return_value=True),
+            patch("xr_ai_vllm._docker.container_exists", side_effect=lambda _name: state["exists"]),
+            patch("xr_ai_vllm._docker.container_running", return_value=True),
+            patch("xr_ai_vllm._docker.container_label", return_value="stale"),
+            patch("xr_ai_vllm._docker.stop_container", return_value=True) as stop,
+            patch("xr_ai_vllm._docker.remove_container", side_effect=remove) as remove_mock,
+            patch("xr_ai_vllm._docker._maybe_ngc_login"),
+            patch("xr_ai_vllm._docker.build_run_argv", return_value=argv),
+            patch("xr_ai_vllm._docker.subprocess.Popen", return_value=process) as popen,
+            patch("xr_ai_vllm._docker._LogStreamer", return_value=MagicMock()),
+            patch("xr_ai_vllm._docker._lifecycle.wait_until_healthy"),
+            patch("xr_ai_vllm._docker._lifecycle.idle_until_stopped"),
+            patch("xr_ai_vllm._docker.signal.getsignal", return_value=None),
+            patch("xr_ai_vllm._docker.signal.signal"),
+        ):
+            run(**kwargs)
+
+        stop.assert_called_once_with("xr-ai-vllm-test")
+        remove_mock.assert_called_once_with("xr-ai-vllm-test")
+        popen.assert_called_once_with(argv, start_new_session=True)
+
+    def test_stale_stopped_container_is_recreated(self, tmp_path):
+        kwargs = _run_kwargs(tmp_path)
+        state = {"exists": True}
         process = MagicMock()
         process.poll.return_value = None
+        argv = ["docker", "run", "fresh-container"]
+
+        def remove(_name):
+            state["exists"] = False
+            return True
 
         with (
             patch("xr_ai_vllm._docker._docker_available", return_value=True),
             patch("xr_ai_vllm._docker._lifecycle.health_ok", return_value=False),
-            patch("xr_ai_vllm._docker.container_exists", return_value=True),
+            patch("xr_ai_vllm._docker.container_exists", side_effect=lambda _name: state["exists"]),
             patch("xr_ai_vllm._docker.container_running", return_value=False),
-            patch("xr_ai_vllm._docker.remove_container", return_value=True) as remove,
+            patch("xr_ai_vllm._docker.container_label", return_value="stale"),
+            patch("xr_ai_vllm._docker.remove_container", side_effect=remove) as remove_mock,
             patch("xr_ai_vllm._docker._maybe_ngc_login"),
             patch("xr_ai_vllm._docker.build_run_argv", return_value=argv),
             patch("xr_ai_vllm._docker.subprocess.Popen", return_value=process) as popen,
@@ -355,20 +484,59 @@ def test_stopped_container_is_removed_and_relaunched(self, tmp_path):
             patch("xr_ai_vllm._docker.signal.getsignal", return_value=None),
             patch("xr_ai_vllm._docker.signal.signal"),
         ):
-            run(
-                image="vllm/vllm-openai:v0.20.0",
-                container_name="xr-ai-vllm-omni",
-                log_prefix="omni",
-                vllm_argv=["vllm", "serve", "model"],
-                host="0.0.0.0",
-                port=8108,
-                model_cache=tmp_path,
-                hf_token=None,
-                cuda_visible_devices="0",
-                extra_env=None,
-                extra_pip=["mamba-ssm"],
-                ready_file=None,
-            )
-
-        remove.assert_called_once_with("xr-ai-vllm-omni")
+            run(**kwargs)
+
+        remove_mock.assert_called_once_with("xr-ai-vllm-test")
         popen.assert_called_once_with(argv, start_new_session=True)
+
+    def test_matching_stopped_container_is_restarted(self, tmp_path):
+        kwargs = _run_kwargs(tmp_path)
+        wait_handle = MagicMock()
+        wait_handle.poll.return_value = None
+
+        with (
+            patch("xr_ai_vllm._docker._docker_available", return_value=True),
+            patch("xr_ai_vllm._docker._lifecycle.health_ok", return_value=False),
+            patch("xr_ai_vllm._docker.container_exists", return_value=True),
+            patch("xr_ai_vllm._docker.container_running", return_value=False),
+            patch("xr_ai_vllm._docker.container_label", return_value=_expected_fingerprint(kwargs)),
+            patch("xr_ai_vllm._docker.start_container", return_value=True) as start,
+            patch("xr_ai_vllm._docker._wait_for_container", return_value=wait_handle) as wait,
+            patch("xr_ai_vllm._docker.build_run_argv") as build,
+            patch("xr_ai_vllm._docker._LogStreamer", return_value=MagicMock()),
+            patch("xr_ai_vllm._docker._lifecycle.wait_until_healthy"),
+            patch("xr_ai_vllm._docker._lifecycle.idle_until_stopped"),
+            patch("xr_ai_vllm._docker.signal.getsignal", return_value=None),
+            patch("xr_ai_vllm._docker.signal.signal"),
+        ):
+            run(**kwargs)
+
+        start.assert_called_once_with("xr-ai-vllm-test")
+        wait.assert_called_once_with("xr-ai-vllm-test")
+        build.assert_not_called()
+
+    def test_matching_running_container_continues_startup(self, tmp_path):
+        kwargs = _run_kwargs(tmp_path)
+        wait_handle = MagicMock()
+        wait_handle.poll.return_value = None
+
+        with (
+            patch("xr_ai_vllm._docker._docker_available", return_value=True),
+            patch("xr_ai_vllm._docker._lifecycle.health_ok", return_value=False),
+            patch("xr_ai_vllm._docker.container_exists", return_value=True),
+            patch("xr_ai_vllm._docker.container_running", return_value=True),
+            patch("xr_ai_vllm._docker.container_label", return_value=_expected_fingerprint(kwargs)),
+            patch("xr_ai_vllm._docker._wait_for_container", return_value=wait_handle) as wait,
+            patch("xr_ai_vllm._docker.start_container") as start,
+            patch("xr_ai_vllm._docker.build_run_argv") as build,
+            patch("xr_ai_vllm._docker._LogStreamer", return_value=MagicMock()),
+            patch("xr_ai_vllm._docker._lifecycle.wait_until_healthy"),
+            patch("xr_ai_vllm._docker._lifecycle.idle_until_stopped"),
+            patch("xr_ai_vllm._docker.signal.getsignal", return_value=None),
+            patch("xr_ai_vllm._docker.signal.signal"),
+        ):
+            run(**kwargs)
+
+        wait.assert_called_once_with("xr-ai-vllm-test")
+        start.assert_not_called()
+        build.assert_not_called()