Keep Inductor's compile cache under ~/.cache so restarts start warm

.claude/skills/benchmarking/SKILL.md

@@ -173,7 +173,7 @@ Comparing against vLLM or SGLang has its own ways to go wrong.
 
 ## Three traps that produce fake regressions
 
-**1. Kernel selection changes across server processes.** Compilation uses `mode="max-autotune-no-cudagraphs"` (`mstar/engine/accelerator_graph_runner.py`). Inductor benchmarks candidate kernels at compile time and keeps the fastest by measured wall time — and those measurements are noisy, so two processes running *identical* code can select different kernels. That changes float accumulation order and perturbs logits in the low bits. Greedy stages are unaffected (argmax rarely flips), but any sampled stage can flip a token, and one flipped token changes the whole downstream generation. Observed on qwen3-omni: identical byte totals across repeats within one process, but mean audio duration 55.6s vs 59.0s across two processes of the same code. It moves throughput too, not only sampled output: the same Whisper code over four restarts gave 368–455 RTFx at c=32, while c=1 stayed within ~1%. So don't attribute a cross-process difference to the branch; sample several processes per side, and treat a high-concurrency delta under ~10% as noise until it survives several restarts. When the change can be switched at runtime, run both arms on one build rather than two. Setting `TORCHINDUCTOR_CACHE_DIR` to a shared persistent path should pin selection after the first compile (worth doing, unverified).
+**1. Kernel selection changes across server processes.** Compilation uses `mode="max-autotune-no-cudagraphs"` (`mstar/engine/accelerator_graph_runner.py`). Inductor benchmarks candidate kernels at compile time and keeps the fastest by measured wall time — and those measurements are noisy, so two processes running *identical* code can select different kernels. That changes float accumulation order and perturbs logits in the low bits. Greedy stages are unaffected (argmax rarely flips), but any sampled stage can flip a token, and one flipped token changes the whole downstream generation. Observed on qwen3-omni: identical byte totals across repeats within one process, but mean audio duration 55.6s vs 59.0s across two processes of the same code. It moves throughput too, not only sampled output: the same Whisper code over four restarts gave 368–455 RTFx at c=32, while c=1 stayed within ~1%. So don't attribute a cross-process difference to the branch; sample several processes per side, and treat a high-concurrency delta under ~10% as noise until it survives several restarts. When the change can be switched at runtime, run both arms on one build rather than two. Inductor's cache now persists across restarts by default (`~/.cache/mstar/torchinductor`), so a restarted server reuses the previous process's picks; give each arm of a cross-branch A/B its own `TORCHINDUCTOR_CACHE_DIR`, and point it at an empty directory when the number you want is a cold start.
 
 **2. The first request of a new shape pays compilation, not inference.** That cost lands entirely on one request and reads like a tail regression. Concrete case: in the VBench I2V set, `req7` is the first landscape image (1024x672; req0-6 are portrait) and takes ~23s where its neighbours take ~7s. `req9` is also landscape and is *not* slow, because req7 already paid. Re-running confirms it — req7 drops from 23.8s to 6.7s. When a p95 or p99 moves on a generation benchmark, check whether one request is carrying a compile, and compare compile cost and steady-state cost separately: a branch can be slower to compile and faster to run.
 

docs/environment_variables.rst

@@ -564,3 +564,11 @@ Compilation
      - ``84``
      - Dynamo's recompile limit, clamped to [8, 256]. Raise it if the logs
        show ``hit config.recompile_limit``.
+   * - ``TORCHINDUCTOR_CACHE_DIR``
+     - ``$XDG_CACHE_HOME/mstar/torchinductor`` (``~/.cache/mstar/torchinductor`` when unset)
+     - torch's own variable; M* sets this default when importing
+       :mod:`mstar.engine` (``_default_compile_cache``), so compiled kernels
+       survive a restart that keeps the cache directory. An exported value wins, and a
+       home that can't be written keeps torch's ``/tmp/torchinductor_<user>``.
+       Point it at an empty directory to measure a cold start. See
+       :ref:`serving:Restarts and the compile cache`.

docs/installation.rst

@@ -89,6 +89,7 @@ interactive shell and check GPU access:
      -v /dev/dri/by-path:/dev/dri/by-path:ro \
      -p 8000:8000 \
      -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
+     -v "$HOME/.cache/mstar:/root/.cache/mstar" \
      mstar:xpu
    python -c "import torch; print(torch.xpu.is_available(), torch.xpu.device_count())"
 

docs/serving.rst

@@ -408,6 +408,30 @@ Pi0.5 DROID variant fixes the action horizon:
 Per-request knobs (``temperature``, ``voice``, ``max_output_tokens``, …) are sent by the
 client instead — see :doc:`clients`.
 
+Restarts and the compile cache
+------------------------------
+
+Most of a first start is compiling: Inductor compiles every capture bucket and autotunes
+its GEMMs, and FlashInfer JIT-builds its kernels. Both cache the results, and a start that
+finds them skips that work. In one measurement, Qwen3.5-0.8B on an H200 took about 22 minutes
+to start with an empty Inductor cache and about 1.5 minutes reusing the previous start's cache.
+
+To keep that across restarts, persist ``~/.cache`` (``$XDG_CACHE_HOME`` if you set it), which
+holds all of it:
+
+- ``~/.cache/mstar/torchinductor``: Inductor's compiled graphs, autotune picks and Triton
+  kernels (M* sets ``TORCHINDUCTOR_CACHE_DIR`` to it unless you export one)
+- ``~/.cache/flashinfer``: FlashInfer's JIT builds
+- ``~/.triton/cache``: Triton kernels compiled outside Inductor
+- ``~/.cache/huggingface``: the weights
+
+In a container, mount these from a volume rather than only ``~/.cache/huggingface``. The
+cache is keyed by torch and Triton version, so an upgrade recompiles rather than reusing
+stale kernels. Autotune picks are measured on the GPU that made them, so give each GPU model
+its own cache if one home serves several. If home is on a network filesystem shared by
+concurrent jobs, export ``TORCHINDUCTOR_CACHE_DIR`` to a node-local path and copy a warm
+cache in at job start.
+
 .. _tensor-transport:
 
 Tensor transport

mstar/engine/__init__.py

@@ -1,7 +1,32 @@
 import logging
 import os
+from pathlib import Path
+
+
+def _default_compile_cache() -> None:
+    """Keep Inductor's cache (compiled graphs, autotune picks, Triton kernels)
+    under ``~/.cache`` beside the weights and FlashInfer's JIT cache, rather
+    than torch's ``/tmp/torchinductor_<user>``, so a restart that keeps
+    ``~/.cache`` starts warm.
+
+    Must run before anything imports ``torch._dynamo``, which writes torch's
+    own default into the environment. An exported value wins, and a home that
+    can't be written keeps torch's default.
+    """
+    if os.environ.get("TORCHINDUCTOR_CACHE_DIR"):
+        return
+    path = Path(os.environ.get("XDG_CACHE_HOME") or Path.home() / ".cache") / "mstar" / "torchinductor"
+    try:
+        path.mkdir(parents=True, exist_ok=True)
+    except OSError:
+        return
+    if os.access(path, os.W_OK):
+        os.environ["TORCHINDUCTOR_CACHE_DIR"] = str(path)
+
+
+_default_compile_cache()
 
-import torch
+import torch  # noqa: E402
 
 logger = logging.getLogger(__name__)
 

test/modular/test_torch_config.py

@@ -1,3 +1,6 @@
+import os
+import subprocess
+import sys
 from concurrent.futures import ThreadPoolExecutor
 
 import pytest
@@ -28,3 +31,17 @@ def test_recompile_limit_env(monkeypatch, raw, expected):
     else:
         monkeypatch.setenv("MSTAR_RECOMPILE_LIMIT", raw)
     assert recompile_limit() == expected
+
+
+# the API server's process, and the module a spawned worker imports to find its target
+@pytest.mark.parametrize("entry", ["mstar.api_server.entrypoint", "mstar.conductor.conductor"])
+def test_compile_cache_defaults_under_the_user_cache(tmp_path, entry):
+    """torch writes its /tmp default into the environment when ``torch._dynamo``
+    is first imported, so mstar's default has to land before any import does."""
+    env = {k: v for k, v in os.environ.items() if k not in ("TORCHINDUCTOR_CACHE_DIR", "XDG_CACHE_HOME")}
+    env["XDG_CACHE_HOME"] = str(tmp_path)
+    code = f"import os, {entry}; print(os.environ['TORCHINDUCTOR_CACHE_DIR'])"
+    out = subprocess.run(
+        [sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=300, check=True,
+    ).stdout.strip().splitlines()[-1]
+    assert out == str(tmp_path / "mstar" / "torchinductor")