[dghosef]

What this does

This keeps Inductor's compile cache under ~/.cache, so a restart in a new container, after a reboot, or after /tmp is cleaned starts warm.

By default torch puts TORCHINDUCTOR_CACHE_DIR at /tmp/torchinductor_<user>. A process restart on the same host already reuses it, but the directory is lost with the container, on reboot, and when /tmp is cleaned. Then every capture bucket is compiled again and its GEMMs are autotuned again.

How much that costs: at the default -O1, Qwen3.5-0.8B on H200 takes about 320 s to start on an empty cache, and 1,294 s at -O2 (#817). With this PR, a restart that loses /tmp takes 96–108 s instead (measured below).

The weights (~/.cache/huggingface) and FlashInfer's JIT builds (~/.cache/flashinfer) already live under ~/.cache. Inductor's cache was the one piece that a ~/.cache volume didn't keep.

How it works

1. mstar/engine/__init__.py sets TORCHINDUCTOR_CACHE_DIR to <user cache dir>/mstar/torchinductor when it isn't already set, and TRITON_CACHE_DIR to <TORCHINDUCTOR_CACHE_DIR>/triton, so exporting the Inductor directory alone moves both. An exported value always wins.
2. The user cache dir comes from one stdlib-only helper, mstar/utils/cache_home.py::user_cache_dir(): $XDG_CACHE_HOME if it is an absolute path, else ~/.cache. It replaces three hand-written XDG lookups (engine, omnilingual_asr, vjepa2). A relative XDG_CACHE_HOME is ignored, and with no usable home both variables stay unset, so torch's /tmp default applies and a compile never fails because of an unwritable cache directory.
3. It's set in mstar.engine because importing torch._dynamo writes torch's own /tmp default into os.environ (torch/_dynamo/package.py calls cache_dir() at module level), so our default has to be set before that first import. mstar.engine is where every entry point first imports it. Platform.process_env_defaults runs too late.
4. The conductor logs the resolved directories once at startup.
5. There's no new mstar knob. These are torch's and Triton's own variables.

Docs

• docs/serving.rst, "Restarts and the compile cache": what to persist, what the cache key covers (torch/Triton versions and, at -O1, the source hash with compute capability and SM count, so keep one cache per GPU model; ptxas and the driver aren't in the key, so delete the directory or set TORCH_COMPILE_CACHE_KEY_TAG after changing them), that the default grows in home and nothing prunes it, and to export a node-local path when home is on NFS or a quota.
• docs/environment_variables.rst: rows for both variables and their defaults.
• XPU docker run (docs/installation.rst) and the Dynamo compose worker use a named mstar-cache volume.
• .claude/skills/benchmarking/SKILL.md: a cross-process A/B or restart sampling uses a fresh, empty cache per boot, since a shared cache pins one arm's autotune picks.

How it was tested

What	Result
ruff check .	clean
CPU Core on 3783c8890 (merged with main f417b902d) vs main	8073 passed, 936 skipped, 2 failed; main 8067 passed with the same 2 failures (test_dump_stacks_retries_with_sudo_only_when_asked, a box-local py-spy install). None fail only on this PR.
Behaviour script, fresh processes on a Linux CPU box	9/9: default under $HOME/.cache/mstar/torchinductor with Triton under it; a second process hits the FX cache (1 hit, 0 misses); exported TORCHINDUCTOR_CACHE_DIR, TRITON_CACHE_DIR and XDG_CACHE_HOME each win; an empty export counts as unset; a relative XDG is ignored; a read-only HOME (user nobody) falls back to /tmp/torchinductor_nobody and the compile works; 8 concurrent compiles on one empty cache all exit 0 and a 9th process hits.
GPU, 1 x H200, Qwen3.5-0.8B and Z-Image, no cache variables exported	Every boot logged Compile caches: Inductor ~/.cache/mstar/torchinductor, Triton .../torchinductor/triton and left nothing in /tmp/torchinductor_*; main wrote to /tmp/torchinductor_root. No tracebacks, every request OK.

Restart timing, Qwen3.5-0.8B at -O1 on 1 x H200 (PR 3783c8890 vs main f417b902d). "Restart" wipes /tmp/torchinductor_* and keeps $HOME, which is what a new container loses:

Boot to /health	cold	restart
this PR	388 s, 535 s	108 s, 96 s
main	547 s, 343 s	343 s, 331 s

• On main a restart costs a full cold start (its cache was in /tmp); with this PR it reused the 326 MB cache and wrote no new files.
• Cold boots land in two clusters (about 345–390 s and 535–550 s) on both arms, so the PR doesn't change cold-start time.
• A restart reproduces the greedy outputs of the boot that wrote the cache (4/4 prompts). Two fresh -O1 compiles can differ on both arms (PR 3/4, main 1/4 identical), which is pre-existing.
• c=1 decode is within the boot-to-boot spread on both arms (PR 641–672, main 637–670 tok/s), and first-request latency is the same (0.16–0.19 s).

Not run: a restart that keeps /tmp (same container), several servers sharing one cache, and Z-Image restarts.

Not in this PR

• A warm cache on brand-new nodes: shipping a prebuilt cache per GPU model and torch version (torch.compiler.save_cache_artifacts), or a shared volume.
• Docs: running M* on AMD GPUs #687, ROCm serving: --gpus sets HIP_VISIBLE_DEVICES, one upload dir per mstar serve, ATEN GEMMs for small buckets #689, Docs: AMD media packages, compile caches and model status #691 and Point Triton at torch's ptxas on Blackwell when its own predates 13.1.115 #910 also touch compile-cache docs; whichever lands second should fold into serving.rst's section, and Point Triton at torch's ptxas on Blackwell when its own predates 13.1.115 #910's key tag belongs there.
• With a persisted cache, -O2's extra compile is paid once per cache rather than once per start, which may be worth revisiting Reduce cold start time with broad compile tuning levels #817's choice of -O1 as the default.

Written with AI assistance (Claude Code); I've reviewed the change and the numbers above.

🤖 Generated with Claude Code

[mstar-review]

Automated review

🟡 1 P2 on 8def7f7, light review

Description

• The startup figures (1,312 s cold, 94 s warm) come from two separate earlier runs, and neither used the new cache directory. A warm restart from ~/.cache/mstar/torchinductor was never timed, yet docs/serving.rst states "about 1.5 minutes with warm ones" as a measured result for this setup. Timing one warm restart would back that sentence.
• The full CPU Core suite was not run; only test/modular/test_torch_config.py was.

What the review checked

• mstar/__init__.py is lazy and imports no torch, so mstar.engine is the first module to set the variable. mstar.cli.main reaches mstar.api_server.entrypoint without importing torch first.
• mstar.conductor.conductor imports plain torch before mstar.engine. That is safe only because import torch loads torch._dynamo lazily, and the new subprocess test checks that this import order is safe.
• mstar/platforms/compile.py imports torch._inductor.config inside a function, not at module import, so it runs after the default is set.
• An exported TORCHINDUCTOR_CACHE_DIR returns early, so the user's value wins. When home can't be written, mkdir hits OSError or os.access fails, and the variable is left unset, so torch falls back to its /tmp default.
• Spawned workers inherit the parent's environment, so the parent's choice carries over to them.
• :ref:serving:Restarts and the compile cache`` resolves because docs/conf.py enables `autosectionlabel` with `autosectionlabel_prefix_document = True`.
• The new test runs in a subprocess with its own XDG_CACHE_HOME, so it doesn't pollute the test runner's environment or the user's real cache.
• Invariant 9: the variable has a row in docs/environment_variables.rst that names its default and the function that sets it.

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}