[cquil11]

TileRT recipes currently depend on the pre-2.0 fork branch. Add native vLLM-prefill/TileRT-decode support to the synced fork using BackendProtocol, StaticRouterFrontend, and schema 2 engine / roles configuration.

• Add TileRT worker commands, native router, topology validation, NIXL/MTP configuration, and worker readiness gates.
• Add optional backend hooks for role-specific images and finite GPU preparation, using the existing Slurm launcher and process registry. Convert weights under a shared lock, validate shards/tokenizer metadata, reuse complete caches, and clean up failed conversions.
• Advertise only real prefill metrics to AIPerf/Tachometer, include TileRT/NIXL fingerprints, and add a GLM-5.1 B200 example plus extension documentation.

Fork main was synced first at 5ea9229c by merging NVIDIA db661efe, preserving the fork's existing commits. The only sync conflict was the cluster example; the fork's launch-plan setting was retained alongside upstream's container-alias layout. This PR contains only the TileRT port on that base.

Validation:

• 19 TileRT behavioral tests pass, including executed conversion scripts, cache reuse/rebuild, failure propagation, native router startup, and metrics generation.
• Existing InferenceX TileRT recipe passes srtctl migrate --verify: v1/v2 resolve identically.
• Example validation/dry-run, Ruff, and generated schema documentation checks pass.
• Full make check: 2,412 tests pass; six failures reproduce on unchanged synced main in this macOS environment (CPU-affinity assumptions, sa-bench shell fixture, and wrapped CLI output). The existing non-blocking type-check diagnostics remain. Removed the affected exact-registry-content assertion rather than pinning another literal; the power-test fixture now uses a real backend.

No Slurm/GPU benchmark was run for this port.

---

Note

Medium Risk
Introduces a new engine/frontend path with Slurm GPU prep, dual-container launches, and shared weight conversion; mistakes in mounts or topology fail at validation but misconfigured clusters could waste GPU time on conversion.

Overview
Adds native TileRT P/D serving to srt-slurm: engine.type: tilert with vLLM prefill workers and TileRT decode workers, paired with frontend.type: tilert-router (no Dynamo/etcd/NATS).

The TileRT backend builds vLLM/NIXL/MTP worker commands, validates topology (single prefill, GLM-5 8-GPU decode, etc.), and runs a pre-worker weight conversion step under a shared filesystem lock with cache reuse. New optional BackendProtocol hooks support per-role container images (prefill_container), finite GPU preparation (BackendPreparation / prepare_backend), and selective Prometheus scraping (prefill only).

Orchestration updates wire preparation before workers, extend AIPerf/Tachometer discovery via get_metrics_port, and add configs/tilert_setup.sh, docs (docs/tilert.md), schema/legacy mappings, and examples/tilert/glm5-disagg.yaml. Contributor docs now cover registering backends/frontends and the extension points.

^{Reviewed by Cursor Bugbot for commit 6bc3f30. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6bc3f30. Configure here.}

[cursor]

Cache hit still fetches HuggingFace source

High Severity

For HuggingFace recipes, _weight_preparation_script calls snapshot_download before taking the convert lock or running complete() on weights_dir. A ready converted-weight cache still needs HuggingFace access, so cache reuse fails when the hub or local HF cache is unavailable even though decode only needs the prepared directory.

 

^{Reviewed by Cursor Bugbot for commit 6bc3f30. Configure here.}

[cursor]

Conversion drops prefill HuggingFace environment

Medium Severity

prepare_backend builds the conversion environment from decode-mode env plus global environment, then get_process_environment. HuggingFace settings such as HF_HOME and HF_TOKEN often live only in prefill env, which _ensure_model_cached already reads via _get_hf_env(). Conversion then cannot see the cache the orchestrator just populated.

Additional Locations (1)

• src/srtctl/cli/do_sweep.py#L193-L214

 

^{Reviewed by Cursor Bugbot for commit 6bc3f30. Configure here.}

[cquil11]

Merged upstream in NVIDIA#529.