feat: onboard models for FPM simulation on target hardware

README.md

@@ -225,6 +225,7 @@ Use the focused SDK documentation instead of treating CLI internals as public
 APIs:
 
 - [Estimator/FPE Python and Rust SDK](docs/core-api.md)
+- [Guided FPM onboarding and local data setup](docs/self-service-support.md)
 - [FPM collection-to-prediction workflow](python/aisimulate/docs/fpm/end-to-end-workflow.md)
 - [Replay SDK and artifact contract](crates/core/src/replay/README.md)
 - [Sweeper SDK](docs/sweeper/overview.md)

docs/self-service-support.md

@@ -0,0 +1,134 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Set up FPM collection for a model and GPU allocation
+
+`aisimulate support` creates a small onboarding request, plans one pure tensor-parallel worker, and produces ordinary `predict` and `recommend` configurations that read your collected FPM data. It builds on AISimulate's existing per-worker FPM support and packaged collector. No other draft PR needs to be merged first.
+
+Planning works before the model has an AISimulate model class or measured FPM timings. A valid request records your choices; model integration, runtime compatibility, and data readiness remain **unchecked**, and accuracy is **not assessed**. This setup does not implement an Inkling model class, run preflight checks, provision GPU resources, or establish measured accuracy. Those steps belong to the broader integration project.
+
+## Create the request
+
+Use an environment installed from this checkout; see [development setup](../DEVELOPMENT.md). Guided setup requires a terminal and starts only when explicitly requested:
+
+```bash
+aisimulate support init --interactive --output support-request.yaml
+```
+
+Enter the actual model identifier or checkpoint path, pinned model revision, dense/MoE kind, pinned vLLM version, GPU system, allocation, and interconnect. Then choose a TP size and pilot workload. Supplied options skip their prompts. Enter accepts displayed defaults; invalid values can be corrected; Ctrl-C or end-of-input cancels without saving. Existing files require `--overwrite`.
+
+Both guided and scripted setup use onboarding. `--profile onboarding` is an optional spelling of the same behavior. For automation, supply the identity flags directly:
+
+```bash
+aisimulate support init \
+  --model /models/your-pinned-checkpoint \
+  --model-revision YOUR_IMMUTABLE_REVISION \
+  --model-kind dense \
+  --framework-version YOUR_PINNED_VLLM_VERSION \
+  --gpu h200_sxm --gpu-count 4 --interconnect nvswitch \
+  --tensor-parallel 2 \
+  --output support-request.yaml
+```
+
+Replace the model and runtime placeholders with the actual inputs. The example does not identify an Inkling checkpoint or claim that four H200s can run your model. Framework support currently selects vLLM. Optional tokenizer, chat-template, and AISimulate revisions are recorded only when supplied.
+
+The default pilot uses 1,024 input tokens, 128 output tokens, concurrency 1, four requests, a 16,384-token context limit, TTFT target 1,000 ms, and TPOT target 100 ms. Scripted setup defaults to TP1 unless `--tensor-parallel` is supplied. These are planning defaults, not measured model capacity or latency. Change them with the corresponding flags shown by `aisimulate support init --help`.
+
+One node is the default; GPUs per node then equals `--gpu-count`. For multiple nodes, supply both `--node-count` and `--gpus-per-node`; their product must equal the total allocation. Each pure-TP worker must fit on one node. Advanced flags include `--request-count`, `--max-candidates`, `--objective`, `--seed`, and `--sm`.
+
+## Plan, preview, and explicitly execute
+
+```bash
+aisimulate support plan \
+  --config support-request.yaml --output-dir ./aisimulate-support
+
+aisimulate support collect-fpm \
+  --config ./aisimulate-support/request.yaml \
+  --output-dir ./aisimulate-support
+```
+
+The first command saves the request, `support-plan.json`, `commands.json`, `predict/pilot.yaml`, `recommend/pilot.yaml`, and a local `systems/` directory. The second prints the collector command without launching it. Generated command vectors and printed next commands use absolute output paths and preserve spaces or shell punctuation. Use a separate output directory for each request. On an existing plan, `--overwrite` can repair missing generated files for the identical request; it rejects changed inputs and preserves existing collected data.
+
+The search uses one selected TP size. By default it evaluates a single worker, so recommendation is not a broad deployment search. `--max-candidates 2` additionally considers the largest count of identical workers that fits the allocation, when that differs from one worker. Each choice gets an independent recommendation config pinned to that replica count with a one-trial budget. The single worker keeps `recommend/pilot.yaml`; the second choice uses `recommend/replicas-N.yaml`, where `N` is its replica count. The plan reports the actual candidate count and lists both config and result paths. Dense collection uses the `tp` preset; MoE uses `pure_tp`; the selected TP size remains exact.
+
+Before execution, prepare the real checkpoint and the pinned runtime using the existing [FPM collection guide](../python/aisimulate/docs/fpm/end-to-end-workflow.md). The packaged collector invokes a Generator-resolved Dynamo/vLLM deployment and needs the corresponding GPU resources, deployment configuration, permissions, and model access. Invoking its command locally does not create that environment. `commands.json` publishes the guarded `aisimulate support collect-fpm --execute` command for collection, alongside a read-only collector planning command.
+
+After those prerequisites are ready, explicitly launch collection from that environment:
+
+```bash
+aisimulate support collect-fpm \
+  --config ./aisimulate-support/request.yaml \
+  --output-dir ./aisimulate-support --execute
+```
+
+Execution requires a matching saved plan. The request records model and runtime revisions; this setup does not download a pinned checkpoint or verify the installed runtime against them. Keep the actual checkpoint and runtime consistent with the request before collecting or predicting.
+
+Set deployment options directly on `support collect-fpm`: `--dynamo-version VERSION`, `--image IMAGE`, `--namespace NAME`, `--model-cache NAME[:MOUNT[:SUBPATH]]`, `--transport nvlink|ib|efa`, and `--image-pull-secret NAME`. The mount, when supplied, is an absolute container path. Prefer an immutable image digest. Supply the same options when previewing, executing, and resuming; deployment settings are part of the collector's frozen-plan identity, so changed settings require a new output directory. Arbitrary collector arguments and engine overrides are not accepted by this command.
+
+For a diagnostic run, add `--execute --smoke`; `--limit N` also requires `--smoke`. Diagnostic smoke and limited runs do not publish formal FPM data. Existing campaign data, raw artifacts, or checkpoints require explicit `--resume` and a readable matching collector checkpoint; otherwise choose a new output directory. A custom `--checkpoint-dir`, if needed, must remain inside the plan's `fpm-checkpoint/` directory. Selecting an empty checkpoint directory does not allow reuse of existing campaign artifacts. Smoke and formal campaigns have separate checkpoints and artifact directories, so an existing smoke run does not prevent the first formal run, or vice versa. The collector verifies the resumed checkpoint's frozen-plan identity.
+
+Planning and collection reject concurrent support operations. The persistent `.support.lock` file uses an OS advisory lock; ownership is released when the process exits, including after an abrupt termination. Leave the file in place. Request validation, saved support-plan checks, and collector input resolution exit 2. Failures after collector execution starts, including a frozen checkpoint identity mismatch, exit 1 with a concise message. Interruption exits 130.
+
+The collector narrows initial prefill sampling with the pilot's input-token and concurrency bounds. Decode uses the collector's existing profile; a four-request synthetic pilot does not imply four timing samples or a short decode campaign. Inspect the generated command and collector plan before committing GPU time. Successful formal collection publishes the FPM Parquet file and metadata pair into the plan's local systems data directory; diagnostic success alone does not provide that pair.
+
+## Run the generated ordinary configurations
+
+After model integration and formal data collection are complete:
+
+```bash
+aisimulate predict \
+  --config ./aisimulate-support/predict/pilot.yaml \
+  --output-dir ./aisimulate-support/predict-results/pilot
+
+aisimulate recommend \
+  --config ./aisimulate-support/recommend/pilot.yaml \
+  --output-dir ./aisimulate-support/recommend-results/pilot
+```
+
+With two candidates, run both replica counts. This loop derives the candidate names from the validated saved request and constructs fixed `aisimulate recommend` commands:
+
+```bash
+python3 - <<'PY'
+import shlex
+import subprocess
+from pathlib import Path
+
+from aisimulate.support.plan import check_plan
+from aisimulate.support.schema import SupportRequest
+
+root = Path("./aisimulate-support").resolve()
+request = SupportRequest.from_yaml(root / "request.yaml")
+check_plan(request, root)
+max_replicas = request.identity.node_count * (request.identity.gpus_per_node // request.search.tensor_parallel)
+replicas = list(dict.fromkeys((1, max_replicas)))[:request.search.max_candidates]
+statuses = []
+for count in replicas:
+    name = "pilot" if count == 1 else f"replicas-{count}"
+    command = [
+        "aisimulate", "recommend",
+        "--config", str(root / f"recommend/{name}.yaml"),
+        "--output-dir", str(root / f"recommend-results/{name}"),
+        "--format", "json",
+    ]
+    result = subprocess.run(command, check=False)
+    statuses.append(result.returncode)
+    print(f"exit {result.returncode}: {shlex.join(command)}", flush=True)
+raise SystemExit(1 if any(statuses) else 0)
+PY
+```
+
+The loop reports each command's exit status and attempts all commands, including when the pilot finds no feasible candidate. It exits 1 after all attempts if any command returned a nonzero status, otherwise 0. Successful outputs remain usable even when the loop exits 1; inspect each result before comparing candidates.
+
+Run either the individual recommendation command or the loop against fresh result directories. The loop also handles a one-candidate plan and does not execute entries from `commands.json`. Results remain separate under `recommend-results/pilot` and, when present, `recommend-results/replicas-N`; compare their objective and latency results for the same workload. This plan does not produce a combined ranking or search additional TP sizes or scheduler settings.
+
+The generated configurations select `engine.workers.aggregated.timing.forward_model: fpm` and set `engine.systems_path` to the plan's absolute local systems directory. The same root supplies hardware and collected FPM data. Recommendation preserves it in exported prediction configs. Moving the plan to another machine requires updating absolute paths or regenerating it there.
+
+Ordinary `predict` and `recommend` commands retain their existing behavior and defaults. Their generated configs can be loaded and edited through the public configuration schema. Missing model integration or data may still prevent execution; a successful simulation is not an accuracy result. Compare its output with an independent run of the same model, runtime, topology, and workload to assess accuracy.
+
+## Hand off a new model architecture
+
+Give an engineer the saved request, the pinned model configuration, and the plan. Follow [How to Add a New Model](../python/aisimulate/docs/add_a_new_model.md) to integrate the architecture and model class, including its operation graph, memory accounting, and KV-cache behavior. FPM supplies measured whole-forward timings; it does not replace the model structure needed by simulation.
+
+The model guide also describes legacy per-operation silicon profiling. Collecting those per-operation timings is not mandatory for this FPM workflow. Integrate the model structure first, collect the matching whole-forward FPM data through the existing collector, and then verify prediction and recommendation on the target deployment.

python/aisimulate/collector/fpm_forward/cli.py

@@ -16,7 +16,7 @@
 from .config import add_fpm_arguments, add_fpm_generator_arguments
 from .entry import resolve_inputs, resolve_run_inputs, run_resolved
 
-_INPUT_ERRORS = (FileNotFoundError, RuntimeError, TypeError, ValueError)
+_INPUT_ERRORS = (OSError, RuntimeError, TypeError, ValueError)
 
 
 def _parser() -> argparse.ArgumentParser:

python/aisimulate/src/aisimulate/compiler.py

@@ -118,14 +118,20 @@ def _deployment(
 
         workers = (engine.workers.prefill, engine.workers.decode)
         for role, worker in zip(("prefill", "decode"), workers, strict=True):
-            if worker is not None and worker.hardware is not None and not load_system_spec(worker.hardware):
+            if (
+                worker is not None
+                and worker.hardware is not None
+                and not load_system_spec(worker.hardware, systems_paths=engine.systems_path)
+            ):
                 raise ValueError(f"unknown workers.{role}.hardware {worker.hardware!r}: no system configuration found")
         if engine.backend_version is None and any(
             worker is not None and worker.hardware is not None for worker in workers
         ):
             versions = {
                 worker.hardware or engine.hardware: resolve_backend_version(
-                    worker.hardware or engine.hardware, engine.backend
+                    worker.hardware or engine.hardware,
+                    engine.backend,
+                    **({"systems_path": engine.systems_path} if engine.systems_path else {}),
                 )
                 for worker in workers
                 if worker is not None
@@ -313,6 +319,7 @@ def _worker_performance_model_metadata(
             "moe_ep_size": parallel.moe_expert if sharded_moe else None,
             "nextn": None,
             "forward_model": worker.timing.forward_model,
+            **({"systems_path": engine.systems_path} if engine.systems_path is not None else {}),
         },
     }
 
@@ -352,6 +359,8 @@ def _worker_engine_args(
     }
     if engine.backend_version is not None:
         payload["aic_backend_version"] = engine.backend_version
+    if engine.systems_path is not None:
+        payload["systems_path"] = engine.systems_path
     if parallel.pipeline != 1:
         payload["aic_pp_size"] = parallel.pipeline
     if parallel.moe_tensor * parallel.moe_expert > 1:

python/aisimulate/src/aisimulate/config/common.py

@@ -11,13 +11,22 @@
 from typing import Annotated, Any, Generic, Literal, TypeVar
 
 import yaml
-from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
+from pydantic import AfterValidator, BaseModel, ConfigDict, Field, field_validator, model_validator
 
 
 class StrictModel(BaseModel):
     model_config = ConfigDict(extra="forbid")
 
 
+def _absolute_systems_path(value: str) -> str:
+    if not value.strip() or "," in value:
+        raise ValueError("systems_path must name one nonempty local directory, not a comma-separated list")
+    return str(Path(value).expanduser().resolve())
+
+
+SystemsPath = Annotated[str, Field(strict=True, min_length=1), AfterValidator(_absolute_systems_path)]
+
+
 T = TypeVar("T")
 PositiveFiniteFloat = Annotated[float, Field(strict=True, gt=0, allow_inf_nan=False)]
 PositiveStrictInt = Annotated[int, Field(strict=True, gt=0)]

python/aisimulate/src/aisimulate/config/engine.py

@@ -9,7 +9,7 @@
 
 from pydantic import Field, field_validator, model_validator
 
-from .common import Choices, IntegerRange, NumericRange, StrictModel
+from .common import Choices, IntegerRange, NumericRange, StrictModel, SystemsPath
 
 PositiveInt = Annotated[int, Field(strict=True, gt=0)]
 NonNegativeInt = Annotated[int, Field(strict=True, ge=0)]
@@ -213,6 +213,7 @@ class EnginePredictionConfig(StrictModel):
     hardware: str
     backend: Backend = "vllm"
     backend_version: str | None = None
+    systems_path: SystemsPath | None = None
     context_length: PositiveInt | Literal["max"] = "max"
     workers: WorkersPredictionConfig = Field(default_factory=WorkersPredictionConfig)
     kv_transfer: KvTransferConfig | None = None
@@ -234,6 +235,7 @@ def _reject_auto_hardware(cls, value: str) -> str:
 
     @model_validator(mode="after")
     def _validate_roles(self) -> EnginePredictionConfig:
+        _validate_systems_path_modes(self, {self.mode})
         _validate_worker_hardware(modes={self.mode}, workers=self.workers)
         if self.mode == "afd":
             _validate_prediction_afd(self)
@@ -388,6 +390,7 @@ class EngineRecommendationConfig(StrictModel):
     hardware: str
     backend: Backend | Choices[Backend] = Field(default_factory=lambda: Choices[Backend](choices=["vllm", "sglang"]))
     backend_version: str | None = None
+    systems_path: SystemsPath | None = None
     context_length: PositiveInt | Literal["max"] = "max"
     workers: WorkersRecommendationConfig = Field(default_factory=WorkersRecommendationConfig)
     kv_transfer: KvTransferConfig | None = None
@@ -403,6 +406,7 @@ def _validate_nonempty(cls, value: str) -> str:
     @model_validator(mode="after")
     def _validate_roles(self) -> EngineRecommendationConfig:
         modes = set(self.mode.choices) if isinstance(self.mode, Choices) else {self.mode}
+        _validate_systems_path_modes(self, modes)
         _validate_worker_hardware(modes=modes, workers=self.workers)
         if "afd" in modes:
             if modes != {"afd"}:
@@ -422,6 +426,15 @@ def _validate_roles(self) -> EngineRecommendationConfig:
         return self
 
 
+def _validate_systems_path_modes(engine, modes: set[str]) -> None:
+    if engine.systems_path is None:
+        return
+    if "afd" in modes:
+        raise ValueError("engine.systems_path does not support AFD")
+    if engine.workers.encoder is not None:
+        raise ValueError("engine.systems_path does not support analytical encoder pools")
+
+
 def _validate_worker_hardware(*, modes: set[str], workers) -> None:
     for role in ("aggregated", "prefill", "decode"):
         worker = getattr(workers, role)

python/aisimulate/src/aisimulate/main.py

@@ -40,6 +40,7 @@
     write_requests,
 )
 from .stack import StackResolutionError, resolve_runner_factory
+from .support.cli import add_support_parser, run_support_command
 from .sweeper.provider import AdapterReplaySpec
 from .sweeper.replay import ReplayOutputRequirements
 
@@ -91,6 +92,7 @@ def build_parser() -> argparse.ArgumentParser:
         action="store_true",
         help="pace prediction against the real wall clock instead of virtual time",
     )
+    add_support_parser(subparsers)
     return parser
 
 
@@ -319,6 +321,13 @@ def _recommend(args: argparse.Namespace, raw: dict[str, Any], factory) -> int:
 def main(argv: Sequence[str] | None = None) -> int:
     parser = build_parser()
     args = parser.parse_args(list(sys.argv[1:] if argv is None else argv))
+    if args.command == "support":
+        try:
+            return run_support_command(args)
+        except (ValidationError, ValueError, OSError) as exc:
+            parser.error(str(exc))
+        except KeyboardInterrupt:
+            return 130
     # Stack resolution deliberately precedes opening the configuration file.
     try:
         factory = resolve_runner_factory(args.stack)

python/aisimulate/src/aisimulate/recommend.py

@@ -112,6 +112,7 @@ def recommendation_to_sweeper(
         "deployment_mode": modes,
         "backend": [str(value) for value in backend_values],
         "backend_version": engine.get("backend_version"),
+        "systems_path": engine.get("systems_path"),
         "model_name": model,
         "hardware_sku": hardware,
         "gpu_budget": optimization.constraints.max_candidate_gpus,
@@ -708,6 +709,8 @@ def _candidate_prediction(
         "context_length": sample.get("context_length") or "max",
         "workers": {},
     }
+    if sample.get("systems_path") is not None:
+        engine["systems_path"] = sample["systems_path"]
     raw_engine = source.engine.model_dump(mode="python", exclude_none=True)
     if deployment.encoder is not None:
         from .config.epd import encoder_prediction_fields