Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/perf-model/fpm-self-service/implementation.md
Original file line number Diff line number Diff line change
Expand Up @@ -699,7 +699,7 @@ Setup reads that file and an adjacent `hf_quant_config.json` ModelOpt sidecar, w

With no parallelism flags, guided setup asks for missing shared precision/layout facts, shows a small topology shortlist with resource reasons, and lets you choose one worker with `--output`, or several with [`--output-dir`](#onboard-multiple-parallel-configurations). Enter selects only the displayed single default, and only when all required profile inputs are resolved and its estimated bytes fit the budget. Otherwise a candidate selection is required before setup resolves the remaining precision and cache geometry. Any explicit parallelism flag bypasses suggestions and retains the existing topology validation. Each chosen topology then follows the same profile review, edit, accept and cancellation flow below.

When present, `text_config` must be one nonempty decoder configuration object. Only its geometry is used; wrapper and encoder dimensions are never merged into it. Otherwise, setup reads the flat decoder fields even when vision or audio metadata is present. Nested decoder architecture/model-type declarations select supported decoder validation and resource estimates. The profile preserves the declared architecture used by collection: nested `architectures` when present, otherwise the wrapper architecture. Provenance records both identities when they differ. A wrapper architecture never establishes an unknown nested decoder's resource layout. Shared outer dtype and quantization metadata are inherited only when the text section does not declare that metadata. The checkpoint identity hint and SHA-256 remain those of the original document. `model_max_length` is accepted as a context-limit alias; conflicting context declarations fail explicitly.
When present, `text_config` must be one nonempty decoder configuration object. Only its geometry is used; wrapper and encoder dimensions are never merged into it. A nonempty `llm_config` object is read the same way. It takes precedence over `text_config` only for wrappers that AISimulate maps to `llm_config`, such as `NemotronH_Omni_Reasoning_V3`. The FPM collector and the profile architecture check choose the decoder key the same way. Without such a decoder object, setup reads the flat decoder fields even when vision or audio metadata is present. Nested decoder architecture/model-type declarations select supported decoder validation and resource estimates. The profile preserves the declared architecture used by collection: nested `architectures` when present, otherwise the wrapper architecture. Provenance records both identities when they differ. A wrapper architecture never establishes an unknown nested decoder's resource layout. Shared outer dtype and quantization metadata are inherited only when the text section does not declare that metadata. The checkpoint identity hint and SHA-256 remain those of the original document. `model_max_length` is accepted as a context-limit alias; conflicting context declarations fail explicitly.

Scripted intake and interactive final review state that FPM models the text decoder only. Config-derived estimates exclude multimodal encoders, projectors, preprocessing and other non-text components. Observed runtime cache capacity accounts for all components actually loaded by the worker, while timing describes the text decoder only. The saved profile preserves that notice in provenance through planning and generated prediction/recommendation configurations. Text-decoder cache layouts that `config.json` cannot establish, such as linear-attention, recurrent state or an architecture that intake does not recognize, are not rejected: their [cache geometry stays pending the runtime probe](#cache-geometry-pending-the-runtime-probe). Other known incompatible layouts remain rejected. MLA/DSA layouts and configs that name no architecture require explicit resource accounting.

Expand Down
22 changes: 14 additions & 8 deletions python/aisimulate/collector/fpm_forward/model_capability.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,11 @@
_get_model_config_path,
_parse_hf_config_json,
)
from aisimulate_core.sdk.fpm_model_metadata import attach_quantization_sidecar, inherit_decoder_precision
from aisimulate_core.sdk.fpm_model_metadata import (
attach_quantization_sidecar,
decoder_config_key,
inherit_decoder_precision,
)

_ATTENTION_SOURCE_OPS = {
"dsa_module": frozenset({"dsa_context_module", "dsa_generation_module"}),
Expand Down Expand Up @@ -69,15 +73,16 @@ def effective_payload(self) -> dict[str, Any]:
"""Return decoder metadata, inheriting only identity and shared precision."""

payload = self.payload
text_config = payload.get("text_config")
if isinstance(text_config, dict):
effective, _ = inherit_decoder_precision(payload, text_config)
key = decoder_config_key(payload)
decoder = payload.get(key)
if isinstance(decoder, dict):
effective, _ = inherit_decoder_precision(payload, decoder)
if effective.get("architectures") is None and payload.get("architectures") is not None:
effective["architectures"] = payload["architectures"]
else:
effective = payload
sidecar = effective.get("hf_quant_config")
if sidecar is None and isinstance(text_config, dict) and isinstance(payload.get("hf_quant_config"), dict):
if sidecar is None and isinstance(decoder, dict) and isinstance(payload.get("hf_quant_config"), dict):
try:
return attach_quantization_sidecar(effective, payload["hf_quant_config"])
except ValueError:
Expand Down Expand Up @@ -155,11 +160,12 @@ def _load_json_mapping(path: Path, *, description: str) -> tuple[dict[str, Any],


def _attach_modelopt_sidecar(payload: dict[str, Any], sidecar: dict[str, Any]) -> dict[str, Any]:
text_config = payload.get("text_config")
key = decoder_config_key(payload)
decoder = payload.get(key)
if payload.get("hf_quant_config") is not None and payload["hf_quant_config"] != sidecar:
raise ValueError("conflicting inline hf_quant_config and adjacent hf_quant_config.json")
if isinstance(text_config, dict):
decoder, _ = inherit_decoder_precision(payload, text_config)
if isinstance(decoder, dict):
decoder, _ = inherit_decoder_precision(payload, decoder)
attach_quantization_sidecar(decoder, sidecar) # Reject contradictory source declarations before planning.
else:
attach_quantization_sidecar(payload, sidecar)
Expand Down
30 changes: 19 additions & 11 deletions python/aisimulate/src/aisimulate/support/config_profile.py
Original file line number Diff line number Diff line change
Expand Up @@ -604,6 +604,10 @@ def _validate_architecture(raw: Mapping[str, Any], architecture: str | None, exp

def load_model_config(path: str | Path) -> ModelConfig:
"""Load local config/ModelOpt metadata with separate byte-level provenance."""
# Imported here, not at module level: tests/unit/test_fpm_config_imports.py and the repository-root
# tests/test_resource_supervision.py fail when aisimulate_core.decoder_config loads before a config is read.
from aisimulate_core.decoder_config import decoder_config_key

path = Path(path).expanduser().absolute()
payload = path.read_bytes()
# JSON Infinity and NaN parse as floats: fields intake reads reject them, and other fields are never read.
Expand All @@ -619,17 +623,20 @@ def load_model_config(path: str | Path) -> ModelConfig:
"for all components actually loaded by the worker; full multimodal latency is not modeled."
)
}
if "text_config" in document:
text_config = document["text_config"]
if not isinstance(text_config, dict) or not text_config:
decoder_key = decoder_config_key(document)
if decoder_key is not None:
decoder = document[decoder_key]
if not isinstance(decoder, dict) or not decoder:
raise ValueError(
"text_config must be a nonempty object describing one text decoder; "
f"{decoder_key} must be a nonempty object describing one text decoder; "
"supply its decoder config or a complete --fpm-profile"
)
if "text_config" in text_config:
raise ValueError("ambiguous nested text_config; supply one text decoder config or a complete --fpm-profile")
raw = dict(text_config)
notes["decoder_config"] = "config text_config; outer model and encoder geometry are excluded"
if decoder_key in decoder:
Comment thread
Arsene12358 marked this conversation as resolved.
Resolved
raise ValueError(
f"ambiguous nested {decoder_key}; supply one text decoder config or a complete --fpm-profile"
)
raw = dict(decoder)
notes["decoder_config"] = f"config {decoder_key}; outer model and encoder geometry are excluded"
inherited = []
for keys, inferred_keys in (
(("dtype", "torch_dtype"), ()),
Expand All @@ -647,7 +654,7 @@ def load_model_config(path: str | Path) -> ModelConfig:
inherited.append(key)
if inherited:
notes["shared_metadata"] = (
f"config-level {', '.join(inherited)} inherited because text_config does not declare that metadata"
f"config-level {', '.join(inherited)} inherited because {decoder_key} does not declare that metadata"
)
for key in _UNTYPED_READ_FIELDS:
_require_finite(raw.get(key), key)
Expand Down Expand Up @@ -682,9 +689,10 @@ def load_model_config(path: str | Path) -> ModelConfig:
f"config wrapper architecture={architecture} retained for checkpoint/collector identity"
)
notes["decoder_architecture"] = (
f"text_config decoder architecture={decoder_architecture}; used for decoder validation/resource estimates"
f"{decoder_key} decoder architecture={decoder_architecture}; used for decoder validation/resource estimates"
if decoder_architecture
else "text_config does not identify a decoder architecture; decoder resource bounds require explicit input"
else f"{decoder_key} does not identify a decoder architecture; "
"decoder resource bounds require explicit input"
)
inkling = architecture == _INKLING_ARCHITECTURE
# A runtime-resolved layout may use zero or absent ordinary attention dimensions.
Expand Down
44 changes: 44 additions & 0 deletions python/aisimulate/src/aisimulate_core/decoder_config.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Choose the key of a checkpoint config that nests the text decoder.

FPM config intake, the FPM collector and the FPM profile check call this function, so all three read one decoder.
The module imports only the standard library. Config intake must stay usable in a process that does not import
aisimulate_core.sdk. The cold-process tests in tests/unit/test_support_topology.py and
tests/unit/test_onboard_topology.py check that. The one exception is a document that holds both text_config and a
nonempty llm_config: for that shape the function reads the registry from aisimulate_core.sdk.common.
"""

from __future__ import annotations

from collections.abc import Mapping
from typing import Any


def decoder_config_key(document: Mapping[str, Any]) -> str | None:
"""Return the key that nests the text decoder in a checkpoint config, or None for a flat config.

The candidates, in order, are the key that MULTIMODAL_TEXT_CONFIG_KEY registers for the wrapper's first
architecture, then text_config, then llm_config. The first candidate that the document holds wins.
text_config is held whenever the key is present, so a malformed one is still chosen and intake reports it.
llm_config is held only as a nonempty object, so a null, empty or non-object value leaves the config flat.
The function never raises.

The registry maps every wrapper to text_config or llm_config. It changes the answer only when the document holds
both keys and the wrapper maps to llm_config, so the registry is imported only in that case.
tests/unit/sdk/test_fpm_model_metadata.py checks that the registry holds no other key.
"""
text_held = "text_config" in document
llm_config = document.get("llm_config")
llm_held = isinstance(llm_config, dict) and bool(llm_config)
Comment thread
Arsene12358 marked this conversation as resolved.
Resolved
if text_held and llm_held:
from aisimulate_core.sdk.common import MULTIMODAL_TEXT_CONFIG_KEY

architectures = document.get("architectures")
wrapper = architectures[0] if isinstance(architectures, list) and architectures else None
if isinstance(wrapper, str) and MULTIMODAL_TEXT_CONFIG_KEY.get(wrapper) == "llm_config":
return "llm_config"
if text_held:
return "text_config"
return "llm_config" if llm_held else None
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

"""Checkpoint precision reconciliation shared by FPM onboarding and collection.
"""Checkpoint decoder selection and precision reconciliation shared by FPM onboarding and collection.

These functions do not change the general SDK's model-loading precedence.
"""
Expand All @@ -10,6 +10,9 @@

from typing import Any

# The helper lives in a standard-library module so that FPM config intake can use it without importing the SDK.
from aisimulate_core.decoder_config import decoder_config_key as decoder_config_key

from .utils import (
_attach_hf_quant_config,
_infer_quantization_fields,
Expand Down
13 changes: 9 additions & 4 deletions python/aisimulate/src/aisimulate_core/sdk/fpm_profile.py
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ def _load_pinned_checkpoint_config(model: str, revision: str) -> dict | None:

def _validate_checkpoint_architecture(profile: FpmModelProfile) -> None:
"""Verify raw checkpoint metadata without loading weights or remote code."""
from aisimulate_core.sdk.fpm_model_metadata import decoder_config_key
from aisimulate_core.sdk.utils import HuggingFaceDownloadError, _load_local_config

checkpoint = Path(profile.model).expanduser()
Expand All @@ -114,10 +115,14 @@ def _validate_checkpoint_architecture(profile: FpmModelProfile) -> None:
) from error
if not isinstance(metadata, dict):
raise ValueError(f"FPM profile checkpoint config.json must be a mapping for {identity}")
# Match the collector's text-model view of checkpoint metadata.
text_config = metadata.get("text_config")
if isinstance(text_config, dict):
metadata = {**metadata, **text_config}
# Match the collector's text-model view of checkpoint metadata. A decoder that declares no architectures,
# absent or null, keeps the wrapper's, as the collector and config intake read it.
decoder = metadata.get(decoder_config_key(metadata))
if isinstance(decoder, dict):
merged = {**metadata, **decoder}
if decoder.get("architectures") is None:
merged["architectures"] = metadata.get("architectures")
metadata = merged
architectures = metadata.get("architectures")
if (
not isinstance(architectures, list)
Expand Down
90 changes: 90 additions & 0 deletions python/aisimulate/tests/unit/collector/test_fpm_forward.py
Original file line number Diff line number Diff line change
Expand Up @@ -1588,6 +1588,96 @@ def test_model_architecture_names_an_unnamed_config_as_intake_does(tmp_path):
assert capability.template_id == "aic_family:deepseekv32:moe_dsa"


_NEMOTRON_SUPER = "nvidia--NVIDIA-Nemotron-3-Super-120B-A12B-FP8"
_NEMOTRON_REASONS = "recurrent-state field mamba_num_heads, recurrent-state field ssm_state_size"


def _nemotron_super(tmp_path, *, nested, decoder_names_itself=True, sidecar=False):
"""The packaged Nemotron 3 Super config, flat or under llm_config in the #562 Nemotron 3.5 Super VL shape.

decoder_names_itself=False deletes the decoder's architectures entry. None writes it as null, as a wrapper's
PretrainedConfig.to_dict() writes a field that is not set.
"""
config = _packaged_config(_NEMOTRON_SUPER)
if decoder_names_itself is None:
config["architectures"] = None
elif not decoder_names_itself:
del config["architectures"]
if nested:
config = {"architectures": ["NemotronH_Omni_Reasoning_V3"], "llm_config": config}
directory = tmp_path / ("nested" if nested else "flat")
directory.mkdir()
(directory / "config.json").write_text(json.dumps(config))
if sidecar:
quant = _MODEL_CONFIGS / f"{_NEMOTRON_SUPER}_hf_quant_config.json"
(directory / "hf_quant_config.json").write_text(quant.read_text())
return directory


@pytest.mark.parametrize(
"nested,decoder_names_itself,architecture",
[
(False, True, "NemotronHForCausalLM"),
(True, True, "NemotronHForCausalLM"),
(True, False, "NemotronH_Omni_Reasoning_V3"),
],
ids=["flat", "llm-config", "llm-config-without-decoder-architecture"],
)
def test_nemotron_super_vl_llm_config_gets_no_attention_template(
tmp_path, monkeypatch, nested, decoder_names_itself, architecture
):
directory = _nemotron_super(tmp_path, nested=nested, decoder_names_itself=decoder_names_itself)
_forbid_templates_and_timing_data(monkeypatch)
with pytest.raises(ValueError) as error:
_layout_capability(directory, native=False)
assert str(error.value) == _LAYOUT_REFUSAL.format(architecture=architecture, reasons=_NEMOTRON_REASONS)
capability = _layout_capability(directory, native=True)
assert capability.architecture == architecture
_assert_native_runtime(capability, _NEMOTRON_REASONS, is_moe=True)


def test_nemotron_super_vl_llm_config_keeps_the_checkpoint_precision(tmp_path):
flat, nested = (
_layout_capability(_nemotron_super(tmp_path, nested=value, sidecar=True), native=True)
for value in (False, True)
)
assert nested.template_id == flat.template_id == GENERIC_RUNTIME_TEMPLATE_ID
assert nested.dtype.to_dict() == flat.dtype.to_dict()
assert nested.dtype.native_kv_cache_dtype == "fp8"


def test_llm_config_decoder_under_an_unregistered_wrapper_is_checked_against_its_sidecar(tmp_path):
directory = _nemotron_super(tmp_path, nested=True, sidecar=True)
document = json.loads((directory / "config.json").read_text())
# This synthetic wrapper is not in MULTIMODAL_TEXT_CONFIG_KEY, so only decoder_config_key finds its decoder.
document["architectures"] = ["ExampleMultimodalForConditionalGeneration"]
document["llm_config"]["quantization_config"] = {"quant_method": "modelopt", "quant_algo": "NVFP4"}
(directory / "config.json").write_text(json.dumps(document))
with pytest.raises(ValueError, match="conflicting.*hf_quant_config"):
load_model_config(str(directory))


@pytest.mark.parametrize("decoder_names_itself", [True, False, pytest.param(None, id="null")])
def test_llm_config_markers_and_architecture_match_config_intake(tmp_path, decoder_names_itself):
from aisimulate.support.config_profile import load_model_config as load_intake_config
from aisimulate_core.sdk.fpm_profile import _validate_checkpoint_architecture

directory = _nemotron_super(tmp_path, nested=True, decoder_names_itself=decoder_names_itself)
intake = load_intake_config(directory / "config.json")
capability = _layout_capability(directory, native=True)
Comment thread
Arsene12358 marked this conversation as resolved.
Resolved
assert ", ".join(intake.runtime_layout) == _NEMOTRON_REASONS
assert capability.support_reason == (
f"config.json cannot describe this cache layout ({_NEMOTRON_REASONS}). The native runtime measures every cell."
)
# Onboarding writes the intake architecture into the profile, and the planner requires the collector's to match.
assert capability.architecture == intake.suggestions["architecture"]
# The prediction-time profile check reads the same checkpoint and must accept that profile architecture.
profile = SimpleNamespace(
model=str(directory), model_revision=None, architecture=intake.suggestions["architecture"]
)
_validate_checkpoint_architecture(profile)


def _detected_layout_plan(tmp_path, monkeypatch, generator_overrides):
config = _unknown_fp8_moe_config()
config["layer_types"] = ["linear_attention", "full_attention"] * 2
Expand Down
Loading
Loading