diff --git a/.agents/skills/adding-a-new-model/SKILL.md b/.agents/skills/adding-a-new-model/SKILL.md
index 0992a2f3..22332558 100644
--- a/.agents/skills/adding-a-new-model/SKILL.md
+++ b/.agents/skills/adding-a-new-model/SKILL.md
@@ -335,6 +335,27 @@ defaults.
 4. Check weight dtype (numpy arrays must be float32)
 5. Compare layer by layer (moderate diff → norm/residual issue; huge → wrong weights)
 
+### 6. HF `_init_weights` corrupting checkpoint values
+
+**Symptom:** HF reference inference is non-deterministic across model loads —
+different argmax each time, despite identical inputs. Affects golden data
+generation and L4 parity tests.
+
+**Root cause:** Some HF models' `_init_weights` re-initialise parameters
+with random values (e.g. `torch.rand`) AFTER `from_pretrained` loads the
+checkpoint. Known cases:
+- **NemotronH**: `_init_weights` clobbers Mamba2 `dt_bias` with `torch.rand()`
+
+**Fix:** `_fix_nemotron_h_dt_bias()` in `mobius._testing.torch_reference`
+reads correct values from safetensors files and patches them in-place.
+Always call after `from_pretrained` for NemotronH models.
+
+**Diagnosis pattern:** Load the model twice and compare outputs. If argmax
+differs between loads, suspect `_init_weights` corruption. Set
+`torch.manual_seed(42)` before `from_pretrained` — if that makes outputs
+deterministic, `_init_weights` is the culprit. Then compare specific
+parameters between the loaded model and the safetensors checkpoint.
+
 > For additional troubleshooting (gated attention split ordering, DeltaNet
 > scaling, identity node folding, fp32 upcast patterns, multi-token prefill,
 > embedding table off-by-one), read
diff --git a/.agents/skills/moe-models/SKILL.md b/.agents/skills/moe-models/SKILL.md
index 662cb582..26f055ad 100644
--- a/.agents/skills/moe-models/SKILL.md
+++ b/.agents/skills/moe-models/SKILL.md
@@ -382,3 +382,138 @@ def _dispatch_moe_fallback(self, op, hidden, router_probs):
         ...
     return output
 ```
+
+## NemotronH MoE (sigmoid routing + shared experts + latent projection)
+
+NemotronH uses a non-standard MoE architecture that differs from the
+standard softmax top-k pattern in several ways.
+
+### Architecture
+
+```
+NemotronHMoEBlock
+ ├── NemotronHMoEGate (sigmoid top-k with correction bias)
+ ├── [optional] fc1_latent_proj (hidden → latent_size)
+ ├── Experts[0..N-1] (non-gated FCMLP: up → act → down)
+ ├── [optional] fc2_latent_proj (latent_size → hidden)
+ └── SharedExperts (FCMLP, all tokens, residual add)
+```
+
+### Classes
+
+| Class | File | Purpose |
+|-------|------|---------|
+| `NemotronHMoEGate` | `models/nemotron_h.py` | Sigmoid routing with e_score_correction_bias |
+| `NemotronHMoEBlock` | `models/nemotron_h.py` | MoE dispatch with shared expert + latent proj |
+| `NemotronHMoELayer` | `models/nemotron_h.py` | Pre-norm → MoE block → residual (stateless) |
+
+### Sigmoid gate with correction bias
+
+NemotronH does NOT use softmax routing. Instead:
+
+1. `router_logits = hidden_states @ gate_weight.T`
+2. `probs = sigmoid(router_logits)` — these become final routing weights
+3. `choice_scores = probs + e_score_correction_bias` — bias affects selection only
+4. `selected = topk(choice_scores)` — select top-k using biased scores
+5. `weights = gather(probs, selected)` — gather from UNBIASED sigmoid probs
+6. Normalize + scale by `routed_scaling_factor`
+
+**Key difference from standard MoE**: The correction bias shifts expert
+selection but does NOT affect final routing weights. The `com.microsoft.MoE`
+op's built-in softmax routing is incompatible — you must use the fallback
+loop or pre-compute routing weights and pass them to a modified MoE call.
+
+### Non-gated FCMLP experts
+
+Unlike Mixtral/Qwen (gated: `gate_proj * up_proj → down_proj`), NemotronH
+experts are simple FCMLPs: `up_proj → activation → down_proj`. This maps
+to `fc1_experts_weights` / `fc2_experts_weights` without a gate projection.
+
+### Latent projection (120B only)
+
+The 120B model has `moe_latent_size=1024` (vs `hidden_size=4096`):
+
+```
+hidden → fc1_latent_proj(4096→1024) → experts(1024→inter→1024) → fc2_latent_proj(1024→4096)
+```
+
+Gate routes on original hidden states (NOT latent). Shared expert operates
+on original hidden_size (no latent projection).
+
+### Shared expert
+
+A single FCMLP that runs on ALL tokens (not routed), added as residual:
+
+```python
+output = routed_expert_output + shared_experts(original_hidden)
+```
+
+### HF weight format
+
+HF stores expert weights as 3D stacked tensors:
+```
+experts.up_proj: [num_experts, intermediate_size, input_size]
+experts.down_proj: [num_experts, hidden_size, intermediate_size]
+```
+
+`preprocess_weights()` splits these into per-expert 2D tensors for the
+loop-based dispatch, or keeps them stacked for the fused MoE op path.
+
+### Config fields (NemotronHConfig)
+
+```python
+config.num_local_experts             # Total experts (128 for 30B, 512 for 120B)
+config.num_experts_per_tok           # Top-k (6 for 30B, 22 for 120B)
+config.moe_intermediate_size         # Per-expert hidden dim
+config.moe_latent_size               # Optional latent projection dim (120B: 1024)
+config.shared_expert_intermediate_size  # Shared expert hidden dim
+config.norm_topk_prob                # Whether to normalize routing weights
+config.routed_scaling_factor         # Post-normalization scale
+```
+
+### com.microsoft.MoE compatibility
+
+**Not compatible with NemotronH.** Three blockers:
+
+1. **Squared ReLU activation**: NemotronH experts use `relu2` (squared ReLU:
+   `relu(x)^2`). The fused MoE op only supports `silu`, `gelu`, `relu`,
+   `none` — no squared ReLU. Since the activation is applied between the
+   two matmuls inside the fused op, there is no way to inject a custom
+   activation.
+
+2. **Sigmoid routing with correction bias**: NemotronH gate uses
+   `sigmoid → add_bias → topk` for expert selection, but final routing
+   weights come from the unbiased sigmoid probs. The fused op has no
+   option to bypass its internal softmax/topk routing.
+
+3. **Shared expert + latent projection**: These must run outside the fused
+   op regardless, adding complexity without eliminating the main bottleneck.
+
+NemotronH uses loop-over-experts dispatch. See `NemotronHMoEBlock` docstring.
+
+### Graph size impact
+
+The loop-over-experts fallback creates one subgraph per expert per MoE layer:
+- 30B: 128 experts × 23 MoE layers = 2,944 expert subgraphs → ~40K nodes
+- 120B: 512 experts × 40 MoE layers = 20,480 expert subgraphs → ~270K nodes
+
+The fused MoE op replaces each per-layer loop with a single op, dramatically
+reducing graph size and enabling batched GPU execution.
+
+### HF dt_bias corruption bug
+
+**Critical**: The NemotronH remote-code `_init_weights` re-initialises
+Mamba2 `dt_bias` parameters with `torch.rand()` AFTER `from_pretrained`
+loads checkpoint weights, silently corrupting the model. HF inference
+becomes non-deterministic — different argmax on each model load.
+
+Fix: Use `_fix_nemotron_h_dt_bias()` from `mobius._testing.torch_reference`
+after loading the HF model. This reads correct `dt_bias` values from the
+safetensors files and patches them in-place. Without the fix, golden
+reference data is unreliable.
+
+```python
+from mobius._testing.torch_reference import _fix_nemotron_h_dt_bias
+model = AutoModelForCausalLM.from_pretrained(model_id, ...)
+_fix_nemotron_h_dt_bias(model, model_id)  # Must call before eval()
+```
diff --git a/scripts/generate_golden.py b/scripts/generate_golden.py
index 9105ad6e..303c439a 100644
--- a/scripts/generate_golden.py
+++ b/scripts/generate_golden.py
@@ -152,6 +152,34 @@ def _extract_logits_golden(
     }
 
 
+# ---- Compat patches ----
+
+
+def _apply_nemotron_h_generate_patch(model: object) -> None:
+    """Patch NemotronH prepare_inputs_for_generation for transformers 5.x.
+
+    The HF remote code accesses ``cache_position[-1]`` without checking
+    for ``None``, which crashes under transformers >=5.x where
+    ``cache_position`` is no longer passed on the first prefill call.
+    We wrap the method to supply a default ``cache_position`` when missing.
+    """
+    cls_name = type(model).__name__
+    if "NemotronH" not in cls_name:
+        return
+
+    import torch
+
+    original_prepare = model.prepare_inputs_for_generation
+
+    def _patched_prepare(input_ids, **kwargs):
+        if kwargs.get("past_key_values") is not None and kwargs.get("cache_position") is None:
+            seq_len = input_ids.shape[-1]
+            kwargs["cache_position"] = torch.arange(seq_len, device=input_ids.device)
+        return original_prepare(input_ids, **kwargs)
+
+    model.prepare_inputs_for_generation = _patched_prepare
+
+
 # ---- Task-specific generators ----
 # Each generator loads a HF model, runs inference, and calls
 # save_golden_ref() from golden.py.  Heavy imports (torch,
@@ -184,6 +212,8 @@ def _generate_causal_lm(case: TestCase, json_path: Path, device: str) -> None:
     if "L5" in case.level:
         import torch
 
+        _apply_nemotron_h_generate_patch(model)
+
         with torch.no_grad():
             gen_output = model.generate(
                 torch.from_numpy(input_ids).to(device),
diff --git a/src/mobius/_configs.py b/src/mobius/_configs.py
index efc5be1c..73ecc231 100644
--- a/src/mobius/_configs.py
+++ b/src/mobius/_configs.py
@@ -2244,6 +2244,7 @@ class NemotronHConfig(ArchitectureConfig):
     mamba_conv_bias: bool = True
     mamba_proj_bias: bool = False
     mamba_time_step_min: float = 0.001
+    moe_latent_size: int | None = None
 
     @classmethod
     def from_transformers(cls, config, parent_config=None) -> NemotronHConfig:
@@ -2253,15 +2254,20 @@ def from_transformers(cls, config, parent_config=None) -> NemotronHConfig:
         layers_block_type = getattr(config, "layers_block_type", None)
         if layers_block_type is None:
             pattern = getattr(config, "hybrid_override_pattern", "")
-            # Map pattern chars: M=mamba2, *=full_attention, -=mlp
-            char_map = {"M": "mamba2", "*": "full_attention", "-": "mlp"}
+            # Map pattern chars: M=mamba2, *=full_attention, -=mlp, E=moe
+            char_map = {
+                "M": "mamba2",
+                "*": "full_attention",
+                "-": "mlp",
+                "E": "moe",
+            }
             layers_block_type = [char_map.get(c, "mamba2") for c in pattern]
         else:
             # Convert HF names to mobius names
             type_map = {
                 "mamba": "mamba2",
                 "attention": "full_attention",
-                "moe": "mlp",
+                "moe": "moe",
             }
             layers_block_type = [type_map.get(t, t) for t in layers_block_type]
 
@@ -2280,8 +2286,24 @@ def from_transformers(cls, config, parent_config=None) -> NemotronHConfig:
         base_fields = {
             k: v
             for k, v in _shallow_fields(base).items()
-            if k not in ("layer_types", "num_hidden_layers", "hidden_act")
+            if k
+            not in (
+                "layer_types",
+                "num_hidden_layers",
+                "hidden_act",
+                "moe_latent_size",
+                "shared_expert_intermediate_size",
+            )
         }
+
+        # Extract shared expert intermediate size (NemotronH uses a dedicated
+        # field name different from the base ArchitectureConfig default).
+        shared_expert_intermediate_size = getattr(
+            config,
+            "moe_shared_expert_intermediate_size",
+            base.shared_expert_intermediate_size,
+        )
+
         return cls(
             **base_fields,
             num_hidden_layers=n,
@@ -2296,6 +2318,8 @@ def from_transformers(cls, config, parent_config=None) -> NemotronHConfig:
             mamba_conv_bias=getattr(config, "use_conv_bias", True),
             mamba_proj_bias=getattr(config, "mamba_proj_bias", False),
             mamba_time_step_min=getattr(config, "time_step_min", 0.001),
+            moe_latent_size=getattr(config, "moe_latent_size", None),
+            shared_expert_intermediate_size=shared_expert_intermediate_size,
         )
 
 
diff --git a/src/mobius/_testing/torch_reference.py b/src/mobius/_testing/torch_reference.py
index 1f16db7a..6211f410 100644
--- a/src/mobius/_testing/torch_reference.py
+++ b/src/mobius/_testing/torch_reference.py
@@ -5,9 +5,93 @@
 
 from __future__ import annotations
 
+import logging
+
 import numpy as np
 import torch
 
+logger = logging.getLogger(__name__)
+
+
+def _fix_nemotron_h_init_weights(model: torch.nn.Module, model_id: str) -> None:
+    """Restore Mamba2 params from checkpoint after HF clobbers them.
+
+    The NemotronH remote-code ``_init_weights`` re-initialises several
+    parameters *after* ``from_pretrained`` loads the checkpoint:
+
+    - ``dt_bias``: overwritten with ``torch.rand(...)``
+    - ``out_proj.weight`` (in Mamba mixer layers): overwritten with
+      ``kaiming_uniform_`` then scaled by ``1/sqrt(n_layers)`` when
+      ``rescale_prenorm_residual`` is True
+
+    This helper reads the original values back from the safetensors
+    files on disk and patches them in-place.
+    """
+    config = getattr(model, "config", None)
+    model_type = getattr(config, "model_type", None)
+    if model_type != "nemotron_h":
+        return
+
+    try:
+        from safetensors import safe_open
+    except ImportError:
+        return
+
+    import glob
+    import os
+
+    from huggingface_hub import snapshot_download
+
+    # Resolve the exact snapshot directory used by HF for this model,
+    # avoiding lexicographic guessing across multiple cached revisions.
+    try:
+        snapshot = snapshot_download(model_id, local_files_only=True)
+    except Exception:
+        logger.warning(
+            "NemotronH init_weights fix: could not resolve snapshot for %s",
+            model_id,
+        )
+        return
+
+    safetensor_files = sorted(glob.glob(os.path.join(snapshot, "*.safetensors")))
+
+    # Collect parameter names that _init_weights corrupts:
+    # 1. All dt_bias params (Mamba2 layers)
+    # 2. mixer.out_proj.weight params (rescale_prenorm_residual)
+    corrupted_suffixes = {"dt_bias"}
+    if getattr(config, "rescale_prenorm_residual", False):
+        corrupted_suffixes.add("mixer.out_proj.weight")
+
+    patched = 0
+    state = model.state_dict()
+    for f in safetensor_files:
+        with safe_open(f, framework="pt") as st:
+            # safe_open objects aren't directly iterable
+            keys = st.keys()
+            for key in keys:
+                if not any(key.endswith(s) for s in corrupted_suffixes):
+                    continue
+                if key not in state:
+                    continue
+                ckpt_val = st.get_tensor(key)
+                param = state[key]
+                with torch.no_grad():
+                    param.copy_(ckpt_val.to(param.device, dtype=param.dtype))
+                patched += 1
+
+    # Write the fixed values back into the live model
+    if patched:
+        model.load_state_dict(state, strict=False)
+        logger.info(
+            "NemotronH: restored %d params from checkpoint (dt_bias + out_proj.weight)",
+            patched,
+        )
+    else:
+        logger.warning(
+            "NemotronH init_weights fix: no corrupted params found in checkpoint for %s",
+            model_id,
+        )
+
 
 def load_torch_model(
     model_id: str,
@@ -27,12 +111,22 @@ def load_torch_model(
     import transformers
 
     tokenizer = transformers.AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+
+    # NemotronH: disable rescale_prenorm_residual before loading to
+    # prevent _init_weights from corrupting out_proj.weight with
+    # random kaiming_uniform_ initialization after checkpoint loading.
+    config = transformers.AutoConfig.from_pretrained(model_id, trust_remote_code=True)
+    if getattr(config, "model_type", None) == "nemotron_h":
+        config.rescale_prenorm_residual = False
+
     model = transformers.AutoModelForCausalLM.from_pretrained(
         model_id,
+        config=config,
         dtype=dtype,
         device_map=device,
         trust_remote_code=True,
     )
+    _fix_nemotron_h_init_weights(model, model_id)
     model.eval()
 
     if tokenizer.pad_token is None:
diff --git a/src/mobius/models/nemotron_h.py b/src/mobius/models/nemotron_h.py
index c33ed232..85ce2b84 100644
--- a/src/mobius/models/nemotron_h.py
+++ b/src/mobius/models/nemotron_h.py
@@ -1,25 +1,28 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 
-"""NemotronH hybrid Mamba2 + Attention + MLP causal language model.
+"""NemotronH hybrid Mamba2 + Attention + MLP + MoE causal language model.
 
-NemotronH interleaves three layer types in a configurable pattern:
+NemotronH interleaves up to four layer types in a configurable pattern:
 - Mamba2/SSD layers for efficient recurrent processing
 - Transformer attention layers for global context
 - Dense MLP layers for feedforward computation
+- MoE layers for sparse expert routing (Nemotron-3 30B/120B)
 
 Each layer is a single-mixer block: RMSNorm → mixer → residual.
 Unlike Jamba/Bamba where every layer has a mixer AND MLP, NemotronH
-treats MLP as a standalone layer type.
+treats MLP and MoE as standalone layer types.
 
 Layer types are specified via ``layers_block_type`` in the HF config:
-``M`` = mamba, ``*`` = attention, ``-`` = mlp (dense feedforward).
+``M`` = mamba, ``*`` = attention, ``-`` = mlp (dense feedforward),
+``E`` = moe (sparse mixture of experts).
 
 State per layer:
     Mamba2: conv_state (batch, conv_dim, d_conv-1)
             ssm_state (batch, num_heads, d_head, d_state)
     Attention: standard KV cache (key + value)
     MLP: stateless — no cache
+    MoE: stateless — no cache
 
 HuggingFace reference: ``NemotronHForCausalLM``.
 """
@@ -194,6 +197,238 @@ def forward(
         return hidden_states, (None, None)
 
 
+# ---------------------------------------------------------------------------
+# MoE components (Nemotron-3 30B/120B)
+# ---------------------------------------------------------------------------
+
+
+class NemotronHMoEGate(nn.Module):
+    """Sigmoid top-k gate with score correction bias (NemotronH style).
+
+    Routing:
+        1. router_logits = linear(hidden_states)  [in float32]
+        2. probs = sigmoid(router_logits)
+        3. choice_scores = probs + e_score_correction_bias
+        4. selected_experts = topk(choice_scores)
+        5. routing_weights = gather(probs, selected_experts)
+        6. normalize + scale
+
+    The correction bias shifts expert selection but does NOT affect the
+    final routing weights (which come from the original sigmoid probs).
+    """
+
+    def __init__(
+        self,
+        hidden_size: int,
+        num_experts: int,
+        top_k: int,
+        *,
+        norm_topk_prob: bool = True,
+        routed_scaling_factor: float = 1.0,
+    ):
+        super().__init__()
+        self.num_experts = num_experts
+        self.top_k = top_k
+        self.norm_topk_prob = norm_topk_prob
+        self.routed_scaling_factor = routed_scaling_factor
+        self.weight = nn.Parameter([num_experts, hidden_size])
+        # Correction bias for expert selection (loaded from checkpoint)
+        self.e_score_correction_bias = nn.Parameter([num_experts])
+
+    def forward(self, op: builder.OpBuilder, hidden_states: ir.Value):
+        # Cast to float32 for numerical stability (eps=1e-20 underflows
+        # in fp16/bf16).  HF does the same: hidden_states.type(torch.float32)
+        # in NemotronHTopkRouter.forward and never casts back.
+        hidden_states = op.Cast(hidden_states, to=1)  # FLOAT32
+
+        weight_t = op.Transpose(self.weight, perm=[1, 0])
+        router_logits = op.MatMul(hidden_states, weight_t)
+
+        # Sigmoid probabilities (these become the final routing weights)
+        probs = op.Sigmoid(router_logits)
+
+        # Add correction bias for expert selection only
+        choice_scores = op.Add(probs, self.e_score_correction_bias)
+
+        # Select top-k experts based on biased scores
+        k = op.Constant(value_ints=[self.top_k])
+        _top_vals, selected_experts = op.TopK(choice_scores, k, axis=-1, _outputs=2)
+
+        # Gather actual routing weights from unbiased probs
+        routing_weights = op.GatherElements(probs, selected_experts, axis=-1)
+
+        if self.norm_topk_prob:
+            weight_sum = op.ReduceSum(routing_weights, [-1], keepdims=True)
+            routing_weights = op.Div(routing_weights, op.Add(weight_sum, 1e-20))
+        if self.routed_scaling_factor != 1.0:  # noqa: RUF069
+            routing_weights = op.Mul(routing_weights, self.routed_scaling_factor)
+
+        # Keep routing_weights in float32 (matching HF which never casts back).
+        # The expert dispatch multiplies these with expert outputs, and ONNX
+        # type promotion handles the mixed-dtype matmul naturally.
+        return routing_weights, selected_experts
+
+
+class NemotronHMoEBlock(nn.Module):
+    """NemotronH MoE block with non-gated experts and shared expert.
+
+    Unlike standard MoE (gated MLP experts), NemotronH uses:
+    - Non-gated FCMLP experts: up_proj → act → down_proj
+    - Optional latent projection wrapping the routed experts
+    - Shared expert (FCMLP) added as residual
+
+    Architecture::
+
+        [optional] hidden → fc1_latent_proj → latent
+        latent → routed experts (FCMLP) → expert_output
+        [optional] expert_output → fc2_latent_proj → hidden
+        output = expert_output + shared_experts(original_hidden)
+
+    Note: ``com.microsoft.MoE`` fused op is **not compatible** with NemotronH
+    because NemotronH uses squared ReLU (relu2) activation, which the fused
+    op doesn't support (only silu/gelu/relu/none). Additionally, NemotronH
+    uses sigmoid routing with correction bias (not softmax), and the fused
+    op has no softmax bypass option. We use loop-over-experts dispatch.
+
+    HuggingFace reference: ``NemotronHMoE``.
+    """
+
+    def __init__(self, config: NemotronHConfig):
+        super().__init__()
+        assert config.num_local_experts is not None
+        assert config.num_experts_per_tok is not None
+        num_experts = config.num_local_experts
+        top_k = config.num_experts_per_tok
+
+        self.gate = NemotronHMoEGate(
+            config.hidden_size,
+            num_experts,
+            top_k,
+            norm_topk_prob=config.norm_topk_prob,
+            routed_scaling_factor=config.routed_scaling_factor,
+        )
+
+        # Expert input/output dimension depends on latent projection
+        expert_input_dim = (
+            config.moe_latent_size
+            if config.moe_latent_size is not None
+            else config.hidden_size
+        )
+        assert config.moe_intermediate_size is not None
+        self.experts = nn.ModuleList(
+            [
+                FCMLP(
+                    expert_input_dim,
+                    config.moe_intermediate_size,
+                    activation=config.hidden_act,
+                    bias=config.mlp_bias,
+                )
+                for _ in range(num_experts)
+            ]
+        )
+
+        # Shared expert processes all tokens (not routed)
+        shared_intermediate = (
+            config.shared_expert_intermediate_size or config.moe_intermediate_size
+        )
+        self.shared_experts = FCMLP(
+            config.hidden_size,
+            shared_intermediate,
+            activation=config.hidden_act,
+            bias=config.mlp_bias,
+        )
+
+        # Optional latent projection (e.g. 120B: 4096 → 1024 → experts
+        # → 1024 → 4096)
+        self._has_latent = config.moe_latent_size is not None
+        if self._has_latent:
+            self.fc1_latent_proj = Linear(
+                config.hidden_size,
+                config.moe_latent_size,
+                bias=config.mlp_bias,
+            )
+            self.fc2_latent_proj = Linear(
+                config.moe_latent_size,
+                config.hidden_size,
+                bias=config.mlp_bias,
+            )
+
+    def forward(self, op: builder.OpBuilder, hidden_states: ir.Value):
+        residual = hidden_states
+
+        # Gate routes on original hidden states
+        routing_weights, selected_experts = self.gate(op, hidden_states)
+
+        # Optional latent projection before expert dispatch
+        if self._has_latent:
+            hidden_states = self.fc1_latent_proj(op, hidden_states)
+
+        # Loop-over-experts dispatch: each expert processes all tokens,
+        # then results are masked and weighted by routing weights
+        result = None
+        for expert_idx, expert in enumerate(self.experts):
+            expert_output = expert(op, hidden_states)
+            expert_id = op.Constant(value_int=expert_idx)
+            # match: True where this expert was selected
+            match = op.Equal(selected_experts, expert_id)
+            match_float = op.CastLike(match, routing_weights)
+            weighted = op.Mul(routing_weights, match_float)
+            # Sum matched routing weights across top_k dim → per-token weight
+            weight = op.ReduceSum(weighted, [-1], keepdims=True)
+            contribution = op.Mul(expert_output, weight)
+            if result is None:
+                result = contribution
+            else:
+                result = op.Add(result, contribution)
+
+        # Optional latent projection back to hidden_size
+        if self._has_latent:
+            result = self.fc2_latent_proj(op, result)
+
+        # Add shared expert output (operates on original hidden states)
+        shared_output = self.shared_experts(op, residual)
+        result = op.Add(result, shared_output)
+        return result
+
+
+class NemotronHMoELayer(nn.Module):
+    """NemotronH MoE layer: RMSNorm → MoE block → residual.
+
+    Single-mixer block — stateless, no cache.
+
+    Args:
+        config: NemotronH architecture config.
+    """
+
+    def __init__(self, config: NemotronHConfig):
+        super().__init__()
+        self.moe = NemotronHMoEBlock(config)
+        self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    def forward(
+        self,
+        op: builder.OpBuilder,
+        hidden_states: ir.Value,
+        attention_bias: ir.Value,
+        position_embeddings: tuple,
+        past_key_value: tuple | None,
+    ):
+        """Forward pass. Returns (hidden_states, (None, None)).
+
+        MoE layers are stateless — the None pair keeps the cache
+        list aligned with all layers.
+        """
+        del attention_bias, position_embeddings, past_key_value  # unused
+
+        # Pre-norm → MoE → residual
+        residual = hidden_states
+        hidden_states = self.norm(op, hidden_states)
+        hidden_states = self.moe(op, hidden_states)
+        hidden_states = op.Add(residual, hidden_states)
+
+        return hidden_states, (None, None)
+
+
 # ---------------------------------------------------------------------------
 # Full model
 # ---------------------------------------------------------------------------
@@ -223,6 +458,8 @@ def __init__(self, config: NemotronHConfig):
                 self.layers.append(NemotronHMambaLayer(config))
             elif ltype == "mlp":
                 self.layers.append(NemotronHMLPLayer(config))
+            elif ltype == "moe":
+                self.layers.append(NemotronHMoELayer(config))
             else:
                 self.layers.append(NemotronHAttentionLayer(config))
 
@@ -314,6 +551,8 @@ def preprocess_weights(
            - mamba: ``mixer.`` → ``mamba.``
            - attention: ``mixer.`` → ``self_attn.``
            - mlp: ``mixer.`` → ``mlp.``
+           - moe: ``mixer.`` → ``moe.``
+        6. MoE stacked 3D expert tensors split into per-expert 2D weights
         """
         layer_types = self.config.layer_types or []
 
@@ -333,7 +572,20 @@ def preprocess_weights(
         new_state_dict: dict[str, torch.Tensor] = {}
         for key, value in state_dict.items():
             new_key = _rename_nemotron_h_weight(key, layer_types)
-            new_state_dict[new_key] = value
+            # Split stacked 3D expert tensors into per-expert 2D weights.
+            # HF stores experts.up_proj as (num_experts, inter, input) and
+            # experts.down_proj as (num_experts, input, inter). We need
+            # individual experts.{i}.up_proj.weight / down_proj.weight.
+            if _is_stacked_expert_tensor(new_key, value):
+                for i, expert_weight in enumerate(value):
+                    # value[i] is (out_dim, in_dim) — standard 2D weight
+                    suffix = new_key.rsplit(".", 1)[-1]  # "up_proj"/"down_proj"
+                    expert_key = (
+                        new_key.rsplit("experts.", 1)[0] + f"experts.{i}.{suffix}.weight"
+                    )
+                    new_state_dict[expert_key] = expert_weight
+            else:
+                new_state_dict[new_key] = value
 
         return new_state_dict
 
@@ -353,6 +605,10 @@ def _rename_nemotron_h_weight(key: str, layer_types: list[str]) -> str:
         {backbone|model}.layers.N.mixer.{in_proj, conv1d, out_proj, norm, A_log, D, dt_bias}  (mamba)
         {backbone|model}.layers.N.mixer.{q_proj, k_proj, v_proj, o_proj}.weight              (attention)
         {backbone|model}.layers.N.mixer.{up_proj, down_proj}.weight                          (mlp)
+        {backbone|model}.layers.N.mixer.gate.{weight, e_score_correction_bias}               (moe gate)
+        {backbone|model}.layers.N.mixer.experts.{up_proj, down_proj}                         (moe experts, 3D stacked)
+        {backbone|model}.layers.N.mixer.shared_experts.{up_proj, down_proj}.weight           (moe shared expert)
+        {backbone|model}.layers.N.mixer.{fc1_latent_proj, fc2_latent_proj}.weight             (moe latent)
         lm_head.weight
 
     ONNX parameter naming:
@@ -362,6 +618,10 @@ def _rename_nemotron_h_weight(key: str, layer_types: list[str]) -> str:
         model.layers.N.mamba.{in_proj, conv1d, out_proj, norm, A_log, D, dt_bias}
         model.layers.N.self_attn.{q_proj, k_proj, v_proj, o_proj}.weight
         model.layers.N.mlp.{up_proj, down_proj}.weight
+        model.layers.N.moe.gate.{weight, e_score_correction_bias}
+        model.layers.N.moe.experts.{i}.{up_proj, down_proj}.weight  (split from 3D)
+        model.layers.N.moe.shared_experts.{up_proj, down_proj}.weight
+        model.layers.N.moe.{fc1_latent_proj, fc2_latent_proj}.weight
         lm_head.weight
     """
     # Global prefix renames (handle both backbone.* and model.* HF names)
@@ -384,6 +644,8 @@ def _rename_nemotron_h_weight(key: str, layer_types: list[str]) -> str:
                 return f"model.layers.{layer_idx}.mamba.{mixer_rest}"
             elif ltype == "full_attention":
                 return f"model.layers.{layer_idx}.self_attn.{mixer_rest}"
+            elif ltype == "moe":
+                return f"model.layers.{layer_idx}.moe.{mixer_rest}"
             else:  # mlp
                 return f"model.layers.{layer_idx}.mlp.{mixer_rest}"
 
@@ -395,3 +657,14 @@ def _rename_nemotron_h_weight(key: str, layer_types: list[str]) -> str:
         return key.replace("backbone.", "model.", 1)
 
     return key
+
+
+def _is_stacked_expert_tensor(key: str, value: torch.Tensor) -> bool:
+    """Check if a weight is a stacked 3D expert tensor that needs splitting.
+
+    HF NemotronH stores expert weights as (num_experts, out_dim, in_dim).
+    These need to be split into per-expert 2D weights.
+    """
+    return (
+        value.ndim == 3 and ".moe.experts." in key and key.endswith((".up_proj", ".down_proj"))
+    )
diff --git a/src/mobius/tasks/_cache_utils.py b/src/mobius/tasks/_cache_utils.py
index 8049aaf1..6a2d6875 100644
--- a/src/mobius/tasks/_cache_utils.py
+++ b/src/mobius/tasks/_cache_utils.py
@@ -220,8 +220,8 @@ def _make_hybrid_cache_inputs(
             )
             flat.append(conv_state)
             pairs.append((conv_state,))  # 1-tuple: conv has no second state
-        elif ltype == "mlp":
-            # MLP-only layers are stateless — no cache inputs needed
+        elif ltype in ("mlp", "moe"):
+            # MLP and MoE layers are stateless — no cache inputs needed
             pairs.append((None, None))
         elif ltype == "mamba":
             conv_state = ir.Value(
@@ -290,8 +290,8 @@ def _register_hybrid_cache_outputs(
     """
     for i, states in enumerate(present_key_values):
         ltype = layer_types[i] if i < len(layer_types) else "full_attention"
-        if ltype == "mlp":
-            continue  # MLP layers produce no cache state
+        if ltype == "mlp" or ltype == "moe":
+            continue  # MLP and MoE layers produce no cache state
         if ltype == "lightning_attention":
             # Single recurrent state only (no conv_state for lightning)
             (state_a,) = states
diff --git a/testdata/cases/causal-lm/nemotron-3-nano-30b.yaml b/testdata/cases/causal-lm/nemotron-3-nano-30b.yaml
new file mode 100644
index 00000000..ff6adf5e
--- /dev/null
+++ b/testdata/cases/causal-lm/nemotron-3-nano-30b.yaml
@@ -0,0 +1,13 @@
+model_id: "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16"
+revision: "main"
+task_type: "text-generation"
+dtype: "float32"
+
+inputs:
+  prompts:
+    - "Here is my poem:"
+
+level: "L4"
+
+ci_skip_reason: "30B MoE model requires ~60GB VRAM; too large for CI."
+notes: "NemotronH Nano 30B-A3B. Hybrid Mamba2 + MoE + Attention from NVIDIA. 128 routed experts, top-6. L5 skipped: HF remote code generate() is broken (cache_position bug)."
diff --git a/testdata/cases/causal-lm/nemotron-3-super-120b.yaml b/testdata/cases/causal-lm/nemotron-3-super-120b.yaml
new file mode 100644
index 00000000..65e1d299
--- /dev/null
+++ b/testdata/cases/causal-lm/nemotron-3-super-120b.yaml
@@ -0,0 +1,13 @@
+model_id: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16"
+revision: "main"
+task_type: "text-generation"
+dtype: "float32"
+
+inputs:
+  prompts:
+    - "Here is my poem:"
+
+level: "L4"
+
+ci_skip_reason: "120B MoE model requires ~240GB VRAM; too large for CI."
+notes: "NemotronH Super 120B-A12B. Hybrid Mamba2 + MoE + Attention from NVIDIA. 512 routed experts, top-22, latent projections. L5 skipped: HF remote code generate() is broken (cache_position bug)."
diff --git a/testdata/cases/causal-lm/nemotron-h-nano-4b.yaml b/testdata/cases/causal-lm/nemotron-h-nano-4b.yaml
index 6650f397..69525c54 100644
--- a/testdata/cases/causal-lm/nemotron-h-nano-4b.yaml
+++ b/testdata/cases/causal-lm/nemotron-h-nano-4b.yaml
@@ -13,5 +13,4 @@ generation:
   max_new_tokens: 20
   do_sample: false
 
-skip_reason: "Requires mamba-ssm package (CUDA-only) for HF inference."
-notes: "NemotronH Nano 4B. Hybrid Mamba2 + Attention + MLP architecture from NVIDIA."
+notes: "NemotronH Nano 4B. Hybrid Mamba2 + Attention + MoE architecture from NVIDIA."
diff --git a/testdata/golden/causal-lm/nemotron-3-nano-30b.json b/testdata/golden/causal-lm/nemotron-3-nano-30b.json
new file mode 100644
index 00000000..c2cf50f3
--- /dev/null
+++ b/testdata/golden/causal-lm/nemotron-3-nano-30b.json
@@ -0,0 +1,41 @@
+{
+  "top1_id": 1256,
+  "top2_id": 10736,
+  "top10_ids": [
+    1256,
+    10736,
+    4410,
+    1429,
+    1766,
+    1576,
+    6421,
+    1032,
+    2129,
+    1364
+  ],
+  "top10_logits": [
+    "0x1.41d4f00000000p+4",
+    "0x1.3785b80000000p+4",
+    "0x1.31355a0000000p+4",
+    "0x1.2fe2d60000000p+4",
+    "0x1.284cfa0000000p+4",
+    "0x1.25cd760000000p+4",
+    "0x1.1d08aa0000000p+4",
+    "0x1.18a7680000000p+4",
+    "0x1.18083e0000000p+4",
+    "0x1.0d2ccc0000000p+4"
+  ],
+  "logits_summary": [
+    "0x1.41d4f00000000p+4",
+    "-0x1.c873080000000p+2",
+    "0x1.f216bd75e2000p+0",
+    "0x1.36aaffa7d12abp+1"
+  ],
+  "input_ids": [
+    11745,
+    1395,
+    2036,
+    28699,
+    1058
+  ]
+}
diff --git a/testdata/golden/causal-lm/nemotron-3-super-120b.json b/testdata/golden/causal-lm/nemotron-3-super-120b.json
new file mode 100644
index 00000000..78afc7b5
--- /dev/null
+++ b/testdata/golden/causal-lm/nemotron-3-super-120b.json
@@ -0,0 +1,41 @@
+{
+  "top1_id": 1256,
+  "top2_id": 1032,
+  "top10_ids": [
+    1256,
+    1032,
+    1362,
+    1429,
+    1319,
+    1531,
+    1766,
+    1576,
+    1349,
+    2129
+  ],
+  "top10_logits": [
+    "0x1.f90a840000000p+5",
+    "0x1.f348f60000000p+5",
+    "0x1.ed63bc0000000p+5",
+    "0x1.eca1d00000000p+5",
+    "0x1.eba2060000000p+5",
+    "0x1.eb9f4a0000000p+5",
+    "0x1.e90f6e0000000p+5",
+    "0x1.e565740000000p+5",
+    "0x1.e1520e0000000p+5",
+    "0x1.e13ed00000000p+5"
+  ],
+  "logits_summary": [
+    "0x1.f90a840000000p+5",
+    "0x1.f497cc0000000p+4",
+    "0x1.6e07380000000p+5",
+    "0x1.b489340000000p+1"
+  ],
+  "input_ids": [
+    11745,
+    1395,
+    2036,
+    28699,
+    1058
+  ]
+}
diff --git a/testdata/golden/causal-lm/nemotron-h-nano-4b.json b/testdata/golden/causal-lm/nemotron-h-nano-4b.json
new file mode 100644
index 00000000..f7786455
--- /dev/null
+++ b/testdata/golden/causal-lm/nemotron-h-nano-4b.json
@@ -0,0 +1,41 @@
+{
+  "top1_id": 1429,
+  "top2_id": 1362,
+  "top10_ids": [
+    1429,
+    1362,
+    1032,
+    1278,
+    1531,
+    2036,
+    2744,
+    4890,
+    1349,
+    1261
+  ],
+  "top10_logits": [
+    "0x1.2657780000000p+3",
+    "0x1.24ff800000000p+3",
+    "0x1.0eb9de0000000p+3",
+    "0x1.0e870c0000000p+3",
+    "0x1.0a38dc0000000p+3",
+    "0x1.0848a20000000p+3",
+    "0x1.0049940000000p+3",
+    "0x1.da4f580000000p+2",
+    "0x1.d24dce0000000p+2",
+    "0x1.ccbf5e0000000p+2"
+  ],
+  "logits_summary": [
+    "0x1.2657780000000p+3",
+    "-0x1.333ce60000000p+3",
+    "-0x1.aee0fa5a37ca0p+1",
+    "0x1.082338f97b2fdp+1"
+  ],
+  "input_ids": [
+    11745,
+    1395,
+    2036,
+    28699,
+    1058
+  ]
+}
diff --git a/testdata/golden/causal-lm/nemotron-h-nano-4b_generation.json b/testdata/golden/causal-lm/nemotron-h-nano-4b_generation.json
new file mode 100644
index 00000000..6d37b0b2
--- /dev/null
+++ b/testdata/golden/causal-lm/nemotron-h-nano-4b_generation.json
@@ -0,0 +1,27 @@
+{
+  "model_id": "nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16",
+  "prompt": "Here is my poem:",
+  "generated_tokens": [
+    1429,
+    1784,
+    11560,
+    15368,
+    1307,
+    39935,
+    1044,
+    1044,
+    1044,
+    1044,
+    1050,
+    1048,
+    1050,
+    1048,
+    1050,
+    1048,
+    1050,
+    1048,
+    1050,
+    1048
+  ],
+  "generated_text": " \"The Great Wall of Beijing,,,,2020202020"
+}
diff --git a/tests/_test_configs.py b/tests/_test_configs.py
index 918f0973..fe874538 100644
--- a/tests/_test_configs.py
+++ b/tests/_test_configs.py
@@ -1165,7 +1165,6 @@ def _base_config(config_cls=None, **overrides) -> ArchitectureConfig:
         False,
     ),
     # nemotron_h: hybrid Mamba2+Attention (requires NemotronHConfig)
-    # NemotronH's "moe" layers are not yet implemented — use only mamba2+attention.
     (
         "nemotron_h",
         {
@@ -1182,6 +1181,36 @@ def _base_config(config_cls=None, **overrides) -> ArchitectureConfig:
         },
         True,
     ),
+    # nemotron_h MoE variant: hybrid Mamba2+MoE+Attention (Nemotron-3 30B/120B)
+    (
+        "nemotron_h",
+        {
+            "hidden_act": "relu2",
+            "layer_types": [
+                "mamba2",
+                "moe",
+                "mamba2",
+                "moe",
+                "full_attention",
+                "moe",
+            ],
+            "_config_cls": NemotronHConfig,
+            "num_hidden_layers": 6,
+            "mamba_n_heads": TINY_KV_HEADS,
+            "mamba_d_head": TINY_HEAD_DIM,
+            "mamba_d_state": 16,
+            "mamba_n_groups": 1,
+            "mamba_d_conv": 4,
+            "mamba_expand": 2,
+            "num_local_experts": 4,
+            "num_experts_per_tok": 2,
+            "moe_intermediate_size": TINY_INTERMEDIATE,
+            "shared_expert_intermediate_size": TINY_INTERMEDIATE * 2,
+            "norm_topk_prob": True,
+            "routed_scaling_factor": 2.5,
+        },
+        True,
+    ),
     # gemma3n_text: all full attention (no sliding window)
     (
         "gemma3n_text",
diff --git a/tests/build_graph_test.py b/tests/build_graph_test.py
index 1604bfb1..52e09d2e 100644
--- a/tests/build_graph_test.py
+++ b/tests/build_graph_test.py
@@ -255,8 +255,8 @@ def test_graph_builds_without_weights(self, model_type: str, config_overrides: d
         layer_types = config.layer_types or []
         for i in range(num_layers):
             ltype = layer_types[i] if i < len(layer_types) else "full_attention"
-            if ltype == "mlp":
-                continue  # MLP layers are stateless — no cache outputs
+            if ltype in ("mlp", "moe"):
+                continue  # MLP and MoE layers are stateless — no cache outputs
             if ltype == "lightning_attention":
                 # Lightning Attention: single recurrent state only (no conv_state)
                 assert f"present.{i}.recurrent_state" in output_names, (
@@ -3507,6 +3507,98 @@ def test_nemotron_h_preprocess_weights(self):
         for key in result:
             assert not key.startswith("backbone."), f"Unrenamed key: {key}"
 
+    def test_nemotron_h_moe_preprocess_weights(self):
+        """Verify stacked 3D MoE expert tensors are split into per-expert 2D weights."""
+        import torch
+
+        from mobius._configs import NemotronHConfig
+        from mobius.models.nemotron_h import NemotronHCausalLMModel
+
+        config = NemotronHConfig(
+            vocab_size=TINY_VOCAB,
+            hidden_size=TINY_HIDDEN,
+            intermediate_size=TINY_INTERMEDIATE,
+            num_hidden_layers=2,
+            num_attention_heads=TINY_HEADS,
+            num_key_value_heads=TINY_KV_HEADS,
+            rms_norm_eps=1e-5,
+            layer_types=["full_attention", "moe"],
+            mamba_n_heads=TINY_KV_HEADS,
+            mamba_d_head=TINY_HEAD_DIM,
+            mamba_d_state=16,
+            mamba_n_groups=1,
+            mamba_d_conv=4,
+            mamba_expand=2,
+            hidden_act="relu2",
+            head_dim=TINY_HEAD_DIM,
+            num_local_experts=4,
+            num_experts_per_tok=2,
+            moe_intermediate_size=TINY_INTERMEDIATE,
+        )
+        module = NemotronHCausalLMModel(config)
+
+        num_experts = 4
+        # Stacked 3D expert tensors (HF format): (num_experts, out_dim, in_dim)
+        up_proj_stacked = torch.randn(num_experts, TINY_INTERMEDIATE, TINY_HIDDEN)
+        down_proj_stacked = torch.randn(num_experts, TINY_HIDDEN, TINY_INTERMEDIATE)
+
+        state_dict = {
+            # Embeddings & norm
+            "backbone.embeddings.weight": torch.zeros(1),
+            "backbone.norm_f.weight": torch.zeros(1),
+            "lm_head.weight": torch.zeros(1),
+            # Layer 0: full_attention
+            "backbone.layers.0.norm.weight": torch.zeros(1),
+            "backbone.layers.0.mixer.q_proj.weight": torch.zeros(1),
+            "backbone.layers.0.mixer.k_proj.weight": torch.zeros(1),
+            "backbone.layers.0.mixer.v_proj.weight": torch.zeros(1),
+            "backbone.layers.0.mixer.o_proj.weight": torch.zeros(1),
+            # Layer 1: moe — stacked expert weights (3D)
+            "backbone.layers.1.norm.weight": torch.zeros(1),
+            "backbone.layers.1.mixer.experts.up_proj": up_proj_stacked,
+            "backbone.layers.1.mixer.experts.down_proj": down_proj_stacked,
+            # MoE gate
+            "backbone.layers.1.mixer.gate.weight": torch.zeros(1),
+            "backbone.layers.1.mixer.gate.e_score_correction_bias": torch.zeros(1),
+            # MoE shared experts
+            "backbone.layers.1.mixer.shared_experts.up_proj.weight": torch.zeros(1),
+            "backbone.layers.1.mixer.shared_experts.down_proj.weight": torch.zeros(1),
+        }
+
+        result = module.preprocess_weights(state_dict)
+
+        # Stacked expert keys must NOT be in the result
+        assert "model.layers.1.moe.experts.up_proj" not in result
+        assert "model.layers.1.moe.experts.down_proj" not in result
+
+        # Per-expert keys must exist with correct shapes
+        for i in range(num_experts):
+            up_key = f"model.layers.1.moe.experts.{i}.up_proj.weight"
+            down_key = f"model.layers.1.moe.experts.{i}.down_proj.weight"
+            assert up_key in result, f"Missing {up_key}"
+            assert down_key in result, f"Missing {down_key}"
+            assert result[up_key].shape == (TINY_INTERMEDIATE, TINY_HIDDEN), (
+                f"{up_key} shape {result[up_key].shape}"
+            )
+            assert result[down_key].shape == (TINY_HIDDEN, TINY_INTERMEDIATE), (
+                f"{down_key} shape {result[down_key].shape}"
+            )
+            # Verify the data matches the original slice
+            torch.testing.assert_close(result[up_key], up_proj_stacked[i])
+            torch.testing.assert_close(result[down_key], down_proj_stacked[i])
+
+        # Gate weights are renamed correctly
+        assert "model.layers.1.moe.gate.weight" in result
+        assert "model.layers.1.moe.gate.e_score_correction_bias" in result
+
+        # Shared expert weights are renamed correctly
+        assert "model.layers.1.moe.shared_experts.up_proj.weight" in result
+        assert "model.layers.1.moe.shared_experts.down_proj.weight" in result
+
+        # No original backbone.* keys should remain
+        for key in result:
+            assert not key.startswith("backbone."), f"Unrenamed key: {key}"
+
 
 # ===========================================================================
 # Hybrid SSM+Attention (Jamba) model tests