Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion tests/scripts/perf-sanity/cache_transceiver_precheck/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,43 @@ starts.
| Same UCX env vars (incl. the `unset UCX_TLS` cases) | `jenkins/scripts/perf/submit.py` builds the precheck commands from the **same** `ucx_tls_cmd` + `$CTX/GEN_WORKER_ENV_VARS` strings as the worker steps; `slurm_precheck_run.sh` sources the same `slurm_env_setup.sh` (the `UCX_TLS=tcp` fixup) as `slurm_run.sh`. |
| Same instance count / parallelism | One precheck `srun` per ctx/gen server with the same `-N/--ntasks/--ntasks-per-node/--mpi=pmix` and the same node slices (`-w`) as the real server steps (`slurm_launch_draft.sh`). TP/PP/CP/attention-DP come from the same `worker_config`. |
| Same transceiver config | `CacheTransceiverConfig(**yaml["worker_config"][role]["cache_transceiver_config"])` — the yaml block is passed through verbatim (backend, `max_tokens_in_buffer`, timeouts, ...). |
| Same KV cache manager version + transceiver runtime | The launch generator forwards the real test's `LLM_MODELS_ROOT` into every precheck process. Explicit per-side `kv_cache_config.use_kv_cache_manager_v2` wins; absent means "auto" and requires a registered model class, then resolves via `get_preferred_kv_cache_manager_version()`. `transceiver_runtime: auto` resolves via `get_preferred_transceiver_runtime()` (NIXL-gated) — both through the same llm_utils code serving uses. An unresolved manager-version `auto` setting fails with INIT_ERROR instead of silently assuming V1. V2 requires the Python transceiver (the C++ one only supports V1); a V2+CPP combination also fails fast. |
| Same KV cache manager version + transceiver runtime | The launch generator forwards the real test's `LLM_MODELS_ROOT` into every precheck process. Explicit per-side `kv_cache_config.use_kv_cache_manager_v2` wins; absent means "auto" and requires a registered model class, then resolves via `get_preferred_kv_cache_manager_version()`. `transceiver_runtime: auto` resolves via `get_preferred_transceiver_runtime()` (NIXL-gated) — both through the same llm_utils code serving uses. The model class comes from the checkpoint; when no checkpoint directory is resolved, `metadata.architecture` supplies a registered architecture hint. An unresolved manager-version `auto` setting fails with INIT_ERROR instead of silently assuming V1. V2 requires the Python transceiver (the C++ one only supports V1); a V2+CPP combination also fails fast. |

Asymmetric layouts (ctx dep4 → gen dep16, ctx pp8 → gen tp32, ...) are
supported: data is seeded per (request, **global** layer) and constant along
the KV-head axis, so the receiver regenerates its expected slice locally under
any TP/PP resharding. KV shape (layers/heads/head_dim, MLA vs GQA, MTP nextn
layers) is read from the real model's `config.json` under `$LLM_MODELS_ROOT`.

For checkpoints unavailable to the network precheck, declare the model architecture:

```yaml
metadata:
architecture: DeepseekV4ForCausalLM
```

The hint accepts one nonempty architecture name, which is looked up in the
model registry; lists are rejected. A resolved checkpoint always takes
precedence; metadata does not replace an unknown or invalid checkpoint
architecture. Model hooks still run through the serving resolvers,
and a hook that needs unavailable checkpoint fields fails closed.

Without a checkpoint, the precheck transfers a generic synthetic KV pool.
A PASS establishes transport availability, not checkpoint readiness or exact
model KV-layout correctness. Serving still requires the model weights and
`config.json` staged under `LLM_MODELS_ROOT` on every worker.

Before creating the KV manager or transceiver, all ranks in an instance must
agree on the resolved settings for both roles and global synthetic KV shape.
The context/generation handshake compares this same contract before any
transfer: effective backend, runtime, manager version, normalized dtype,
tokens per block, and global layers/heads/head dimension/MLA mode. Comparing
both role resolutions preserves configured asymmetric manager and parallelism
choices while rejecting inconsistent checkpoint visibility or environment
resolution. Local checkpoint paths and diagnostic source labels are excluded.
Architecture declarations, manager settings, and dtype inputs also participate
in the shared plan fingerprint.

## Enabling / disabling

**On by default** for every disaggregated perf-sanity test. To opt out:
Expand Down Expand Up @@ -86,6 +115,14 @@ csv/ctx_<i>/<uuid>_<rank>.csv # Python transceiver per-task perf
# (KVSendTask throughput on the ctx side)
```

Every status JSON written by the runner includes `model_dir_resolved` and
`synthetic_kv_shape`, including intermediate records and failure results.
`model_dir_resolved` means a directory containing `config.json` was resolved;
it does not establish that model weights are available or valid.
`synthetic_kv_shape` is true for a missing checkpoint, a simplified layout,
or a generic fallback shape, including fallback from a resolved directory.
These fields qualify the transport result without changing the gate's verdict.

## Manual runs

```bash
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@

# Fallback KV shape for dry-runs and explicitly selected manager versions when
# the model directory cannot be resolved. Manager-version "auto" resolution
# fails fast instead of silently pairing this shape with V1.
# requires a registered checkpoint or declared architecture instead of assuming V1.
FALLBACK_KV_SHAPE = {
"num_layers": 32,
"num_kv_heads": 8,
Expand Down Expand Up @@ -339,6 +339,39 @@ def gate_library_content(draft_launch_sh, llm_src):
)


def declared_architecture(cfg: Mapping[str, Any]) -> str | None:
"""Read one architecture hint for prechecks without a local checkpoint.

Args:
cfg: Parsed disaggregated benchmark YAML.

Returns:
The stripped architecture name, or None when the hint is omitted.

Raises:
ValueError: If a supplied declaration is not a nonempty string.
"""
metadata = cfg.get("metadata") or {}
if "architecture" not in metadata:
return None
value = metadata["architecture"]
if not isinstance(value, str) or not value.strip():
raise ValueError("metadata.architecture must be a nonempty string")
return value.strip()


def normalized_kv_dtype(dtype: str) -> str:
"""Return the storage dtype used by the synthetic KV pool.

Args:
dtype: Per-role KV dtype from the benchmark configuration.

Returns:
Canonical fp8, fp16, or bf16 spelling; unsupported/auto uses bf16.
"""
return {"fp8": "fp8", "fp16": "fp16", "half": "fp16"}.get(dtype.lower(), "bf16")


def resolve_plan(cfg, benchmark_mode="e2e"):
"""Build the shared precheck plan both roles must agree on.

Expand Down Expand Up @@ -439,6 +472,7 @@ def resolve_plan(cfg, benchmark_mode="e2e"):
),
"rendezvous_timeout_s": rendezvous_timeout_s,
"verify_data": bool(knobs["verify_data"]),
"architecture": declared_architecture(cfg),
}
for role, side, xcvr in (("ctx", ctx_side, ctx_xcvr), ("gen", gen_side, gen_xcvr)):
kv_cfg = side.get("kv_cache_config") or {}
Expand All @@ -457,6 +491,11 @@ def resolve_plan(cfg, benchmark_mode="e2e"):
def plan_fingerprint(plan):
"""Stable string both sides must agree on before transferring."""
keys = (
"architecture",
"ctx_use_kv_cache_manager_v2",
"gen_use_kv_cache_manager_v2",
"ctx_kv_dtype",
"gen_kv_dtype",
"num_ctx_servers",
"num_gen_servers",
"ctx",
Expand All @@ -479,6 +518,7 @@ def side_plan(plan, role):
"""Per-role view: this role's parallelism + transceiver/kv config."""
return {
"role": role,
"architecture": plan["architecture"],
"parallel": plan[role],
"cache_transceiver_config": plan[f"{role}_cache_transceiver_config"],
"kv_dtype": plan[f"{role}_kv_dtype"],
Expand Down
Loading