Skip to content

fix(recipes): make the Solar Open2 250B benchmark and docs link usable #14956

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ynpandey-nv merged 1 commit into from
Sep 17, 2026
Merged

fix(recipes): make the Solar Open2 250B benchmark and docs link usable #14956

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 0 additions & 5 deletions .lycheeignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,6 @@
^https://github\.com/ai-dynamo/dynamo/tree/main/recipes/deepseek-v4/deepseek-v4-pro-0813/perf$
^https://github\.com/ai-dynamo/dynamo/blob/main/recipes/deepseek-v4/deepseek-v4-pro-0813/perf/README\.md$

# The Solar Open2 250B recipe page is introduced by the same PR as its docs link,
# so the rendered docs URL 404s from CI until the page is published. Remove this
# entry once the page is live.
^https://docs\.nvidia\.com/dynamo/latest/recipes/model-recipes/solar-open2-250b$

# The DeepSeek-V4.1-Flash docs page is published by the same PR that adds this
# link, so the URL returns 404 until the docs deploy. Remove this entry after
# the page is live.
Expand Down
6 changes: 3 additions & 3 deletions docs/fern/pages/recipes/model-recipes/solar-open2-250b.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -251,9 +251,9 @@ below 5 s and p50 user output throughput at or above 50 tok/s.
| Chat (15% subset) | Disaggregated 1P:2D, round-robin | vLLM | B200 | 44 | 206.20 | 51.69 | 2277 |

Figures are measured with the profile's benchmark manifest (`perf.yaml` for

@dmitry-tokarev-nv dmitry-tokarev-nv Sep 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] The published table was not measured with these manifests. docs/fern/pages/recipes/model-recipes/solar-open2-250b.mdx:253. The numbers in the table above come from the old image and the removed cache_bust block. The PR body reports 213.74 tok/s/GPU and 325 ms for aggregated. Please update the table, or say which client produced the figures.

Measured: aiperf 0.12.0 rejects the old configuration, accepts the new one

Both ConfigMap bodies extracted from the manifests and checked with aiperf config validate on a real 0.12.0 install (pip, macOS ARM64):

tree perf.yaml perf-disagg.yaml
merge base rejected rejected
this PR valid valid

The rejection on the merge base is the same on both files:

benchmark.datasets.0.file.cache_bust.target
  Input should be 'none', 'system_prefix', 'system_suffix',
  'first_turn_prefix' or 'first_turn_suffix'
  [type=enum, input_value='warmup_isolation_system', input_type=str]

So the change is necessary and correct: warmup_isolation_system is a 0.13.0 enum value and does not exist in 0.12.0.

The point of this comment is narrower. The table at lines 250-251 still reads 216.88 / 52.02 / 308 ms and 206.20 / 51.69 / 2277 ms, and the sentence at line 253 tells the reader those figures come from perf.yaml and perf-disagg.yaml. Those two files no longer configure warmup isolation and no longer use the client that produced the figures. A reader who reproduces the run gets the PR body numbers, not the table numbers.

The PR body says every figure sits inside the run-to-run spread. I could not check that claim. The spread is not stated, and I have no B200 cluster. Unverified.

Registry access, checked anonymously with a token from nvcr.io/proxy_auth:

image manifest HTTP status
nvcr.io/nvstaging/ai-dynamo/aiperf:0.13.0rc2 (merge base) 401
nvcr.io/nvidia/ai-dynamo/aiperf:0.12.0 (this PR) 200

aggregated, `perf-disagg.yaml` for disaggregated). That configuration isolates the
warmup phase from profiling; it does not clear server-side KV cache. For runs that
must start from a cold cache, restart the worker pods between them.
aggregated, `perf-disagg.yaml` for disaggregated). Neither manifest clears
server-side KV cache, so a run inherits whatever the workers already hold. For runs
that must start from a cold cache, restart the worker pods between them.

## Source

Expand Down
2 changes: 1 addition & 1 deletion recipes/solar-open2-250b/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ SPDX-License-Identifier: Apache-2.0
# Solar Open2 250B NVFP4 Recipes

Documentation for these recipes is published at
<https://docs.nvidia.com/dynamo/latest/recipes/model-recipes/solar-open2-250b>.
<https://docs.nvidia.com/dynamo/dev/recipes/solar-open2-250b>.
5 changes: 1 addition & 4 deletions recipes/solar-open2-250b/perf/perf-disagg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,6 @@ data:
format: mooncake_trace
sampling: sequential

cache_bust:
target: warmup_isolation_system

synthesis:
max_isl: ${MAX_ISL:1048576}
max_osl: ${CAP_OSL:12000}
Expand Down Expand Up @@ -70,7 +67,7 @@ spec:

containers:
- name: aiperf
image: nvcr.io/nvstaging/ai-dynamo/aiperf:0.13.0rc2
image: nvcr.io/nvidia/ai-dynamo/aiperf:0.12.0
imagePullPolicy: IfNotPresent

command: ["/bin/bash", "-lc"]
Expand Down
5 changes: 1 addition & 4 deletions recipes/solar-open2-250b/perf/perf.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,6 @@ data:
format: mooncake_trace
sampling: sequential

cache_bust:
target: warmup_isolation_system

synthesis:
max_isl: ${MAX_ISL:1048576}
max_osl: ${CAP_OSL:12000}
Expand Down Expand Up @@ -70,7 +67,7 @@ spec:

containers:
- name: aiperf
image: nvcr.io/nvstaging/ai-dynamo/aiperf:0.13.0rc2
image: nvcr.io/nvidia/ai-dynamo/aiperf:0.12.0
imagePullPolicy: IfNotPresent

command: ["/bin/bash", "-lc"]
Expand Down
Loading