Skip to content

docs(blog): add Dynamo shadow engine failover post - #13124

Draft
mohammedabdulwahhab wants to merge 2 commits into
mainfrom
mabdulwahhab/blog-dynamo-bulwark
Draft

docs(blog): add Dynamo shadow engine failover post#13124
mohammedabdulwahhab wants to merge 2 commits into
mainfrom
mabdulwahhab/blog-dynamo-bulwark

Conversation

@mohammedabdulwahhab

@mohammedabdulwahhab mohammedabdulwahhab commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Revives #9381 (closed; its branch was deleted from origin). The old post sat at
docs/digest/dynamo-bulwark/dynamo-bulwark.md, a tree that no longer exists — #12373 moved
blog content to docs/fern/pages/blog/. Rather than rebase, this rebuilds the post on the
current Fern blog convention, replaces its placeholder benchmark section with a real
measurement, and applies a round of editorial review.

Structure — follows docs/fern/pages/blog/README.md: MDX under pages/blog/2026/,
figures under _assets/bulwark/, <BlogStyles/> + <BlogArticleMeta/>, no body H1,
<Frame> + <Info className="fig-caption"> figures, plus the nav entry, landing-page card
and sidebar date label.

Benchmarks — the previous section carried projected TRT-LLM numbers on Qwen3-235B with an
"illustrative, will be replaced before publication" warning. Replaced with a measured
Kimi-K2.6 NVFP4 three-node cascade (3× B200, TP=8, code-agent trace at 100k/1k, concurrency 24;
three engine kills 60 s apart after steady state):

Metric Baseline Shadow failover
Serving through the cascade blackout, ~150 s continuous
Truly-failed requests 409 17
Requests completed 867 1,017
Requests inside the original SLA 699 907

The harness emits light-canvas Matplotlib; _assets/bulwark/tools/darkify.py inverts lightness
in HLS while preserving hue and saturation so the charts sit on the site's dark canvas. No data
point, axis, or label is altered, only pixel colour.

Editorial review (second commit) — renamed throughout from "Dynamo Bulwark" to
Shadow Engine Failover, since "Bulwark" appears nowhere in main and the shipped feature
has a name. The pod-level unit is now "failover worker" rather than a coined product noun.
Also:

  • Intro leads with zero-copy weight sharing through GMS, so the "does this double my HBM?"
    question is answered before it forms.
  • New gms-overview.svg diagram: two engines, own contexts, one physical copy in HBM, GMS
    off the access path. Fills a TODO placeholder the original PR never produced.
  • Added the intuition for why a GMS-backed weight read costs no more than an engine-allocated
    one: mapping is one-time at startup and GMS is absent at read time.
  • New callout marking KV cache under GMS as active work, explicitly not shipped.
  • Dropped the scratch-aliased KV section as too confusing, and repaired the cross-reference
    it left behind.
  • Explained the 17 failures as the cutover window; quantified a parked shadow at ~6.2 GB per GPU.
  • Removed the TTFT and decode table rows whose near-ties read as a baseline win.
  • Added the DRA prerequisite to the caveats.

2,304 words, down from 2,766.

Validation

  • fern check --local --warnings0 errors; 11 warnings, all pre-existing on main.
  • check_asset_paths.py, check_style_components.py, check_component_imports.py → pass.
  • Post compiles under @mdx-js/mdx; the blog components esbuild-parse.
  • All asset paths and relative doc links resolve on disk.
  • Rendered locally via fern docs dev.

Known issue for reviewers

The two request-outcome charts have mismatched y-axes (baseline 0–200, shadow failover 0–35),
which makes them hard to compare by eye. Fixing it needs a re-plot from the benchmark harness;
the per-bin counts are not recoverable from the committed images (a reconstruction attempt
missed the known category totals by 3–75% depending on method, so it was rejected rather than
shipped). Options are to regenerate on a shared axis, substitute the cumulative_successes
pair from the same run, or drop the pair. Until then the caption states the axis difference
explicitly.

Open questions

  1. Benchmark link. benchmarks/kimi-cascade-failover/ lives on a separate branch, so a
    blob/main/... link would 404. The post is self-contained for now.
  2. Placement. Per the team publishing guide, a benchmark-driven narrative like this is
    Primary on the NVIDIA Tech Blog with the GitHub Digest carrying a short link-out. Filed here
    as the working draft; if it goes to the Tech Blog, this PR should shrink to a blurb.

Draft while the post is out for editorial review.

Rebuild the Dynamo Bulwark blog post on the current Fern blog layout
(docs/fern/pages/blog/2026 + _assets, MDX, BlogStyles/BlogArticleMeta)
and replace the placeholder benchmark section with the measured
Kimi-K2.6 NVFP4 three-node cascade result.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: furkhan324@berkeley.edu <furkhan324@berkeley.edu>
@copy-pr-bot

copy-pr-bot Bot commented Aug 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added docs documentation Improvements or additions to documentation labels Aug 12, 2026
@datadog-official

datadog-official Bot commented Aug 12, 2026

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 3 Pipeline jobs failed

Pre Merge | pre-commit   View in Datadog   GitHub Actions

🔧 Fix in code. This looks caused by changes in this PR. Pre-commit hook failed due to code formatting changes required in 'darkify.py'.

Docs link check | lychee   View in Datadog   GitHub Actions

See error HTTP 404 errors encountered for PyTorch documentation links during job execution.

Pre Merge | pre-merge-status-check   View in Datadog   GitHub Actions

📋 Copy prompt for your agent
CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: mabdulwahhab/blog-dynamo-bulwark

Pre Merge | pre-commit
Commit: 0056220983cc439a62c6bc4fe58278af935f9ab6
Error (code / quality):
Pre-commit hook failed due to code formatting changes required in 'darkify.py'.
CI job: https://github.com/ai-dynamo/dynamo/actions/runs/31834941028/job/94879031966

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0056220 | Docs | Datadog PR Page | Give us feedback!

Rename throughout from "Dynamo Bulwark" to Shadow Engine Failover, retitle,
and add Vikram and Maksim to the byline. Lead the intro with zero-copy weight
sharing through GMS, add a GMS overview diagram and the intuition for why a
GMS-backed weight read costs no more than an engine-allocated one, and note
that KV cache under GMS is active work rather than shipped behaviour.

Drop the scratch-aliased KV section as too confusing, explain the 17 failed
requests as the cutover window, quantify a parked shadow at ~6.2 GB per GPU,
remove the two results rows whose near-ties read as a baseline win, and add
the DRA prerequisite to the caveats.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: furkhan324@berkeley.edu <furkhan324@berkeley.edu>
@mohammedabdulwahhab mohammedabdulwahhab changed the title docs(blog): add Dynamo Bulwark failover post docs(blog): add Dynamo shadow engine failover post Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs documentation Improvements or additions to documentation size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant