feat(thunderagent): include HiCache retention capacity

[ishandhanani]

ThunderAgent now derives its retention budget from SGLang worker metadata as GPU KV plus native HiCache host capacity, rather than device KV alone. This keeps program-level pause/resume aligned with the memory SGLang can actually retain while preserving SGLang as the authority for cache admission and movement.

CLOSES: DYN-3366

How This Was Implemented

• Publish the realized GPU KV and HiCache host capacities in each SGLang worker's model deployment card.
• Add native host capacity to ThunderAgent's per-worker retention budget; exclude Mooncake because its content-addressed entries are conditional rather than guaranteed program residency.
• Keep the policy advisory: ThunderAgent does not call cache prefetch, demotion, eviction, restore, or Mooncake APIs.
• Use Harbor's DynamoPi adapter so every trial sends one stable session ID and the ThunderAgent arm sends an explicit terminal lifecycle request.
• Add a two-worker TP4 MiniMax ta|kv launcher, a scored Verified walkthrough, and a dedicated 32-way nop image-preparation command.

Walkthrough

Mental model

SGLang knows the realized device and host cache sizes after startup, so it publishes both through the existing model deployment card. ThunderAgent consumes that card when it computes utilization; the session ID remains an identity hint that groups turns into a program, not a cache command.

flowchart LR
  S["SGLang: GPU KV + HiCache host capacity"] --> M["Worker model deployment card"]
  M --> T["ThunderAgent retention budget"]
  H["Harbor trial"] -->|"stable session ID on every Pi turn"| T
  T -->|"ordinary routed requests"| S

Loading

Capacity and lifecycle

The capacity observer sums GPU KV tokens and native HiCache host tokens for each worker. The scheduler applies its existing thresholds to that sum, so a 0.95 pause threshold means 95% of the memory that can retain the program locally; the existing GPU-only behavior remains when host capacity is absent.

Harbor runs Pi inside the SWE-bench task container and maps the trial ID to DYN_AGENT_SESSION_ID. Pi sends the same header on every model turn, and the ThunderAgent arm sends one x-dynamo-session-final request at trial completion so program state can be released; the stock KV arm omits only that terminal request while retaining the same identity headers.

Reproduction

run_minimax_8xh100.sh ta|kv owns the repeated two-worker TP4 MiniMax stack and loads MiniMaxAI/MiniMax-M2.7 from Hugging Face behind the MiniMaxAI/MiniMax-M2 API alias. The README shows one scored Verified smoke and the complete 500-task run.

Image preparation is a separate upstream Harbor nop install-only job: authenticate to Docker Hub, run 32 trials concurrently with host networking, retain stopped containers to avoid per-trial Compose teardown, then prune once after all 500 task images exist. The measured run uses DynamoPi and the same host-network overlay, avoiding one Docker bridge network per trial.

Boundaries and limitations

Mooncake capacity is deliberately excluded because matching content can be absent or evicted. The change does not make session identity sticky in the stock router, does not change SGLang cache correctness, and does not claim pause/resume activity from an unsaturated one-task smoke.

Validation

• pytest -q components/src/dynamo/thunderagent_router/tests/test_capacity.py components/src/dynamo/sglang/tests/test_runtime_metadata.py — 24 passed.
• npm run check and npm test -- --run in agent-plugins/pi-plugin — TypeScript clean, 6 passed.
• Clean H100 source setup completed with the documented Dynamo, Harbor 0.16.0, and agent-plugins revisions.
• The documented ta and kv launch commands each registered two TP4 workers from MiniMaxAI/MiniMax-M2.7; the public model endpoint became ready.
• The exact one-task Harbor command passed astropy__astropy-12907 in both arms with reward 1.0 and zero exceptions. ThunderAgent logged terminal release with zero programs remaining; stock KV omitted the lifecycle request.
• The dedicated pre-pull command completed all 500 Verified tasks in 15m32s with zero exceptions. It produced 500 unique runnable task-image references and image IDs with zero missing; Docker stayed at three networks.
• bash -n components/src/dynamo/thunderagent_router/run_minimax_8xh100.sh, fern docs broken-links, applicable pre-commit hooks, and git diff --check passed.

Summary by CodeRabbit

• New Features

  • Added HiCache host-token capacity reporting to improve worker retention budgeting and routing decisions.
  • Added a new host-network deployment option and updated launcher support for ThunderAgent or KV modes.
  • Added Pi setup guidance for Dynamo-compatible endpoints.

• Bug Fixes

  • Improved capacity calculations by including valid HiCache host totals when present.
  • Added cleanup and readiness checks to make local launches more reliable.

• Documentation

  • Expanded ThunderAgent and agent-harness docs with updated Harbor/Pi workflows and revised terminology.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-019f33a0-c289-7652-8ec3-b6e20ae1796f.docs.buildwithfern.com/dynamo/dev

[coderabbitai]

Walkthrough

This PR introduces HiCache host token capacity metadata published by the SGLang engine and consumed by ThunderAgent's retention-budget calculation, updates documentation to reflect this budget model, reworks the MiniMax launch script into a policy-parameterized script, and replaces MiniMax-M2 reproduction docs with a Harbor/Pi A/B walkthrough.

Changes

HiCache Retention Budget

Layer / File(s)	Summary
SGLang HiCache capacity publishing components/src/dynamo/sglang/register.py, components/src/dynamo/sglang/tests/test_runtime_metadata.py	Adds a runtime-config key and helper to extract/validate hicache_host_total_tokens from scheduler_info, wired into engine-specific runtime config, with unit tests.
ThunderAgent retention budget consumption components/src/dynamo/thunderagent_router/capacity.py, components/src/dynamo/thunderagent_router/tests/test_capacity.py	Updates WorkerCapacityProvider/_parse_pool_tokens to output retention_tokens, adding HiCache host token capacity into the computed budget, with new tests.
Retention budget documentation components/src/dynamo/thunderagent_router/README.md, docs/agents/thunderagent-router.md	Documents retention budget derivation from GPU+host capacities, pause-threshold semantics, and exclusion of Mooncake.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Harbor/Pi Reproduction and Launch Script

Layer / File(s)	Summary
Policy-parameterized launch script components/src/dynamo/thunderagent_router/run_minimax_8xh100.sh, components/src/dynamo/thunderagent_router/harbor-host-network.yml	Reworks the launcher to accept a ta/kv policy argument, add cleanup/trap handling, configurable env vars, conditional router/frontend setup, and readiness polling; adds a host-network compose file.
Harbor/Pi walkthrough documentation components/src/dynamo/thunderagent_router/README.md, docs/agents/thunderagent-router.md, docs/agents/agent-harnesses.md	Replaces the MiniMax-M2 reproduction procedure with a Harbor/Pi A/B walkthrough and a Pi setup section, updating reference links.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Related Issues: None referenced in the provided changes.

Related PRs: None referenced in the provided changes.

Suggested labels: documentation, enhancement, sglang, thunderagent-router

Suggested reviewers: None specified.

Poem:
A rabbit hopped through cache and code,
Counting tokens down the host-side road.
Scripts now branch by ta or kv's call,
While Harbor and Pi replace the old walkthrough all.
Retention budgets tallied, tests in tow —
Hop, hop, hooray, the metadata flows! 🐇

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description is detailed, but it misses the required template sections for reviewer start and a properly formatted Related Issues block.	Add a 'Where should the reviewer start?' section and replace 'CLOSES: DYN-3366' with the required Related Issues template format.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding HiCache retention capacity to ThunderAgent.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

Devin Review found 2 new potential issues.