Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion docs/cookbook/autoregressive/Meta/MuseSparkOpen.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ Choose your **Hardware Platform**, **Strategy**, and **Quantization** below. The
import { Deployment } from "/src/snippets/_deployment.jsx";
import { config } from "/src/snippets/configs/meta-models/muse-spark.jsx";

<Deployment config={config} />
<Deployment config={config} agenticLink="agentic-long-context-deployment" />

## Playground

Expand All @@ -35,6 +35,19 @@ import { Playground } from "/src/snippets/_playground.jsx";

<Playground config={config} />

## Agentic Long-Context Deployment

Agentic workloads, such as coding agents and other multi-turn, tool-using assistants, send long prompts whose prefixes grow and repeat from turn to turn. Serving them well depends on reusing the KV cache across turns, offloading it beyond GPU memory, and keeping each session on the workers that already hold its prefix. This section gives complete launch commands for such deployments, each benchmarked in [SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) and linked to the run that measured it. Pick a hardware platform, checkpoint, deployment shape, concurrency, KV cache offloading mechanism and router to get every process of the deployment as a raw command, one tab per process in launch order.

- **Router.** Each configuration was submitted behind one front door: Dynamo (`dynamo.frontend` with `dynamo.sglang` workers) or SGLang (`sglang.launch_server`, behind `sglang_router` when there are several workers or DP-attention ranks).
- **KV offload.** Choose among these KV cache offloading mechanisms: none (GPU KV only), HiCache (host DRAM), or a Mooncake store attached through the external linker. The Mooncake option needs an image whose SGLang build includes `--enable-unified-cache-external-linker`.
- **Container image.** Run the commands inside the image listed for each configuration; flag names follow that image.

import { AgentX } from "/src/snippets/_agentx.jsx";
import { agentx } from "/src/snippets/agentx/meta-models/muse-spark-open.jsx";

<AgentX data={agentx} />

## 1. Model introduction

This cookbook provides deployment recipes for the Muse Spark Open 816B-A42B checkpoints:
Expand Down
Loading
Loading