docs: add a multimodal pipeline concept page

[slin1237]

Multimodal was the largest area of change in v1.11.0, about 60 smg commits, and the site had no multimodal page. Only a short tensor-transport table in the configuration reference covered it. This adds a concept page on how images and video flow through SMG with gRPC workers, and where noted with the direct ZMQ backends.

Changes

• New concepts/architecture/multimodal.md ("Multimodal Pipeline")
  • Request flow: which APIs accept media, the seven gRPC-path steps, and where the processor config comes from.
  • Where media is processed:
    • --mm-processing auto|router|worker and the media-reference contract for vLLM gRPC workers (feat(vllm-grpc): media_refs wire contract and worker-side media processing smg#2399, #2400).
    • The vLLM servicer's media flags (#2626).
    • The Redis media-processing sidecar (#2401), including how to run it and the worker/sidecar fingerprint matching.
    • Failure classification (#2596, #2624).
  • Supported models: per-family image and video limits from the processor registry, and which families support worker-side expansion on vLLM. Covers Qwen4-Exp (#2327), MiniMax-M3 (#2371, #2581), GLM-5.3-Flash (#2349), and DeepSeek-V4.1 (#2526).
  • Engine × modality matrix.
  • Video:
    • Decode backends and the opencv-video feature.
    • Sampling at the model's reference rate (#2579) with reference frame placement (#2584).
    • The fps and max_long_side_pixel extensions.
  • Limits and safety:
    • Count overrides (#2153, #2381), size caps, and fetch timeout.
    • The in-flight budget (#2585).
    • A table of every multimodal error code and status.
  • Performance: fetch-once dedup, the data-URL fast path, parallel preprocessing, and the pixel cache.
  • Tensor transport: inline/shm/auto/rdma, per-engine support, the RDMA pixel lane, and encoder input width.
  • ZMQ, PD, and EPD paths, including the language-model-only decode codes (#2640).
  • Observability, a configuration summary, and a runnable image_url example.
• Nav and concepts index: the page is listed under Architecture, after gRPC Pipeline.

The worker spec has multimodal_tensor_transport and multimodal_shm_min_bytes fields, and the transport code reads them. In v1.11.0, though, worker registration never copies them onto the worker. The page says so instead of presenting them as working overrides; #43 has the details.

Verification

• Checked against smg v1.11.0 (3be823a7): routers/grpc/multimodal/* (plan, refs, settings, inflight, transport, assemble, process, pixel cache), crates/multimodal (registry, media, tracker, processors), grpc_servicer/smg_grpc_servicer/vllm/* (mm_processor, mm_sidecar, servicer), and observability/metrics.rs.
• pnpm lint, pnpm check (0 errors), and pnpm build pass.
• The link check reports only links that other PRs in this series resolve: zmq-workers.md, configuration.md#multimodal-configuration, and admin.md#worker-spec (docs(api): document the full worker spec for the worker management API #43).
• The page and the concepts index render with status 200 under wrangler pages dev.

This is one of a series of docs PRs that bring the site up to smg v1.11.0, one per area.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

Next included review available in 23 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available. Your 65 included PR review attempts over the past 7 days set your current allowance at 4 reviews per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 30959840-ee0e-48ad-86ab-c84ef198bc2e

📥 Commits

Reviewing files that changed from the base of the PR and between 7362b40 and 1fb19e1.

📒 Files selected for processing (3)

• src/lib/config/concepts-nav.ts
• src/lib/content/concepts/architecture/multimodal.md
• src/lib/content/concepts/index.md

---

_{Comment @coderabbitai help to get the list of available commands.}