Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
165 commits
Select commit Hold shift + click to select a range
5ba1aad
[model] feat: add deepseek_v41_ngram SeedOmni module
Coach257 Sep 16, 2026
805a32d
[model] feat: add deepseek_v41 vision and text_encoder SeedOmni modules
Coach257 Sep 16, 2026
c6185c2
[model] feat: add deepseek_v41_llm config and Engram connection
Coach257 Sep 16, 2026
52de97d
[model] fix: declare gate_temp on DeepseekV41LlmConfig and note MoE r…
Coach257 Sep 16, 2026
c489dcb
[model] feat: add deepseek_v41 llm attention stack
Coach257 Sep 16, 2026
b8b56b7
[model] feat: train the deepseek_v41 indexer with an auxiliary KL
Coach257 Sep 16, 2026
e725c2e
[model] feat: add DeepSeek-V4.1 mHC wiring, MoE routing and decoder l…
Coach257 Sep 16, 2026
8a03700
[model] feat: add the deepseek_v41_llm backbone and its graph hooks
Coach257 Sep 16, 2026
e0bbcab
[model] feat: wire the deepseek_v41 n-gram node into the training graph
Coach257 Sep 16, 2026
89f3f3a
[model] feat: add deepseek_v41 checkpoint dequantization
Coach257 Sep 16, 2026
482bcf0
[model] feat: add the deepseek_v41 checkpoint converter
Coach257 Sep 16, 2026
3a93bb4
[model] docs: record the deepseek_v41 Step 2 landing
Coach257 Sep 16, 2026
1203fe9
Merge remote-tracking branch 'origin/omniv2' into szl.omniv2-deepseek…
Coach257 Sep 16, 2026
c453d3e
[docs] docs: record the dev box's proxy and nvm PATH facts
Coach257 Sep 16, 2026
75bcf86
[omni, model, config] feat: add the DeepSeek-V4.1-Flash training conf…
Coach257 Sep 16, 2026
19a359f
[config] fix: point the v41 convert command at the shared entry point
Coach257 Sep 16, 2026
80aa4ea
[omni, ckpt] fix: stage converter shards when the destination has no …
Coach257 Sep 16, 2026
0a6d286
[omni, model, ckpt] fix: separate the ngram module's two vocabulary s…
Coach257 Sep 16, 2026
62e146b
[omni, ckpt] fix: dequantize the engram tables in row chunks, and acc…
Coach257 Sep 16, 2026
0d25da1
[model] feat: add a streaming teacher-forcing forward for the convert…
Coach257 Sep 17, 2026
b3edda7
[omni, model, docker] fix: let the CUDA image and the V4.1 ViT build …
Coach257 Sep 17, 2026
da8e8e2
[omni, model, ops] feat: add the decode-step attention the V4.1 gener…
Coach257 Sep 17, 2026
f8fe60c
[docker] revert: restore Dockerfile.cu130, which already built for arm64
Coach257 Sep 17, 2026
0d3fdcf
[model] feat: per-layer decode cache for DeepSeek-V4.1 sparse attention
Coach257 Sep 17, 2026
0646fba
[model] feat: n-gram decode cache for DeepSeek-V4.1
Coach257 Sep 17, 2026
6398e83
[model] feat: backbone decode step for DeepSeek-V4.1
Coach257 Sep 17, 2026
da8c497
[model, config] feat: wire DeepSeek-V4.1 into the inference FSM
Coach257 Sep 17, 2026
d764787
[config, model] feat: inference config for DeepSeek-V4.1 and a genera…
Coach257 Sep 17, 2026
2889bef
[ops] feat: port fp8_gemm and fp4_gemm TileLang kernels
Coach257 Sep 17, 2026
9914827
[model] feat: add a native-precision conversion mode for DeepSeek-V4.1
Coach257 Sep 17, 2026
48b79fa
[model] feat: add a quantized linear for DeepSeek-V4.1 inference
Coach257 Sep 18, 2026
5cf066b
[seed_omni] feat: add config-only export to refresh omni checkpoint m…
Coach257 Sep 18, 2026
ee317d6
[seed_omni] feat: run DeepSeek-V4.1 dense projections from the releas…
Coach257 Sep 18, 2026
ed1d705
[seed_omni] feat: run the grouped output projection and shared expert…
Coach257 Sep 18, 2026
5373ed2
[seed_omni] feat: run the routed experts from the released fp4 slabs
Coach257 Sep 18, 2026
3465ac3
[seed_omni] feat: keep the n-gram tables in the fp8 they were release…
Coach257 Sep 18, 2026
e922902
[seed_omni] feat: expert-parallel inference over the fp4 slabs, and i…
Coach257 Sep 18, 2026
bb0df84
[seed_omni] fix: a module asking only for ep or emb was built as if u…
Coach257 Sep 18, 2026
009eac0
[seed_omni] fix: pad the n-gram tables into a row count that can be s…
Coach257 Sep 18, 2026
bdd0796
[seed_omni] fix: shard a quantized weight's scales alongside the weight
Coach257 Sep 18, 2026
a080ce2
[arguments,distributed] fix: let a module overlay's ep_size reach the…
Coach257 Sep 18, 2026
3906352
[distributed,arguments,seed_omni] feat: shard inference by plan alone…
Coach257 Sep 18, 2026
85b4ab1
[seed_omni,docs] feat: add a DeepSeek-V4.1 image chat script
Coach257 Sep 18, 2026
8f21237
[seed_omni] fix: make the ranks agree on the sampled token
Coach257 Sep 18, 2026
15cafc4
[seed_omni] fix: make generation reproducible across ranks
Coach257 Sep 20, 2026
c80043a
[seed_omni] feat: dequant V4.1 at load and capture KV on batched prefill
Coach257 Sep 21, 2026
bb99143
[seed_omni] feat: hang DSpark on generate and convert the tied MTP te…
Coach257 Sep 21, 2026
6469b1f
[seed_omni,docs] feat: drop full-determinism and text-encoder emb fro…
Coach257 Sep 21, 2026
0495682
[seed_omni] feat: skip DSpark hits and CED-prefill without changing t…
Coach257 Sep 22, 2026
2ffcfc1
[seed_omni] feat: fold indexer KL into train loss and remap the CED s…
Coach257 Sep 23, 2026
1a78a29
[seed_omni] feat: smoke a toy four-module train step and meter the n-…
Coach257 Sep 23, 2026
894f473
[seed_omni] feat: freeze the first N backbone layers and bind fp8/fp4…
Coach257 Sep 23, 2026
0446bb0
[dist] fix: size ExtraParallel shards locally instead of through DTensor
Coach257 Sep 28, 2026
5fb2f23
[seed_omni] fix: keep released fp8 n-gram tables fp8 under mixed prec…
Coach257 Sep 28, 2026
395daf9
[seed_omni] fix: initialize the raw mHC, router and expert parameters
Coach257 Sep 28, 2026
7e80bb4
[seed_omni] fix: bind the fused MoE and mHC-post slots on the V4.1 llm
Coach257 Sep 28, 2026
ad76614
[seed_omni] feat: load the CED-kept released layers into a shrunk V4.…
Coach257 Sep 28, 2026
fa16abb
[seed_omni] feat: train DSpark/MTP with a packed teacher-forced pass …
Coach257 Sep 28, 2026
9b83824
[seed_omni] feat: add V4.1 8xH20 overfit, corpus smoke and MTP train …
Coach257 Sep 28, 2026
e562049
[dist] fix: reduce ExtraParallel-only grads the way the FSDP2 wrap would
Coach257 Sep 28, 2026
981bf31
[seed_omni] feat: stream-dequantize the released fp8 n-gram tables fo…
Coach257 Sep 28, 2026
eccf413
[dist] fix: skip frozen plan shards when scaling ExtraParallel-only g…
Coach257 Sep 28, 2026
c038ba5
[seed_omni] fix: give each DeepSeek-V4.1 layer its own shared attenti…
Coach257 Sep 28, 2026
f3dd3bc
[seed_omni] fix: keep the FSDP2 input cast off the DeepSeek-V4.1 blocks
Coach257 Sep 28, 2026
1125984
[seed_omni] feat: add multi-node DeepSeek-V4.1 full and 64-card train…
Coach257 Sep 28, 2026
707f48a
[seed_omni] fix: let a float64 DeepSeek-V4.1 pre mix through the narr…
Coach257 Sep 28, 2026
e039b84
[seed_omni] feat: support sequence parallelism in the DeepSeek-V4.1 llm
Coach257 Sep 28, 2026
1af207a
[seed_omni] fix: train DeepSeek-V4.1 on tulu samples that fit max_seq…
Coach257 Sep 28, 2026
48d043f
[ops] fix: stop recompiling the V4 sparse attention backward per quer…
Coach257 Sep 28, 2026
00389fd
[seed_omni] fix: train DeepSeek-V4.1 DSpark as decode's block slot 0
Coach257 Sep 29, 2026
6164cbe
[seed_omni] feat: add DeepSeek-V4.1 64-card configs that train every …
Coach257 Sep 29, 2026
1b50a28
[seed_omni] fix: accept data-layer uint8 image tensors in the DeepSee…
Coach257 Oct 5, 2026
6deff18
[docs] chore: note that SeedOmni preprocessors get uint8 image tensor…
Coach257 Oct 5, 2026
90992f6
[seed_omni] fix: fold the DeepSeek-V4.1 dummy image into the backbone…
Coach257 Oct 6, 2026
5d0ef43
[dist] fix: give every planned ExtraParallel module its own FSDP unit
Coach257 Oct 6, 2026
d0dc31d
[docs] chore: add a hang triage recipe to the debug skill
Coach257 Oct 6, 2026
00ad20c
[config] feat: add a long-context data mix for DeepSeek-V4.1-Flash
Coach257 Oct 7, 2026
d5b27cf
[misc] fix: record the allocator history only over the profiled steps
Coach257 Oct 7, 2026
865e54f
[model] feat: stream-load the bf16 DeepSeek-V4.1 llm from its quantiz…
Coach257 Oct 7, 2026
1351d8d
[config] feat: add a 4k-32k token text mix for DeepSeek-V4.1-Flash se…
Coach257 Oct 7, 2026
771e727
[config] fix: name the 4k-32k tulu source after its SeedOmni preproce…
Coach257 Oct 7, 2026
f06fe7a
[data] fix: keep DataLoader workers from racing on the HF stream epoch
Coach257 Oct 7, 2026
5bc0932
[agent] chore: record the HF stream epoch race in data constraints
Coach257 Oct 7, 2026
a628a6f
[model] fix: keep DeepSeek-V4.1 dense sparse attention backward linea…
Coach257 Oct 7, 2026
9698505
[ckpt] fix: consolidate HF safetensors exports on every rank
Coach257 Oct 7, 2026
a217965
[ckpt] fix: shard unwrapped ExtraParallel params on their own mesh di…
Coach257 Oct 7, 2026
a6eb856
[data] fix: train DeepSeek-V4.1 on a row-shuffled tulu-3 copy
Coach257 Oct 7, 2026
14892fc
[ckpt] fix: wait for HF safetensors consolidation on a long-timeout g…
Coach257 Oct 7, 2026
fe3f463
[ckpt] fix: remove the sharded staging copy after HF safetensors cons…
Coach257 Oct 7, 2026
51eae12
[model] fix: checkpoint DeepSeek-V4.1 DSpark stages like decoder layers
Coach257 Oct 7, 2026
521e580
[model] perf: run DeepSeek-V4.1 DSpark training attention on the Tile…
Coach257 Oct 7, 2026
bf4105d
[model] perf: chunk the DeepSeek-V4.1 DSpark draft CE so long SP rows…
Coach257 Oct 7, 2026
4116a85
[omni] feat: log each SeedOmni V2 loss term as module.method.loss_name
Coach257 Oct 8, 2026
117f1ef
[omni, model] merge: sync github/omniv2 into the DeepSeek-V4.1 branch
Coach257 Oct 13, 2026
95794d1
[ops] fix: restore loss_reduction_group and explicit shift_labels in …
Coach257 Oct 13, 2026
75d986d
[ops] fix: make chunk_loss honour the ForCausalLMLoss call contract
Coach257 Oct 13, 2026
612d1b9
[ops] feat: optional per-row logits_bias in chunk_loss
Coach257 Oct 13, 2026
f96f932
[dist] fix: let a ShardedEmbedding dispatch outside any FSDP wrap
Coach257 Oct 13, 2026
0da02b7
[omni] feat: let a family converter write its own module subfolders
Coach257 Oct 13, 2026
9b5b065
[omni] test: give ModuleRuntime stand-in args needs_distributed_build
Coach257 Oct 13, 2026
f967603
[docs] fix: describe the _loss dict and rank-agreed sampling in omni …
Coach257 Oct 13, 2026
4812f05
[model] fix: port DeepSeek-V4.1 to the omniv2 module contracts
Coach257 Oct 13, 2026
b990d9d
[model] fix: carry the released engram layout to the V4.1 llm config
Coach257 Oct 13, 2026
2bd9d54
[model] refactor: score the V4.1 DSpark draft CE with chunk_loss
Coach257 Oct 13, 2026
90232a5
[model] feat: shard the V4.1 backbone over the cp mesh
Coach257 Oct 13, 2026
035cf6d
[seed_omni] feat: add a 128-card DeepSeek-V4.1 full-parameter config
Coach257 Oct 13, 2026
986d2f0
[seed_omni] fix: stream the llm's expert shards in the 256-card config
Coach257 Oct 13, 2026
dfe845a
[ops] perf: project only supervised rows when chunk_loss has a logits…
Coach257 Oct 13, 2026
93378ab
[seed_omni] docs: state the emb sizing rule and frozen-layer saving c…
Coach257 Oct 13, 2026
7534ec0
[misc] docs: refresh notes made stale by V4.1 CP and the chunk_loss c…
Coach257 Oct 13, 2026
8e95bb3
[data] test: restore the HF epoch privacy test lost in the omniv2 merge
Coach257 Oct 13, 2026
7856ffd
[ckpt] feat: keep oversized emb-sharded tables split in the HF export…
Coach257 Oct 13, 2026
fb4185e
[omni] docs: note the n-gram merge step for a V4.1 HF export
Coach257 Oct 13, 2026
8f42cc8
[ckpt] fix: stream and validate the offline merge of deferred HF tables
Coach257 Oct 13, 2026
4d0c848
[ckpt] docs: scope the deferred HF merge and describe its checks
Coach257 Oct 13, 2026
509fec0
[ckpt] fix: harden the offline merge of deferred HF tables
Coach257 Oct 13, 2026
fe93594
[ckpt] docs: note the deferred HF merge needs extra_parallel_only
Coach257 Oct 13, 2026
1039cb2
[omni, model] merge: sync origin/omniv2 (#1310)
Coach257 Oct 16, 2026
bfe4034
[docs] fix: point rank-0 broadcast loader anchor to line 1307
Coach257 Oct 16, 2026
569566f
Merge 'szl.dsv41-hf-layout' into 'szl.omniv2-deepseek-v4.1-flash'
Coach257 Oct 16, 2026
afcb266
[omni, ckpt] fix: shard large carried tensors and consolidate HF sour…
Coach257 Oct 17, 2026
2526543
[omni, distributed] fix: average extra_parallel_only replicated grads…
Coach257 Oct 19, 2026
3ee068e
[docs] docs: add DeepSeek-V4.1-Flash verification report with B token…
Coach257 Oct 20, 2026
1056f7a
[omni, parallel] feat: add Ulysses sequence parallelism to deepseek_v…
Coach257 Oct 20, 2026
fa81f0a
Merge branch 'omniv2' into szl.omniv2-deepseek-v4.1-flash
Coach257 Oct 20, 2026
2ee61e9
[omni, parallel] feat: support ViT Ulysses SP + LLM CP in DeepSeek-V4.1
Coach257 Oct 20, 2026
5f09fc2
Merge branch 'omniv2' into szl.omniv2-deepseek-v4.1-flash
Coach257 Oct 20, 2026
f92839d
[omni, ops] refactor: migrate deepseek_v41 to module-level VeomniOp
Coach257 Oct 20, 2026
e4d04e8
[docs] docs: update deepseek_v41 verification report for f92839dc2 Ve…
Coach257 Oct 20, 2026
5f653cf
support dynbsz
Coach257 Oct 22, 2026
bc10ec5
[docs, agent, omni] docs: document SeedOmni dyn_bsz contract and _omn…
Coach257 Oct 22, 2026
019f083
[model, ckpt] feat: unify DeepSeek-V4 FP8/FP4 export across VeRL and …
Coach257 Oct 22, 2026
10e05a6
[model, ops] feat: support DeepSeek-V4 DSpark MTP training and chunk_…
Coach257 Oct 22, 2026
589f3b8
[model, ops] fix: address review feedback on DeepSeek-V4 DSpark MTP a…
Coach257 Oct 22, 2026
c33eb85
[model, ckpt] fix: synthesize release fqn mapping and skip non-owner …
Coach257 Oct 22, 2026
5849ad4
Merge remote-tracking branch 'github/main' into szl.dsv4-export-quant
Coach257 Oct 23, 2026
eefb484
Merge remote-tracking branch 'github/main' into szl.dsv4-mtp-training
Coach257 Oct 23, 2026
26a6df0
[model, ops] fix: resolve op-handle CI check and review comments on D…
Coach257 Oct 23, 2026
184f1d7
[model, ckpt] fix: bound consolidation stream cache and address revie…
Coach257 Oct 23, 2026
a12b802
[model, ops] fix: preserve 3-tuple liger empty backward and clean Dee…
Coach257 Oct 23, 2026
34d3246
[model, ckpt] fix: add preflight SM90 check for quantized DeepSeek-V4…
Coach257 Oct 23, 2026
de8f5d5
[model, docs] fix: refine DeepseekV4Config from_dict overrides, forwa…
Coach257 Oct 23, 2026
4153fa6
[ckpt] fix: guard save_hf_weights and lora_config lookups on stub con…
Coach257 Oct 23, 2026
1b0d853
[ckpt] fix: gate preflight export quant check on distributed HF save …
Coach257 Oct 23, 2026
8dde08a
[model] fix: preserve DSpark full sliding window on unpacked path and…
Coach257 Oct 23, 2026
251ea7b
[model] fix: apply has_next_valid MTP mask only when targets are deri…
Coach257 Oct 23, 2026
ef64117
Merge branch 'szl.dsv4-export-quant' into szl.omniv2-deepseek-v4.1-flash
Coach257 Oct 23, 2026
27a94a3
Merge branch 'szl.dsv4-mtp-training' into szl.omniv2-deepseek-v4.1-flash
Coach257 Oct 23, 2026
17bde2f
[omni, model, ckpt] refactor: reuse deepseek_v4 DSpark and dequant/ex…
Coach257 Oct 23, 2026
3e5ac59
[ckpt] fix: gather rank-partitioned save_state sizes in _size_balance…
Coach257 Oct 23, 2026
6ba73ca
[omni, model] fix: restore _apply_gate on DeepseekV41QuantizedExperts…
Coach257 Oct 26, 2026
69f5772
[ops, omni] fix: skip plain-linear swiglu_mlp fusion on quantized pro…
Coach257 Oct 26, 2026
0b24da3
[omni, model, ckpt, ops] merge: sync main (#1330, #1344, #1345) into …
Coach257 Oct 26, 2026
3ae6b57
[docs] docs: add DeepSeek-V4.1-Flash-1017 8-GPU CP alignment and 128-…
Coach257 Oct 26, 2026
aee27b6
[omni, model] merge: sync omniv2 into szl.omniv2-deepseek-v4.1-flash
Coach257 Oct 26, 2026
446f7ee
[omni] fix: restore needs_distributed_build and configure_optimizer i…
Coach257 Oct 26, 2026
f35ea0c
[docs, ci] chore: register DeepSeek-V4.1 verification report in docs …
Coach257 Oct 26, 2026
88e5a12
[omni, ckpt, docs, ci] fix: address pre-PR review findings on DeepSee…
Coach257 Oct 26, 2026
f1277d3
[omni, model] merge: sync latest omniv2 into szl.omniv2-deepseek-v4.1…
Coach257 Oct 26, 2026
e4036fc
[omni, model, data] feat: support video frame expansion and ViT gradi…
Coach257 Oct 26, 2026
8b0955b
[omni] fix: restore ep_sharded_stream_load support on HF layout views…
Coach257 Oct 26, 2026
23401b6
[omni, model, ckpt, docs] fix: address CodeRabbit and FoolPlayer revi…
Coach257 Oct 26, 2026
55f717f
[omni, model] merge: sync latest omniv2 into szl.omniv2-deepseek-v4.1…
Coach257 Oct 26, 2026
8d79535
[omni, model, ckpt] feat: adapt deepseek_v41 for deepgemm_fp8 MoE tra…
Coach257 Oct 26, 2026
aaa00e4
[docs] docs: add DeepSeek-V4.1-Flash-1017 video SFT with 3L MTP, ViT …
Coach257 Oct 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .agents/knowledge/dev_box_env.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Dev Box Environment

Two things about the ByteDance dev boxes that make an agent shell behave
differently from the user's terminal. Both look like "the network is down" or
"the tool is gone" and are neither.

## External network needs the proxy

The box has egress, but only through the corporate relay. An agent shell does
not inherit it, so `curl https://huggingface.co` and `lark-cli` fail while the
same commands succeed in the user's terminal.

```bash
P='<corporate-http-proxy-url>'
export https_proxy="$P" http_proxy="$P" HTTPS_PROXY="$P" HTTP_PROXY="$P"
export no_proxy='localhost,127.0.0.1' NO_PROXY='localhost,127.0.0.1'
```

DNS resolves without the proxy (NAT64 `64:ff9b::` addresses), so name lookup
succeeding proves nothing — the connect is what times out. `curl -x "$P"`
returning any HTTP status, including 404, means the path works.

Tests that reach the Hub fail without this. `tests/data/test_classification_data_processor.py`
pulls a real Qwen3 tokenizer and reports `[Errno 101] Network is unreachable`.

## Tools installed under nvm are off an agent's PATH

`lark-cli` is the global npm package `@larksuite/cli` at
`~/.nvm/versions/node/<version>/bin/`. The user's interactive shell gets it from
`nvm.sh` sourced in `~/.bashrc`; a non-interactive agent shell skips `~/.bashrc`
entirely, so the binary appears not to exist. Its credentials live separately in
`~/.local/share/lark-cli/` and are unaffected.

```bash
export PATH="$(echo ~/.nvm/versions/node/*/bin | tr ' ' ':'):$PATH"
```

Before concluding a CLI is missing, check `~/.nvm/versions/node/*/bin` — and
scope `find` with `-xdev`, pruning `/mnt`: the HDFS mount makes a bare `find /`
hang long enough to look like a failure.
11 changes: 11 additions & 0 deletions .agents/skills/veomni-debug/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,17 @@ Phase 5: Knowledge capture -> pending
- FSDP2: verify sharding plan matches model structure (`veomni/distributed/parallel_plan.py`).
- Sequence parallel: check that attention inputs are properly split/gathered (`veomni/distributed/sequence_parallel/`).
- MoE: verify expert routing and load balancing (`veomni/distributed/moe/`).
- Hang with no traceback: `py-spy dump --pid` every rank's main thread and
count distinct top frames; the few odd ranks out are the lead.
- Ranks in *different* collectives mean program order diverged, e.g. a
module whose output reaches the loss only on some ranks skips its FSDP
backward collectives elsewhere (fold dummy outputs in with `0 * x.mean()`).
- Ranks in the *same* collective (compare group size, numel and FSDP
`target_fqn` via `py-spy dump --locals`) mean the stall is below Python:
`py-spy dump --native`. A stack in `ncclCollPreconnect` /
`ncclTransportP2pSetup` is NCCL's lazy connection setup over the network;
the call never returns, so the NCCL watchdog never sees the work and the
job hangs indefinitely instead of aborting.

### Phase 2: Pattern Analysis

Expand Down
11 changes: 11 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,9 @@ dmypy.json
# wandb
wandb/

# Local experiment outputs
outputs/

# ckpt
config.json
generation_config.json
Expand Down Expand Up @@ -148,6 +151,7 @@ preprocessor_config.json
special_tokens_map.json
vocab.json
processor_config.json
chat_template.jinja

# trace file
*.pt.trace.json
Expand All @@ -172,3 +176,10 @@ nsysoutput/

# Agents worksspace should not be tracked by the primary repo.
.agents_workspace/

# Official released dump. VeOmni's port is veomni/models/seed_omni/modules/deepseek_v41/.
DeepSeek-V4.1-Flash/

# Mermaid diagrams written by scripts/visualize_omni_graph.py. The authoritative
# copies are the ones OmniConfig.save_pretrained writes into the model directory.
/graphs/
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ On session start, read the following:
- `.agents/knowledge/cpu_only_env.md` — what can be verified without a GPU/NPU; read when running on a CPU-only machine

Read on demand:
- `.agents/knowledge/dev_box_env.md` — proxy and PATH facts that make network or CLI failures look like outages; read when a download, a Hub-backed test, or an installed CLI fails
- `.agents/knowledge/testing.md` — how CI selects tests, and how to decide whether a change needs one; read before adding or wiring a test
- `.agents/knowledge/multimodal_metadata.md` — multimodal metadata precompute contract; read before touching VLM/Omni collators, ViT forwards, or the model metadata hooks
- `.agents/knowledge/veomni-seed-omni.md` — SeedOmni runtime architecture, config resolution, and SP/FSDP/DDP/resume constraints; read before touching `veomni/models/seed_omni/`, `veomni/trainer/omni/`, `veomni/data/seed_omni/`, or `tasks/omni/`
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# DeepSeek-V4.1-Flash inference — image + text understanding (I2T / VQA).
#
# Two states, and the split is between "seen once" and "seen per token":
#
# prompt_encode the image is encoded, the prompt is embedded, the whole
# prompt is hashed and the backbone prefills. No sampling
# happens here — the body's last text-encoder edge is a
# feedback edge, so the node does not run a second time.
# text_ar samples from the previous hidden states, hashes, steps the
# backbone, then DSpark drafts the next block
# (``text_encoder.draft``). A DSpark hit writes the rest of
# the block onto the carrier so one body verifies several
# tokens; a miss still costs one backbone step.
#
# Body edges are walked in declaration order under a feed-forward fan-in rule:
# an edge into a node counts only when it appears *before* that node's first
# use as a source; later ones are feedback and do not gate it. That is what
# lets `deepseek_v41_text_encoder` both open the body and receive the
# backbone's output at the end of it without forming a cycle.
#
# The dependencies are the training graph's, for the same reasons:
# vision -> ngram an image span's length is however many tokens the ViT
# produced, and the n-gram node emits one row per position
# of the sequence, image spans included.
# ngram -> llm the gathered rows ride the returned dict as
# `ngram_embeds`; the backbone refuses to run without them.
# text_encoder -> llm the backbone reads embeddings off the carrier, so the
# prompt has to be embedded before it packs.
initial: prompt_encode

states:
prompt_encode:
body:
- { from: deepseek_v41_vision, to: deepseek_v41_ngram }
- { from: deepseek_v41_ngram, to: deepseek_v41_llm }
- { from: deepseek_v41_text_encoder, to: deepseek_v41_llm }
- { from: deepseek_v41_llm, to: deepseek_v41_text_encoder }
- { from: deepseek_v41_text_encoder, to: end }
transitions:
- { condition: { type: module_signal, key: text_done }, next_state: done }
- { condition: { type: default }, next_state: text_ar }

text_ar:
body:
- { from: deepseek_v41_text_encoder, to: deepseek_v41_ngram }
- { from: deepseek_v41_ngram, to: deepseek_v41_llm }
- { from: deepseek_v41_llm, to: deepseek_v41_text_encoder.draft }
- { from: deepseek_v41_text_encoder.draft, to: end }
transitions:
- { condition: { type: module_signal, key: text_done }, next_state: done }
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Per-module overrides for eager (unsharded) inference -- DeepSeek-V4.1-Flash.
#
# Eager means one rank holds each module whole, which is a statement about the
# *model*, not the config: in bf16 these four are ~750 B parameters, and the
# n-gram tables alone are 196.6 B of frozen rows. Nothing runs this file on one
# device. It is the fragment for the small-config smoke runs and for the
# sharded file to diff against, and `modules_infer_fsdp.yaml` is what a real
# checkpoint needs.
#
# Gradient checkpointing is off everywhere. It trades compute for activation
# memory during a backward, and there is no backward here -- left on it would
# recompute every layer for nothing. Training turns it on for `llm` only, which
# is the one module where it pays.
#
# Ulysses and expert parallelism are absent rather than set to 1: the training
# file raises `ep_size` for the backbone's 543.4 B of routed experts, and a
# single-token decode step touches only `num_experts_per_tok` of them. What that
# working set should be sharded as is a question for the sharded file.

deepseek_v41_vision:
model_path: deepseek_v41_vision
ops_implementation:
# The ViT falls back to an SDPA varlen path when FA2 is missing, which is
# what an aarch64 host (GB200) gets -- there are no FA2 wheels for it.
attn_implementation: flash_attention_2
accelerator:
fsdp_config:
fsdp_mode: eager
gradient_checkpointing:
enable: false

deepseek_v41_text_encoder:
model_path: deepseek_v41_text_encoder
accelerator:
fsdp_config:
fsdp_mode: eager
gradient_checkpointing:
enable: false

deepseek_v41_ngram:
model_path: deepseek_v41_ngram
model_config:
# Frozen in training and immaterial here, but the field also drives
# `freeze_model`, so leaving it out would build the tables with grads.
freeze: true
accelerator:
fsdp_config:
fsdp_mode: eager
gradient_checkpointing:
enable: false

deepseek_v41_llm:
model_path: deepseek_v41_llm
accelerator:
fsdp_config:
fsdp_mode: eager
gradient_checkpointing:
enable: false
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# Per-module overrides for sharded inference -- DeepSeek-V4.1-Flash, 8 ranks.
#
# Sized for the converted checkpoint, which keeps the released fp8 and fp4
# weights. Training dequantizes the llm at load; inference does not. Measured
# on disk:
#
# vision 0.9 GiB bf16, replicated (eager)
# text_encoder 2.5 GiB bf16, replicated (eager) -- too small to vocab-shard
# ngram 188.8 GiB fp8 rows, emb=8 -> 23.6 GiB a rank
# llm 275.7 GiB fp4 experts ep=8 (33.6 GiB) + ~6.7 GiB dense replica
# ------------------------------------
# total 468 GiB -> ~67 GiB resident a rank
#
# Against 8 x H20 that is ~67 of 97 GiB, leaving room per rank for the KV
# cache and activations. Dequantizing the llm to bf16 at load is the training
# path; it does not fit this inference layout.
#
# Gradient checkpointing is off everywhere, as in the eager file: it trades
# compute for activation memory across a backward, and there is no backward.

deepseek_v41_vision:
model_path: deepseek_v41_vision
ops_implementation:
# Falls back to an SDPA varlen path where FA2 has no wheels (aarch64/GB200).
attn_implementation: flash_attention_2
accelerator:
# 0.9 GiB, so sharding it saves nothing worth an all-gather per image.
fsdp_config:
fsdp_mode: eager
gradient_checkpointing:
enable: false

deepseek_v41_text_encoder:
model_path: deepseek_v41_text_encoder
accelerator:
# 2.5 GiB of wte + lm_head. Vocab-parallel would AllToAll every embed and
# every head; vLLM with the usual TP=1 + EP recipe keeps the vocab replica
# too. Training still shards it (modules_train.yaml).
fsdp_config:
fsdp_mode: eager
gradient_checkpointing:
enable: false

deepseek_v41_ngram:
model_path: deepseek_v41_ngram
broadcast_model_weights_from_rank0: false
ep_sharded_stream_load: true
model_config:
# Also drives `freeze_model`; without it the tables would be built with
# grads for 196.6 B rows that are never stepped.
freeze: true
accelerator:
# Row-sharded over 8 ranks -> 23.6 GiB each. A lookup dispatches the ids to
# the rank owning each row and ships the fp8 row and its scales back, so the
# result is bit-identical to a full-table gather.
extra_parallel_sizes: [8]
extra_parallel_names: ["emb"]
extra_parallel_placement_innermost: [false]
fsdp_config:
fsdp_mode: fsdp2
# FSDP's all-gather buffer for one 12.3 GiB table shard was the
# 11.45 GiB allocation that went over the card.
extra_parallel_only: true
# The tables are frozen and read once per step; re-sharding them after the
# lookup would buy back memory that was never gathered in the first place.
reshard_after_forward: false
gradient_checkpointing:
enable: false

deepseek_v41_llm:
model_path: deepseek_v41_llm
broadcast_model_weights_from_rank0: false
ep_sharded_stream_load: true
accelerator:
# The routed experts are 269 of the module's 275.7 GiB, so this is the
# sharding that matters: 33.6 GiB a rank. Tokens are dispatched to the rank
# owning their expert and the fp4 GEMMs run there.
ep_size: 8
ep_outside: false
# The remaining ~6.7 GiB is dense fp8, replicated. FSDP2 cannot shard it
# here whatever we ask: one decoder layer holds four storage dtypes (int8
# packed fp4, fp8_e4m3, fp8_e8m0 scales, fp32 hyper-connections) and an FSDP
# parameter group requires a single original dtype. So the wrap is dropped
# and the plan does the sharding -- which at inference is all FSDP was
# contributing anyway, there being no gradients or optimizer state.
fsdp_config:
fsdp_mode: fsdp2
extra_parallel_only: true
reshard_after_forward: false
gradient_checkpointing:
enable: false
Loading
Loading