[AgentX] DeepSeek-V4 B300 SGLang update - #2701
Conversation
…nterval (+28%) Retunes the existing B300 SGLang AgentX recipe. The headline change is --prefill-decode-interval 20 (sgl-project/sglang#35017): under speculative decoding plus DP attention SGLang synchronises decode globally, so a rank with no prefill work runs an idle batch and the busiest rank sets the clock for all eight. Measured at conc 128 over 1800 s on 8xB300, that takes output throughput from 2,433-2,470 to 3,127-3,161 tok/s and closes the gap to the vLLM recipe from 1.32x to 1.03x. Every engine-side change is scoped to the DP-attention path; the TP-only path is left exactly as upstream had it, because all measurements here ran with DP attention enabled. Also restructures the search space from 47 points to 12 and raises the CPU tier to match the vLLM agentic lane on the same runner.
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
1 similar comment
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32435332678 |
No functional change; retriggers CI.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| CUDA_GRAPH_MAX_BS=$CONC | ||
| [ "$CUDA_GRAPH_MAX_BS" -gt 64 ] && CUDA_GRAPH_MAX_BS=64 | ||
|
|
||
| # --cuda-graph-max-bs is an alias whose dest is cuda_graph_max_bs_decode, so the | ||
| # two forms below are the same knob and must not both be passed. | ||
| CUDA_GRAPH_ARGS=(--cuda-graph-max-bs "$CUDA_GRAPH_MAX_BS") | ||
| SWA_FULL_TOKENS_RATIO=0.1 | ||
| if [ "$DP_ATTENTION" = "true" ]; then | ||
| # Decode graphs must cover the padded MTP batch across all DP ranks, which | ||
| # exceeds CONC; capping at 64 would fall back to eager decode. | ||
| CUDA_GRAPH_ARGS=(--cuda-graph-max-bs-decode 544) | ||
| SWA_FULL_TOKENS_RATIO=0.075 |
There was a problem hiding this comment.
🟡 On the DP-attention path, --cuda-graph-max-bs-decode is hardcoded to 544 regardless of concurrency, but this same PR raises the DEP8+hicache sweep's top concurrency to conc=576 (configs/nvidia-master.yaml conc-list [...,512,576]), up from the prior max of 512. Since 544 was clearly sized as headroom over the old 512 ceiling, the new conc=576 point — the top of this PR's own sweep — will exceed the captured CUDA graph and silently fall back to eager decode, undercutting the throughput measured at the highest concurrency. Raise the cap to cover the new max (e.g. ~608) or trim the sweep back to 512.
Extended reasoning...
The cap --cuda-graph-max-bs-decode 544 at dsv4_fp4_b300_sglang_mtp.sh:174 is unconditional on the DP-attention path — it does not scale with TP, EP_SIZE, or CONC. The adjacent comment explains why a fixed value is used at all: decode CUDA graphs on this path must cover "the padded MTP batch across all DP ranks, which exceeds CONC," and capping at the TP-only default of 64 would fall back to eager decode. That reasoning is sound, but the specific constant chosen (544) is a poor fit for the new search space this very PR introduces.
544 = 512 + 32. That is a strong signal it was sized as headroom over the previous ceiling of the DEP8+dp-attn row (max conc was 512 before this PR, per the old conc-list: [52, 72, 100, 128, 144, 196, 512]). This same diff replaces that row with conc-list: [64, 128, 256, 384, 512, 576] in configs/nvidia-master.yaml:1172 — pushing the ceiling to 576, which exceeds the hardcoded cap of 544.
The padded decode batch that the comment warns about tracks total concurrency (via --enable-dp-attention-local-control-broadcast, which pads every DP rank's batch to the busiest rank's request count, further inflated by MTP's speculative-num-draft-tokens=4 verify batch). Whatever the exact scaling constant, at conc=576 the effective padded batch will exceed the 544-sized graph that was calibrated for the 512-max sweep. When a decode step's batch exceeds the captured cuda_graph_max_bs_decode, SGLang does not error — it silently falls back to eager decode for that step, which is materially slower. This is the exact fallback behavior the neighboring comment already describes for the CONC-capped-at-64 case, just triggered from the other direction (cap too low) instead of being caught.
The result: the conc=576 data point — the newest, highest-concurrency point in this PR's own new search space — would run a meaningful fraction of its decode steps in eager mode instead of using the captured graph, producing an artificially depressed throughput number for exactly the regime this PR is trying to characterize at the top of the sweep.
Concrete walk-through: (1) Old DEP8+dp-attn row's max conc was 512; 544 covers that with 32 headroom for MTP padding — a deliberate, tight fit. (2) This PR bumps the row's conc-list to end at 576 instead of 512. (3) The cap at line 174 is left untouched at 544. (4) At runtime, when CONC=576, MAX_RUNNING_REQUESTS=1152 and the DP-attention broadcast pads every rank's decode batch toward the busiest rank's count, scaling with CONC — pushing the padded batch past 544. (5) SGLang detects the decode batch exceeds cuda_graph_max_bs_decode and falls back to eager decode for those steps, silently, with no startup or runtime error. (6) The measured throughput at conc=576 is now a mix of graph-captured and eager-decode steps, understating the true captured-graph throughput at that concurrency — the opposite of what a "flagship" top-of-sweep data point is meant to show.
Fix: raise the DP-path cap to cover the new ceiling (e.g. size it the same way — some headroom over 576, such as ~608 — or compute it from CONC the same way CUDA_GRAPH_MAX_BS is derived on the TP-only path) so the constant tracks whatever the sweep's max concurrency is, instead of being a value hand-tuned for a ceiling this PR itself moves. Alternatively, cap the new sweep's row at conc=512 if 576 isn't actually needed.
This is not a crash and doesn't block the PR's headline throughput claims — the cited gains in the changelog are measured at conc=128, well under 544 and unaffected. It does quietly corrupt one exploratory data point at the very top of the new sweep, so it is worth fixing but not blocking.
| # DP_ATTENTION: | ||
| # dp-attn: false -> TP-only + flashinfer_mxfp4 + chunked-prefill 8192 | ||
| # + EAGLE (3,1,4) + mem-fraction 0.90 | ||
| # dp-attn: true -> DP-attn + flashinfer_mxfp4 + chunked-prefill 32768 | ||
| # + EAGLE (1,1,2) + mem-fraction 0.92 + max-running 256 | ||
| # + mem-fraction 0.88 + swa-full-tokens-ratio 0.1 | ||
| # dp-attn: true -> DP-attn + megamoe + fp4 indexer | ||
| # + chunked-prefill 65536 + mem-fraction 0.90 | ||
| # + swa-full-tokens-ratio 0.075 | ||
| # + prefill-decode-interval 20 |
There was a problem hiding this comment.
🟡 The rewritten comment above dsv4-fp4-b300-sglang-mtp (configs/nvidia-master.yaml:1180) states the dp-attn: true path uses a fixed 'chunked-prefill 65536', but the script sets CHUNKED_PREFILL_SIZE=$((8192 * TP)), so it's 65536 only for TP=8; for this config's own DEP4 row (tp:4, ep:4) it's 32768. Suggest wording it as 'chunked-prefill 8192*TP' to match the perf-changelog, which already documents the scaling correctly.
Extended reasoning...
The comment block above dsv4-fp4-b300-sglang-mtp was rewritten in this PR to summarize the two SGLang recipe paths (TP-only vs DP-attn). For the dp-attn: true branch it states a single fixed value: "chunked-prefill 65536". But the script change in this same PR (benchmarks/single_node/agentic/dsv4_fp4_b300_sglang_mtp.sh:147) replaces the old fixed CHUNKED_PREFILL_SIZE=16384 with CHUNKED_PREFILL_SIZE=$((8192 * TP)). That's a formula, not a constant, and it only evaluates to 65536 when TP=8.
Step-by-step proof: The dsv4-fp4-b300-sglang-agentic-hicache-mtp config directly above (same file, lines ~1171-1172) sweeps both DEP shapes under dp-attn: true: a { tp: 4, ep: 4, ... } row and a { tp: 8, ep: 8, ... } row. Plugging TP into the new formula: TP=4 → 81924 = 32768; TP=8 → 81928 = 65536. So the comment's "65536" is correct only for the DEP8 shape and is off by 2x for the actively-swept DEP4 shape.
One verifier raised a reasonable objection: the same comment block also states "mem-fraction 0.90" for the dp-attn path even though the script lowers it to 0.89 at conc>=512, so maybe "65536" is likewise just a representative headline value rather than a precise spec. I don't think that analogy holds up, though. MEM_FRACTION_STATIC=0.9 genuinely is the default in the script — 0.89 is a narrow override for one tail condition (conc>=512) within a single TP shape, so "0.90" accurately describes the common case. Chunked-prefill has no such default: CHUNKED_PREFILL_SIZE=$((8192 * TP)) is unconditional, and both TP=4 and TP=8 are core, equally-weighted rows in this PR's own search space (not an edge-case exception). There's no single representative value to pick here — the comment needs the formula, not a number.
This is purely a documentation/comment inaccuracy: the script itself computes the value correctly, so nothing breaks at runtime and no benchmark result is affected. The PR's own perf-changelog entry already states the fix correctly ("Scale chunked-prefill-size with dp_size (8192 * TP) instead of pinning 65536"), so the config comment is simply out of sync with the changelog and the code it's describing. A one-line wording fix (e.g. "chunked-prefill 8192*TP") would resolve it.
… weights floor DEP4 shards the model over half the node, so per-rank weight memory roughly doubles and the weights-only floor rises above 0.9. The engine refuses to start with 'Loaded weights leave no GPU memory for the KV cache' and reports a minimum viable 0.9013. Keep upstream's 0.95 there; raise DEP8 to 0.93 (0.92 at the conc>=512 tail).
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32435670884 |
hicache capacity is a host/device token ratio, so host bytes scale with device KV and therefore with mem-fraction-static -- the two knobs multiply. ratio=4 at mem-fraction 0.93 left only 5.84 GB free on a 2,964 GB node and the V4 paged pool failed to allocate (requested 8.70 GB). ratio=3 keeps the tier near 2 TB with headroom.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32440084204 |
MegaMoE's transient workspace sits outside the static allocation and needs a single ~7 GB contiguous block, so required headroom grows with batch size. At conc 256, 0.93 (~16 GB free) and 0.95 (~11 GB free) both OOM one DP rank, which hangs the engine in the MLP-sync collective; 0.835 (~42 GB free) runs. Ladder: 0.93 at conc 64/128, 0.9 at 256, 0.89 at 384, 0.875 at 512/576.
AgentX concurrency counts session trees, not requests, and the recipe already sets max-running-requests to 2*CONC to allow fan-out. Capturing decode graphs only up to CONC therefore dropped every larger batch to eager decode on the conc 1/4/8 rows. Capture to 4*CONC (still capped at 64); the runtime clamps to the request-pool size, so it cannot over-capture.
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32443647675 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32454825757 |
0.95 leaves only ~11 GB of GPU headroom -- the same margin that OOM'd a rank at DEP8 conc 256 and hung the engine. 0.93 gives ~16 GB, matching what DEP8 conc 128 runs with at an identical per-rank load (max-running/dp = 32).
…val 10 + load-back fix validation at conc 512 完整复制 #2701 的 DP-attention 路径改动并将 prefill-decode-interval 从 20 改为 10；临时将 agentic sweep 缩减到 conc 512 单点，recipe 脚本在启动 sglang 前把 sglang-loadback-multipin.patch（sgl-project/sglang#35880，即 #34975 cherry-pick 到 dev-nightly-0820 基线并保留 #34519 write-back 门控） 打到镜像内 editable 安装的 sglang 源码上，验证 unified_cache commit_load_back 单 pin 断言的修复。仅为验证用途，不用于合入。
…c 512 内容与 #2701 完全一致（prefill-decode-interval 保持 20），唯一差异： 1) 临时将 agentic sweep 缩减到 conc 512 单点；2) recipe 脚本在启动 sglang 前把 sglang-loadback-multipin.patch（sgl-project/sglang#35880，即 #34975 cherry-pick 到 dev-nightly-0820 基线并保留 #34519 write-back 门控） 打到镜像内 editable 安装的 sglang 源码上，验证 unified_cache commit_load_back 单 pin 断言的修复。仅为验证用途，不用于合入。
…c 512 内容与 #2701 完全一致（prefill-decode-interval 保持 20），唯一差异： 1) 临时将 agentic sweep 缩减到 conc 512 单点；2) recipe 脚本在启动 sglang 前把 sglang-loadback-multipin.patch（sgl-project/sglang#35880，即 #34975 cherry-pick 到 dev-nightly-0820 基线并保留 #34519 write-back 门控） 打到镜像内 editable 安装的 sglang 源码上，验证 unified_cache commit_load_back 单 pin 断言的修复。仅为验证用途，不用于合入。
DEP8 at conc 384/512 dies mid-run on an assertion in unified_tree_core.py commit_load_back: load_back_pending_id holds a single anchor, but a node can legitimately be pinned by two concurrent H->D load-backs. Eight schedulers exit together and the surviving DP ranks block forever in the MLP-sync collective, which surfaces as a frozen in-flight count with zero errors. sgl-project/sglang#34975 fixes it (the pin becomes a set); it is still open, so this points at sglang-staging:dev-cu13-pr-35880 -- build commit 2fca6c4e35, which is dev-nightly-0820 (92eeed41d7) with #35880 cherry-picked on top. Same CUDA 13.0.3, same layer count. Swap to a released tag once it merges.
# Conflicts: # perf-changelog.yaml
The removed rows total 42 points (7 + 12 + 6 + 10 + 7), not 47.
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32514214332 |
DEP4 cannot serve this trace. Sharding DeepSeek-V4 over half the node leaves ~627k KV tokens per rank at mem-fraction 0.93 (measured from the hicache host pool / ratio 8), against prompts reaching 950,812 tokens -- the longest requests cannot be held at all. Raising mem-fraction far enough to hold them leaves too little room for the mega-MoE workspace, so the two constraints barely overlap; every DEP4 point failed. Search space is now TP-8 no-offload at conc 1/4/8/16/32 and DEP8+hicache at conc 32/64/128/256/384/512/576 -- still 12 points.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32514214332 |
…ill-decode-interval 10 at conc 512 基于 #2701 的 recipe（prefill-decode-interval 改为 10），临时将 agentic sweep 缩减到 conc 512 单点，recipe 脚本在启动 sglang 前把 sglang-loadback-multipin.patch（sgl-project/sglang#35880，即 #34975 cherry-pick 到 dev-nightly-0820 基线并保留 #34519 write-back 门控） 打到镜像内 editable 安装的 sglang 源码上，验证 unified_cache commit_load_back 单 pin 断言的修复。仅为验证用途，不用于合入。
加一行注释说明 interval=10 是相对 #2701 的 A/B 对照，同时用真实改动重新触发 CI。
…action 0.86 将 prefill-decode-interval 从 10 改为 5，DEP8 conc512 档 mem-fraction-static 从 0.875 降到 0.86（对齐 #2701 最新档位），继续 conc 512 单点 A/B。 追加对应 perf-changelog 条目。
…cannot abort a run
The recipe launched sglang_router with --disable-retries. A single
transient router->engine send failure ("error sending request") then
surfaces as a 500, and AgentX treats a failed root warmup request as
fatal: "ProfileAborted: A root AgentX warmup request failed, so
profiling was not started." The engine stays healthy throughout -- it
keeps logging prefill batches and serving /metrics 200 OK -- so the run
dies with no result and no crash to point at.
Measured on 8xB300 / DeepSeek-V4-Pro at conc 512: three separate 2h15m
arms were killed this way, each by exactly one failed request (2 ERROR
lines in router.log). After enabling retries, one 3600s run logged 22
such transients spread over all 8 DP workers with zero client-visible
500s and errors=0 -- every one recovered.
Retry is safe here: "error sending request" means the request was never
delivered, so there is no partial state, and it cannot inflate
throughput because it only fires on a failed send. Runs that never hit
a transient are unaffected (the three completed baseline arms logged
zero router errors either way).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # perf-changelog.yaml
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32695861783 |
2 similar comments
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32695861783 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32695861783 |
|
/stage-results 32695861783 |
|
@cquil11 staged run 32695861783: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-24~r32695861783 This run remains available across future |
|
@cquil11 staged run 32695861783: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-24~r32695861783 This run remains available across future |
Resolve the perf-changelog.yaml tail conflict: main's three new entries (#2684, #2701, #2687) keep their original bytes, and the dsv4-fp4-b200-dynamo-trt entry moves to the physical end of the file per the append-only invariant. 解决 perf-changelog.yaml 末尾冲突：main 的三个新条目(#2684、#2701、#2687) 保留原有字节，dsv4-fp4-b200-dynamo-trt 条目按仅追加约束移至文件物理末尾。 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
中文：为 #2701 回滚条目填入规范的 pr-link。
…新 DSV4 B300 SGLang AgentX 镜像和 HiCache 并发网格 (#2759) * chore(dsv4-b300): move AgentX HiCache MTP lane to a published nightly image Bump dsv4-fp4-b300-sglang-agentic-hicache-mtp from the one-off staging tag lmsysorg/sglang-staging:dev-cu13-pr-35880 to the published nightly lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1, so every point is reproducible from a public image. Image-only change: the write policy, search space, dram-utilization and all serving flags from #2701 are left untouched. --prefill-decode-interval is retained because sgl-project/sglang#35017 merged before this nightly's build commit. The HiCache load-back fix (sgl-project/sglang#34975 and its cherry-pick #35880) is still unmerged, so the DEP8 conc 384/512/576 crash is a known risk; this is recorded in perf-changelog.yaml. 将 dsv4-fp4-b300-sglang-agentic-hicache-mtp 的镜像从一次性构建的 staging 标签 lmsysorg/sglang-staging:dev-cu13-pr-35880 切换到已发布的 nightly lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1，使所有数据点均可基于公开 镜像复现。 本次仅改动镜像：#2701 引入的写策略、搜索空间、dram-utilization 及全部服务参数 均保持不变。由于 sgl-project/sglang#35017 已在该 nightly 的构建提交之前合并， --prefill-decode-interval 得以保留。HiCache load-back 修复（sgl-project/sglang#34975 及其 cherry-pick #35880）仍未合并，因此 DEP8 并发 384/512/576 存在已知的崩溃 风险，该风险已记录在 perf-changelog.yaml 中。 Co-Authored-By: Claude <noreply@anthropic.com> * chore(changelog): point the entry at PR #2759 Replace the placeholder pr-link now that the PR number exists. PR 号确定后，将 changelog 条目中的占位 pr-link 替换为实际链接。 Co-Authored-By: Claude <noreply@anthropic.com> * fix(dsv4-b300): drop DEP8 concurrency 64 and 128 from the hicache row Concurrency 128 failed on run 33051183882 with a CUDA OOM inside deep_gemm fp8_fp4_paged_mqa_logits (5.35 GiB requested, 4.55 GiB free) on DP ranks 3, 4 and 5, crashing scheduler_0 and aborting AIPerf during warmup. Concurrency 64 shares the same mem-fraction-static 0.93 tier and is dropped with it. The changelog entry also corrects the previous entry's prediction: DEP8 concurrency 384, 512 and 576 passed on this image, so the HiCache load-back failure did not reproduce. 并发 128 在 run 33051183882 上因 deep_gemm fp8_fp4_paged_mqa_logits 内的 CUDA OOM 失败（请求 5.35 GiB，仅剩 4.55 GiB），DP rank 3、4、5 同时报错，导致 scheduler_0 崩溃并使 AIPerf 在 warmup 阶段中止。并发 64 与其同属 mem-fraction-static 0.93 档位，一并移除。 changelog 同时修正了上一条目的预测：DEP8 并发 384、512、576 在该镜像上均通过， HiCache load-back 失败并未复现。 Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cam Quilici <cjquilici@gmail.com>
2 participants
DeepSeek-V4 B300 SGLang update on Agentx
Note
Medium Risk
Benchmark and container-image changes only, but they alter memory, HiCache, and router retry behavior on long agentic runs where mis-tuning can OOM or hang DP collectives.
Overview
Updates the dsv4-fp4-b300-sglang-agentic-hicache-mtp AgentX lane: staging SGLang image (prefill-decode interval + HiCache load-back fix), dram-utilization 0.95, and a 12-point search space (TP8 baseline + DEP8+hicache only; TP4/DEP4 rows removed).
The benchmark script retunes DP-attention serving: MegaMoE with FP4 indexer and matching DeepGEMM env flags, prefill delayer / prefill-decode-interval 20, chunked-prefill scaled as
8192 * TP, concurrency-aware mem-fraction-static, HiCache ratio 3 on TP≥8, decode CUDA graphs at 544, and router retries so transient send failures do not abort long runs. The TP-only path keeps FlashInfer MoE but sizes decode graphs at 4×CONC (cap 64) for subagent fan-out.Docs in perf-changelog capture measured throughput/OOM/TTFT tradeoffs and align the nvidia-master recipe comments with the new flags.
Reviewed by Cursor Bugbot for commit 471e9db. Bugbot is set up for automated code reviews on this repo. Configure here.