fix(capture): post_block captures true block output (residual + mlp), not bare residual - #202
Conversation
…lp), not bare residual
…mma3 (capture true block output)
Multi-architecture validation (GPU, node1 RTX 3090)Tested across architectures — and the second commit fixes a real bug this surfaced. Bug found in Gemma: #174's hook placement (which I initially mirrored) put Results:
MoE models share the dense residual skeleton (MoE only swaps the FFN internals), so Qwen3 covers them structurally. gemma2 is fixed by analogy to gemma3 (identical sandwich-norm structure; no cached checkpoint to GPU-test). Unit 4/4; 100 steering/capture regression tests pass (1 pre-existing CUDA Triton-warmup failure, identical on baseline). |
Problem
The
post_mlp→post_blockrename (#198) renamed the hook but kept the old semantics:post_blockcaptured the bare post-attentionresidual. In vLLM's deferred-add scheme a decoder layer's MLP output isn't folded intoresidualuntil the next layer'sinput_layernorm, sopost_blockwas byte-identical topost_attn— anyone readingpost_blockactivations got the pre-MLP residual, not the block output. Silent footgun, affecting both capture and steering at that hook.This is the remaining substantive piece of #174 (the rest landed via #195/#198/#200/#201).
Fix
New
apply_block_steering(module, hidden_states, residual): capture consumers now observeresidual + hidden_states— the true block output, HF'shidden_states[L+1]. The sum is computed only when a capture manager is active (gated byget_active_capture_manager()), so non-capture servers pay nothing andtorch.compiletraces it as a constant branch.Steering propagation is unchanged — the steering delta still rides
residualinto the next layer's fused add, identical to the old behavior. So generation outputs are unaffected; only the captured/observed value changes.Converted the 41 deferred-add model decoder layers (those firing
POST_BLOCKonresidual) toapply_block_steering. Models that fire the hook on a tensor that is already the block output were left onapply_layer_steering(matching #174).Validation
tests/model_executor/layers/test_block_steering.py: capture seesresidual + hidden_states; gated off with no manager;post_blockdiffers frompost_attnby exactly the MLP branch; steering ridesresidual.post_block[L]differs frompost_attn[L](was identical before) and equalspre_attn[L+1]exactly — i.e. it is the true block outputhidden_states[L+1].