Skip to content

fix(capture): post_block captures true block output (residual + mlp), not bare residual - #202

Merged
RhizoNymph merged 2 commits into
feat/integrationfrom
fix/post-block-true-output
Jun 24, 2026
Merged

fix(capture): post_block captures true block output (residual + mlp), not bare residual#202
RhizoNymph merged 2 commits into
feat/integrationfrom
fix/post-block-true-output

Conversation

@RhizoNymph

Copy link
Copy Markdown
Owner

Problem

The post_mlppost_block rename (#198) renamed the hook but kept the old semantics: post_block captured the bare post-attention residual. In vLLM's deferred-add scheme a decoder layer's MLP output isn't folded into residual until the next layer's input_layernorm, so post_block was byte-identical to post_attn — anyone reading post_block activations got the pre-MLP residual, not the block output. Silent footgun, affecting both capture and steering at that hook.

This is the remaining substantive piece of #174 (the rest landed via #195/#198/#200/#201).

Fix

New apply_block_steering(module, hidden_states, residual): capture consumers now observe residual + hidden_states — the true block output, HF's hidden_states[L+1]. The sum is computed only when a capture manager is active (gated by get_active_capture_manager()), so non-capture servers pay nothing and torch.compile traces it as a constant branch.

Steering propagation is unchanged — the steering delta still rides residual into the next layer's fused add, identical to the old behavior. So generation outputs are unaffected; only the captured/observed value changes.

Converted the 41 deferred-add model decoder layers (those firing POST_BLOCK on residual) to apply_block_steering. Models that fire the hook on a tensor that is already the block output were left on apply_layer_steering (matching #174).

Validation

  • New unit test tests/model_executor/layers/test_block_steering.py: capture sees residual + hidden_states; gated off with no manager; post_block differs from post_attn by exactly the MLP branch; steering rides residual.
  • Regression: 100 steering/capture layer tests pass (1 pre-existing CUDA Triton-warmup failure, identical on baseline).
  • GPU e2e (Qwen3-0.6B, real capture): post_block[L] differs from post_attn[L] (was identical before) and equals pre_attn[L+1] exactly — i.e. it is the true block output hidden_states[L+1].

@RhizoNymph

Copy link
Copy Markdown
Owner Author

Multi-architecture validation (GPU, node1 RTX 3090)

Tested across architectures — and the second commit fixes a real bug this surfaced.

Bug found in Gemma: #174's hook placement (which I initially mirrored) put apply_block_steering before post_feedforward_layernorm in gemma2/gemma3, so it captured residual + raw_mlp instead of the true block output residual + post_ffn_norm(mlp). The block-output invariant caught it: |post_block[L] - pre_attn[L+1]| = 2368 (should be ~0). Fixed by applying the post-FFN norm before the hook. Only gemma2/gemma3 are affected (post-FFN sandwich norm); a scan confirms all 41 converted models now go straight from the hook to return.

Results:

check Qwen3 (Llama-style) Gemma3 (post-FFN-norm)
invariant post_block[L]==pre_attn[L+1], !=post_attn[L] PASS PASS (after fix; 0.0 exact)
parity (baseline tokens == branch tokens) PASS PASS
cudagraph (eager tokens == compiled tokens) PASS
decode (all_generated capture, 5 rows, post_block≠post_attn, finite) PASS

MoE models share the dense residual skeleton (MoE only swaps the FFN internals), so Qwen3 covers them structurally. gemma2 is fixed by analogy to gemma3 (identical sandwich-norm structure; no cached checkpoint to GPU-test). Unit 4/4; 100 steering/capture regression tests pass (1 pre-existing CUDA Triton-warmup failure, identical on baseline).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant