[Feat] dnnl build for AVX2 W8A8 Int8

[tianmu-li]

Purpose

The CPU backend's W8A8 INT8 quantization ops (static_scaled_int8_quant, dynamic_scaled_int8_quant, onednn_scaled_mm) were gated behind __AVX512F__ and completely absent from the _C_AVX2 shared library. Running a compressed-tensors W8A8 INT8 model on an AVX2-only host (E.g: Xeon-6 with E-cores) resulted in a missing-symbol error at runtime. This PR links _C_AVX2 against the existing dnnl_ext and adds avx2 operators needed for quantization. int8 quantization is especially beneficial to AVX2, as bf16/fp16 models run at fp32 rate on AVX2.

Note: dnnl_ext now compiles with -mavx2. onednn detects isa and jit-compiles kernels during runtime, so I don't expect it to be a problem.

Test Plan

Test platform: an AVX2-enabled platform
Server

export VLLM_CPU_KVCACHE_SPACE=20
export VLLM_CPU_OMP_THREADS_BIND=0-47
MODEL_NAME={RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 or meta-llama/Llama-3.1-8B-Instruct}
vllm serve $MODEL_NAME \
  --served-model-name meta-llama/Llama-3.1-8B-Instruct \
  --port 8868 --host 0.0.0.0 \
  --no-enable-prefix-caching \
  --max-model-len=16384 \
  --max-num-batched-tokens=8192

Client

MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct
vllm bench serve --model $MODEL_NAME \
  --num-prompt 50 \
  --port=8868 \
  --random-input-len 128 \
  --random-output-len 128

Test Result

Dtype	Before throughput (toks/s)	After throughput (toks/s)
bf16	69	72.5
int8	DNR	172.7

AI assistance

This PR was developed with Claude Code assistance. All changed lines have been reviewed by the submitting author.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[gemini-code-assist]

Code Review

This pull request enables oneDNN support for AVX2 architectures by updating CMake configurations and providing AVX2-compatible implementations for vector operations, including masked stores, clamping, and reductions. It also fixes a loop increment bug in the dynamic quantization kernel where the index was being incremented by one instead of the vector element count. I have no feedback to provide.

[bigPYJ1151]

Thanks! LGTM :)

[mergify]

Hi @tianmu-li, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[mergify]

Hi @tianmu-li, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?

mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally:

# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

[tianmu-li]

Found some issues in an apple silicon smoke test https://github.com/tianmu-li/vllm/actions/runs/25149522905/job/73716695058#logs, will need to merge/rebase after #41387. Also, some potential compilation issues on ARM using dnnl that needs fixing

[louie-tsai]

loop @louie-tsai