[MLAS/CPU EP] Improve performance of Silu activation path within the QuickGelu CPU kernel

[hariharans29]

Description

The Silu activation is basically the same as QuickGelu but with the scaling factor (alpha) as 1. In cusomer models containing Silu, the graph optimizer suite correctly fuses the nodes into a QuickGelu with alpha = 1. This optimizes the implementation of QuickGelu when alpha = 1 by avoiding the scaling and vectorizes the subsequent elementwise multiplication.

Tests:
There are already tests for QuickGelu with alpha = 1 and there are no new tests necessary (

onnxruntime/onnxruntime/test/contrib_ops/activation_op_test.cc

Line 126 in f98c756

// Silu = x*sigmoid(x), i.e., alpha = 1.0f.

)

Performance improvements measured:
Gives about 2.5% throughput boost for a customer model that has a lot of Silu activations.

Motivation and Context

Some low hanging fruit perf improvements that give instant easy perf wins

[github-actions]

You can commit the suggested changes from lintrunner.

[Copilot]

Pull request overview

This PR optimizes the CPU implementation of the QuickGelu activation function for the special case where alpha=1.0 (equivalent to the Silu activation). The optimization avoids unnecessary scaling operations and adds a vectorized element-wise multiplication function to improve performance.

• Adds vectorized MlasEltwiseMul function for efficient element-wise multiplication
• Optimizes QuickGelu computation by skipping scaling when alpha=1.0
• Replaces scalar multiplication loop with vectorized MlasEltwiseMul call

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
onnxruntime/core/mlas/lib/eltwise.cpp	Implements vectorized element-wise multiplication function MlasEltwiseMul<float> following the pattern of existing MlasEltwiseAdd
onnxruntime/core/mlas/inc/mlas.h	Adds template declaration for MlasEltwiseMul function
onnxruntime/contrib_ops/cpu/activations.h	Modifies QuickGelu kernel to branch on alpha value, avoiding scaling for alpha=1.0 and using vectorized multiplication for final step

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[tianleiwu]

The PR optimizes the QuickGelu activation in the CPU Execution Provider, specifically improving the performance for the SiLU (Sigmoid Linear Unit) case where alpha_ is 1.0. It also introduces a new MLAS function MlasEltwiseMul for vectorized element-wise multiplication.

Key Changes

1. QuickGelu Optimization (activations.h)

• Condition for alpha_: Added a check for alpha_ == 1.0f.
  • Previous behavior: Always multiplied input by alpha_ in a loop, then computed Logistic, then element-wise multiplied input and output.
  • New behavior:
    • If alpha_ != 1.0f: Maintains the original logic (scaling input before Logistic).
    • If alpha_ == 1.0f (SiLU): Skips the scaling loop and directly computes Logistic on the input.
• Vectorization: Replaced the final scalar loop p_output[i] = p_input[i] * p_output[i] with a call to the new MlasEltwiseMul, enabling SIMD acceleration.

2. New MLAS Function: MlasEltwiseMul (mlas.h, eltwise.cpp)

• Interface: void MlasEltwiseMul(const T* left, const T* right, T* output, size_t N)
• Implementation (Float):
  • Uses MLAS_FLOAT32X4 intrinsics to process 4 elements at a time.
  • Includes a scalar fallback for remaining elements.
  • This allows for efficient vectorization (SSE/AVX depending on the platform) of the final multiplication step in QuickGelu.

3. Testing (test_eltwise.cpp)

• Added MlasEltwiseMulTest to verify the correctness of the new MlasEltwiseMul function.
• Tests with random inputs and compares against a reference scalar implementation.
• Covers both QuickGelu scaling use case (implied) and general element-wise multiplication.

Code Review Comments

• Performance: The changes correctly identify and optimize the SiLU case by removing one pass of memory access (reading/writing for alpha_ scaling). The introduction of MlasEltwiseMul further improves performance by vectorizing the final multiplication.
• Correctness: The logic for splitting the alpha_ cases preserves the mathematical equivalence. The vectorization logic in eltwise.cpp correctly handles constraints and tail elements.
• Maintainability: The changes are localized and clean. The new MLAS function is reusable for other operators that might need element-wise multiplication.

Conclusion

The PR looks solid. It effectively optimizes a common activation function (SiLU) and adds a useful primitive to MLAS. The added tests ensure correctness.

Status: Approved (Pending CI results)