feat: Add aibrix profile for E2E testing framework #688

yehudit1987 · 2025-11-18T07:28:05Z

Add AIBrix profile implementing 5-stage deployment:
- Semantic Router (Helm)
- AIBrix Dependencies (Envoy Gateway CRDs)
- AIBrix Core (Gateway Plugins, Metadata Service, Controller)
- Demo LLM + Gateway API Resources
- Environment verification with stabilization wait
Implement robust error handling:
- Track deployment state at each stage
- Partial cleanup on failure to prevent orphaned resources
- Panic recovery with defer-based cleanup
Add configurable AIBrix version via AIBRIX_VERSION env var
Integrate into CI with matrix strategy for parallel testing
- Both ai-gateway and aibrix profiles tested on every PR
- Separate test reports per profile
Support all test cases: chat-completions, domain-classify,
semantic-cache, pii-detection, jailbreak-detection, stress tests

Resolve [E2E] Add aibrix profile for E2E testing framework #660

netlify · 2025-11-18T07:28:12Z

✅ Deploy Preview for vllm-semantic-router ready!

Name	Link
🔨 Latest commit	`dd72c08`
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/691cc4f5e89d1f0008791024
😎 Deploy Preview	https://deploy-preview-688--vllm-semantic-router.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

github-actions · 2025-11-18T07:28:35Z

👥 vLLM Semantic Team Notification

The following members have been identified for the changed files in this PR and have been automatically assigned:

📁 `e2e`

Owners: @Xunzhuo
Files changed:

e2e/profiles/aibrix/profile.go
e2e/README.md
e2e/cmd/e2e/main.go

📁 `Root Directory`

Owners: @rootfs, @Xunzhuo
Files changed:

.github/workflows/integration-test-k8s.yml

📁 `deploy`

Owners: @rootfs, @Xunzhuo
Files changed:

deploy/kubernetes/aibrix/semantic-router-values/values.yaml

📁 `tools`

Owners: @yuluo-yx, @rootfs, @Xunzhuo
Files changed:

tools/make/e2e.mk

🎉 Thanks for your contributions!

This comment was automatically generated based on the OWNER files in the repository.

Copilot

Pull Request Overview

This PR adds a new AIBrix profile to the E2E testing framework to enable testing of Semantic Router with vLLM AIBrix integration. The implementation follows the existing pattern established by the ai-gateway profile while adding robust error handling and cleanup mechanisms.

Key changes:

Implemented 5-stage deployment for AIBrix (Semantic Router, AIBrix Dependencies, AIBrix Core, Demo LLM, environment verification)
Added comprehensive error handling with partial cleanup and panic recovery
Integrated into CI with matrix strategy for parallel testing of both ai-gateway and aibrix profiles

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
e2e/profiles/aibrix/profile.go	New AIBrix profile implementation with 5-stage deployment, error handling, and cleanup logic
e2e/cmd/e2e/main.go	Registered the new aibrix profile in the profile factory
tools/make/e2e.mk	Updated help text to document the aibrix profile
e2e/README.md	Updated documentation to list aibrix as a supported profile (moved from future to current)
.github/workflows/integration-test-k8s.yml	Added matrix strategy to test both profiles in parallel with separate artifact names

Comments suppressed due to low confidence (2)

.github/workflows/integration-test-k8s.yml:28

Go version '1.24' does not exist as of January 2025. The latest stable Go version is 1.23. This should be corrected to a valid Go version like '1.23' or '1.22'.

          go-version: '1.24'

.github/workflows/integration-test-k8s.yml:33

Rust version 1.90 does not exist as of January 2025. The latest stable Rust version is around 1.84. This should be corrected to a valid Rust version or use 'stable' instead.

          toolchain: 1.90

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

e2e/profiles/aibrix/profile.go

rootfs · 2025-11-18T16:28:43Z

@yehudit1987 this is ready to go, can you resolve the conflict so we can run the latest e2e test. Thanks

Signed-off-by: Yehudit Kerido <[email protected]>

rootfs · 2025-11-18T17:48:53Z

reading the log, PII policy is disabled and that caused the test failure

================================================================================
TEST RESULTS
================================================================================
✅ PASSED - chat-completions-request (2.439804138s)
✅ PASSED - chat-completions-stress-request (56.247452209s)
❌ FAILED - domain-classify (2m54.006573145s)
  Error: domain classification test failed: 0% accuracy (0/280 correct)
✅ PASSED - semantic-cache (14.79522197s)
❌ FAILED - pii-detection (50.703493022s)
  Error: PII detection test failed: 0% accuracy (0/100 correct)
✅ PASSED - jailbreak-detection (59.186211484s)
✅ PASSED - chat-completions-progressive-stress (32.544681604s)

Co-authored-by: Copilot <[email protected]> Signed-off-by: yehudit1987 <[email protected]>

yehudit1987 · 2025-11-18T18:40:48Z

reading the log, PII policy is disabled and that caused the test failure

================================================================================
TEST RESULTS
================================================================================
✅ PASSED - chat-completions-request (2.439804138s)
✅ PASSED - chat-completions-stress-request (56.247452209s)
❌ FAILED - domain-classify (2m54.006573145s)
  Error: domain classification test failed: 0% accuracy (0/280 correct)
✅ PASSED - semantic-cache (14.79522197s)
❌ FAILED - pii-detection (50.703493022s)
  Error: PII detection test failed: 0% accuracy (0/100 correct)
✅ PASSED - jailbreak-detection (59.186211484s)
✅ PASSED - chat-completions-progressive-stress (32.544681604s)

All tests are passing now.

================================================================================
TEST RESULTS
================================================================================
✅ PASSED - chat-completions-request (2.703768398s)
✅ PASSED - chat-completions-stress-request (1m44.397821511s)
✅ PASSED - domain-classify (5m50.878113089s)
✅ PASSED - semantic-cache (21.788123278s)
✅ PASSED - pii-detection (1m18.554515807s)
✅ PASSED - jailbreak-detection (2m2.531104517s)
✅ PASSED - chat-completions-progressive-stress (48.51256606s)
================================================================================
Total: 7 | Passed: 7 | Failed: 0

rootfs · 2025-11-18T19:11:52Z

thanks @yehudit1987 let run the latest code for one last time

Xunzhuo

thanks a lot, nice work! would you like to try adding more testcases to make sure the new signal-decsion engines works well?

yehudit1987 · 2025-11-19T07:17:15Z

thanks a lot, nice work! would you like to try adding more testcases to make sure the new signal-decsion engines works well?

Hi @Xunzhuo sure, I will and thanks for the feedback. Do we have an open issue for that? Want me to create one?

yehudit1987 · 2025-11-19T07:37:01Z

@Xunzhuo will start to work on #692 let me know if the requirements are different or need some modifications.

…m-project#688 The aibrix E2E profile was designed to use ModernBERT for PII detection (as configured in PR vllm-project#688), but during the merge it was incorrectly changed to use the LoRA PII model which isn't available in the aibrix deployment. This commit reverts the aibrix PII model configuration to use: - model_id: "models/pii_classifier_modernbert-base_presidio_token_model" - use_modernbert: true This restores the original working configuration from PR vllm-project#688 and allows the aibrix E2E tests to pass while keeping the LoRA PII auto-detection feature available for other profiles that explicitly configure it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Yossi Ovadia <[email protected]>

The previous policy.go changes added a default_decision fallback mechanism that broke aibrix profile. Aibrix uses static YAML config and doesn't have a default_decision defined. Reverting to the upstream version from PR vllm-project#688 that passed aibrix tests successfully. Related to vllm-project#648 Signed-off-by: Yossi Ovadia <[email protected]>

…m-project#688 The aibrix E2E profile was designed to use ModernBERT for PII detection (as configured in PR vllm-project#688), but during the merge it was incorrectly changed to use the LoRA PII model which isn't available in the aibrix deployment. This commit reverts the aibrix PII model configuration to use: - model_id: "models/pii_classifier_modernbert-base_presidio_token_model" - use_modernbert: true This restores the original working configuration from PR vllm-project#688 and allows the aibrix E2E tests to pass while keeping the LoRA PII auto-detection feature available for other profiles that explicitly configure it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Yossi Ovadia <[email protected]>

github-actions bot assigned rootfs and Xunzhuo Nov 18, 2025

yehudit1987 force-pushed the aibrix_profile branch 4 times, most recently from d29de37 to 294e4df Compare November 18, 2025 10:40

yehudit1987 marked this pull request as ready for review November 18, 2025 11:53

yehudit1987 requested review from Xunzhuo and rootfs as code owners November 18, 2025 11:53

yehudit1987 force-pushed the aibrix_profile branch from 294e4df to e13f277 Compare November 18, 2025 13:55

rootfs requested a review from Copilot November 18, 2025 14:01

Copilot started reviewing on behalf of rootfs November 18, 2025 14:02 View session

Copilot finished reviewing on behalf of rootfs November 18, 2025 14:04

Copilot AI reviewed Nov 18, 2025

View reviewed changes

e2e/profiles/aibrix/profile.go Outdated Show resolved Hide resolved

e2e/profiles/aibrix/profile.go Outdated Show resolved Hide resolved

yehudit1987 force-pushed the aibrix_profile branch 2 times, most recently from 76ddf31 to 931c983 Compare November 18, 2025 14:48

feat: Add aibrix profile for E2E testing framework

4e3f91e

Signed-off-by: Yehudit Kerido <[email protected]>

yehudit1987 force-pushed the aibrix_profile branch from 931c983 to 749431c Compare November 18, 2025 17:17

Update e2e/profiles/aibrix/profile.go

a4a5e93

Co-authored-by: Copilot <[email protected]> Signed-off-by: yehudit1987 <[email protected]>

yehudit1987 force-pushed the aibrix_profile branch from 749431c to a4a5e93 Compare November 18, 2025 18:06

Merge branch 'main' into aibrix_profile

dd72c08

Xunzhuo approved these changes Nov 19, 2025

View reviewed changes

Xunzhuo merged commit 6a4ebb9 into vllm-project:main Nov 19, 2025
19 checks passed

yehudit1987 deleted the aibrix_profile branch November 19, 2025 07:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add aibrix profile for E2E testing framework #688

feat: Add aibrix profile for E2E testing framework #688

Uh oh!

yehudit1987 commented Nov 18, 2025 •

edited

Loading

Uh oh!

netlify bot commented Nov 18, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 18, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

rootfs commented Nov 18, 2025

Uh oh!

rootfs commented Nov 18, 2025

Uh oh!

yehudit1987 commented Nov 18, 2025 •

edited

Loading

Uh oh!

rootfs commented Nov 18, 2025

Uh oh!

Xunzhuo left a comment

Uh oh!

Uh oh!

yehudit1987 commented Nov 19, 2025

Uh oh!

yehudit1987 commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: Add aibrix profile for E2E testing framework #688

feat: Add aibrix profile for E2E testing framework #688

Uh oh!

Conversation

yehudit1987 commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for vllm-semantic-router ready!

Uh oh!

github-actions bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👥 vLLM Semantic Team Notification

📁 e2e

📁 Root Directory

📁 deploy

📁 tools

🎉 Thanks for your contributions!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

rootfs commented Nov 18, 2025

Uh oh!

rootfs commented Nov 18, 2025

Uh oh!

yehudit1987 commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rootfs commented Nov 18, 2025

Uh oh!

Xunzhuo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yehudit1987 commented Nov 19, 2025

Uh oh!

yehudit1987 commented Nov 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yehudit1987 commented Nov 18, 2025 •

edited

Loading

netlify bot commented Nov 18, 2025 •

edited

Loading

github-actions bot commented Nov 18, 2025 •

edited

Loading

📁 `e2e`

📁 `Root Directory`

📁 `deploy`

📁 `tools`

yehudit1987 commented Nov 18, 2025 •

edited

Loading