[model-gateway] reducing cpu overhead in various of places

[slin1237]

Summary

This PR reduces CPU overhead in the HTTP router hot path through a series of targeted optimizations across multiple layers: routing, validation, middleware, worker management, and load balancing policies.

Changes by Layer

HTTP Router (src/routers/)

• Worker lookup caching: Reduced redundant worker_registry.get_by_url() calls from 4+ to 1 per request
• SIMD stream scanning: Replaced windows(12) byte scanning with memchr::memmem::find() for data: [DONE] detection
• Header pre-filtering: Filter headers once before worker loop instead of inside each iteration
• Conditional debug logging: Wrap expensive serde_json::to_string() in tracing::enabled!() check
• String comparison: Use eq_ignore_ascii_case() instead of to_lowercase() throughout header handling

Request Validation (src/protocols/)

• Request ID generation: Changed from chars().nth() (O(n) per char) to byte array indexing (O(1))
• Text extraction: Rewrote extract_text_for_routing() to use single buffer instead of Vec<String>.join()
• Zero-copy iterators: Added StringOrArray::iter() returning &str references instead of cloning

Middleware (src/middleware.rs)

• Latency recording: Changed from format!("{:?}", latency) to latency.as_micros() as u64
• WASM string caching: Pre-compute method/path/query strings before WASM module loop

Worker Management (src/core/)

• RuntimeType parsing: Use eq_ignore_ascii_case() instead of to_lowercase().as_str()
• worker_to_info(): Cache worker_type(), connection_mode(), url() to avoid redundant clones
• normalised_url(): Remove redundant contains("@") check before rfind('@')
• WorkerRegistry::stats(): Iterate DashMap directly instead of cloning all workers via get_all()
• Model counting: Count models by iteration instead of allocating Vec<String>

Policy Registry (src/policies/registry.rs)

• Double lock fix: Cache prefill_policy and decode_policy to avoid acquiring prefill_policy lock twice in get_all_power_of_two_policies()

Load Balancing Policies (src/policies/)

• power_of_two.rs:
  • Change info! to debug! for per-request logging
  • O(1) deterministic second random selection instead of retry loop
• cache_aware.rs: Compute min/max load in single pass without Vec allocation
• bucket.rs: Change info! to debug! for per-request status logging

Checklist

• No breaking API changes
• All optimizations are backward compatible
• Debug logging preserved for troubleshooting (just demoted from info level)
• No changes to external behavior

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.
• Work with maintainers to merge your PR. See the PR Merge Process

[gemini-code-assist]

Summary of Changes

Hello @slin1237, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly reduces CPU overhead in the HTTP router's hot path by implementing a series of targeted optimizations across various layers of the application. These changes focus on minimizing allocations, reducing redundant computations, and improving the efficiency of string operations, data structures, and logging, ultimately leading to a more performant and resource-efficient gateway.

Highlights

• HTTP Router Optimizations: Reduced redundant worker lookups, improved stream scanning with SIMD (memchr), pre-filtered headers once per request, added conditional debug logging for expensive operations, and used eq_ignore_ascii_case for header comparisons to avoid allocations.
• Request Validation Enhancements: Optimized request ID generation using byte array indexing (O(1)), rewrote text extraction to use a single buffer, and introduced zero-copy iterators for StringOrArray.
• Middleware Efficiency: Changed latency recording to use u64 microseconds directly and pre-computed WASM method/path/query strings to avoid repeated allocations.
• Worker Management Refinements: Switched RuntimeType parsing to eq_ignore_ascii_case, cached worker properties in worker_to_info, removed redundant checks in normalised_url, and optimized WorkerRegistry::stats() by iterating directly and counting models without intermediate Vec allocations.
• Policy Registry Lock Reduction: Cached prefill_policy and decode_policy to prevent double-locking in policy retrieval logic.
• Load Balancing Policy Improvements: Demoted per-request logging from info! to debug!, implemented O(1) deterministic second random selection in power_of_two.rs, and computed min/max load in a single pass in cache_aware.rs.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This is an excellent pull request that delivers a comprehensive suite of performance optimizations across the model gateway. The changes are well-targeted at reducing CPU overhead in hot paths, including routing, validation, middleware, and worker management. Key improvements like replacing expensive string operations, caching frequently accessed data, using SIMD-accelerated libraries, and improving algorithms are all implemented correctly and idiomatically. The fix for potential lock contention in the policy registry is also a valuable improvement. Overall, these changes significantly enhance the performance and efficiency of the gateway with high-quality, well-documented code.