Repository navigation
Add Support for API Key Authentication - #230
Merged
Merged
Conversation
Merged
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
Garrybest
pushed a commit
to Garrybest/sglang
that referenced
this pull request
Jan 9, 2026
* separate the kv cache from the forward batch
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
Comprehensive scenario suite to CHARACTERIZE + DOCUMENT the design-ideal system (not to cut features). Deterministic CPU harness sweeps the arm matrix (policy x pressure x hint-delay x reuse x migrate) in seconds. Contents: README.md (full characterization design + UQ1-6 + slices + metrics + arm mapping), harness/sim.py (trace-replay eviction simulator: L1 inline + L2 hint-steering-with-injectable-staleness + L3 imperative demote + lock protection; models scheduling logic+timing, not GPU kernels), harness/verify.py (6 invariants, green), harness/sweep.py (headline tables), RESULTS.md (interpreted findings + honest caveats). Tier-1 findings: * UQ1/UQ4 steering-vs-LRU value is pressure-dependent: +49% re-prefill reduction under headroom -> +3% at saturation. Quantitatively explains the live A3 observation (saturated -> ours ~= baseline, migrate ~= 6): at 99% HBM almost nothing the policy can do beats LRU. The daemon binds in under-/critically-loaded regimes. * UQ2 (headline) hint-latency budget = the predictable-reuse lead time (knee ~= phat_lead). Hints staler than one reuse-lead-time can't protect an about-to-return prefix; inline eviction races them. => inline/hint is latency-critical (fresh within a reuse lead time), consistent with the "100ms batching not unconditionally harmless" analysis. * UQ3 imperative-migrate-as-eviction ~= 0 over inline steering (matches migrate=6); caveat: harness models migrate only as demote, not the promotion/cross-tier/predictive roles where it earns value (Tier-2). * UQ5 steering shines for imminent+frequent reuse (+41%) and one-shot (+100%), least for long-tool-gap reuse. design stays aim-for-ideal; this maps where each mechanism binds. Tier-2 e2e (hint-delay knob + real-stack arms) is the next increment.
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
… + arm wiring)
Builds the real-stack arm infrastructure for the eviction/hint-latency
characterization (Tier-1 harness already landed). RUNS need GPUs; this is
the build, ready for when the do-no-harm campaign frees 4-7.
* daemon/kv_scheduler.py: AGINFER_HINT_DELAY_MS knob — defers hint
DELIVERY (computed now, delivered stale) via loop.call_later, so the
S2 latency-budget arms can sweep the freshness knee on hardware. 0 =
off (production byte-identical). Daemon inherits the env from run_k.
* baselines/sglang_adapter.py: const_v_u_score — uniform inline score
(plumbing live, signal stripped) for the S1 ablation isolating
"does V_u content matter" from "does the daemon scorer matter at all".
* scenarios/_shared/run_k.sh: respect a pre-set SGLANG_KV_POLICY_MODULE
so arms swap the inline scorer (lru/const/ours) without forking;
default unchanged when unset.
* scenarios/5_eviction_characterization/e2e/: run_sweep.sh (arm matrix:
lru / constvu / ours-fresh / ours-delay-{100,250,500,1000}ms over a
pressure regime, N cycles) + parse.py (per-arm cache-hit / rejects /
migrate / resolved table — the on-hardware hint-latency budget).
Tests: verify/t40 B4 pins the knob (defers under delay, immediate when
off); 18/18 green. kv_scheduler_value_rule 25/25, outbound_coalesce 7/7.
design-ideal unchanged; this only adds measurement knobs.
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
…/d250/d1000) as the agreed first cut
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
…gged single-flight) Live a3 cycle-2 evidence: a migrate BURST (134 in one cycle, vs cycle-1's 6) dispatched INDIVIDUALLY re-clogged the single-flight outbound channel exactly like the pre-sgl-project#228 hint flood — oldest-age climbed to ~700ms and remove_hbm_not_device_leaf rejects returned (the migrates aged in the queue). sgl-project#228 had coalesced only hints; under high migrate volume the imperative path's own traffic saturates the channel. Fix: migrates now coalesce like hints — the worker merges a wake's fresh migrate actions into ONE POST (latest decision per unit hash) after the sgl-project#227 stale-drop. A burst is one round-trip, not N; the single-flight ceiling now bounds how OFTEN a (coalesced) batch is sent, not how many actions it carries — which is what makes the burst case land. This also unifies the design (all three endpoints coalesce). The deeper limit (the single-flight ceiling itself) is the imperative path's structural bound, characterized in sgl-project#230 — coalescing does not remove it, it just stops bursts stacking N round-trips. Escalation (sgl-project#164): now counts failed POSTs per WAKE, not per enqueued batch. The sustained-unreachable fatal still fires (sustained failed dispatches + backlog age) — coalescing reduces POST count, not the signal. t164 B0/C0 prove fatal-on-sustained-failure via a 3-endpoint wake (3 failed POSTs → consec=3 → exit 1 + forensic dump). Tests (all rewritten for the coalesced contract, comprehensive): * outbound_coalesce A1/A2: migrate burst→1 POST, latest-per-hash, stale-drop-before-merge, freshness=0 keeps both. * t164 A0/A1/A2: escalation accounting tested per-dispatch via _dispatch_one (decoupled from drain); B0/C0: 3-endpoint wake → fatal; C3: slow stub proves live-peek in-flight age. * t36 A1/A2/A3/A4: content-delivered (not POST-count), cross-endpoint order, survive-errors-across-wakes. 18/18 touched suites green. DESIGN §6 updated.
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
…-1/Tier-2); t40 README documents B4 hint-delay knob + sgl-project#229 stub refresh
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
…oject#208) Real-stack do-no-harm verdict for the sgl-project#224+sgl-project#227+sgl-project#228 fix stack on TP=4 A3 (harbor/terminus-2 swebenchpro). Per-task agent_execution time (the LLM serving phase the daemon affects; docker env-setup + verifier excluded): a3 (ours) n=74 mean=734.2s median=681.2s std=296.5s a3_kvoff n=81 mean=695.2s median=628.2s std=275.6s Delta = +5.6% mean, NOT significant (std ~290s >> 39s gap) -> HOLDS No fatals/crashes, identical task outcomes (reward 0, 12 broken-env both arms). Benefit absent at saturation (expected per sgl-project#230 Tier-1). The cyc2 80-reject burst (migr=155) is the pre-migrate-coalescing single-flight wall that sgl-project#228-complete (f88c885) fixes; rejects were do-no-harm regardless.
rucnyz
added a commit
to rucnyz/sglang
that referenced
this pull request
Jun 9, 2026
… hardcoded ours_greedy_score The sgl-project#230 characterization sweep swaps SGLANG_KV_POLICY_MODULE per-arm (lru_score / const_v_u_score). run_k's startup invariant hardcoded ours_greedy_score and HALTed (exit 5) the moment a non-ours arm loaded its correct policy — every lru/constvu arm fast-failed in ~5min with idle GPUs, silently emitting rc=0 and zero results. Pin the check to the exported SGLANG_KV_POLICY_MODULE (default ours_greedy_score).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces the ability to secure the server using API key authentication, adding an extra layer of security for server access. By requiring an API key for incoming requests, we ensure that only authorized users can interact with the server's endpoints.
The
APIKeyValidatorMiddlewareclass has been introduced to perform the API key check. It intercepts incoming requests, checks for the API key in theX-API-Keyheader, and validates it against the expected key.If the API key is missing or incorrect, the middleware immediately returns a
403response, blocking access to the server's endpoints.This feature is opt-in, ensuring backward compatibility for existing deployments. Users can enable API key authentication by setting the appropriate variable.