Example file for docker compose and k8s by LucienShui · Pull Request #1006 · sgl-project/sglang

LucienShui · 2024-08-09T18:50:27Z

Motivation

More convenience to start with docker.

Modification

Add compose.yml for serving as service.
Add k8s example file.
Update README.md for corresponding changes.

merrymercy · 2024-08-10T19:01:50Z

Could you revert your changes to the entry points?

zhyncs · 2024-08-12T06:25:05Z

@LucienShui May you rebase the latest main branch? Thanks. ref #1033

zhyncs · 2024-08-12T16:37:12Z

cc @LucienShui

ref cmd

# first sync fork's main

# next
git checkout main
git pull
git checkout  chore/docker_entrypoint
git rebase main
git push -f

zhyncs · 2024-08-13T08:47:15Z

Hi @LucienShui I noticed that you updated your PR. Have the compose and k8s conf been locally verified?

LucienShui · 2024-08-13T08:59:39Z

Hi @LucienShui I noticed that you updated your PR. Have the compose and k8s conf been locally verified?

Yes, both compose and k8s have been verified locally, but I want to simplify these files as much as possible to make them more beginner-friendly so I added [WIP] in PR's title.

Feel free to merge it now but it would be better after I make more changes.

BTW, I need to work during the day so I only have time in the evening, my timezone is UTC+8.

zhyncs · 2024-08-13T09:01:55Z

Got it. Thanks!

LucienShui · 2024-08-13T16:28:29Z

This PR is ready to merge, thanks for your time and patience!

zhyncs · 2024-08-13T17:05:24Z

@@ -0,0 +1,66 @@
+apiVersion: node.k8s.io/v1


May we add health check, image pull policy and network policy?

Sorry, I've just started learning k8s for 4 days, after searching, I still have no idea about what the network policy should do here.

Health check and pull policy was added.

It's basically done, and I currently don't have the appropriate environment. It might need to wait for another reviewer to do a simple local verification. Thank you for your contribution!

cc @merrymercy @Ying1123

merrymercy · 2024-08-13T22:08:07Z

@LucienShui Thanks for the contribution!

…flash Adapt failed execution test cases

Boot 65 got further than any boot in this window -- past the ring family, into decode -- and died as "output with shape [1] doesn't match the broadcast shape [3]" out of cuda_graph_buffer_registry._foreach_copy. That sentence does not say WHICH field, and every enabled slot goes into ONE torch._foreach_copy_ call, so one bad pair fails the batch anonymously. The tier selection is exonerated on the way: the live capture list was bs=[1, 2, 4] (trimmed from the declared [1,2,4,8,12,16,24] by available memory, logged at capture), and _pad_to_bucket returns the smallest bucket >= raw_size, so raw_bs=3 pads to 4, not to 1. The [1] is a slot buffer's own width, not padded_bs. `axis == "none"` slots take the whole buffer unsliced, which is the shape a per-rank field under PP=3 would collide with. So the failure path walks the pairs it just handed to the grouped call and names every dst/src shape disagreement, with raw_bs, padded_bs and both token counts. Nothing runs on the happy path -- the walk happens only after the grouped call has already raised. This is an instrument and it is deliberate: without the name any fix here is a guess, and guessing has been the expensive move in this window. One boot turns "[1] vs [3]" into a field name. Evidence: desk, executed. py_compile; the naming exists once and appends once, and it sits inside the except branch. Belegstufe: DESK-BEWIESEN.

…n, die es nicht gibt == DER BEFUND (gemessen, nicht vermutet) == Boot_855_704bgroup2: 57 Prefill-Batches, NULL Decode-Batches, eine 6-Token-Probe -- und 42 Flips. Die Gruende standen im Log, es hatte sie nur nie jemand gezaehlt (`PHASE-POLICY arming`, scheduler.py:13627 -- die Zeile existierte bereits, ich musste kein Instrument bauen): 21x pp_to_tp: "idle Ns >= Ns, returning to the decode resting layout" 21x tp_to_pp: "pending prefill N tok > 0 (purity: prefill cannot run in tp, nothing decoding)" 21 Rundtrips auf einer LEEREN Kiste. Die zwei Regeln zertifizierten einander: diese hier flippte fuer nichts nach TP, die Prefill-Arbeit der Health-Probe konnte dort nicht laufen, die tp-waertige Regel flippte sofort zurueck. Bei gemessenen 8,07 s je Rundtrip (4,016 + 4,050 s, RECONCILED `flips`-Tabelle) sind das ~169 s Naht fuer nichts. == DIE WURZEL == `idle` ist an dieser Stelle der STARKE Leere-Term (`decode_work_bs() == 0 AND not work_exists()`) -- es ist also BEWIESEN, dass weder ein residenter Decode-Bundle noch ein Prefill irgendwo geschuldet ist. Die Regel flippte trotzdem, um sich in der Ruhe-Schicht zu POSITIONIEREN. Diese Praemisse ist fuer eine KALTE Ankunft verkehrt herum, und eine andere bekommt eine leere Kiste nicht: ein neuer Request braucht ZUERST einen Prefill-Pass, und Prefill laeuft unter strict purity nicht in TP. In TP zu ruhen heisst, die naechste Ankunft zahlt tp_to_pp zum Prefillen UND pp_to_tp zum Dekodieren. In PP zu ruhen ist fuer genau diese Ankunft strikt besser. KEINE Dwell-Erhoehung: ein laengerer Dwell macht die Schleife langsamer, nicht abwesend, und bei NULL Arbeit ist kein Timer lang genug, um den Handel positiv zu machen (sgl-project#819-Preisfrage, die ein Timer per Konstruktion nicht beantwortet). == FEHLERKLASSEN-PASSENDER CHECK (Direktaufruf von decide(), 3 Faelle) == Klasse a) Format-Spec auf einem Conditional im f-String: beide Zweige ausgefuehrt -> "idle 0.0s" / "idle 11.5s". AST + Import gruen. Klasse b) falscher Zweig / Verhungern: CASE1 idle+leer in PP -> direction=None, Reason traegt sgl-project#1011 PASS CASE2 Prefill geschuldet -> "prefilling in pp (50000 tok pending)" PASS (nimmt den idle-Zweig NICHT) CASE3 Decode-Arbeit da -> direction=pp_to_tp via DRAINED PASS <- das ist der sgl-project#858/sgl-project#1006-Verhungerungsbeweis: mit echter Arbeit feuert der Flip weiterhin. Belegt, nicht argumentiert. == ZWEISEITIGE ABNAHME AUF METALL (boot_855_1011idle) == (i) LEERLAUF, 11 min 39 s, Deadman aktiv, 7 health_generate-Proben bedient: Flips 0 | PHASE-POLICY armings 0 | sgl-project#1011-Refusals 77 | health 200 Vorher auf vergleichbarem Leerlauf: 42 Flips. JETZT NULL. (ii) LAST, conc=4 x 200 tok: 4/4 fertig, 0 Fehler, TTFT min 0,30 / med 0,31 / MAX 0,31 s -- keine Verhungerung Completion 11,38-11,85 s Flips: GENAU 1, armiert via DRAINED "N req decoding" -- der Flip feuert, wenn echte Arbeit wartet. == WAS DIESER SCHNITT NICHT TUT -- und meine eigene frueher zu grosse Behauptung == Ich hatte A als groessten Hebel gegen die 68-%-Flip-Steuer benannt. GEMESSEN FALSCH, und die Korrektur gehoert hierher: conc=4, 600 tok: 88,64 tok/s (vorher 90,47) -- unveraendert in der Streuung bs1, 600 tok: 25,09 tok/s (vorher 25,37) -- unveraendert Unter DAUERLAST sind die Flips nachfrage-getrieben (bs1 zahlt weiter ~2 Flips je Request: 12 Flips auf 6 Requests), und genau das ist die 68-%-Steuer. Der Leerlauf-Befund bleibt gueltig und der Schnitt beseitigt reine Verschwendung ohne Preis -- aber er kauft die Last-Steuer NICHT zurueck. BELEG-STUFE: BOOT-BEWIESEN fuer beide Abnahmehaelften (Zahlen und n oben). Der Drain-and-Flip-Kontrakt (sgl-project#925) ist unberuehrt; nur der Leerlauf-Zweig mit `rest_phase == PHASE_TP` aendert sein Verdikt.

merrymercy reviewed Aug 9, 2024

View reviewed changes

Comment thread docker/Dockerfile Outdated

zhyncs suggested changes Aug 10, 2024

View reviewed changes

Comment thread compose.yml Outdated

Comment thread README.md Outdated

LucienShui force-pushed the chore/docker_entrypoint branch from f7ec3d1 to bd4b7ec Compare August 12, 2024 16:42

LucienShui changed the title ~~Add entrypoint to Dockerfile, add compose.yml~~ [WIP] Example file for docker compose and k8s Aug 12, 2024

LucienShui changed the title ~~[WIP] Example file for docker compose and k8s~~ Example file for docker compose and k8s Aug 13, 2024

LucienShui added 3 commits August 14, 2024 00:25

Add ENTRYPOINT in Dockerfile

b596d00

Update README.md, add compose.yml

211cf7f

Revert Dockerfile, add k8s service and compose file

652f2a0

LucienShui force-pushed the chore/docker_entrypoint branch from 8c0e903 to 652f2a0 Compare August 13, 2024 16:28

zhyncs reviewed Aug 13, 2024

View reviewed changes

Add health check, and env variable in compose and k8s configuration

0b082f5

LucienShui force-pushed the chore/docker_entrypoint branch from b71fbdf to 0b082f5 Compare August 13, 2024 18:09

zhyncs reviewed Aug 13, 2024

View reviewed changes

Comment thread README.md Outdated

zhyncs requested review from Ying1123, hnyls2002 and merrymercy August 13, 2024 18:15

README.md polished by ChatGPT

1ca561d

zhyncs approved these changes Aug 13, 2024

View reviewed changes

merrymercy enabled auto-merge (squash) August 13, 2024 22:07

merrymercy disabled auto-merge August 13, 2024 22:07

merrymercy merged commit 312e849 into sgl-project:main Aug 13, 2024

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Example file for docker compose and k8s (sgl-project#1006)

57492db

chx96642264 pushed a commit to chx96642264/sglang that referenced this pull request Jul 15, 2026

Merge pull request sgl-project#1006 from pllimax/pr_branch_plli_dsv4_…

d66b43b

…flash Adapt failed execution test cases

Uh oh!

Conversation

LucienShui commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modification

Uh oh!

Uh oh!

Uh oh!

Uh oh!

merrymercy commented Aug 10, 2024

Uh oh!

zhyncs commented Aug 12, 2024

Uh oh!

zhyncs commented Aug 12, 2024

Uh oh!

zhyncs commented Aug 13, 2024

Uh oh!

LucienShui commented Aug 13, 2024

Uh oh!

zhyncs commented Aug 13, 2024

Uh oh!

LucienShui commented Aug 13, 2024

Uh oh!

zhyncs Aug 13, 2024

Choose a reason for hiding this comment

Uh oh!

LucienShui Aug 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhyncs Aug 13, 2024

Choose a reason for hiding this comment

Uh oh!

zhyncs Aug 13, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

merrymercy commented Aug 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LucienShui commented Aug 9, 2024 •

edited

Loading

LucienShui Aug 13, 2024 •

edited

Loading