Update README.md by Ying1123 · Pull Request #927 · sgl-project/sglang

Ying1123 · 2024-08-05T06:00:32Z

No description provided.

…t when Full triggers it Boot 2f died 40 s in, 14 s after the first real cache hit. Not a wedge -- the scheduler RAISED at on_idle: pool memory leak detected! [full] total=465669, available=132894, evictable=346, protected=8192, session_held=0, uncached=0, withheld=324357, double_owned=8129, double_owned_src=live THE DEFECT IS EXCLUSIVITY, NOT ACCOUNTING, and the ledger is only what noticed. `double_owned_src=live` means the number came from `_live_double_claimed_rows`, which is `len(free_rows & cached_rows)` over the allocator's real free set and `all_values_flatten()` -- "the SIZE OF AN ENUMERATED INTERSECTION, the ids are known, not estimated". So 8129 rows were genuinely on the free list AND named by live tree nodes. Those ids stay matchable, which is the sgl-project#767 direction: a later prefix hit can serve KV out of rows that have already been reissued. ROOT, one line. `full_component.evict_component` (:115-118) frees the device rows on ANY cascade that reaches it and deliberately leaves `cd.value` set -- correctly, because `free_swa` has to read it first, so the clear is deferred to `_cascade_evict`. That deferred tombstone then asked trigger.component_type == BASE_COMPONENT_TYPE which is a DIFFERENT QUESTION from "were Full's rows freed". A cascade triggered by MAMBA or SWA (`mamba_component.py:529`, `swa_component.py:441`) that reaches Full freed its rows and never tombstoned them, so the node went on naming ids the allocator had handed back -- permanently, not for a window. The fix asks the question the deferral actually owes: `base_rows_freed`, set when Full is the trigger OR when the cascade loop evicts Full, and the tombstone fires on that. The opposite direction is preserved and pinned: a cascade that never reaches Full must NOT clear `value`, because stranding live KV is a deficit -- the worse defect in the other sign. SIBLING SWEEP, and the class is a single instance: `mamba_component.py:484` and `swa_component.py:393` both clear `cd.value = None` INLINE, immediately after freeing. Only Full defers, only Full needed a completion condition, and only Full got it wrong. Nothing else to fix. WHY NOW, AND WHY IT IS OLD (#wurzel-vor-wirkung): the intersection can only be as large as the tree. Until the mamba checkpoint grid stopped vetoing every anchor, no prefix match ever succeeded -- the tree stayed at `evictable=1` and this term measured about ONE row, which is exactly the 2c precedent that motivated sgl-project#912's live reading. The first real 8538-row cached prefix turned it into 8129. The defect is not in a516b37's diff; a516b37 only made hits reachable. THE ARITHMETIC, since it names the sign: PP0's raw ledger is 132894+346+8192+324357 = 465789, a 120-row SURPLUS over 465669. Subtracting double_owned=8129 makes it an 8009-row DEFICIT, and `_check_pool_invariant` raises on `!=`, so either sign is fatal. The two peer ranks sit at a 64-row surplus with double_owned=0; the 56-row difference is exactly PP0's extra free rows. All three trees hold the same 8538 rows (8192+346 on PP0) -- only the lock and the free-list overlap differ, which is why the rank that matched is the rank that died. RELATIONSHIP TO THE NEIGHBOURS, checked rather than assumed: sgl-project#916 (lawful tree/request share) and sgl-project#922 (stale _resident_rows) are different populations and are untouched. sgl-project#912 is the DETECTOR, and it worked exactly as designed -- it was built to catch a row claimed by two owners and it caught one. This is not an indicator defect; the indicator was right. Tests, hermetic, CUDA_VISIBLE_DEVICES="": test_cascade_tombstone_927.py, 5 passed. Red-first proven: restoring the old trigger-gated condition turns `test_the_tombstone_is_not_gated_on_who_triggered` RED, then reverted. SPECIMEN 2 ARRIVED MID-BUILD AND CHANGED THE VERDICT. The targeted repro (boot_2f_a516b3750b_0827_0649.log) reproduced it with double_owned=8192 -- the WHOLE hit prefix, so the first specimen's 63-row gap was incidental, not a discriminator. And its ledger BALANCES EXACTLY: available 125865 + protected 8192 + evictable 1111 = 135168 = the post-cutover free pool ("POOL CENSUS post-cutover ... free=135168") 135168 + withheld 305265 = 440433 = total -> surplus ZERO So on that run there is no leak and nothing to correct: the raise is caused ENTIRELY by subtracting 8192 from an exact ledger. That means the two free readings disagree by exactly the protected count -- `double_owned` is derived from `free_reading.rows` (an ENUMERATION) and subtracted from a ledger built on `available_size()` (a COUNT), and `FreeRowReading`'s own docstring warns that "HOW MANY" and "WHICH ONES" are different questions on this allocator family. Nothing compared them. SO THIS COMMIT DOES NOT CLAIM TO CLOSE 2f, and says so rather than letting the tombstone fix take credit for a crash it may not cause. The second change is a DIAGNOSTIC, not a gate: when the live reading fires, it now logs the enumeration size against the count and the delta. Whether the enumeration over-reports or `available_size()` under-reports decides which side is the defect, and the fix differs on each side -- so it names the disagreement and leaves the verdict to the reader instead of guessing. The tombstone fix stands on its own merits, independent of 2f: a live node naming freed rows is wrong whatever the ledger says about it, and the sibling parity argument (both peers clear inline) is what proves it rather than the specimen. OBSERVABLES for the next boot: the new "sgl-project#927 FREE READING DISAGREES WITH ITSELF" line and its delta -- if it prints delta == protected, the enumeration is the defect and the correction term must be built from the count; if it does not print at all while double_owned is still large, the rows really are doubly claimed and the tombstone path is the one to keep pulling.

…ker per abort The abort path is the leading candidate for the rank-local slot divergence behind the PP output-ring wedges, and the code says why itself. `abort_request` carries sgl-project#631 pin 4 verbatim: "an abort applied on one rank before its peers DIVERGES THE REPLICATED LIVE SET mid-flip" -- and the guard that prevents it is conditional, `if window is not None and window.active`. The `phase_flip_abort_window` is activated only on arming and drained after the cutover, so OUTSIDE an armed flip `_abort_request_now` runs directly, on whichever rank processes it, in whichever pass it lands. The hazard is recognised, named, and closed only inside the flip window. THE CHECK COULD NOT BE MADE, WHICH IS THE POINT OF THIS COMMIT. Grepping the three wedge logs for an abort near the onset returns nothing -- and that nothing is worthless. Every marker inside `_abort_request_now` is `logger.debug` and the boots run `log_level='info'`: measured, ZERO DEBUG lines in boot_accept2e0827_0827_0454, boot_accept2e0827b6_0827_0613 and boot_802f_staged1_0822_1716. So "no abort before the wedge" is a LOGGING GAP, not a finding. This is the marker-absence trap sgl-project#843 hit on this same path ("the only refusal marker was a logger.debug on a boot running log_level='info', so it could never appear"), and reporting the null as evidence would have been the second instance. ONE SUMMARY LINE, not the per-request ones. `abort_all` can name the whole live set, so promoting those would trade a blind spot for a flood -- the sgl-project#801 void-streak shape in reverse. This says an abort happened, when, whether the flip window deferred it, and what it targeted, which is everything the discriminator needs to place it against a wedge onset. `deferred` is now read once into a local and used for both the log and the branch, so the line cannot disagree with the path taken. WHAT THE SAME SWEEP DID FIND, recorded because it is real and does NOT support the abort hypothesis: * Boot 1's wedge onset (last progress 05:08:59) is preceded by EIGHT SECONDS by `sgl-project#905 HOST-POOL DOUBLE-FREE about to raise ... 8192 of 8192 index(es) are in range but not allocated, span [3530, 14937]` on all three ranks, and the `[sgl-project#703 flip-writeback]` drain fails with it. Boot-1-specific: zero such lines in boot 6 and zero in 1712. The magnitude is the same 8192 the sgl-project#927 hit prefix carries, on the HOST pool rather than the device one, which is worth linking rather than filing twice. * Specimen 1712 shows a rank-local admission-congruence retraction (`sgl-project#797 PP-ADMISSION pass voided on rank 1 ... told=147456 local=139264`) -- that is the in-loop site, already covered by the void relay whose default sgl-project#801 widened. * So the three wedges do NOT share one proximate cause in these logs, and the single-class assumption should not be carried further without evidence. NO BEHAVIOUR CHANGE. The branch is byte-identical; only the marker is new. Import smoke + wiring check green (the marker precedes both branches, and the branch reads the same `deferred` the line printed).

…g, and say so 698cd39 claimed to fix a real ownership defect. IT WAS A NO-OP, the boot agent's third crash on that SHA is consistent with that, and this reverts it rather than leaving a confident commit message standing in front of an open defect. THE CLAIM WAS: `_cascade_evict`'s deferred tombstone asked whether Full was the TRIGGER, while `full_component.evict_component` frees Full's rows on ANY cascade that reaches it -- so a MAMBA- or SWA-triggered cascade would free those rows and never tombstone them, leaving a live node naming freed ids. IT CANNOT HAPPEN, and the priority lattice is the proof. On the DEVICE target this function is reached with a non-BASE trigger from exactly two places, both on INTERNAL nodes: `mamba_component.py:529` and `swa_component.py:441`. Internal priorities are "full=2 > swa=1 > mamba=0" (`tree_component.py:292`), and the cascade loop admits a component only when `eviction_priority(is_leaf) <= trigger_priority` -- so Full at 2 is unreachable from a trigger at 0 or 1. The two remaining call sites pass `target=EvictLayer.HOST` and the tombstone block is gated on DEVICE. The leaf path does not use this function at all (`_evict_device_leaf` loops the components directly, `:2132`), and `_evict_to_host` -- the ONLY path that leaves an evicted node in the tree -- passes the BASE component as the trigger explicitly (`:2065-2071`). "Full was the trigger" and "Full's rows were freed in this cascade" therefore name the same set of cascades. `base_rows_freed` was a rename. The finding is recorded at the site so the next reader does not re-derive it. THE RED-FIRST PROOF WAS AN ARTEFACT, and that is the more useful half. The original suite asserted on `inspect.getsource` strings; the mutant I ran to "prove" it red had restored the original's exact multi-line formatting, which is what `assertNotIn` matched. A one-line mutant leaves all five green -- as the boot agent measured. A source-string test observes the source, not the system, and structurally cannot go red for a behavioural change. It should not have been checked in as a red-first proof and the rule that forbids it is my own. The suite is replaced by behavioural tests that build a real cache on CPU and assert on tree state -- and they are labelled as an INVARIANT PIN, not as a red-first proof, because a one-line mutant leaves them green too and now the reason is understood: there is nothing there to catch. What they pin is worth keeping on its own account and is exactly what `double_owned` measures: after an eviction, the intersection of the allocator's free list with the rows the tree still names must be EMPTY. THE sgl-project#927 DIAGNOSTIC WAS ALSO VACUOUS and is corrected in the same commit. It compared `len(free_reading.rows)` against `free_reading.count` -- the same number by construction (`read_free_rows` builds the enumerated reading as `rows=rows, count=len(rows)`, kv_row_ownership.py:1007-1009). Worse, the ledger's `available` IS that same reading (`full_available_size = free_reading.count if free_reading.is_enumerable`), so the "enumeration over-reports" branch was unreachable from the start. THE LINE PRINTING ZERO TIMES THEREFORE PROVED NOTHING, and a verdict was being built on that silence. It now compares the enumeration against the allocator's own `available_size()` (`ps.full_available_size`), which is an independent source, and states the alternative explicitly when they agree. The conclusion that verdict reached still holds, but it is derivable without the instrument: `available` counts the enumerated free set while `protected`/`evictable` count the tree, so a non-zero intersection means those rows are counted twice in the raw sum. Specimen 2's raw sum equalled `total` exactly, so the double-count is masked by an equal number of rows with NO owner. The rows are genuinely doubly claimed AND there is a real 8192-row hole; the exact balance was two errors cancelling. WHERE sgl-project#927 ACTUALLY IS, now that eviction is excluded: something frees the matched prefix's rows while the tree still holds them. The trigger is known ("cached prefix + fresh final chunk in the SAME request"), and the candidate that fits it is the insert-time duplicate free, `_insert_helper` `dup_start = max(0, params.prev_prefix_len - total_prefix_length); free(value_slice[dup_start:consumed_from])` -- because on a HIT `req.prefix_indices` ARE the tree's row ids, not copies, so `prev_prefix_len` (= `req.cache_protected_len`) is the only thing standing between that free and the tree's own rows. Not yet proven, and NOT fixed here; recorded as the next place to look rather than the next thing to guess. Tests: test_cascade_tombstone_927.py 3 passed (invariant pin, CPU, hermetic).

…h setters WHAT THE FIELD MEANS, stated once because two sites were deriving it and only one of them was right: `cache_protected_len` is HOW MANY LEADING ROWS OF THIS REQUEST'S KV THE TREE OWNS. It is not a length of anything the request owns. Two consumers depend on exactly that reading -- `_insert_helper`'s duplicate free (`dup_start = max(0, prev_prefix_len - total_prefix_length)`) and sgl-project#824's `retention_shrinks_protected` -- and both are unsafe if it under-reports. THE HAZARD, REPRODUCED BEHAVIOURALLY rather than argued. On a prefix HIT `req.prefix_indices` ARE the tree's row ids; the request reuses them, it does not copy them. So in the prefix region `_insert_helper`'s `value_slice` holds the TREE's ids, and `prev_prefix_len` is the only thing standing between them and `token_to_kv_pool_allocator.free`. With `prev_prefix_len=0` and a full prefix hit, every row of the prefix ends up in the free list AND in the tree at once -- counted, ids compared, not read off the source. That set is precisely the `double_owned` population (`free_rows & cached_rows`) the on-idle ledger reports as `src=live`. THE ROOT: the two setters guessed differently, because neither was told. `MatchResult.cache_protected_len` defaults to None and `UnifiedRadixCache` never populates it, so on this rig the field's value depended on which site touched the request last: * `Req.init_next_round_input` (schedule_batch.py:1351-1354) -- has an `else` and falls back to `len(self.prefix_indices)`. CORRECT. * `match_prefix_for_req` (schedule_policy.py:148-149) -- had NO else, so the branch never fired and the field kept its previous value, which is 0 for a fresh Req (schedule_batch.py:1677). Meanwhile the same function assigns `req.prefix_indices = match_result.device_indices` UNCONDITIONALLY. A request could therefore carry the tree's rows while claiming none of them were tree-owned. * `UnifiedRadixCache.cache_unfinished_req` (:1261) -- `len(new_indices)`. CORRECT. The sibling is given the same fallback here, so the two cannot diverge again. Both callers pass `include_req=True` over the WAITING queue, so the value can only be (re)derived for requests that are not yet in flight -- it cannot unprotect anything mid-prefill. AND THE VALUE THE LIVE LOG SHOWS IS RIGHT, WHICH MATTERS MORE THAN THE FIX. 2g-1 reads `cache_protected_len 8192` with a mamba `tracked position 4096`, and sgl-project#824 declines the anchor. That decline is CORRECT and the 8192 is not a symptom: * with `--chunked-prefill-size 4096` and a ~9447-token prompt, `cache_unfinished_req` publishes the protected length at each chunk boundary -- 4096, then 8192, then 9447. At the chunk-2 boundary the tree genuinely owns 8192 leading rows, so 8192 IS the true value at that instant. * `tracked position 4096` is a DIFFERENT AXIS: it is mamba's `cache_len` after the ReplaySSM `write_pos` subtraction, i.e. the last FLUSH boundary of the recurrent state. The mamba state lags the KV by a chunk. * so sgl-project#824 is refusing to file a state captured at 4096 under a key of 8192, which is the sgl-project#767 pairing direction exactly. Refusing is right. THE WARNING THAT FOLLOWS, for whoever owns the anker/decline chain: do NOT "fix" the decline by lowering `cache_protected_len` to meet the tracked position. That would re-open the duplicate-free hazard above AND pair a recurrent state with a depth it was not captured at -- both directions of the same corruption at once. The number is right; the lag is the defect. Tests, hermetic, CUDA_VISIBLE_DEVICES="": test_insert_dup_free_927.py, 3 passed. Combined mem_cache+managers lane: 17 failed / 7473 passed, and all 17 are NAME-IDENTICAL at f1a3391 (arena_high_water_631 x7, restore_never_rebuild_677 x4, phase_flip_rotation_wiring_809 x4, acceptance_emitters_758 RefillTiming x2) -- zero new failures. Genuinely red-first this time, and checked with the mutant shape that defeated the last suite: reverting the `else` turns `test_match_prefix_for_req_states_the_protected_len` RED while the other two stay green, then restored. The hazard case is a CHARACTERISATION (it asserts the tree's rows ARE freed at prev_prefix_len=0) rather than a red-first pin, and is labelled as such. THE RE-ADMIT PATH, TRACED, because the observed crashes all run through it (PP prefill -> retract at the cutover -> re-admission in TP as a full prefix hit, `ADMIT prefix_lens=9447 phase=tp #cached-token: 9447`): * the hazard condition IS created there. `Req.reset_for_retract` (schedule_batch.py:1611) sets `prefix_indices = empty`, `last_node = None` and `cache_protected_len = 0`, and the request is requeued at the front of the waiting queue. A request that then takes a FULL prefix hit is exactly `prev_prefix_len=0` + full hit -- this file's characterisation case. * but it is CLOSED again before the insert. `get_new_batch_prefill` calls `req.init_next_round_input(self.tree_cache)` (scheduler.py:8723) on every admitted request, and that is the sibling that HAS the `len(prefix_indices)` fallback. So the value reaching `_insert_helper` on the observed path was already correct, and the crash is NOT this hazard firing. * SO sgl-project#927 IS NOT CLOSED BY THIS COMMIT. What this closes is the window where `match_prefix_for_req` is the last setter -- real, but not the observed instance. Said plainly so the ticket is not marked done on it. ONE ADJACENT GAP FOUND WHILE TRACING, recorded rather than fixed blind: under `pp_size > 1`, `scheduler.py:8749-8773` truncates `req.prefix_indices` to the PP-agreed `told` (sgl-project#791 admission uniformity) and does NOT update `req.cache_protected_len` with it -- zero mentions of the field in that block. After `init_next_round_input` set them equal, the truncation leaves `cache_protected_len > len(prefix_indices)`. That direction is SAFE for the duplicate free (a larger `dup_start` frees less), which is why it has not shown up as a double-claim; it is the direction that feeds `assert req.cache_protected_len <= len(new_indices) + page_size - 1` (unified_radix_cache.py:1231). Not touched here because the safe direction does not warrant a blind edit on the admission path, and because it wants its own red-first. NOT CLAIMED: that this closes the 2f/2g crash. The live value is already correct via the sibling that had the fallback, so this closes a WINDOW -- the path where `match_prefix_for_req` is the last setter -- not necessarily the observed instance. What it does settle is the reachability question and the meaning of the field, and it removes the disagreement so the next reader is not choosing between two answers.

@staticmethod

…he allocator per access sgl-project#927 IS A CHECKER-SIDE DEFECT, AND THE on_idle RAISE WAS A FALSE POSITIVE. Nothing in the KV pool is repaired here, because the evidence now says nothing in it was broken: the guard read the wrong object. THE MECHANISM, and it is constructed rather than inferred. `SchedulerInvariantChecker` stored `token_to_kv_pool_allocator` as a dataclass field taken once at construction (`scheduler.init_invariant_checker`). The phase flip REBINDS that allocator -- `hicache_phase_binding._stamp` sets `token_to_kv_pool_allocator = incoming.allocator`, and `phase_pools_for` takes that object from the incoming phase's OWN worker stack (`:341-343`), so it is a DIFFERENT OBJECT per phase. But `readers_of` names exactly three readers -- scheduler, tree_cache, cache_controller -- and the checker is not one of them. Its own docstring states the consequence in advance: "a reader this function forgets is a reader the rebind silently leaves behind." Nothing else refreshed it either; `vram_dial._refresh_capacity_snapshots` touches only `max_total_num_tokens`. So after the first cutover, `cache_controller.load` allocated load-back rows from the INCOMING allocator (it IS rebound) while the ledger read the BOOT one. Both address the same id space, so those rows read as FREE to the checker while the tree legitimately named them, and `_live_double_claimed_rows` reported the overlap as `double_owned src=live` -- in the magnitude of the loaded-back prefix, on the rank that matched, at the instant `load_back` filled the nodes' `value`. Every measured property of the crash, including the timing that refuted four earlier candidates. WHY THE TIMING FITS EXACTLY, which is what makes this the answer rather than another candidate: the tree was refilled long before the hit, by `_insert_helper_host`, which creates nodes carrying ONLY `host_value` (`:1788`) -- `value` stays None, and `all_values_flatten` reads `value`. So the nodes were invisible to the ledger from the cutover until `load_back` populated them. That is why `double_owned` read 0 on every census and then jumped at the first hit. THE FIX IS PER-ACCESS RESOLUTION, AND A `readers_of` ENTRY WOULD NOT HAVE WORKED AT ALL. That was the smaller-looking option and it is unavailable: all three affected components are `@dataclass(kw_only=True, slots=True, frozen=True)`, and `_stamp` moves a reader by `setattr`. On a frozen instance that raises, `rebind` catches it mid-set and escalates to `RebindIncoherent` ("One reader failed mid-way: the set is now split") -- so adding them to the dict would have converted a silent stale read into a hard flip failure. Per access is the only shape that works here, and it is also the one the next component to hold a binding cannot be forgotten out of. `_allocator()` derives the allocator when it reads, from a callable the scheduler supplies -- the idiom this class already uses for `get_last_batch` / `get_running_batch`. Absent getter falls back to the field, so constructions outside the phase-flip boot are byte-identical. SIBLING SWEEP, and the class had THREE members, not one. The class is "construction-time reference to a rebindable object, on a component outside `readers_of`": * `SchedulerInvariantChecker` (invariant_checker.py:92) -- FIXED * `SchedulerPoolStatsObserver` (pool_stats_observer.py:144) -- FIXED; this one feeds the SAME ledger (`session_held`, the `available` fallback), so it was reading the stale pool alongside the checker. * `SchedulerDPAttnAdapter` (dp_attn.py:270) -- FIXED All three now resolve per access and are wired from their construction sites. BOOT PREDICTION, stated so it can be falsified: the three on_idle crashes should DISAPPEAR with this commit, with nothing in the pool having been repaired. If they do not, this is not the producer and the reframing above is wrong. MANDATORY RE-READ BEFORE ANYONE BUILDS ON THE OLD NUMBERS. If the ledger has been auditing the boot phase's allocator since the first cutover, then every post-flip reading it produced is suspect -- sgl-project#913's standing "live rows already unmapped" lines on every boot, and the sgl-project#912 `withheld`/`available` readings, included. They must be re-read on the next boot AFTER this fix before any of them is treated as a measurement. This warning belongs in both task contexts. ONE REGRESSION OF MY OWN, CAUGHT BY THE GATE AND FIXED IN THE DOUBLE. The first full run came back 20 failed against 17 known pre-existing, and all three new ones were `test_kv_page_invariants.py` with `AttributeError: '_FakeChecker' object has no attribute '_allocator'`. That double binds `_check_kv_page_invariants` off the real class but carried only the FIELDS, so once the production method resolved through `self._allocator()` the double stopped modelling production -- the drift class this tree has been bitten by before ("the suite's own double had the attribute and not the method, exactly backwards from production"). Fixed in the DOUBLE, not in production: it now binds `_allocator` off the real class alongside the method under test. Swept the siblings -- every other test touching these three classes uses `_check_pool_invariant`, a @staticmethod with explicit args, so no other double carries the same risk. Tests, hermetic, CUDA_VISIBLE_DEVICES="": test_checker_reads_bound_pool_927.py 4 passed; the three sgl-project#927 files together 10 passed; test_kv_page_invariants.py 5 passed. Combined mem_cache+managers gate: 17 failed / 7477 passed, and the 17 are NAME-IDENTICAL to f1a3391's -- zero new failures. ruff: 105 errors at base and 105 now across the four touched files, none added. Genuinely red-first -- mutating `_allocator` back to the construction reference turns `test_the_checker_resolves_the_live_allocator` RED while the rest stay green, then restored. The characterisation test reproduces the crash as arithmetic: rows allocated from the INCOMING allocator and held by the tree read as N doubly-claimed against the BOOT allocator and 0 against the bound one. OWNERSHIP BOUNDARY held: the mamba twin of the same host re-population (`mamba_exist=True` out of `_insert_helper_host`) is the sgl-project#928 agent's, shared and not touched here.

Update README.md

d8a5a23

Ying1123 merged commit 399cad9 into main Aug 5, 2024

Ying1123 deleted the Ying1123-patch-1 branch August 5, 2024 06:01

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025

Update README.md (sgl-project#927)

2ec09d5

Navigation Menu

Uh oh!

Update README.md - #927

Update README.md#927
Ying1123 merged 1 commit into
mainfrom
Ying1123-patch-1

Ying1123 commented Aug 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Ying1123 commented Aug 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant