Add CoT feature for LLava in SGLang with demo rather than multi-turn talk by fisher75 · Pull Request #520 · sgl-project/sglang

fisher75 · 2024-06-10T08:32:56Z

I add a CoT(Chain of Thought) feature into SGLang with the use and inspiration of s+=.
In this case, the information of the picture and the dialog of the questions and answers can be used for the latest question.
The result demonstrated a clear CoT.

Refined and added demo
#365

questions_cot.jsonl:
{"image": "demo_cot.jpg", "text1": "Identify potential reasons why a city with yellow taxis would be experiencing a scenario where a person is ironing clothes on top of a cab. Provide a detailed chain of thought to explain each potential reason.", "text2": "Based on the identified reasons, analyze how this situation could impact the traffic flow and city dynamics. Provide a detailed chain of thought for each identified impact.", "category": "cot", "question_id": 0}

answers_cot.jsonl:
{"question_id": 0, "prompt1": "Identify potential reasons why a city with yellow taxis would be experiencing a scenario where a person is ironing clothes on top of a cab. Provide a detailed chain of thought to explain each potential reason.", "text1": "There could be several reasons for a person ironing clothes on top of a yellow taxi in a city with yellow taxis. Here are some potential reasons and a detailed chain of thought for each:\n\n1. Time constraints: The person might be in a hurry and needs to get their clothes ironed quickly. They might have chosen the taxi as a convenient and fast means of transportation to reach their destination.\n\nChain of thought: The person might have a tight schedule, and they need to get their clothes ironed before attending an important event or meeting. The taxi provides a quick and efficient way to reach their destination, allowing them to complete the ironing task while on the move.\n\n2. Limited access to ironing facilities: The person might not have access to a proper ironing facility, such as a laundromat or a dry cleaner. In this case, they might have chosen the taxi as a temporary solution to get their clothes ironed.\n\nChain of thought: The person might not have a nearby laundromat or dry cleaner, or they might not have the time or resources to visit one. The taxi provides a convenient and flexible option to get their clothes ironed while on the go.\n\n3. Unusual circumstances: The person might be experiencing an unusual or unexpected situation, such as a power outage or a broken iron, which has led them to iron their clothes on top of the taxi.\n\nChain of thought: The person might have encountered an issue with their iron or power supply, and they have chosen the taxi as a temporary solution to get their clothes ironed. The taxi provides a convenient means of transportation to reach their destination, where they can continue with their ironing task.\n\n4. Cultural or personal preference: The person might have a cultural or personal preference for ironing clothes on the go, and they might find it more enjoyable or relaxing to do so.\n\nChain of thought: The person might have a unique approach to ironing clothes, and they might find it more enjoyable or relaxing to do so while on the move. The taxi provides a comfortable and flexible space for them to iron their clothes.\n\nIn conclusion, the person ironing clothes on top of a yellow taxi in a city with yellow taxis could be doing so due to time constraints, limited access to ironing facilities, unusual circumstances, or personal preferences. Each potential reason has a chain of thought that explains the situation and provides context for the person's actions.", "prompt2": "Based on the identified reasons, analyze how this situation could impact the traffic flow and city dynamics. Provide a detailed chain of thought for each identified impact.", "text2": "The situation of a person ironing clothes on top of a yellow taxi in a city with yellow taxis could have several impacts on traffic flow and city dynamics. Here are some potential impacts and a detailed chain of thought for each:\n\n1. Traffic congestion: The person ironing clothes on top of the taxi might cause traffic congestion, as other vehicles on the road might need to slow down or change lanes to avoid the unusual sight.\n\nChain of thought: The presence of a person ironing clothes on top of a taxi could distract other drivers, leading to slower traffic flow and potential traffic jams. Drivers might need to slow down or change lanes to avoid the unusual sight, causing delays and congestion.\n\n2. Safety concerns: The person ironing clothes on top of the taxi might pose safety concerns for both the person and other road users. For example, the person might lose balance and fall off the taxi, causing injury or accidents.\n\nChain of thought: The person ironing clothes on top of the taxi might not be wearing appropriate safety gear, such as a seatbelt or helmet, increasing the risk of injury in case of an accident. Additionally, the person's actions could distract other drivers, leading to accidents or near misses.\n\n3. Impact on taxi services: The person ironing clothes on top of the taxi might cause delays or disruptions to the taxi service, as the driver might need to stop frequently to allow the person to iron their clothes.\n\nChain of thought: The person ironing clothes on top of the taxi might cause delays or disruptions to the taxi service, as the driver might need to stop frequently to allow the person to iron their clothes. This could lead to longer wait times for other passengers and potential loss of revenue for the taxi service.\n\n4. Impact on city infrastructure: The person ironing clothes on top of the taxi might cause damage to the taxi or other vehicles on the road, leading to costly repairs or replacements.\n\nChain of thought: The person ironing clothes on top of the taxi might cause damage to the taxi or other vehicles on the road, leading to costly repairs or replacements. This could strain the city's infrastructure and resources, as well as increase insurance costs for taxi drivers.\n\nIn conclusion, the situation of a person ironing clothes on top of a yellow taxi in a city with yellow taxis could have several impacts on traffic flow and city dynamics. These impacts include traffic congestion, safety concerns, disruptions to taxi services, and potential damage to city infrastructure. Each potential impact has a chain of thought that explains the situation and provides context for the person's actions.", "model_id": "/home/users/ntu/chih0001/scratch/model/llava-v1.5-7b", "answer_id": 0, "metadata": {}}

merrymercy · 2024-06-21T03:32:23Z

+    """
+    图像问答函数
+
+    Args:
+        s: sglang对象
+        image_file: 图像文件名
+        question_1: 问题1
+        question_2: 问题2
+        question_3: 问题3
+
+    Returns:
+        s: sglang对象


use english comments.

merrymercy · 2024-06-21T03:32:43Z

+    parser.add_argument("--question-file", type=str, default="/home/users/ntu/chih0001/scratch/VLM/sglang_fork/benchmark/llava_bench/questions_cot.jsonl")
+    parser.add_argument("--answer-file", type=str, default="/home/users/ntu/chih0001/scratch/VLM/sglang_fork/benchmark/llava_bench/test_answers.jsonl")


do not include personal path

merrymercy · 2024-06-21T03:34:04Z

+from sglang.utils import read_jsonl, dump_state_text
+from PIL import Image
+
+# This function is adapted to handle two questions, simulating a Chain of Thought (CoT) reasoning process.


multi-turn is different from CoT. They are two different concepts.

Can you change the filename/description to multi_turn?

merrymercy · 2024-07-04T07:40:14Z

The git history is messed up. Please address comments and rebase. Close this for now. Feel free to open new ones.

…ve a field the WORKER re-derives (sgl-project#499-B) Generalizes sgl-project#499's fix. sgl-project#499 aligned the hibernate manifest identity's MATCH (launcher, inside __post_init__) and PARK (worker, long after) banks across materialize_declarations, but named one residual it could not reach: a field a WORKER re-derives AFTER the match, computed by a process that does not exist yet when the match runs. Two live instances found, both fixed: - ModelRunner._sm80_dtype_fallback declares dtype="float16" on a card without bfloat16 (sm75 hetero host, gfx900-class ROCm) -- the predicted sgl-project#499 residual. - spec_worker.match_target_context_length (NOT predicted by sgl-project#499): every speculative worker pins context_length to the target model's resolved context_len at construction. All four spec worker implementations (eagle_worker_v2.py, standalone_worker_v2.py, frozen_kv_mtp_worker_v2.py, multi_layer_eagle_worker_v2.py) write this through the SAME source string via the existing ServerArgs.override() entry point -- no call site needed a code change, only the new source needed classifying. This one writes the SHARED server_args that BaseSpecWorker.model_runner reads from, so sgl-project#89's fast restore was unreachable on every speculative boot, not only the sm75 host, and nothing refuses spec under hibernate to warn about it. Fix: identity now reads through launch_view (arg_groups/overrides.py) -- resolved_view plus the un-applied writes of IDENTITY_TRANSPARENT_SOURCES. A source is identity-transparent when its write RE-DERIVES a field from state the fingerprint already pins (the rank's card, re-checked by NVML UUID and presence-gated at match time; the checkpoint at model_path). A source that changes WHAT IS LOADED stays deliberately absent from the set and must keep showing through -- model_runner.update_weights is the case sgl-project#499 argued must never be normalized away. The superseded launch value is recorded once, at the single post-resolution mutation point (ServerArgs.override), first writer wins. HIBERNATE_VERSION stays 2 for the same reason as sgl-project#499. The classification is a MAINTAINED sweep, not an eyeball check: an AST scan over srt/ classifies every literal-source post-resolution writer of an identity field as transparent or explicitly opaque (8 sites at this tip), so a new worker-side writer turns the sweep red instead of silently reopening the hazard. Residual, named and not fixable at the identity level: on heterogeneous hardware only rank 0 writes the identity, so a per-rank re-derivation describes rank 0 only (the per-rank NVML-UUID re-check is what pins the others); the end-to-end proof (park on the sm75 host, relaunch, confirm the 50s -> 8-14s restore) still needs a real mixed-hardware boot. FEATURE_CATALOG.md: replaces sgl-project#499's stale "Residual, named and NOT fixed" clause about _sm80_dtype_fallback (now fixed) with the full sgl-project#520 account. Clean auto-merge, no conflict -- this branch's base is exactly this worktree's pre-merge tip (a5eff26), so there was no divergence to reconcile. All 17 sections and all 21 previously-preserved additive paragraphs verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_hibernate_identity_520.py alone: 12 passed. - test_hibernate_identity_499.py (pre-existing sgl-project#499 test): still 7 passed, unaffected. - test/registered/unit/model_loader/ + unit/server_args/ + unit/spec/ (full, run in parallel against both this merge and the pre-merge tip a5eff26): 33 failed on both sides. Diffed the complete failing-test-ID lists (not counts): byte-identical, empty diff -- confirmed zero regression, matching the pre-stated 33-failure baseline exactly. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all 5 touched/added files.

… rule in code, not in the reader sgl-project#482 established that accumulating H2D counters are only comparable across arms when the counted work matches, and fixed the READER (ARM3_COMPUTE analysis) to apply that rule. sgl-project#523 moves the rule into the code that produces the counters, so a future reader cannot reintroduce the mistake by construction. --against is now the single cross-arm comparison path in read_arm.py: it requires an explicit reference arm and refuses a bare percentage delta between two runs with different completed-request counts, rather than silently accepting whatever the two logs happened to record. Off-by-one pins added where request/round counting is adjacent to a boundary. expert_compute_placement.py and planner/rejected.py get corrected number strings that had drifted from the actual measured values documented in ARM3_COMPUTE.md/ANALYSE_456_dsv4f_matrix_sweep.md/ROADMAP_456. New fixture set under test/registered/unit/layers/moe/fixtures/work_matched_523/ (compute/equal x tp0/1/2, in green_final/legacy_nowork/window_gap variants) backs test_work_matched_counters_523.py's hermetic coverage of the --against gate itself, plus a companion update to test_expert_compute_placement_439.py. Clean auto-merge -- this branch's base (a5eff26) predates sgl-project#520, but sgl-project#520 never touched §1/§16 or any of this branch's files (confirmed before merging), so no conflict. All 17 catalog sections and all 22 previously-preserved additive paragraphs verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_expert_compute_placement_439.py + test_work_matched_counters_523.py: 125 passed, 0 failed. - test_s12_log_analyse.py (s12_log_analyse.py is touched): same 4 pre-existing failures as the established baseline (TestBefund x2, TestBericht, TestHarnessLuecke), byte-identical, no new red. - test/registered/unit/layers/moe/ + tests/moe_offload/ (broader cross-check): 2 failed (both in test_gguf_moe_offload.py, the long-standing pre-existing issue already bisected back to before this session in the prior sgl-project#462/sgl-project#494 merge), 641 passed, 34 skipped, 395 subtests. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all touched/added files including the new PROVENANCE.md fixture doc.

…utation, sgl-project#471/sgl-project#33407/sgl-project#33312/sgl-project#33098/sgl-project#33271 ports, sgl-project#481 PP fixes Eight commits, largest bundle this session: - sgl-project#479 (two commits): the GGUF-MoE offload coverage test and its refusal message named MXFP4/type 39 as THE example of a ggml type with no MoE kernel -- sgl-project#398 made that false (type 39 has a full native kernel set on a wheel that carries it). Traced the ACTIVE DSV4-Flash MXFP4 path end to end and confirmed it is native, not an untraced fallback; the refusal message now names the layer's own declared types instead of a stale hardcoded example, so it cannot go stale again the same way. docs/dev/NOTE_479_mxfp4_active_driver_path.md has the trace. - sgl-project#471: ports upstream sglang#32320 -- mask the SM120 SWA page-split to touched pages only, instead of the full page range. docs/dev/TICKET_471_masked_page_split.md. - sgl-project#33407 port: SM120 sparse-MLA buckets the topk width instead of using it unbucketed, falling back to Triton outside the bucketed range. - sgl-project#33312 port: DSpark draft head resolves and loads fused shared experts. - sgl-project#33098 port: DSpark draft fills the EP token-accounting fields. - sgl-project#481: three pipeline-parallel defects from the sgl-project#445 window fixed in server_args.py/resident_fraction.py/expert_stats.py. - sgl-project#33271 port: auto-round GPTQ MoE completes the MoeWNA16 delegation. FEATURE_CATALOG.md §8 conflict-risk as flagged, but no actual conflict -- this branch's base (a5eff26) is now several merges behind (sgl-project#520, sgl-project#523, design-453 landed since), yet none of those touched §8, so git's 3-way merge combined everything automatically. All 17 sections and all 22 previously-preserved additive paragraphs verified present. **Explicit verification requested and confirmed**: the two "ancient pre-existing" failures in tests/moe_offload/test_gguf_moe_offload.py (test_materialize_declines_uncovered_ggml_type, test_guard_still_refuses_an_unstaged_gguf_layer) that this session bisected back to before this session even began -- sgl-project#479's fix to expert_offload.py and the type-39 assumption in gguf.py directly addresses their root cause. Full file run: 20 passed, 0 failed, both confirmed GREEN now. One test-isolation quirk found and run to ground, not a code bug: the three flash_mla test files this branch adds/touches (test_flash_mla_backends.py, test_flash_mla_page_split_mask_471.py, test_flash_mla_sm120_topk_buckets.py) interfere with each other when run in the same pytest process -- a shared Triton-decorated function object resolves to JITFunction instead of the expected InterpretedFunction once another file in the same process has already imported flash_mla_sm120.py under real (non-interpret) conditions. Verified this is NOT a merge regression: reproduced identically on the unmerged branch tip in isolation (fresh archive, no other code involved), and every one of the 30 tests across the three files passes cleanly when its file is run alone. Branch-inherent test-ordering fragility, not in scope for this merge to fix. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - tests/moe_offload/test_gguf_moe_offload.py alone: 20 passed (was 18 passed/2 failed before this merge, per this session's own bisection). - The eight new/touched test files, run individually where the isolation quirk above requires it: all pass in full (dspark 442: 18; auto_round: 17; dspark shared-expert 33312: 21; gguf mxfp4 479: 18; pp defects 481: 26; flash_mla backends/topk-buckets/page-split-mask: 10/25/10 each alone). - test/registered/unit/quantization/ + unit/models/ + spec/dspark/: 28 failed both before and after (diffed by test ID, not count -- the pre-merge archive showed 35 because 7 of its failures were an artifact of an incomplete git-archive extraction missing sgl-kernel/, confirmed by re-checking those 7 IDs against the working tree directly). The remaining 28 are dspark_kernel_parity's CUDA-only subtests plus test_gguf_dequant_scratch_budget.py, both pre-named as pre-existing. - test/registered/unit/layers/ + tests/moe_offload/: 3 failed (the pre-named test_deterministic_fp8_gemm.py x2 + test_flashinfer_workspace_zeroing.py x1), 1168 passed -- confirms test_gguf_moe_offload.py's contribution to this count is now 0 failures. - test/registered/unit/distributed/ (full, server_args.py touched): 24 failed, byte-identical established baseline. - ruff check --select=F401,F821,UP037: 15 findings, all in expert_offload.py, all confirmed pre-existing (same set already verified at cc5a92e in the earlier sgl-project#494 merge) and outside this branch's two diff hunks (2112-2138, 2616-2648) -- auto_round.py and every other touched file clean. - codespell: clean on all 21 touched/added files.

… in-tree C1 (results/RESULTS_C1_hibernate.md), the full park -> process exit -> restore round trip on hardware, TP=3 uneven, GGUF Q3_K_M: - POST /hibernate -> {"message":"hibernate: weights parked to disk."} - 1202 params parked per rank on all 3 ranks, 15 GiB image, manifest written. Every per-rank artifact is keyed by the card's NVML UUID, which is the identity mechanism itself rather than a naming convention. - Restore: "matching manifest found ... -> load_format='hibernate' (fast restore)", 1202 params restored per rank. - Cold 249 s -> restore 193 s = 1.29x. Recorded as measured and deliberately NOT compared against the catalog's 50s->8-14s, which is a different case. - Post-restore greedy smoke coherent. C3 PASS with it: the fast-restore path was reached BY NAME, which is the sgl-project#499/ sgl-project#520 signal. Only the MATCH branch is exercised; a moved card is untested. C2 NOT VERIFIED (no sparse-write clause observed, no dense-vs-sparse sha256). Worth noting the image lands on ZFS, where DESIGN_456 already measured the byte win as exactly zero because compression had taken it. Also lands the window's combination matrix in docs/dev/ as the durable record. Its final tally: 8 cells passed on hardware, 7 decided at the desk without a GPU, and the rest carried as explicitly named open states rather than silence. Three findings change what "it works" means for this feature family, and all three are code-cited in the matrix: 1. The KV pressure ladder has NO actuator on the production flag set -- wired_relief_features() returns empty unless --kv-reshard-vectors, --enable-kv-session-offload or --max-running-requests-ceiling is set, and production sets none. Flips are still logged; they move nothing, and the trailing field says "no actuator declared". Confirmed in both directions: with kvso wired, the same rung flips with "actuator WIRED: admission limit -> 1 (changed=True)". 2. Byte-equality of generated text is not a valid instrument on this rig: two IDENTICAL loads against one process, zero spills, already give identical=0 diverged=6. Correctness on the spill path needs decode-class bands, not byte identity. 3. sgl-project#363 act mode has zero flip targets by construction (planner_candidates is called without solve_fn), so it cannot actuate regardless of flags. Serving restored afterwards: pgid 63368, health 200 after 90 s, completion smoke correct, corridor 2260/4403/2778 MiB free. Locks released, holder handed back to a SERVING line.

fisher75 and others added 3 commits April 5, 2024 00:28

Add CoT feature for Llava in SGLang,

5f4e78c

add demo;

46c1643

Add CoT feature and demo scripts for Llava in SGLang

588c304

merrymercy requested changes Jun 21, 2024

View reviewed changes

updates;

3868642

Ying1123 force-pushed the main branch from d530a1c to c7709d3 Compare July 3, 2024 21:59

merrymercy force-pushed the main branch from 48546b9 to 63fbef9 Compare July 4, 2024 06:19

merrymercy closed this Jul 4, 2024

Navigation Menu

Uh oh!

Add CoT feature for LLava in SGLang with demo rather than multi-turn talk - #520

Add CoT feature for LLava in SGLang with demo rather than multi-turn talk#520
fisher75 wants to merge 4 commits into
sgl-project:mainfrom
fisher75:main

fisher75 commented Jun 10, 2024

Uh oh!

merrymercy Jun 21, 2024

Uh oh!

merrymercy Jun 21, 2024

Uh oh!

Uh oh!

merrymercy Jun 21, 2024 •

edited

Loading

Uh oh!

merrymercy commented Jul 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		parser.add_argument("--question-file", type=str, default="/home/users/ntu/chih0001/scratch/VLM/sglang_fork/benchmark/llava_bench/questions_cot.jsonl")
		parser.add_argument("--answer-file", type=str, default="/home/users/ntu/chih0001/scratch/VLM/sglang_fork/benchmark/llava_bench/test_answers.jsonl")

Uh oh!

Conversation

fisher75 commented Jun 10, 2024

Uh oh!

merrymercy Jun 21, 2024

Choose a reason for hiding this comment

Uh oh!

merrymercy Jun 21, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

merrymercy Jun 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

merrymercy commented Jul 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

merrymercy Jun 21, 2024 •

edited

Loading