feat: Allow passing any arguments to vllm and sglang engines

[grahamking]

Put the arguments in a JSON file:

{
    "dtype": "half",
    "trust_remote_code": true
}

Pass it like this:

dynamo-run out=sglang ~/llm_models/Llama-3.2-3B-Instruct --extra-engine-args sglang_extra.json

Requested here #290 (dtype) and here #360 (trust_remote_code).

[piotrm-nvidia]

When I pass invalid file name, then no error is displayed. How can I know that something is wrong?

root@eos0001:/workspace# dynamo-run out=vllm Qwen/Qwen2.5-3B-Instruct --extra-engine-args invalid_file_name.json
INFO 03-25 10:30:37 __init__.py:190] Automatically detected platform cuda.
INFO 03-25 10:30:37 nixl.py:16] NIXL is available
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  3.38it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00,  2.20it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00,  2.32it/s]
Capturing CUDA graph shapes: 100%|██████████| 35/35 [00:06<00:00,  5.22it/s]
2025-03-25T17:31:00.319867Z  INFO dynamo_run::input::text: Ctrl-c to exit

[grahamking]

It does not raise FileNotFoundError? What is the fix?

[dmitry-tokarev-nv]

Should catch the exception and print a message to the user to stdout and then quit

[piotrm-nvidia]

Also, when I pass corrupted file no error is printed:

root@eos0001:/workspace# cat sglang_extra.json 
INVALID JSON SYNTAX
root@eos0001:/workspace# dynamo-run out=vllm Qwen/Qwen2.5-3B-Instruct --extra-engine-args sglang_extra.json 
INFO 03-25 10:32:36 __init__.py:190] Automatically detected platform cuda.
INFO 03-25 10:32:36 nixl.py:16] NIXL is available
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  3.33it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00,  2.18it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00,  2.30it/s]
Capturing CUDA graph shapes: 100%|██████████| 35/35 [00:07<00:00,  4.92it/s]
2025-03-25T17:33:00.328763Z  INFO dynamo_run::input::text: Ctrl-c to exit
? User › 

[grahamking]

Can you give an example of corrupted file?

[piotrm-nvidia]

INVALID JSON SYNTAX

[piotrm-nvidia]

The only JSON this code hates is with some magic Unicode in the file name:

File content hereroot@eosRUST_BACKTRACE=full  dynamo-run out=vllm Qwen/Qwen2.5-3B-Instruct --extra-engine-args1:/workspace# cat 'evil_'$'\377\376''_bad_filename.json'

thread 'main' panicked at library/std/src/env.rs:871:51:
called `Result::unwrap()` on an `Err` value: "evil_\xFF\xFE_bad_filename.json"
stack backtrace:
   0:     0x555556df78e9 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::he089f96442833f67
   1:     0x555555f81933 - core::fmt::write::h2f210ed4c94745cb
   2:     0x555556df6cf2 - std::io::Write::write_fmt::h7de08171ab770fb2
   3:     0x555556df7743 - std::sys::backtrace::BacktraceLock::print::h810fbd31421329e6
   4:     0x555556df7218 - std::panicking::default_hook::h24e207139139d40a
   5:     0x555556df6b87 - std::panicking::rust_panic_with_hook::ha9131beeb2ddc506
   6:     0x555556e368a8 - std::panicking::begin_panic_handler::{{closure}}::h1bba0eaeb6da506f
   7:     0x555556e36809 - std::sys::backtrace::__rust_end_short_backtrace::h1d1ca3eade483f4c
   8:     0x555556e3777c - rust_begin_unwind
   9:     0x5555556e26df - core::panicking::panic_fmt::h896a0727a1a943f9
  10:     0x5555556e2a55 - core::result::unwrap_failed::h1b5ed8541c7bebd6
  11:     0x555555b77c45 - dynamo_run::main::h0ad5976c308a53fa
  12:     0x555555969833 - std::sys::backtrace::__rust_begin_short_backtrace::hdf20cb83f67d21df
  13:     0x555555b790a4 - main
  14:     0x7ffff71911ca - __libc_start_call_main
                               at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  15:     0x7ffff719128b - __libc_start_main_impl
                               at ./csu/../csu/libc-start.c:360:3
  16:     0x555555768f25 - _start
  17:                0x0 - <unknown>

This file is fine and here it works nicely for cat:

root@eos0001:/workspace# cat 'evil_'$'\377\376''_bad_filename.json'
File content here