Skip to content

[Benchmark] Benchmark structured output with datasets #10557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mgoin merged 16 commits into from
Dec 4, 2024
Merged

[Benchmark] Benchmark structured output with datasets #10557

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
f0b0c0d
benchmark: add guided decoding script
aarnphm Nov 5, 2024
c62b55b
chore: add warmup args
aarnphm Nov 5, 2024
e64a701
chore: run format accordingly
aarnphm Nov 7, 2024
a0c46f1
chore: add @mgoin's suggestion
aarnphm Nov 15, 2024
91d9efc
chore: run format
aarnphm Nov 15, 2024
8e67db0
Merge branch 'pr10046_structured_output' into benchmark_structured_ou…
xuechendi Nov 22, 2024
18c245c
Add xgrammar similiar dataset
xuechendi Nov 22, 2024
9bec7fc
Add grammar, regex, choice, json - json support using file path
xuechendi Nov 22, 2024
ad531ae
Asycn engine save results
xuechendi Nov 22, 2024
b6164b7
Add latency for first and Next token - async engine
xuechendi Nov 26, 2024
b5fd6a2
Add correctness check
xuechendi Nov 26, 2024
d67fc48
Add mix request using guided_decoding_ratio
xuechendi Nov 26, 2024
28d69b4
only check with format
xuechendi Nov 26, 2024
badf679
Fix correctness check error when no json detected
xuechendi Nov 26, 2024
54678bb
fix warmup per suggestion
xuechendi Nov 28, 2024
4196660
using describe for request latency
xuechendi Nov 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading