Skip to content

Implement integration tests in CI pipeline#639

Merged
jcyang43 merged 1 commit intomainfrom
vllm/sting_integration_test
Oct 7, 2025
Merged

Implement integration tests in CI pipeline#639
jcyang43 merged 1 commit intomainfrom
vllm/sting_integration_test

Conversation

@CienetStingLin
Copy link
Copy Markdown
Collaborator

@CienetStingLin CienetStingLin commented Sep 4, 2025 •
edited by jcyang43
Loading

Description

Brings accuracy test to CI.

Based on this upstream PR, we have changed the lm_eval version used to git+https://github.com/EleutherAI/lm-evaluation-harness.git@206b7722158f58c35b7ffcd53b035fdbdda5126d#egg=lm-eval[api].

Tests

buildkite

Checklist

Before submitting this PR, please make sure:

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Sep 4, 2025

Description

Start with a short description of what the PR does and how this is a change from
the past.

The rest of the description includes relevant details and context, examples:

  • why is this change being made,
  • the problem being solved and any relevant context,
  • why this is a good solution,
  • some information about the specific implementation,
  • shortcomings of the solution and possible future improvements.

If the change fixes a bug or a Github issue, please include a link, e.g.,:
FIXES: b/123456
FIXES: #123456

Tests

Please describe how you tested this change, and include any instructions and/or
commands to reproduce.

Checklist

Before submitting this PR, please make sure:

  • I have performed a self-review of my code.
  • I have necessary comments in my code, particularly in hard-to-understand areas.
  • I have made or will make corresponding changes to any relevant documentation.

@CienetStingLin CienetStingLin force-pushed the vllm/sting_integration_test branch from 2f4fa05 to a509587 Compare September 4, 2025 08:05
jcyang43
jcyang43 reviewed Sep 4, 2025
View reviewed changes
Comment thread docker/Dockerfile Outdated
jcyang43
jcyang43 reviewed Sep 4, 2025
View reviewed changes
Comment thread requirements_benchmarking.txt Outdated
jcyang43
jcyang43 reviewed Sep 4, 2025
View reviewed changes
Comment thread scripts/vllm/integration/conftest.py Outdated
jcyang43
jcyang43 reviewed Sep 4, 2025
View reviewed changes
Comment thread scripts/vllm/integration/test_accuracy.py Outdated
jcyang43
jcyang43 reviewed Sep 4, 2025
View reviewed changes
Comment thread scripts/vllm/integration/test_accuracy.py Outdated
jcyang43
jcyang43 reviewed Sep 4, 2025
View reviewed changes
Comment thread scripts/vllm/integration/test_accuracy.py Outdated
jcyang43
jcyang43 reviewed Sep 4, 2025
View reviewed changes
Comment thread scripts/vllm/integration/test_accuracy.py Outdated
jcyang43
jcyang43 reviewed Sep 4, 2025
View reviewed changes
Comment thread scripts/vllm/integration/test_accuracy.py Outdated
jcyang43
jcyang43 reviewed Sep 26, 2025
View reviewed changes
Comment thread .buildkite/buildkite_ci_model_template.yml Outdated
jcyang43
jcyang43 reviewed Sep 26, 2025
View reviewed changes
Comment thread .buildkite/buildkite_ci_model_template.yml Outdated
jcyang43
jcyang43 reviewed Sep 26, 2025
View reviewed changes
Comment thread docker/Dockerfile Outdated
jcyang43
jcyang43 reviewed Sep 26, 2025
View reviewed changes
Comment thread scripts/vllm/integration/conftest.py Outdated
jcyang43
jcyang43 reviewed Sep 26, 2025
View reviewed changes
Comment thread tests/e2e/benchmarking/test_accuracy.sh
jcyang43
jcyang43 reviewed Sep 26, 2025
View reviewed changes
Comment thread tests/e2e/benchmarking/test_accuracy.sh Outdated
@meiyeh123 meiyeh123 force-pushed the vllm/sting_integration_test branch 2 times, most recently from 5577585 to 4e04557 Compare September 30, 2025 09:59
@meiyeh123
Copy link
Copy Markdown
Collaborator

meiyeh123 commented Sep 30, 2025

the new commit replace some args with env, and remove yaml files in this PR.
(buildkite test)

@jcyang43 jcyang43 force-pushed the vllm/sting_integration_test branch from 4e04557 to 8953129 Compare October 1, 2025 17:09
@meiyeh123
accomplish integration test, this change adds a new test accuracy on …
5305c98 
…TPU to the CI pipeline.

Signed-off-by: StingLin <sting.lin@cienet.com>

Signed-off-by: dennis yeh <dennis.yeh@cienet.com>
@meiyeh123 meiyeh123 force-pushed the vllm/sting_integration_test branch from d3e310f to 5305c98 Compare October 7, 2025 08:00
@jcyang43 jcyang43 merged commit 577cbda into main Oct 7, 2025
1 of 2 checks passed
jcyang43 added a commit that referenced this pull request Oct 7, 2025
@jcyang43
Revert "Implement integration tests in CI pipeline (#639)"
b1ee2d7 
This reverts commit 577cbda.
@jcyang43 jcyang43 mentioned this pull request Oct 7, 2025
Merged
helloleah pushed a commit that referenced this pull request Oct 8, 2025
@CienetStingLin @meiyeh123 @helloleah
Implement integration tests in CI pipeline (#639)
754d330 
Signed-off-by: dennis yeh <dennis.yeh@cienet.com>
Co-authored-by: dennis yeh <dennis.yeh@cienet.com>
@jcyang43 jcyang43 deleted the vllm/sting_integration_test branch November 24, 2025 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jcyang43 jcyang43 jcyang43 left review comments

@vipannalla vipannalla vipannalla approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@CienetStingLin @meiyeh123 @jcyang43 @vipannalla