Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] regular update #2431

Merged
merged 37 commits into from
Sep 18, 2024
Merged

[ci] regular update #2431

merged 37 commits into from
Sep 18, 2024

Conversation

zhulinJulia24
Copy link
Collaborator

@zhulinJulia24 zhulinJulia24 commented Sep 6, 2024

  1. [tested] add evaluation function test into daily test, only run 50 cases for each benchmark dataset
  2. [tested] react benchmark test, because more than one logs need to attached to benchmark report so move allure report attachment action into utils
  3. [tested] fix do_sample param related testcase
  4. [tested] react env paramter of env.COV_PARAM in daily test
  5. [tested] change installation method of autoawq_kenerls
  6. [tested] add more test models into benchmark test
  7. [Not fully verified] try to use pytest_cache when rerun pipelines

pipe = pipeline(model_path, backend_config=backend_config)
gen_config = GenerationConfig(random_seed=1, top_k=40, do_sample=True)
response = pipe(['Shanghai is'] * 3, gen_config=gen_config)
result = response[0].text == response[1].text and response[
Copy link
Collaborator

@lvhan028 lvhan028 Sep 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is possible that the reponse.text is different for batch inference, because they are performed in continuous batching rather than fix batch size
Let's change it to the following:

response = [pipe('Shanghai is', gen_config=gen_config) for i in range(3)]
result = response[0].text == response[1].text and response[1].text == response[2].text

@lvhan028 lvhan028 merged commit dd49877 into InternLM:main Sep 18, 2024
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants