[CI] Replace large models with tiny alternatives in tests

[tahsintunan]

Purpose

This PR replaces large language Llama models with tiny alternatives in test files to reduce CI execution time.

• Partially addresses [CI]: Replace use of models with smaller models where possible #23456.
• Only targeted the big, gated meta-llama models in this PR. More PRs will follow.
• Made sure we're only modifying tests that do NOT test model-specific behavior.

Key changes:

• Replace big (>1B) meta-llama models with smaller (e.g., EleutherAI/pythia-14m, JackFram/llama-68m) models.
• Refactor some test implementations (e.g., test_shutdown, test_basic_correctness, test_sampling_params_e2e)

Test Plan

Ran the following (modified) tests and had them all pass.

# Basic Correctness Tests
pytest -xvs tests/basic_correctness/test_basic_correctness.py
pytest -xvs tests/basic_correctness/test_cpu_offload.py
pytest -xvs tests/basic_correctness/test_cumem.py

# Distributed Tests (Requires multiple GPUs)
pytest -xvs tests/distributed/test_sequence_parallel.py
pytest -xvs tests/entrypoints/llm/test_collective_rpc.py

# API Tests
pytest -xvs tests/entrypoints/openai/test_shutdown.py
pytest -xvs tests/entrypoints/openai/test_run_batch.py
pytest -xvs tests/entrypoints/openai/test_serving_models.py

# Sampling Tests
pytest -xvs tests/samplers/test_ignore_eos.py
pytest -xvs tests/samplers/test_no_bad_words.py

# V1 Tests
VLLM_USE_V1=1 pytest -xvs tests/v1/sample/test_sampling_params_e2e.py
VLLM_USE_V1=1 pytest -xvs tests/v1/core/test_scheduler_e2e.py
VLLM_USE_V1=1 pytest -xvs tests/v1/engine/test_engine_core.py
VLLM_USE_V1=1 pytest -xvs tests/v1/shutdown/test_delete.py
VLLM_USE_V1=1 pytest -xvs tests/v1/shutdown/test_forward_error.py
VLLM_USE_V1=1 pytest -xvs tests/v1/shutdown/test_startup_error.py

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

[tahsintunan]

After further consideration, I've removed the pre-commit check from the Buildkite pipeline. Since the pre-commit checks already run on GitHub Actions (and it's way faster), I think it'd be simpler to just let Buildkite run only after GitHub's pre-commit validation passes. Perhaps we can configure Buildkite to skip builds with failing commit status, or use branch protection to enforce this order.

Not related to this PR, though. Earlier, I added the pre-commit check to Buildkite since it was a relatively small change and addressed #23452, but now I think there are better ways to handle this.

[njhill]

Thanks @tahsintunan! Re any changes to the pre-commit flow, it would be good to keep in separate PRs (I know that's maybe n/a now since you reverted it).

[njhill]

Thanks @tahsintunan, looks great! It may overlap slightly with other PRs e.g. #23896, but I think that's fine, we can just try to get them all merged quicky.

We were hoping to standardize on hmellor/tiny-random-LlamaForCausalLM as the small model, do you think you could update the PR to use that?

[tahsintunan]

@njhill Hey! I've updated the PR to use hmellor/tiny-random-LlamaForCausalLM where possible. Couldn't replace it in 3 tests due to specific requirements:

• test_sequence_parallel - No decoder-only support
• test_no_bad_words - Doesn't have add_prefix_space tokenizer support
• test_basic_correctness - tokenization mismatches when using prompt embeddings (missing whitespace)

Edit: hmellor/tiny-random-LlamaForCausalLM also doesn't work with TP and SP

[hmellor]

@tahsintunan could you please look into the remaining failures?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @tahsintunan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[tahsintunan]

@hmellor All failures are fixed

[hmellor]

All the other changes LGTM, just one question about test_shutdown.py

[njhill]

Thanks a lot @tahsintunan @hmellor!