Cleanup basic and entrypoint test organisation

[hmellor]

• Don't call basic test files individually
• Splits the long entrypoints test group into 2 shards
• Move the unit tests into a unit directory so we don't have to manually ignore everything else
• Move openai/tool_parsers into the unit tests because that's what they are

[gemini-code-assist]

Code Review

This pull request focuses on reorganizing the test structure for better maintainability and efficiency. It involves moving unit tests into a dedicated directory, splitting a large test group into shards, and updating test pipeline configurations. The review identifies a critical issue related to potential test failures due to missing multi-GPU configurations.

[gemini-code-assist]

The test_collective_rpc test is marked as a multi-GPU test, but the pytestmark is defined after the imports. This could lead to tests being run without the multi-gpu setup if pytest runs before the mark is applied. It's better to define pytestmark at the top of the file to ensure it's always applied. This is a critical issue as it can lead to tests passing incorrectly when they should be failing due to insufficient GPU resources.

pytestmark = pytest.mark.multi_gpu_test(num_gpus=2)


from ...utils import create_new_process_for_each_test


@pytest.mark.parametrize("tp_size", [1, 2])

[hmellor]

We don't do this anywhere else that we use pytestmark

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Removing device-count guard lets multi-GPU test run on single GPU

The test previously skipped when torch.cuda.device_count() < tp_size, but the new module-level pytestmark = pytest.mark.multi_gpu_test(num_gpus=2) does not perform that check—the multi_gpu_test skip logic lives in the decorator in tests/utils.py, not in a pytest marker. As a result, on environments with only one GPU this test will now run and attempt to create an LLM with tensor_parallel_size=2, causing a failure instead of a skip. To restore the behaviour, either keep the explicit device-count guard or apply the multi_gpu_test decorator (or its marks) so the skip happens before the LLM is constructed.

Useful? React with 👍 / 👎.

[bbrowning]

• Move openai/tool_parsers into the unit tests because that's what they are

This is mostly correct, except test_hermes_tool_parser.py that actually spins up vLLM in some of its tests. Those tests may need to get split out from the rest here, and generally tool parser tests that spin up a server are under tests/tool_use instead. The hermes test file has some of both, so it could either be moved entirely for now or split into two files in the appropriate place.

[hmellor]

Thanks for the additional context. My reasoning for moving them there was that they were run in the Entrypoints Unit Tests job.

I'll move the ones that start the server into tests/tool_use

[hmellor]

Too stale, superseded by #31228