tests : speed-up server test suite 3x

[ggerganov]

Overview

• Reduce long sleep when waiting for server to start
• Run load_server_presets() once per session instead of once per module
• Print top 30 slowest tests

On M2 Ultra:

time ./tools/server/tests.sh

# before: 9m39.395s
# after:  3m12.399s

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[ggerganov]

@ngxson The unit/test_router.py seems a bit slow - after this change it takes about 1/3 of the total time. Not sure if it can be optimized. Alternatively, we can move it to a separate workflow as it is independent of the backend (i.e. CUDA, Metal, etc.)

[ggerganov]

These are currently the top 30 slowest tests:

13.03s call     unit/test_router.py::test_router_queue_is_fifo
9.49s call     unit/test_router.py::test_router_queue_coalesces_requests_for_same_model
8.84s call     unit/test_router.py::test_router_queue_does_not_evict_busy_model
8.37s setup    unit/test_basic.py::test_server_start_simple
7.69s call     unit/test_router.py::test_router_queue_client_disconnect_keeps_model
6.94s call     unit/test_router.py::test_router_models_max_evicts_lru
5.35s call     unit/test_router.py::test_router_download_model
4.73s call     unit/test_stream.py::test_stream_resumes_after_reload_during_model_load
4.71s call     unit/test_stream.py::test_stream_resume_and_stop_with_slashed_model_name
4.70s call     unit/test_router.py::test_router_no_models_autoload
4.69s call     unit/test_router.py::test_router_chat_completion_stream[ggml-org/tinygemma3-GGUF:Q8_0-True]
4.67s call     unit/test_router.py::test_router_api_key_required
3.75s call     unit/test_router.py::test_router_remote_preset
2.64s call     unit/test_router.py::test_router_unload_model
2.62s call     unit/test_stream.py::test_stream_stop_during_model_load
2.24s call     unit/test_completion.py::test_cancel_request
2.19s call     unit/test_mcp_servers.py::test_mcp_slow_tool_call_slot_release
2.16s call     unit/test_sleep.py::test_server_sleep
1.68s call     unit/test_compat_anthropic.py::test_anthropic_vision_base64_with_multimodal_model
1.48s call     unit/test_basic.py::test_load_split_model
1.45s call     unit/test_speculative.py::test_different_draft_min_draft_max
1.29s call     unit/test_speculative.py::test_with_ctx_shift
1.03s teardown unit/test_router.py::test_router_download_model
1.02s call     unit/test_vision_api.py::test_vision_chat_completion[What is this:\n-IMG_URL_0-True-(cat)+]
0.99s call     unit/test_rerank.py::test_rerank_top_n[4-4]
0.99s call     unit/test_rerank.py::test_rerank_tei_top_n[4-4]
0.99s call     unit/test_rerank.py::test_rerank_tei_top_n[2-2]
0.99s call     unit/test_rerank.py::test_rerank_top_n[None-4]
0.99s call     unit/test_rerank.py::test_rerank
0.99s call     unit/test_rerank.py::test_rerank_tei_top_n[None-4]

[ngxson]

Thanks! Nice improvement, I haven't thought about this.

Btw, I've also been thinking about another improvement: currently, server need to restart on every single test; we can tweak the test framework such that a group of test cases can reuse the same instance. Although, that's a more complicated change than the current PR.

[ngxson]

@ngxson The unit/test_router.py seems a bit slow - after this change it takes about 1/3 of the total time. Not sure if it can be optimized. Alternatively, we can move it to a separate workflow as it is independent of the backend (i.e. CUDA, Metal, etc.)

yes I think some test cases for router is too extensive, it can definitely be optimized

[ggerganov]

Thanks! Nice improvement, I haven't thought about this.

Btw, I've also been thinking about another improvement: currently, server need to restart on every single test; we can tweak the test framework such that a group of test cases can reuse the same instance. Although, that's a more complicated change than the current PR.

We can also probably run the test suite in parallel with multiple servers. This should be relatively simple change to the tests.sh script.