[Benchmark] Fix client seed synchronization in multi-turn benchmark#28512
Conversation
There was a problem hiding this comment.
Code Review
This pull request correctly fixes a critical flaw in the multi-turn benchmark where clients were synchronized due to using the same random seed. The implemented solution of creating a unique seed for each client based on its ID is a good step forward. My review includes one suggestion to further improve the statistical robustness of the benchmark by using a more advanced seeding strategy (numpy.random.SeedSequence) designed for parallel applications. This would ensure the random behavior of clients is not just unique but also statistically independent, which is crucial for the validity of the benchmark results.
|
@DarkLight1337 could you have a look? |
|
Nice catch |
All clients were using the same seed, causing them to generate identical random intervals for request timing. This led to synchronized behavior where all clients would sleep for the same duration. Fix by setting unique seed per client: args.seed + client_id + 1 The +1 ensures no client uses the same seed as the main process. Since each client runs in its own process (via multiprocessing.Process), each has its own RNG state, so we just need to seed them differently. Simple 2-line change: - random.seed(args.seed + client_id + 1) - np.random.seed(args.seed + client_id + 1) Signed-off-by: ai-jz <aijz.xplr@gmail.com>
Head branch was pushed to by a user without write access
fd3bc12 to
faf52f6
Compare
|
@DarkLight1337 I updated the commit to include the DCO sign-off, which seems a new requirement of vLLM. This update has no code change - only the commit message was amended with "Signed-off-by". |
…llm-project#28512) Signed-off-by: ai-jz <aijz.xplr@gmail.com>
…llm-project#28512) Signed-off-by: ai-jz <aijz.xplr@gmail.com>
Problem
All clients in multi-turn benchmark use the same seed value, causing them to generate identical random sequences. This leads to synchronized behavior where all clients sleep for the same duration and make requests at the same time, defeating the purpose of Poisson-distributed request timing.
Evidence
Real logs showing the synchronization issue:
All clients sleep for exactly 0.796 seconds because they use the same random seed.
Solution
Add
client_idto the seed value to ensure each client has a unique but reproducible random sequence:Impact