Skip to content

server : use different seeds for child completions #18700

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ggerganov merged 3 commits into from
Jan 9, 2026
Merged

server : use different seeds for child completions #18700

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f358562
server : use different seeds for child completions
ggerganov Jan 8, 2026
c43134b
cont : handle default seed
ggerganov Jan 9, 2026
5975ed6
cont : note [no ci]
ggerganov Jan 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions tools/server/server-context.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
#include "server-task.h"
#include "server-queue.h"

#include "arg.h"
#include "common.h"
#include "llama.h"
#include "log.h"
Expand All @@ -16,7 +15,6 @@
#include <cstddef>
#include <cinttypes>
#include <memory>
#include <unordered_set>
#include <filesystem>

// fix problem with std::min and std::max
Expand Down Expand Up @@ -2927,9 +2925,11 @@ std::unique_ptr<server_res_generator> server_routes::handle_completions_impl(
if (task.params.n_cmpl > 1) {
task.n_children = task.params.n_cmpl - 1;
for (size_t j = 0; j < task.n_children; j++) {
server_task child = task.create_child(
task.id,
rd.get_new_id());
server_task child = task.create_child(task.id, rd.get_new_id());

// use different sampling seed for each child
child.params.sampling.seed += j + 1;

@ngxson ngxson Jan 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we also handle the case where default seed is used?

@ngxson ngxson Jan 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah nevermind, I think we don't need to care about that case here

@ggerganov ggerganov Jan 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, you might be right. If we don't handle the default seed here, then 1 child will have random seed and the rest will get the same seeds.

tasks.push_back(std::move(child));
}
}
Expand Down
1 change: 0 additions & 1 deletion tools/server/tests/unit/test_chat_completion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -503,5 +503,4 @@ def test_chat_completions_multiple_choices():
assert len(res.body["choices"]) == 2
for choice in res.body["choices"]:
assert "assistant" == choice["message"]["role"]
assert match_regex("Suddenly", choice["message"]["content"])
assert choice["finish_reason"] == "length"
Loading