Skip to content

Add kv_cache_fixed_to_max_length search option (AMD RyzenAI) #2156

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Oleksandr Kholodnyi (akholodnamdcom) wants to merge 2 commits into from
Closed

Add kv_cache_fixed_to_max_length search option (AMD RyzenAI) #2156

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e6eb629
Add kv_cache_fixed_to_max_length search option (AMD RyzenAI)
akholodnamdcom May 12, 2026
645daa9
Potential fix for pull request finding
akholodnamdcom May 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions src/config.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1264,6 +1264,8 @@ struct Search_Element : JSON::Element {
v_.early_stopping = JSON::Get<bool>(value);
} else if (name == "blank_penalty") {
v_.blank_penalty = static_cast<float>(JSON::Get<double>(value));
} else if (name == "kv_cache_fixed_to_max_length") {
v_.kv_cache_fixed_to_max_length = JSON::Get<bool>(value);
Comment thread
akholodnamdcom marked this conversation as resolved.
} else {
throw JSON::unknown_value_error{};
}
Expand Down Expand Up @@ -1335,6 +1337,18 @@ struct Engine_Element : JSON::Element {
};

void SetSearchNumber(Config::Search& search, std::string_view name, double value) {
// AMD RyzenAI fixed-buffer mode pins kv-cache size to max_length from genai_config.json,
// so runtime overrides that would change max_length must be ignored to preserve the pre-sized buffer.
// Same-value sets are silent no-ops; differing values are warned and dropped.
if (name == "max_length" && search.kv_cache_fixed_to_max_length) {
const int new_value = static_cast<int>(value);
Comment thread
akholodnamdcom marked this conversation as resolved.
if (new_value != search.max_length) {
Log("warning", "Ignoring max_length=" + std::to_string(new_value) +
" override: kv_cache_fixed_to_max_length is enabled in genai_config.json (kv-cache pinned to max_length=" +
std::to_string(search.max_length) + ").");
Comment thread
akholodnamdcom marked this conversation as resolved.
Outdated
}
return;
}
try {
Comment thread
akholodnamdcom marked this conversation as resolved.
Search_Element(search).OnValue(name, value);
} catch (...) {
Expand Down
1 change: 1 addition & 0 deletions src/config.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,7 @@ struct Config {
int random_seed{-1}; // -1 = Seed with random device, otherwise use value to seed RNG
std::optional<size_t> chunk_size; // Chunk size for prefill chunking during context processing. If present, chunking is enabled with the chunk size > 0.
float blank_penalty{}; // Penalty applied to blank token logits in CTC/RNNT decoding. Default 0 means no penalty.
bool kv_cache_fixed_to_max_length{}; // AMD RyzenAI only: kv-cache is allocated as a fixed buffer sized to max_length from genai_config.json. When set, max_length cannot be overridden via SetSearchOption.
} search;

struct Engine {
Expand Down
Loading