Add kv_cache_fixed_to_max_length search option (AMD RyzenAI)

[Oleksandr Kholodnyi (akholodnamdcom)]

Summary

Adds a new kv_cache_fixed_to_max_length boolean to the search section of genai_config.json. Some AMD RyzenAI models require the kv-cache to be allocated as a fixed buffer sized to max_length; this option marks the model as having that constraint so OGA can preserve the
invariant at runtime.

Behavior

• When kv_cache_fixed_to_max_length is true, runtime overrides of max_length via OgaGeneratorParamsSetSearchNumber (and equivalents in C#/Python/Java/Objective-C) that would change the value are dropped with a yellow warning log entry, preserving the pre-sized buffer.
• Same-value overrides are silent no-ops.
• When the option is unset (default), behavior is unchanged.

Why a warning instead of an error

Throwing on every override would break clients that pass max_length through defensively (e.g. model_benchmark's --max_length flag, which is set by default to prompt_length + generation_length). Logging a warning and ignoring the override keeps existing tools working
while still surfacing that the value was not applied.

Files changed

• src/config.h — new bool kv_cache_fixed_to_max_length{} field at the end of Config::Search.
• src/config.cpp — JSON parser branch for the new key (last in the chain, before the unknown-key throw), and a guard at the top of SetSearchNumber that warns and drops differing max_length overrides.

Test plan

• model_benchmark -ml 1024 with kv_cache_fixed_to_max_length: true and max_length: 131072 in genai_config.json → warning printed, benchmark runs with max_length=131072.
• model_benchmark -ml 131072 (matches config) → no warning, benchmark runs.
• model_benchmark -ml -1 (no override call) → no warning, benchmark runs.
• Verified on Windows x64, RelWithDebInfo, CPU EP, with Llama-3.2-1B-Instruct (Q4F16).

Sample warning output:

[warning]  Ignoring max_length=1024 override: kv_cache_fixed_to_max_length is enabled in genai_config.json (kv-cache pinned to max_length=131072).

[Oleksandr Kholodnyi (akholodnamdcom)]

Baiju Meswani (@baijumeswani) kunal-vaishnavi could you please take a look at the fix

[Copilot]

Pull request overview

Adds a new model configuration flag to represent a hardware constraint (AMD RyzenAI) where the KV-cache must remain allocated at the genai_config.json max_length, and updates runtime search-option overrides to preserve that invariant.

Changes:

• Adds search.kv_cache_fixed_to_max_length to Config::Search and parses it from genai_config.json.
• Updates SetSearchNumber to ignore runtime max_length overrides when kv_cache_fixed_to_max_length is enabled (with a warning on differing values).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
src/config.h	Adds the kv_cache_fixed_to_max_length flag to the search config struct.
src/config.cpp	Parses the new JSON key and adds a guard to drop incompatible runtime max_length overrides.

[Baiju Meswani (baijumeswani)]

This pull-request introduces a new config setting kv_cache_fixed_to_max_length that controls whether or not to use fixed shape for the kv-cache regardless of what the user may set with max_length. A few issues I think this introduces:

• This seems like a temporary work-around. The longer-term solution might be to update the models to support dynamic kv-cache sizes. The problem with this temporary work-around is that it is user facing. Removing this setting in the future will be difficult.
• It ignores the max_length set by the user. Moreover, it is prone to errors in case the user/application updates the value of kv_cache_fixed_to_max_length themselves.
• It makes the settings more clunky.

Here are some alternative proposals:

1. Update the model to accept dynamic axes for the kv-cache. This might be a good solution if the fixed shape is not a hard requirement but a by-product of the maximum number of tokens the model can support. If the max number of tokens the model can support is fixed, you could consider making the context length = 4096 which would limit the number of tokens generated + input tokens to 4096.
2. Read the session information to derive the fixed length of the kv-cache from graph outputs. If the model has a fixed sequence length index, use that to make necessary buffer allocations instead of exposing the setting through the genai_config.json.

[Baiju Meswani (baijumeswani)]

Closing in favor of #2166