tests : add support for qwen3 SSM archs

[ggerganov]

Overview

Enable test-llama-archs for Qwen3 architectures using SSM.

Additional information

|       qwen3next|Apple M2 Ultra|   MoE|  OK (8.53e-08)|       OK|
|       qwen3next|    Accelerate|   MoE|  OK (1.00e-17)|       OK|
|       qwen3next|Apple M2 Ultra|   MoE|  OK (9.61e-14)|       OK|
|       qwen3next|          Meta|   MoE|  OK (8.53e-08)|     SKIP|
|          qwen35|Apple M2 Ultra| Dense|  OK (8.53e-08)|       OK|
|          qwen35|    Accelerate| Dense|  OK (0.00e+00)|       OK|
|          qwen35|Apple M2 Ultra| Dense|  OK (9.18e-14)|       OK|
|          qwen35|          Meta| Dense|  OK (8.53e-08)|     SKIP|
|       qwen35moe|Apple M2 Ultra|   MoE|  OK (8.53e-08)|       OK|
|       qwen35moe|    Accelerate|   MoE|  OK (0.00e+00)|       OK|
|       qwen35moe|Apple M2 Ultra|   MoE|  OK (9.49e-14)|       OK|
|       qwen35moe|          Meta|   MoE|  OK (8.53e-08)|     SKIP|

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: NO

[JohannesGaessler]

Suggested change

	bool try_scalar = false;
	try {
	ml.get_key_or_arr(LLM_KV_FULL_ATTENTION_INTERVAL, hparams.recurrent_layer_arr, hparams.n_layer, false);
	} catch (...) {
	try_scalar = true;
	}
	if (try_scalar) {
	const uint32_t n_main = hparams.n_layer - hparams.nextn_predict_layers;
	uint32_t full_attn_interval = 4;
	ml.get_key(LLM_KV_FULL_ATTENTION_INTERVAL, full_attn_interval, false);
	for (uint32_t i = 0; i < hparams.n_layer; ++i) {
	hparams.recurrent_layer_arr[i] = (i < n_main) && ((i + 1) % full_attn_interval != 0);
	}
	try {
	ml.get_key_or_arr(LLM_KV_FULL_ATTENTION_INTERVAL, hparams.recurrent_layer_arr, hparams.n_layer, false);
	} catch (...) {
	const uint32_t n_main = hparams.n_layer - hparams.nextn_predict_layers;
	uint32_t full_attn_interval = 4;
	ml.get_key(LLM_KV_FULL_ATTENTION_INTERVAL, full_attn_interval, false);
	for (uint32_t i = 0; i < hparams.n_layer; ++i) {
	hparams.recurrent_layer_arr[i] = (i < n_main) && ((i + 1) % full_attn_interval != 0);
	}

I think that this would be slightly simpler but either way is fine I think.

[JohannesGaessler]

What is the intended scope of this PR, will there be more changes?

[ggerganov]

It's ready now.

[JohannesGaessler]

I'm not sure "is_swa_impl" is a good choice for the variable name. I'm reading it as "is SWA implementation" but then you have the code of the individual models manipulating it which to me would intuitively seem like the models messing with the internals of llama_hparams. Maybe "swa_pattern" to be consistent with set_swa_pattern?

[CISC]

I get the is_swa part, to match the function, but agreed, it's confusing, maybe is_swa_layer?

[ggerganov]

I'll follow-up with more refactoring of the hparams after this to avoid this PR growing. The main goal here is to get recurrent models enrolled in test-llama-archs to be able to generate small dummy models for testing purposes.

[JohannesGaessler]

Be aware though that currently there is no implementation for creating a dummy vocab for those models - I have a poor understanding of the related code and did not want to delay the unit tests for TP. But this means that you cannot just use the dummy models for e.g. llama-perplexity or llama-completion.

[ggerganov]

Yes, I noticed that. I'll be using it with test-save-load-state and I can rework it to not require a vocab.

[JohannesGaessler]

For dummy models, wouldn't it be fine to just map ASCII characters to int? I would intuitively assume that that would not be too difficult to implement, the problem for me was just that I would have to read up on the vocab code first.

[ggerganov]

Yes probably. It would be definitely useful to generate some dummy vocabs too. Will take a look.