vulkan: fix and complete turbo3 KV cache support

[Titaniumtown]

Overview

The PR for initial vulkan support (#33), was not fully working for me, it was resulting in gibberish output on Gemma 4 being gibberish. I put claude in a closed loop with real hardware to solve the problem. It based it's work on the cuda kernel that exists.

Things done:

• Fix block size: 32 -> 128
• add WHT shader
• add flash attention support
• fix dequantization for turbo3 on vulkan
• Rewrite SET_ROWS path to be correct also 12.7x faster (on a380)

This PR also adds some tests for vulkan + turbo3 that would've caught the issues in #33.

Additional information

#33

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Claude Opus 4.6 was used in a closed loop with real hardware.

[TheTom]

Thanks for the Vulkan fix and the test coverage — built and ran locally on M5 Max (Metal) and the Vulkan-only code is properly isolated, no impact on CUDA or Metal at runtime. The expanded FA test matrix (adding TURBO3_0 and hsk=128 for non-F16 types) is a win: all 6256 FA tests pass on Metal.

One blocker before merging: the 27 new TURBO_WHT and TURBO_WHT_ROUNDTRIP tests fail on Metal due to tolerance set too tight. Errors land in the 3.25e-7 to 1.015e-6 range, but the default NMSE threshold is 1e-7, which is tighter than f32 SIMD reduction precision on GPU.

[TURBO_WHT] ERR = 0.000000325 > 0.000000100   TURBO_WHT(head_dim=128,n_heads=1,direction=0): FAIL
[TURBO_WHT] ERR = 0.000000387 > 0.000000100   TURBO_WHT(head_dim=128,n_heads=4,direction=0): FAIL
...
  0/27 tests passed
  Backend MTL0: FAIL

Math is correct, just a tolerance problem. You already did the right thing on test_set_rows_turbo3 (max_nmse_err() = 0.05 because turbo3 is lossy), just need the same treatment on the pure WHT tests:

struct test_turbo_wht : public test_case {
    // ...
    double max_nmse_err() override {
        return 1e-5;  // f32 SIMD reduction precision on GPU backends
    }
};

struct test_turbo_wht_roundtrip : public test_case {
    // ...
    double max_nmse_err() override {
        return 1e-5;
    }
};

1e-5 is conservative — you could probably go tighter if you want. Once that's in, happy to merge.

Minor side note (not a blocker): the 22 new SET_ROWS_TURBO3 tests report "not supported" on Metal and get skipped (0/0 tests passed on that backend). Metal clearly supports the op in production via kernel_set_rows_turbo3_i32/i64 templates, so the test leaf-tensor setup is probably mismatched against what the Metal supports_op check expects. Worth investigating separately so the tests actually exercise Metal, but doesn't block this PR.

[Titaniumtown]

Let me know if that works. I don't have any apple devices to test.

[TheTom]

Verified on M5 Max (Metal):

• TURBO_WHT: 27/27 pass ✅
• TURBO_WHT_ROUNDTRIP: pass ✅
• FLASH_ATTN_EXT: 6256/6256 pass ✅

The 1e-5 tolerance is the right call — actual errors land around 3-5e-7, so you've got plenty of headroom. Thanks for the quick turnaround, merging now.

Note: SET_ROWS_TURBO3 still reports "not supported" on Metal (skipped 0/0). Not a blocker since Metal clearly supports the op in production, but worth a separate look later if you want the tests to actually exercise Metal.