Skip to content

docs : document what the test suites do and do not cover - #275

Merged
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
giveen:document-2
Aug 8, 2026
Merged

docs : document what the test suites do and do not cover#275
TheTom merged 1 commit into
TheTom:feature/turboquant-kv-cachefrom
giveen:document-2

Conversation

@giveen

@giveen giveen commented Aug 8, 2026

Copy link
Copy Markdown

Overview

Update to AGENT.md document to cover test suite. Helped by AI to ensure that it matches actual test suite.

  • AI usage disclosure: Assisted-by: Claude

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 8, 2026
@TheTom

TheTom commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Merging. This is the section I asked for on #272 and it is better than what I described, because you went and checked the mechanisms rather than writing the general warning.

Fact-checked the specific claims against the tree, since a doc that is confidently wrong about coverage is worse than no doc:

claim verdict
the SET_ROWS view variant uses r/2 rows, so r=1 writes nothing correct, test-backend-ops.cpp:2270, :2326, :2417
mat-vec decode is selected only at src2->ne[1] <= 8 correct, ggml-vulkan.cpp:10629
the backend verdict is tests_passed == tests_total, so 0/0 passes correct
test-quantize-fns skips TURBO2/3/4 because dequant output stays in the rotated domain correct
llama-bench never checks correctness correct

The r=1 observation is the one I had not made myself. A view variant that silently writes zero rows and reports OK for every type in all_types is a coverage hole that looks exactly like coverage, and it is why the TQ4_1S SET_ROWS cases appeared to pass before #260 fixed the shader.

Two things I would single out as worth more than the rest:

Tying each limit to the bug it actually caused, with issue numbers, is what will make someone believe it. "A green run means the cases that ran passed" is easy to nod along to and ignore; "this is how the wave64 ballot bug shipped, FLASH_ATTN_EXT passed while SET_ROWS was silently skipped on GCN4" is not.

And the -o filter note is the practical one. -o matching ggml_op_desc means the dedicated turbo write tests need their own names, and someone filtering on SET_ROWS while testing a turbo write path will get a clean run over nothing at all. That has now bitten at least twice in this repo.

One addition worth making later, not blocking: #276 just landed a fourth instance, where the harness itself wrote a non-contiguous view packed instead of strided and poisoned the CPU reference, so the reference was wrong and every other backend looked guilty. Same family as the rest: the failure did not present where the fault was.

@TheTom
TheTom merged commit 6c2e5f2 into TheTom:feature/turboquant-kv-cache Aug 8, 2026
1 check passed
@giveen
giveen deleted the document-2 branch August 19, 2026 17:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants