Skip to content

Trading model catalog: Qwen3.8-27B row native_proven from li26 (#342) — micro-F1 0.9909, C2 26.28 tok/s, bounded - #354

Merged
seathatflowsinourveins merged 1 commit into
mainfrom
claude/models-qwen-li26-20260926
Sep 27, 2026
Merged

seathatflowsinourveins merged 1 commit into
mainfrom
claude/models-qwen-li26-20260926

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Owner

What

This records the li26 result merged in #342 on the Qwen/Qwen3.8-27B row of the trading model catalog (catalogs/us-equities/models.json and models.md). The row moves from source_review to native_proven, for one host and one task:

  • Measured: the unsloth/Qwen3.8-27B-GGUF UD-Q4_K_M quant at revision 4ca7207, served by llama.cpp b11146 at 8192 context. On the frozen 360-filing 8-K item-extraction task (2020-03-02 cohort) it scored micro-F1 0.9909 against the filer-declared header items, and every reply was valid JSON.
  • Production now serves the C2 profile: --gpu-layers 99 --n-cpu-ffn 20 --spec-type draft-mtp --spec-draft-n-max 3. It decodes at a median 26.28 tokens/s with a median TTFT of 4,355 ms, against C0's 5.25 tokens/s and 11,378 ms at the same F1.
  • Limitations kept on the row:
    • The labels are header ITEMS, not hand-checked.
    • Decoding was greedy with thinking off, about 17 tokens per filing, one run per arm.
    • The MTP gain holds only for this protocol.
    • There is no free-memory guard, and the rollback to C0 was not exercised.
    • MiMo-9B failed non-inferiority (lower bound −0.0560). B and X are runtime_unsupported on b11146.
  • decision stays conditional, and the manifest-bound checked_at stays 2026-09-19. The update is dated inside the row.

Evidence

  • evidence/receipts/local-inference-c2-serving-switch-20260926.json (kind native_model_e2e) satisfies the catalog validator's native_proven rule. results/decision.json is also cited.
  • scripts/validate_catalogs.py and scripts/validate.py passed. tests.test_catalogs: 47 OK. tests.test_host_receipts: 180 OK.
  • Cross-family review (GPT-6 gpt-6-astra, effort max, read-only, of a35316e): VERDICT: accept, with no findings. "Numbers, identifiers, pins, flags, and outcomes match the merged evidence", and the limitations "adequately bound native_proven and the production claim".

SOTA sources

🤖 Generated with Claude Code

#342 merged the preregistered li26 comparison and its native_model_e2e receipt.
The Qwen/Qwen3.8-27B row moves from source_review to native_proven for that one
host and task: micro-F1 0.9909 against filer-declared header items on the
360-filing 2020-03-02 8-K cohort, and the C2 serving profile now in production
(26.28 tokens/s median decode). Limitations keep the label source, protocol,
memory and rollback boundaries, and the MiMo and B/X outcomes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@seathatflowsinourveins seathatflowsinourveins added the lane:trading Trading lane: sim, paper and live north star label Sep 26, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@seathatflowsinourveins
seathatflowsinourveins merged commit 5af288d into main Sep 27, 2026
24 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/models-qwen-li26-20260926 branch September 27, 2026 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:trading Trading lane: sim, paper and live north star

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant