docs: SWE-bench Lite technical report — 198/300 (66.0%) #1 on leaderboard

[tamirdresher]

SWE-bench Lite Technical Report

Squad v0.9.6 achieved 198/300 (66.0%) on SWE-bench Lite — #1 on the leaderboard (current top: Claude Opus 4.6 at 62.7%).

This post is the required technical report for the leaderboard submission, covering:

• System architecture (coordinator + code expert multi-agent)
• Full methodology (pass@1, no test knowledge, no web browsing)
• Results breakdown by repository
• Error analysis
• Integrity verification (12-point check)
• Reproduction instructions

Checklist

• Follows existing blog post format (YAML frontmatter + markdown)
• Numbered correctly (033)
• All claims verified against eval report data
• No unverifiable claims

[github-actions]

🛫 PR Readiness Check

ℹ️ This comment updates on each push. Last checked: commit fa0eea3

PR Scope: 🔧 Infrastructure

⚠️ 3 item(s) to address before review

Status	Check	Details
❌	Single commit	3 commits — consider squashing before review
✅	Not in draft	Ready for review
✅	Branch up to date	Up to date with main
❌	Copilot review	No Copilot review yet — it may still be processing
✅	Changeset present	No source files changed — changeset not required
✅	Scope clean	No .squad/ or docs/proposals/ files
✅	No merge conflicts	No merge conflicts
✅	Copilot threads resolved	No Copilot review threads
❌	CI passing	6 check(s) failing: samples-build, test, docs-quality, sdk-exports-validation, Policy Gates, changes

Files Changed (1 file, +285 −0)

File	+/−
docs/src/content/blog/033-swe-bench-lite-results.md	+285 −0

Total: +285 −0

---

This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.

[bradygaster]

Hey @tamirdresher_microsoft — great post! 🎉 I did a consistency review and found two things worth addressing before we merge:

1. Timeout count discrepancy (draft vs this PR)

Your earlier draft (in the EMU repo) states 31/300 (10.3%) hit the 30-minute timeout, but this PR version says Timeout Cases (20/300, 6.7%). If this was an intentional reclassification (e.g., some of those 31 produced partial patches that fall into the 'apply error' bucket), it'd be worth a brief note explaining the change. Otherwise one of the numbers needs correcting.

2. Wording inconsistency within the PR

In Key Parameters you write:

Of the 20 tasks that produced no patch, most were timeout-related

But then in Error Analysis you have:

Timeout Cases (20/300, 6.7%) — The 20 'no generation' instances are primarily tasks that exceeded the 30-minute timeout.

The first implies fewer-than-20 timeouts; the second equates all 20 with timeouts. Suggest aligning these — either 'all' or give the actual timeout count vs other causes.

---

Still verifying the leaderboard scores against public sources — will follow up if anything surfaces there. Everything else (per-repo sums, category partitioning, percentages) checks out perfectly. 🧮✅

[tamirdresher]

Thanks Brady! Pushed a fix:

1. Timeout reconciliation —
   un_metadata.json\ shows 31 timeouts total. Reconciled: 20 produced no patch (the 'no generation' bucket), 11 produced partial patches that then failed to apply (subset of the 38 'patch apply errors'). Updated both Key Parameters and Error Analysis to state this consistently.

2. Wording aligned — Removed the 'most were timeout-related' hedging. Now states the actual breakdown clearly.

3. Added a Limitations & Attribution section — Acknowledges that GPT-4o single-agent scores ~40-45% vs Squad's 66%, so the model contributes but orchestration adds ~20+ points. Commits to a single-agent ablation as follow-up.

4. Linked public data repo — All raw data (predictions, eval report, 1284 worker logs, runner code) at tamirdresher/squad-swe-bench for full reproduction and verification.

[bradygaster]

Follow-up: External Fact-Check Results 🔍

I've verified the post's claims against public sources. The math is solid internally, but there are factual issues with the leaderboard comparison table and benchmark description that need attention before publishing:

❌ Leaderboard Competitor Names

Claimed System	Issue
Claude Opus 4.6	This model does not exist. Anthropic's naming is 'Claude 4.5 Opus', 'Claude Opus 4.1', etc. — there is no '4.6' release.
MiniMax M2.5	Does not exist. MiniMax's current model is MiniMax-M1 (56.0% on SWE-bench Verified). The '56.3%' score is suspiciously close to M1's real score with a fabricated version number.

Claude Haiku 4.5 and GPT-5 are real models, but their specific Lite scores couldn't be independently confirmed.

⚠️ Score Plausibility

Claude 4.5 Opus already scores 76.8% on SWE-bench Verified (confirmed from live leaderboard data at swebench.com). If Squad is using gpt-4o as its base model and scoring 66% on the easier Lite benchmark, claiming #1 ahead of frontier models that score 70-77% on the harder Verified benchmark needs very careful framing.

❌ '12 repositories' → should be 11

SWE-bench Lite officially covers 11 of the 12 full SWE-bench repos (confirmed at swebench.com/lite.html and HuggingFace dataset page). The post lists 12 repos in the per-repo breakdown — one of those repos (likely seaborn or flask based on instance counts) shouldn't be there if this is truly Lite.

However — the per-repo table shows data for all 12, which means either:

1. The run was actually against the full 12-repo set (not standard Lite), or
2. The post's '12 repos' framing is correct for this specific run and the intro text should just avoid calling it the standard Lite set

✅ Confirmed Accurate

• 300 instances ✅
• Evaluation methodology (Docker + \swebench.harness.run_evaluation) ✅
• Pass@1 / no test knowledge / no hints — standard and correctly described ✅

---

Recommendation: The leaderboard comparison table needs correction before publishing — either verify the exact model names/scores against the live leaderboard, or remove the comparison table and just report Squad's own score. The '12 vs 11 repos' question also needs a quick check against what was actually in the predictions.json.

[tamirdresher]

Re: External Fact-Check Results

Thanks for the thorough review! Here's my response with sources:

✅ Model Names — Confirmed Correct

I verified against two independent leaderboard aggregators:

1. pricepertoken.com/leaderboards/benchmark/swe-bench-lite — Shows Claude Opus 4.6 at 62.7%, MiniMax M2.5 at 56.3%, GPT-5 at 54.3%, Claude Haiku 4.5 at 54.3%
2. llmrun.dev/benchmark/swe-bench-lite — Same scores confirmed

Screenshot of the live leaderboard (June 23, 2026):

⚠️ 12 vs 11 Repos — Our Data Has 12

Our run loaded princeton-nlp/SWE-bench_Lite from HuggingFace (300 instances, test split). The dataset itself contains instances from 12 repos:

astropy: 6, django: 114, matplotlib: 23, mwaskom/seaborn: 4, pallets/flask: 3, psf/requests: 6, pydata/xarray: 5, pylint-dev: 6, pytest-dev: 17, scikit-learn: 23, sphinx-doc: 16, sympy: 77

This is what the official dataset provides. If the documentation says "11 repos" but the actual dataset has 12, that's a docs discrepancy upstream — not an issue with our run. I'll add a footnote acknowledging this.

❌ Verified vs Lite — Apples to Oranges

The 76.8% score you cite is from SWE-bench Verified (500 harder instances, different benchmark). Our #1 claim is specifically about the Lite leaderboard. The comparison table in the post compares only Lite scores.

---

All raw data is now publicly available at tamirdresher/squad-swe-bench — predictions, eval report, 1284 worker logs, runner code. Anyone can re-run the evaluation to verify.

[bradygaster]

Tamir addressed all fact-check items with sources. LGTM — nice work on the report! 🚀

[bradygaster]

@dependabot rebase