docs: add SWE-bench Lite results blog post (#032) - #1374
Closed
bradygaster wants to merge 1 commit into
Closed
Conversation
Adds Tamir Dresher's technical report on Squad v0.9.6 achieving #1 on SWE-bench Lite with a 66% resolve rate (198/300 issues). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
🟢 Impact Analysis — PR #1374Risk tier: 🟢 LOW 📊 Summary
🎯 Risk Factors
📦 Modules Affecteddocs (1 file)
This report is generated automatically for every PR. See #733 for details. |
Contributor
🛫 PR Readiness Check
PR Scope: 🔧 Infrastructure
|
| Status | Check | Details |
|---|---|---|
| ✅ | Single commit | 1 commit — clean history |
| ✅ | Not in draft | Ready for review |
| ✅ | Branch up to date | Up to date with dev |
| ❌ | Copilot review | No Copilot review yet — it may still be processing |
| ✅ | Changeset present | No source files changed — changeset not required |
| ✅ | Scope clean | No .squad/ or docs/proposals/ files |
| ✅ | No merge conflicts | No merge conflicts |
| ✅ | Copilot threads resolved | No Copilot review threads |
| ❌ | CI passing | 8 check(s) still running |
Files Changed (1 file, +260 −0)
| File | +/− |
|---|---|
docs/src/content/blog/032-swe-bench-lite-results.md |
+260 −0 |
Total: +260 −0
This check runs automatically on every push. Fix any ❌ items and push again.
See CONTRIBUTING.md and PR Requirements for details.
Owner
Author
|
Closing as duplicate of #1373 — Tamir already submitted this directly. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new published blog post documenting Squad v0.9.6’s reported SWE-bench Lite results and providing a technical report-style writeup (architecture, configuration, evaluation methodology, error analysis, and reproduction links).
Changes:
- Adds blog post
032-swe-bench-lite-results.mdwith results summary and per-repo breakdown. - Documents multi-agent architecture and runner configuration/methodology.
- Includes integrity checks, error analysis, and reproduction pointers.
Show a summary per file
| File | Description |
|---|---|
| docs/src/content/blog/032-swe-bench-lite-results.md | New blog post detailing SWE-bench Lite results and how the run was executed/reproduced |
Copilot's findings
- Files reviewed: 1/1 changed files
- Comments generated: 2
Comment on lines
+11
to
+13
| Squad v0.9.6 just completed its first official run on [SWE-bench Lite](https://www.swebench.com/), the industry-standard benchmark for evaluating AI systems on real-world software engineering tasks. The result: **198 out of 300 issues resolved (66.0%)**, which would place Squad at **#1 on the SWE-bench Lite leaderboard**. | ||
|
|
||
| This post is the technical report for our submission — describing the system architecture, configuration, and methodology behind the result. |
|
|
||
| ## What is SWE-bench? | ||
|
|
||
| SWE-bench Lite is a curated subset of 300 real GitHub issues from 12 popular open-source Python repositories (Django, Sympy, Matplotlib, Scikit-learn, etc.). Each task requires the system to: |
bradygaster
deleted the
bradygaster-squad-add-tamir-swe-bench-blog-post
branch
August 20, 2026 07:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Tamir Dresher's technical report on Squad v0.9.6 achieving #1 on SWE-bench Lite with a 66% resolve rate (198/300 issues).
Summary
Blog post
032-swe-bench-lite-results.mdcovers:Source:
tamirdresher_microsoft/squad-blog-draft(EMU repo)