Add vally eval suite (copilot-sdk executor, real agent evals)

.github/workflows/evals.yml

@@ -0,0 +1,50 @@
+name: Evals
+
+on:
+  pull_request:
+    paths:
+      - 'evals/**'
+      - 'packages/squad-sdk/src/**'
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  validate:
+    name: Validate eval schemas
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+          cache: 'npm'
+      - run: npm ci
+      - run: npx vally lint --eval evals/
+
+  run-evals:
+    name: Run eval suites
+    runs-on: ubuntu-latest
+    needs: validate
+    if: github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, 'run-evals')
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+          cache: 'npm'
+      - run: npm ci
+      - name: Run evals
+        # Requires Copilot access on the repository. The copilot-sdk executor
+        # authenticates via GITHUB_TOKEN when running in GitHub Actions.
+        run: npx vally eval --eval-spec evals/routing/eval.yaml --eval-spec evals/skill-invocation/eval.yaml --eval-spec evals/task-completion/eval.yaml --output-dir results --junit
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+      - name: Upload results
+        uses: actions/upload-artifact@v4
+        with:
+          name: eval-results
+          path: results/
+        if: always()

evals/README.md

@@ -0,0 +1,103 @@
+# Squad Evals
+
+End-to-end evaluations for the Squad SDK using [@microsoft/vally](https://www.npmjs.com/package/@microsoft/vally-cli).
+
+## Philosophy
+
+These evals test **real agent behavior** — not SDK functions in isolation.
+Each eval sends real user prompts to a Copilot agent configured with Squad
+skills and fixtures, then grades the agent's actual responses using built-in
+vally graders.
+
+**What these evals are NOT:**
+- Unit tests of SDK functions (those belong in `packages/squad-sdk/test/`)
+- Reading comprehension tests ("given X.md, what should happen?")
+- Mock-based synthetic checks
+
+## Eval Suites
+
+| Suite | What it tests |
+|-------|---------------|
+| `routing` | Given real prompts, does the agent route to the correct squad member? |
+| `skill-invocation` | Given task prompts, does the agent apply the right skills? |
+| `task-completion` | Given development prompts, does the agent produce working solutions? |
+
+## Running Evals
+
+### Prerequisites
+
+- Node.js 22.5+
+- `@microsoft/vally-cli` installed (workspace devDependency)
+- Copilot SDK credentials (for `copilot-sdk` executor)
+
+### Run all evals
+
+```bash
+npx vally eval --eval-spec evals/routing/eval.yaml --eval-spec evals/skill-invocation/eval.yaml --eval-spec evals/task-completion/eval.yaml
+```
+
+### Run a single suite
+
+```bash
+npx vally eval --eval-spec evals/routing/eval.yaml
+npx vally eval --eval-spec evals/skill-invocation/eval.yaml
+npx vally eval --eval-spec evals/task-completion/eval.yaml
+```
+
+### Schema validation (no credentials needed)
+
+```bash
+npx vally lint --eval evals/
+```
+
+## Structure
+
+```
+evals/
+├── README.md
+├── routing/
+│   ├── eval.yaml           # Eval definition
+│   └── fixtures/           # Squad workspace fixtures
+│       ├── team.md
+│       ├── routing.md
+│       └── agents/         # Agent charters
+├── skill-invocation/
+│   ├── eval.yaml
+│   └── fixtures/
+│       ├── team.md
+│       ├── routing.md
+│       └── skills/         # Skill definitions (SKILL.md)
+└── task-completion/
+    ├── eval.yaml
+    └── fixtures/
+        ├── team.md
+        ├── routing.md
+        └── agents/
+```
+
+## Adding New Evals
+
+1. Create a directory under `evals/`
+2. Add an `eval.yaml` following the [vally eval schema](https://www.npmjs.com/package/@microsoft/vally)
+3. Add fixtures in a `fixtures/` subdirectory
+4. Use built-in graders: `output-matches`, `output-contains`, `skill-invocation`, `tool-call`, `prompt`, `pairwise`
+5. Write real user prompts — no "Given the rules in X.md" preambles
+
+## Grading
+
+| Grader | Use for |
+|--------|---------|
+| `output-matches` | Regex match on agent output (fast, deterministic) |
+| `output-contains` | Substring check (simpler than regex) |
+| `prompt` | LLM-as-judge using rubric criteria |
+| `pairwise` | Comparative grading between outputs |
+| `skill-invocation` | Check which skills were/weren't called |
+| `tool-call` | Check which tools were/weren't called |
+
+## CI
+
+Evals run in a dedicated workflow (`.github/workflows/evals.yml`) that:
+- Triggers on changes to `evals/` or `packages/squad-sdk/src/`
+- Requires Copilot SDK credentials (repository secret)
+- Reports scores as PR comments
+- Does NOT block merge — evals are advisory

evals/routing/eval.yaml

@@ -0,0 +1,101 @@
+name: squad-routing
+description: |
+  Evaluates Squad's work routing — given real user prompts, does the agent
+  route to the correct squad member based on routing.md rules?
+type: capability
+config:
+  timeout: 3m
+environment:
+  skills:
+    - ../../packages/squad-cli/src/cli/skills
+  files:
+    - src: fixtures/team.md
+      dest: .squad/team.md
+    - src: fixtures/routing.md
+      dest: .squad/routing.md
+    - src: fixtures/agents/lead/charter.md
+      dest: .squad/agents/lead/charter.md
+    - src: fixtures/agents/developer/charter.md
+      dest: .squad/agents/developer/charter.md
+    - src: fixtures/agents/tester/charter.md
+      dest: .squad/agents/tester/charter.md
+    - src: fixtures/agents/scribe/charter.md
+      dest: .squad/agents/scribe/charter.md
+scoring:
+  threshold: 0.7
+stimuli:
+  - name: Routes bug fix to developer
+    prompt: |
+      There's a null reference exception in the login handler when the
+      user's email field is missing. Can you fix it?
+    graders:
+      - type: output-matches
+        config:
+          pattern: (developer|Developer)
+      - type: prompt
+    rubric:
+      - The response identifies this as a bug fix task and routes to the Developer agent
+      - The response does not route to Lead, Tester, or Scribe for a simple bug fix
+
+  - name: Routes architecture review to lead
+    prompt: |
+      We need to redesign the event bus to support fan-out to multiple
+      consumers. Can you review the current architecture and propose changes?
+    graders:
+      - type: output-matches
+        config:
+          pattern: (lead|Lead)
+      - type: prompt
+    rubric:
+      - The response identifies this as an architecture/design task and routes to the Lead agent
+      - Architecture reviews and design decisions go to Lead, not Developer
+
+  - name: Routes test coverage request to tester
+    prompt: |
+      The auth module has zero test coverage. Write integration tests for
+      the token refresh flow and session expiration handling.
+    graders:
+      - type: output-matches
+        config:
+          pattern: (tester|Tester)
+      - type: prompt
+    rubric:
+      - The response identifies this as a testing task and routes to the Tester agent
+      - Test writing, coverage, and quality assurance tasks go to Tester
+
+  - name: Routes documentation update to scribe
+    prompt: |
+      The API reference is out of date — the /users endpoint now accepts
+      a pagination cursor. Update the docs to reflect the new parameter.
+    graders:
+      - type: output-matches
+        config:
+          pattern: (scribe|Scribe)
+      - type: prompt
+    rubric:
+      - The response identifies this as a documentation task and routes to the Scribe agent
+      - Documentation updates, README changes, and reference maintenance go to Scribe
+
+  - name: Routes new feature implementation to developer
+    prompt: |
+      Add webhook retry support with exponential backoff. When a webhook
+      delivery fails, retry up to 3 times with increasing delays.
+    graders:
+      - type: output-matches
+        config:
+          pattern: (developer|Developer)
+      - type: prompt
+    rubric:
+      - The response identifies this as a feature implementation task and routes to Developer
+      - New feature coding tasks go to Developer, not Lead (who handles design only)
+
+  - name: Handles ambiguous request with reasoning
+    prompt: |
+      The CI pipeline is failing on the main branch. The error seems related
+      to a flaky test in the payment module.
+    graders:
+      - type: prompt
+    rubric:
+      - The response acknowledges ambiguity — this could be a test fix (Tester) or a bug fix (Developer)
+      - The response provides reasoning for its routing decision
+      - The response does not silently pick one agent without explaining why

evals/routing/fixtures/agents/developer/charter.md

@@ -0,0 +1,16 @@
+# Developer Agent Charter
+
+## Role
+Feature implementation specialist handling bug fixes, new features, and code refactoring.
+
+## Responsibilities
+- Implement new features from specs
+- Fix bugs and handle error cases
+- Refactor code for maintainability
+- Write clean, tested, production-ready code
+
+## Expertise
+- TypeScript/JavaScript full-stack development
+- Node.js APIs and tooling
+- Git workflows and PR best practices
+- Performance optimization

evals/routing/fixtures/agents/lead/charter.md

@@ -0,0 +1,17 @@
+# Lead Agent Charter
+
+## Role
+Technical lead responsible for architecture decisions, design reviews, and team coordination.
+
+## Responsibilities
+- Review and approve architectural changes
+- Design system boundaries and module interfaces
+- Plan technical roadmap and sprint priorities
+- Set coding standards and patterns for the team
+- Triage ambiguous tasks to appropriate team members
+
+## Expertise
+- System architecture and design patterns
+- API design and interface contracts
+- Performance and scalability planning
+- Cross-cutting concerns (logging, auth, error handling)

evals/routing/fixtures/agents/scribe/charter.md

@@ -0,0 +1,16 @@
+# Scribe Agent Charter
+
+## Role
+Documentation specialist maintaining all written artifacts for the project.
+
+## Responsibilities
+- Update API references when endpoints change
+- Maintain README and getting-started guides
+- Write changelogs and release notes
+- Document architectural decisions (ADRs)
+
+## Expertise
+- Technical writing and documentation
+- API documentation standards (OpenAPI, JSDoc)
+- Markdown and documentation tooling
+- Developer experience (DX) writing

evals/routing/fixtures/agents/tester/charter.md

@@ -0,0 +1,16 @@
+# Tester Agent Charter
+
+## Role
+Quality assurance specialist responsible for test coverage, test strategy, and validation.
+
+## Responsibilities
+- Write unit and integration tests
+- Identify coverage gaps
+- Design test strategies for new features
+- Fix flaky tests and improve test reliability
+
+## Expertise
+- Vitest and testing frameworks
+- Test-driven development (TDD)
+- Mocking strategies and test doubles
+- CI/CD test pipeline optimization

evals/routing/fixtures/routing.md

@@ -0,0 +1,16 @@
+# Routing Rules
+
+## Routing Table
+
+| Work Type | Route To | Examples |
+|-----------|----------|----------|
+| architecture-review | Lead | System design, architecture decisions, module boundaries |
+| design-review | Lead | API design, interface design, pattern review |
+| planning | Lead | Sprint planning, roadmap, technical direction |
+| bug-fix | Developer | Null reference, crash fix, error handling |
+| feature-dev | Developer | New endpoint, new module, feature implementation |
+| refactoring | Developer | Code cleanup, extract function, rename |
+| test-writing | Tester | Unit tests, integration tests, test coverage |
+| quality-assurance | Tester | Test review, coverage gaps, flaky tests |
+| documentation | Scribe | README update, API docs, changelog |
+| api-reference | Scribe | Endpoint docs, parameter docs, examples |

evals/routing/fixtures/team.md

@@ -0,0 +1,16 @@
+# Team Roster
+
+## Squad: Eval Demo Team
+
+| Agent | Role | Capabilities |
+|-------|------|--------------|
+| Lead | lead | Architecture, design reviews, planning, code standards |
+| Developer | developer | Feature implementation, bug fixes, refactoring, code writing |
+| Tester | tester | Test writing, coverage analysis, QA, integration testing |
+| Scribe | scribe | Documentation, API references, READMEs, changelogs |
+
+## Governance
+
+- **Routing**: Work is assigned based on the routing table in routing.md
+- **Escalation**: Ambiguous tasks go to Lead for triage
+- **Reviews**: All PRs require at least one team member review