Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .claude/skills/ironclaw-reborn-testing/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
---
name: ironclaw-reborn-testing
description: Use when adding or reviewing tests for Reborn behavior — choosing a test tier, covering a bug fix, testing model/tool-choice behavior, touching tests/support/reborn or tests/fixtures/llm_traces, or when a test needs Postgres, Docker, or a live LLM.
description: Use when adding or reviewing tests for Reborn behavior — choosing a test tier, covering a bug fix, testing model/tool-choice behavior, touching tests/integration or tests/fixtures/llm_traces, or when a test needs Postgres, Docker, or a live LLM.
---

# Reborn Testing

Pick the tier first; everything else follows. The repo's tier knowledge lives in `tests/support/reborn/CLAUDE.md` (396 lines — read it before writing harness tests); this skill is the decision layer plus the traps.
Pick the tier first; everything else follows. The repo's tier knowledge lives in `tests/integration/CLAUDE.md` (read it before writing harness tests); this skill is the decision layer plus the traps.

## Tier decision tree

1. **Pure logic, no gated side effect** → unit test in the crate (`mod tests` / crate `tests/`).
2. **A helper gates a side effect (HTTP, DB write, egress body, approval, dispatch)** → you also need a caller-path test driving the real entry point (`*_handler`, facade method, adapter, coordinator). Helper-only is insufficient — `.claude/rules/testing.md` has the bug catalog. Gold standard: `tests/reborn_group_approvals/scenario_gate_then_approve.rs` asserts the approved write **exists on disk**.
3. **Whole-turn Reborn behavior (submit → runner → loop → reply), deterministic** → the in-process scripted-model harness (`tests/support/reborn/`, run as `cargo test --test reborn_<name>`, zero setup, offline). **Mock only at the vendor-SDK seam** (`TraceLlm`): the real `ironclaw_llm` decorator chain (retry/failover/circuit-breaker) must execute. Mocking at the gateway seam skips it — that's the gateway-seam replay tier's job (`RebornBinaryE2EHarness` / `RebornTraceReplayModelGateway`), not yours by default.
2. **A helper gates a side effect (HTTP, DB write, egress body, approval, dispatch)** → you also need a caller-path test driving the real entry point (`*_handler`, facade method, adapter, coordinator). Helper-only is insufficient — `.claude/rules/testing.md` has the bug catalog. Gold standard: `tests/integration/group_approvals/scenario_gate_then_approve.rs` asserts the approved write **exists on disk**.
3. **Whole-turn Reborn behavior (submit → runner → loop → reply), deterministic** → the in-process scripted-model harness (`tests/integration/`, run as `cargo test --test reborn_integration_<name>`, zero setup, offline). **Mock only at the vendor-SDK seam** (`TraceLlm`): the real `ironclaw_llm` decorator chain (retry/failover/circuit-breaker) must execute. Mocking at the gateway seam skips it — that's the gateway-seam replay tier's job (`RebornBinaryE2EHarness` / `RebornTraceReplayModelGateway`), not yours by default.
4. **Model tool-choice / request-shape is the behavior under test** → recorded QA fixtures (`tests/fixtures/llm_traces/reborn_qa/` + `tests/reborn_qa_recorded_behavior.rs`: ignored live recorder → hermetic contract assertions → hermetic replay). Fixtures must pass `scripts/ci/check-reborn-qa-fixtures.sh` (secret/PII scrub). Never commit unscrubbed traces.
5. **Browser-visible** → `tests/e2e/` Playwright (`reborn_v2_*` fixtures for WebChat v2). **Live LLM** → `#[ignore]` canary tier; supplemental only, never the PR gate.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ Living companions to the tier tree in `../SKILL.md`. Each exemplar is a real in-

## 1. Side-effect proof at the caller

`tests/reborn_group_approvals/scenario_gate_then_approve.rs` — drives a scripted `builtin.write_file` through the **real** stack: first-party runtime → `PermissionMode::Ask` → `TurnStatus::BlockedApproval` → real `ApprovalResolver::approve_dispatch` (lease issued) → `coordinator.resume_turn` → `Completed` — and then **asserts the file exists on disk**. The assertion target is the side effect itself, not a mock's call count. When your change gates a side effect, this is the shape: drive the public entry point, assert the world changed. Re-verify: `ls tests/reborn_group_approvals/`.
`tests/integration/group_approvals/scenario_gate_then_approve.rs` — drives a scripted `builtin.write_file` through the **real** stack: first-party runtime → `PermissionMode::Ask` → `TurnStatus::BlockedApproval` → real `ApprovalResolver::approve_dispatch` (lease issued) → `coordinator.resume_turn` → `Completed` — and then **asserts the file exists on disk**. The assertion target is the side effect itself, not a mock's call count. When your change gates a side effect, this is the shape: drive the public entry point, assert the world changed. Re-verify: `ls tests/integration/group_approvals/`.

## 2. The scripted-model harness seam

The in-process harness (`tests/support/reborn/`, spec in its `CLAUDE.md`) fakes exactly one thing: the vendor SDK at the bottom (`TraceLlm`). Everything else — product workflow, coordinator, scheduler, agent loop, the real `ironclaw_llm` retry/failover/circuit-breaker chain — executes for real, and assertions read *persisted state* (filesystem, thread history), never internals.
The in-process harness (code in `tests/integration/support/`, spec in `tests/integration/CLAUDE.md`) fakes exactly one thing: the vendor SDK at the bottom (`TraceLlm`). Everything else — product workflow, coordinator, scheduler, agent loop, the real `ironclaw_llm` retry/failover/circuit-breaker chain — executes for real, and assertions read *persisted state* (filesystem, thread history), never internals.

- **Right**: mock at the vendor-SDK seam; assert from durable state; `cargo test --test reborn_<name>` runs offline with zero setup.
- **Right**: mock at the vendor-SDK seam; assert from durable state; `cargo test --test reborn_integration_<name>` runs offline with zero setup.
- **Wrong**: mocking at the gateway seam (skips the whole `ironclaw_llm` chain — that's the separate binary-replay tier's job); hand-building `TraceStep`s; asserting on internal structs.

## 3. The declare/enforce contract pair
Expand Down
2 changes: 1 addition & 1 deletion .claude/skills/reborn-feature/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ For a feature with N endpoints, expect to touch (in dependency order):
| HTTP | `ironclaw_webui_v2` | route constants + pattern + `*_descriptor()` (use `read_policy`/`mutation_policy`) + add to `webui_v2_routes()`; thin handler over `state.services()`; mount in `router.rs`; **update `tests/webui_v2_descriptors_contract.rs`** (it locks the table) |
| Wiring | `ironclaw_reborn_composition` + `ironclaw_reborn_cli` | thread inputs through `RebornRuntimeInput`/`RebornRuntime`; attach in `build_webui_services`; pass from `serve.rs` |
| Frontend | `ironclaw_webui_v2_static` | call endpoints via `apiFetch` in `static/js/pages/*/lib/*-api.js`; consume in hooks. No build step — `node --check <file>.js` to syntax-check |
| Tests | `tests/support/reborn/` + crate tests | for whole-turn behavior, add a scripted-model harness case (see `tests/support/reborn/CLAUDE.md` — mock only at the vendor-SDK seam); facade changes extend `crates/ironclaw_product_workflow/tests/reborn_services_contract.rs`, and handler changes extend `crates/ironclaw_webui_v2/tests/webui_v2_handlers_contract.rs` |
| Tests | `tests/integration/` + crate tests | for whole-turn behavior, add a scripted-model harness case (see `tests/integration/CLAUDE.md` — mock only at the vendor-SDK seam); facade changes extend `crates/ironclaw_product_workflow/tests/reborn_services_contract.rs`, and handler changes extend `crates/ironclaw_webui_v2/tests/webui_v2_handlers_contract.rs` |

## Boundary rules (the guardrails that will reject your PR)

Expand Down
153 changes: 0 additions & 153 deletions .github/workflows/reborn-coverage.yml

This file was deleted.

Loading
Loading