Skip to content

docs(testing): document epic confidence workflows #6928

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
serrrfirat merged 1 commit into from
Jul 30, 2026
Merged

docs(testing): document epic confidence workflows #6928

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
147 changes: 141 additions & 6 deletions docs/internal/testing-playbook.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,35 @@ Run:

This keeps the development loop fast while preserving outside-in coverage.

## How local tests map to CI

The test types above describe *what evidence to add*. CI lanes describe *when
that evidence runs*. Keep those decisions separate: a browser test does not
become a live canary because it runs nightly, and a deterministic provider
journey remains hermetic even when the deep lane repeats it in more orders.

| Lane | Purpose | What engineers should expect |
| --- | --- | --- |
| Pull request feedback | Fast, scoped signal on the proposed change | Relevant deterministic subsets, evidence gates, and changed-code checks may run before review. |
| Merge queue | Production gate on the merged result | Queue-covered deterministic checks run in the same shape required before `main`; path scope is computed by workflow jobs rather than trigger filters. |
| Push to `main` | Confirm the queue result and publish shared evidence | Repeats queue-covered checks, runs the documented post-merge-only Windows, benchmark, and legacy snapshot checks, warms caches, and publishes reports. |
| Deep scheduled | Exercise expensive breadth | Higher property-test counts, reversed and isolated journey order, mutation audits, browser shards, stress, soak, and live drift run outside the ordinary merge gate. |
| Release artifact | Verify what will actually ship | Smoke the exact built archive or binary rather than treating a development build as release evidence. |

The authoritative workflow contract, required-check names, and scheduling
details live in [`.github/workflows/README.md`](../../.github/workflows/README.md).
Its "Known accepted gaps" section names deterministic and informational checks
that are deliberately not merge-gating. Re-derive the current lanes before
changing CI:

```bash
rg -n "pull_request:|merge_group:|push:|schedule:|workflow_dispatch:" \
.github/workflows
```
Comment on lines +180 to +189

@coderabbitai coderabbitai Bot Jul 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

cd docs
mint dev
mint broken-links

Repository: nearai/ironclaw

Length of output: 233

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'Repo files matching CLAUDE/AGENTS/rules:\n'
git ls-files | rg '(^|/)(CLAUDE\.md|AGENTS\.md|\.claude/)' || true

printf '\nRelevant guideline lines:\n'
for f in CLAUDE.md AGENTS.md .claude/rules; do
  [ -f "$f" ] && { printf '\n-- %s --\n' "$f"; rg -n "docs/\*\*|\*/testing-playbook|mint dev|mint broken-links" "$f" || true; }
done

printf '\nChanged/internal doc file reference:\n'
git diff --name-only 2>/dev/null | rg '^docs/internal/testing-playbook\.md$' || true

Repository: nearai/ironclaw

Length of output: 5407

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'AGENTS.md doc guideline section:\n'
nl -ba AGENTS.md | sed -n '1,120p'

printf '\nCLAUDE.md doc guideline section:\n'
nl -ba CLAUDE.md | sed -n '1,160p'

printf '\nCitation snippets:\n'
rg -n "docs/internal/|docs/\*\*|Test documentation changes|mint dev|mint broken-links|broken-links" AGENTS.md CLAUDE.md .claude/rules -S || true

Repository: nearai/ironclaw

Length of output: 226

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'AGENTS.md doc guideline section:\n'
awk '{printf "%6d\t%s\n", NR, $0}' AGENTS.md | sed -n '1,120p'

printf '\nCLAUDE.md doc guideline section:\n'
awk '{printf "%6d\t%s\n", NR, $0}' CLAUDE.md | sed -n '1,160p'

printf '\nCitation snippets:\n'
rg -n "docs/internal/|docs/\*\*|Test documentation changes|mint dev|mint broken-links|broken-links" AGENTS.md CLAUDE.md .claude/rules -S || true

Repository: nearai/ironclaw

Length of output: 19298

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'AGENTS.md doc guideline lines 150-190:\n'
awk '{printf "%6d\t%s\n", NR, $0}' AGENTS.md | sed -n '150,190p'

printf '\nMint-related files:\n'
git ls-files | rg '(^|/)mint(\.lock|\.json|mintlify\.json)?$|docs/package\.json|docs/mint' || true

Repository: nearai/ironclaw

Length of output: 2714

Run the required docs/mint validation.

docs/**/* requires mint dev and mint broken-links run from docs/. The repository currently has no docs/mint project/config, so this path has failing guardrail contract. Either add the Mint setup or replace this guidance with the actual docs validation command.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/internal/testing-playbook.md` around lines 180 - 189, Update the docs
validation guidance in the testing playbook: remove the required docs/mint
validation unless a valid docs/mint project/config is added, and document the
repository’s actual docs validation command instead. Ensure the instructions
specify running the command from the correct directory and align with the
authoritative workflow contract.

Source: Coding guidelines


A green pull request does not imply that scheduled, live, backend, or release
tiers ran. Name the tiers actually exercised in the pull request test card.

## Where tests belong

| Behavior | Location | Notes |
Expand Down Expand Up @@ -519,17 +548,107 @@ representative compositions across ingress, execution, provider state, and
delivery. A `JourneyCase` is evidence metadata, not a workflow DSL: execution
logic, provider setup, and readback remain in their owning test modules.

Each case names its trace when it has one, isolated provider worlds, ingress,
execution lane, delivery target, observable assertions, and an exact Pytest or
Cargo test declaration. `test_journey_coverage.py` verifies that the evidence
still exists and is executable. It also derives inbound and outbound channel
surfaces from shipped first-party manifests, so adding a production channel
without representative journey evidence fails CI.
Each case names isolated provider worlds, ingress, execution lane, delivery
target, observable assertions, and an exact Pytest or Cargo declaration.
Provider journeys also bind their recorded trace, replay facts, and scheduled
live evidence. Product journeys may bind exact delivery addresses and browser
evidence when the cited test proves those claims. Do not populate an optional
field from intent: name it only when the referenced test asserts it.

`test_journey_coverage.py` verifies that the evidence still exists and is
executable. It also derives inbound and outbound channel surfaces from shipped
first-party manifests, so adding a production channel without representative
journey evidence fails CI.

Prefer one representative whole-path case per supported ingress and delivery
mechanism. Do not multiply every provider operation by every ingress or move
provider-specific assertions into the generic registry.

Run the registry gate directly after adding or changing a journey:

```bash
cd tests/e2e
pytest scenarios/test_journey_coverage.py -q
```

## Generated lifecycle and interaction coverage

`tests/e2e/state_machine_coverage.py` projects journeys, provider operations,
provider faults, and focused Reborn integration tests onto the lifecycle
dimensions that matter across boundaries. It is an evidence inventory, not a
second runtime state machine and not a Cartesian-product test generator.

Update it when production adds or changes a supported:

- ingress, authentication, policy, operation, provider-outcome, lifecycle, or
delivery class;
- trigger, retry, cancellation, duplicate, restart, or concurrent-submit
sequence;
- terminal-stability, at-most-once-effect, actor-isolation,
truthful-uncertainty, or no-orphan-resource invariant;
- high-risk interaction between two of those dimensions.

Prefer projecting an existing `JourneyCase`, `ProviderOperationCase`, or
provider fault. Add a focused row only when no existing registry owns the
evidence, and cite the exact executable Pytest or Cargo declaration. Add a
required pair only for an interaction whose combined behavior carries more
risk than either dimension alone.

Re-derive the supported dimensions, sequences, invariants, and selected pairs,
then run the fail-loud gate:

```bash
rg -n \
"SUPPORTED_DIMENSIONS|REQUIRED_EQUIVALENCE_PAIRS|SequenceClass|StateMachineInvariant" \
tests/e2e/state_machine_coverage.py
cd tests/e2e
pytest scenarios/test_state_machine_coverage.py -q
```

## Promoting failures into regression tests

When a production, live-canary, or QA failure is reproducible, promote it to
the lowest deterministic seam that proves the broken rule:

1. State the user-visible failure and the expected outcome.
2. Remove credentials, personal data, provider identifiers, and irrelevant
transcript content before committing any fixture.
3. Reproduce the failure with a unit/contract, Reborn integration, recorded
fixture, provider operation, journey, or browser test.
4. Confirm the regression test fails for the original reason.
5. Apply the fix and confirm the same test passes.
6. Keep a live canary only when real model or provider drift remains a distinct
risk.

The commit hook and
[`regression-test-check.yml`](../../.github/workflows/regression-test-check.yml)
require test changes for conventionally named fixes and selected high-risk
paths. `[skip-regression-check]` and the matching label are review-visible
exceptions for genuinely infeasible cases, not substitutes for a reproducible
test. Explain the missing deterministic seam and compensating evidence in the
pull request test card.

## Mutation audits test the assertions

Coverage proves that code ran; it does not prove that a test would detect the
wrong result. Use a mutation audit when a critical invariant, escaped defect,
or suspiciously broad coverage needs assertion-strength evidence.

Start with one file or package, triage every viable survivor, and verify a
`real-gap` fix against both unmodified and sabotaged code:

```bash
./scripts/mutation-audit.sh -p OWNING_CRATE path/to/production.rs
./scripts/mutation-verify-fix.sh -p OWNING_CRATE \
'copy the exact mutant string from the triage queue'
```

Do not optimize a workspace mutation score. Equivalent mutants and unclear
product contracts are explicit outcomes, and broad mutation work belongs in
the scheduled frontier. Follow
[`docs/internal/mutation-audit.md`](mutation-audit.md) for environment
isolation, verdicts, acceptance criteria, and the fail-loud self-test.

## Product-surface coverage report

The Reborn E2E lane publishes `product-surface-coverage-<sha>` as JSON and
Expand All @@ -546,3 +665,19 @@ silently become passing evidence. A harvested live-QA fixture is not current
live evidence unless a stable live result artifact binds back to its typed row.
Scheduled live cells name the exact workflow, job, case id, and result artifact
and remain `scheduled` until a consumer inspects that result.

Generate the same bird's-eye view locally instead of maintaining a separate
capability or journey spreadsheet:

```bash
cd tests/e2e
python product_surface_coverage.py \
--json ../../artifacts/product-surface-coverage/matrix.json \
--markdown ../../artifacts/product-surface-coverage/matrix.md
```

Open `artifacts/product-surface-coverage/matrix.md` for the human-readable
matrix. In GitHub Actions, download the
`product-surface-coverage-<source-commit>` artifact from the Reborn E2E run.
External dashboards or Notion pages may summarize or link this report, but the
typed registries and generated matrix remain authoritative.
Loading