Skip to content

docs(#5670): add fixture authoring section to behaviour-testing guide - #5673

Merged
rh-hemartin merged 1 commit into
mainfrom
agent/5670-fixture-authoring-docs
Jul 30, 2026
Merged

docs(#5670): add fixture authoring section to behaviour-testing guide#5673
rh-hemartin merged 1 commit into
mainfrom
agent/5670-fixture-authoring-docs

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

  • Adds a "Fixture authoring" section to docs/guides/dev/behaviour-testing.md documenting that every BT scenario dispatching an agent stage must include a write_fixture row emitting output/agent-result.json with schema-valid content
  • Includes a 5-step checklist for new scenarios, a fixture inventory table of existing fixtures and their target schemas, guidance on downstream CLI validation (e.g. full-length SHAs), and a copy-and-adapt example from the fork-bash-routing scenario

Context

The code agent's first attempt on PR #5654 omitted the write_fixture row for agent-result.json, causing a harness validation failure. The existing behaviour-testing guide documented the write_fixture syntax but did not explain the requirement for output/agent-result.json or which schemas fixtures must conform to. This section fills that gap for both agents and human contributors.

Testing

  • Documentation-only change — no code modified
  • Verified fixture inventory table matches actual files under e2e/behaviour/fixtures/
  • Verified AGENTS.md already indexes the behaviour-testing guide (no index update needed)
  • Verified VitePress sidebar config already has a manual entry for this page

Closes #5670

Post-script verification

  • Branch is not main/master (agent/5670-fixture-authoring-docs)
  • Secret scan passed (gitleaks — a3dd597c77dc90dc183f844b5aa9badea57a7bd4..HEAD)
  • PR body secret scan passed (gitleaks — no-git)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

Add a "Fixture authoring" section documenting the requirement that
every BT scenario dispatching an agent stage must include a
write_fixture row emitting output/agent-result.json with content
conforming to the stage's result schema. The section includes:

- A checklist for new scenarios (identify role, create fixture,
  add write_fixture row, add assertion, verify downstream
  validation)
- A fixture inventory table listing existing fixtures and their
  target schemas
- Guidance on downstream CLI validation beyond JSON schema (e.g.
  full-length hex SHAs)
- A copy-and-adapt example using the fork-bash-routing scenario

Note: pre-commit could not run in sandbox (network restriction).
The post-script runs pre-commit authoritatively on the runner.

Closes #5670
@fullsend-ai-coder
fullsend-ai-coder Bot requested a review from a team as a code owner July 28, 2026 12:00
@fullsend-ai-coder fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Jul 28, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 28, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 12:02 PM UTC · Completed 12:13 PM UTC
Commit: 32d1d32 · View workflow run →

@github-actions

Copy link
Copy Markdown

Site preview

Preview: https://29c9fc07-site.fullsend-ai.workers.dev

Commit: 32d1d32888a4333a8dc89f82f8eff525a45da384

@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

Copy link
Copy Markdown

Review

Findings

Low

  • [technical accuracy] docs/guides/dev/behaviour-testing.md:93 — The doc states head_sha "must be a full-length 40-character hex SHA" but the downstream CLI validation (internal/cli/postreview.go:31) accepts both 40 and 64 character hex strings: ^[0-9a-fA-F]{40}$|^[0-9a-fA-F]{64}$. The 40-char guidance is practical (SHA-1 is standard on GitHub) and the example value is correct, but the stated constraint is technically incomplete — it omits SHA-256 (64-char) as an alternative.

Labels: PR adds documentation for behaviour test fixture authoring, touching e2e testing guidance

Comment thread docs/guides/dev/behaviour-testing.md
@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge component/docs User-facing documentation component/e2e End-to-end tests labels Jul 28, 2026
@rh-hemartin
rh-hemartin added this pull request to the merge queue Jul 30, 2026
Merged via the queue into main with commit eeabe26 Jul 30, 2026
34 of 35 checks passed
@rh-hemartin
rh-hemartin deleted the agent/5670-fixture-authoring-docs branch July 30, 2026 07:07
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jul 30, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 7:10 AM UTC · Completed 7:24 AM UTC
Commit: 32d1d32 · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #5673 — docs(#5670): add fixture authoring section to behaviour-testing guide

Workflow overview

This PR was the product of a fully autonomous retro→triage→code→review pipeline. The retro agent on PR fullsend-ai/fullsend#5654 identified a documentation gap that caused a first-attempt BT failure (missing write_fixture row for agent-result.json), and filed issue fullsend-ai/fullsend#5670. The pipeline completed the full cycle in ~24 minutes of agent time with zero rework iterations:

Stage Run Duration Outcome
Triage 30356310757 ~5 min Classified as docs enhancement, added ready-to-code
Code 30356640064 ~7 min Created PR with 56 lines of documentation (single file)
Review 30357165319 ~12 min APPROVED with 1 low-severity finding
Human rh-hemartin ~1.8 days APPROVED (no comments)

Cost: $2.02 for the code agent session. The code agent spent 3m11s in the Claude session (45% of wall time); the remaining 55% was infrastructure overhead (sandbox creation, Go build from source, post-script).

What went well

  • Zero rework. The code agent produced clean, well-structured documentation on the first attempt — checklist, fixture inventory table, validation guidance, and a copy-and-adapt example.
  • Accurate review finding. The review agent correctly identified that the documentation's SHA constraint ("must be a full-length 40-character hex SHA") was technically incomplete — the CLI validation at internal/cli/postreview.go:31 also accepts 64-character SHA-256 hashes. The agent appropriately rated this as low severity.
  • Cross-validation with sibling issue. The review agent's SHA finding independently corroborates issue fullsend-ai/agents#621 (filed by the same retro run), which proposes tightening the schema to match the CLI's regex.

Evidence for existing issues (no new proposals needed)

  • #1286 (auto-file tracking issues for unresolved low-severity review findings): PR docs(#5670): add fixture authoring section to behaviour-testing guide #5673 merged with the review agent's valid SHA-256 finding unaddressed — another data point for this proposal.
  • #5669 (tighten head_sha validation in review-result.schema.json): The review agent's finding on this PR is independent confirmation that the 40-char-only documentation is incomplete relative to the CLI's actual validation.
  • #427 in fullsend-ai/agents (track docs-only autonomy evidence): This PR is another clean data point — docs-only single-file change, review agent approved, human approved without comments, zero rework.
  • #1475 (fast-track triage for retro-filed issues): The triage agent spent ~5 minutes largely restating the retro agent's already well-structured issue body. Pre-triaged issues from the retro agent could skip or fast-path this step.

Conclusion

This workflow operated cleanly end-to-end. The retro→triage→code→review pipeline produced a correct docs change on the first attempt with no human intervention before the final approval. All observations are covered by existing open issues — no new proposals filed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/docs User-facing documentation component/e2e End-to-end tests ready-for-merge All reviewers approved — ready to merge ready-for-review Triggers review agent dispatch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add fixture authoring checklist to behaviour-testing.md

1 participant