Skip to content

docs(#6757): document credential context separation in install pkg - #6758

Merged
rh-hemartin merged 1 commit into
mainfrom
agent/6757-credential-context-docs
Sep 8, 2026
Merged

docs(#6757): document credential context separation in install pkg#6758
rh-hemartin merged 1 commit into
mainfrom
agent/6757-credential-context-docs

Conversation

@fullsend-ai-coder

Copy link
Copy Markdown
Contributor

Summary

Documents the credential context separation between the test suite's e2e installation token and dispatch's per-repo GITHUB_TOKEN in the pkg/behaviourtest/drivers/install package. This knowledge gap caused both a human developer (PR #6703) and an autonomous agent (PR #6709) to independently attempt the same incorrect suite-side GetCollaboratorPermission polling approach for #6701.

Related Issue

Closes #6757

Changes

  • Add doc.go with package-level documentation explaining the two-token model: the suite's e2e GitHub App installation token vs dispatch's per-repo GITHUB_TOKEN, their independent permission propagation graphs, and why suite-side permission probing is unreliable
  • Add inline note on awaitCreation in ensure.go warning against adding GetCollaboratorPermission polling, with a pointer to doc.go
  • Add "Credential context separation" paragraph in docs/guides/dev/behaviour-testing.md under the "Repo allocation" section, cross-referencing the package doc and issue behaviourtest: wait for permission propagation after pool repo recreate (dispatch sees no role, matrix empty) #6701

Testing

  • go build and go vet pass on the changed package
  • go test ./pkg/behaviourtest/drivers/install/... passes
  • gofmt produces no changes
  • lychee and lint-docs-links pass on the changed markdown
  • No production code changes — documentation only

Checklist

  • PR title follows Conventional Commits (correct type, ! for breaking changes)
  • No secrets or sensitive data in the diff

Closes #6757

Post-script verification

  • Branch is not main/master (agent/6757-credential-context-docs)
  • Secret scan passed (gitleaks — dc0f8e15488fc816f9464c63e644e17de549edd1..HEAD)
  • PR body secret scan passed (gitleaks — no-git)

The test suite's e2e installation token and dispatch's per-repo
GITHUB_TOKEN are distinct credential contexts with independent
permission propagation graphs. After pool repo recreation, the
suite can confirm the repo exists (GetRepo) but cannot observe
or predict when dispatch-side collaborator permissions are ready.
This was empirically validated in #6701: both a human (PR #6703)
and an agent (PR #6709) independently attempted suite-side
GetCollaboratorPermission polling and both failed.

Add a package-level doc comment (doc.go) explaining the two-token
model, an inline note on awaitCreation warning against permission
polling, and a cross-reference in the behaviour-testing guide.

Closes #6757
@github-actions

Copy link
Copy Markdown

Site preview

Preview: https://21773da4-site.fullsend-ai.workers.dev

Commit: 440cfd4d5655194be6326f00b806e695c4b88881

@fullsend-ai-review

fullsend-ai-review Bot commented Aug 29, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:29 PM UTC · Completed 6:43 PM UTC

Commit: 440cfd4 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $3.59

@codecov

codecov Bot commented Aug 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Aug 29, 2026
@fullsend-ai-review

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Small documentation-only PR by a bot adding package docs and guide entries with no protected paths, security concerns, or dependency changes; moderate churn history on touched files is the only uplift signal.

@fullsend-ai-review

Copy link
Copy Markdown

Looks good to me


Labels: PR adds documentation for e2e behaviour test credential context separation

@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge documentation component/e2e End-to-end tests labels Aug 29, 2026
@rh-hemartin
rh-hemartin added this pull request to the merge queue Sep 4, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 4, 2026
@rh-hemartin
rh-hemartin added this pull request to the merge queue Sep 8, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 8, 2026
@rh-hemartin
rh-hemartin added this pull request to the merge queue Sep 8, 2026
Merged via the queue into main with commit 7226c2e Sep 8, 2026
102 of 103 checks passed
@rh-hemartin
rh-hemartin deleted the agent/6757-credential-context-docs branch September 8, 2026 11:59
@fullsend-ai-retro

fullsend-ai-retro Bot commented Sep 8, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 12:01 PM UTC · Completed 12:08 PM UTC

Commit: 440cfd4 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $1.88

@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #6758 — Document credential context separation in install pkg

The workflow for PR #6758 was a clean end-to-end success demonstrating the full retro-to-merge pipeline.

Timeline

  1. Retro agent created issue #6757 identifying that both a human (PR #6703) and a code agent (PR #6709) independently attempted the same incorrect fix for #6701 due to undocumented credential context separation.
  2. Triage agent classified it as medium-severity documentation (cost: $0.62, run 33267606127).
  3. Code agent produced 55 lines of well-structured documentation across 3 files — a new doc.go, an inline comment on awaitCreation, and a guide section — in a single iteration with no rework (cost: $2.17, run 33267867307).
  4. Review agent approved with zero findings ("Looks good to me"), risk moderate (2/5), using opus at high effort (cost: $3.59, run 33268317345).
  5. Human reviewer (rh-hemartin) approved with no comments 6 days later.
  6. PR merged 10 days after creation.

Total agent cost: $6.38. Rework iterations: 0.

Assessment

This is the system working as designed — a retro agent identified a knowledge gap that caused repeated incorrect fixes, and the pipeline produced quality documentation to close that gap.

Evidence for existing issues

  • #6891 (pre-flight scope heuristic): This PR is a strong data point. The review agent used opus at high effort ($3.59) on a 55-line docs-only PR and produced zero findings. A scope heuristic detecting docs-only/trivially-scoped PRs could have routed this to a lighter model or lower effort, potentially reducing review cost by 3-5x.
  • #5289 (autonomy readiness for docs-only): This PR matches the same pattern — the review agent fully covered the human review (both approved, zero inline comments from either). Human review added no incremental signal but added 6 days of latency.

No new proposals

The improvement opportunities identified in this retro are already tracked by existing open issues. No new proposals are warranted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/e2e End-to-end tests documentation ready-for-merge All reviewers approved — ready to merge ready-for-review Triggers review agent dispatch risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Document credential context separation in behaviourtest pool repo lifecycle

1 participant