docs(#138): track mechanical-consistency autonomy class for review agent

[fullsend-ai-coder]

Add evidence-driven autonomy classes to the autonomy-spectrum problem doc, introducing a framework for granting review agents higher authority on narrow, well-defined categories of findings backed by empirical evidence. The first class is mechanical consistency in documentation repos (typos, numbering gaps, cross-reference errors, scope coherence).

Add tracking data and validation plan to applied/konflux-ci/ based on evidence from konflux-ci/architecture PR #367, where the review agent caught 2 medium-severity mechanical consistency issues that 5 human reviewers missed. Both merged without fix because findings were posted at COMMENT level.

Add cross-reference in code-review.md open questions linking to the autonomy class evidence.

Note: pre-commit could not run in the sandbox due to a shellcheck network policy error (infrastructure issue, not code-related). Markdown link checker (lychee) passed with 0 errors across 1473 links.

---

Closes #138

Post-script verification

• Branch is not main/master (agent/138-doc-review-autonomy-class)
• Secret scan passed (gitleaks — 6757be26681aa98414a535299f9c279931d88c7e..HEAD)
• Pre-commit hooks passed (authoritative run on runner)
• Tests ran inside sandbox

[guyoron1]

/fs-review

[fullsend-ai-review]

Review — comment

Documentation-only PR (+100/−0) adding evidence-driven autonomy classes to the autonomy-spectrum problem doc, tracking data to applied/konflux-ci, and a cross-reference in code-review.md open questions. PR scope matches issue #138 authorization. No security findings. All cross-reference anchor links resolve correctly. The content is well-structured and introduces a useful framework backed by concrete evidence.

Three medium-severity findings worth noting — none blocking.

---

Findings

1. Medium — Glossary entry missing for "autonomy class"

File: docs/glossary.md

The glossary preamble states: "PRs that introduce new terminology should add to this glossary as part of the change." This PR introduces "autonomy class" (or "evidence-driven autonomy class") as a distinct classification concept — a named category of repos or change-types assigned based on observable evidence signals. No glossary entry exists for this term.

Remediation: Add a glossary entry for "Autonomy class" pointing to the new ## Evidence-driven autonomy classes section in autonomy-spectrum.md.

2. Medium — Autonomy class framework definition leans prescriptive for a problem document

File: docs/problems/autonomy-spectrum.md

AGENTS.md states: "Problem documents should present multiple options with trade-offs, not prescribe single solutions." The Proposed autonomy change subsection correctly presents 3 options with trade-offs (CHANGES_REQUESTED, auto-file issues, acknowledgment prompt). However, the autonomy class framework itself (the 5-part template: qualifying criteria, evidence, validation plan, proposed change, boundary conditions) is presented as the singular approach to sub-repo authority elevation with no alternatives considered (e.g., flat confidence thresholds per finding, deferring to ADRs, or per-finding calibration). The existing doc's binary model section has similar prescriptive framing, so this matches precedent — but the gap is worth noting for future iterations.

Remediation: Consider adding a brief note acknowledging alternative approaches to evidence-based authority elevation, even if this framework is the preferred direction. This could be a sentence in the introductory paragraph or an additional open question.

3. Medium — Concrete class criteria in general doc derived from single org's evidence

File: docs/problems/autonomy-spectrum.md

AGENTS.md states: "Keep core problem documents organization-agnostic." The mechanical-consistency-doc-repo class with its specific qualifying criteria and evidence thresholds was derived entirely from the konflux-ci/architecture repo. The criteria are stated generically ("documentation-only repo"), and the tracking data correctly lives in the applied doc — but the class definition itself is bootstrapped from a single org observation. Once validated across more repos/orgs, this concern dissolves.

Remediation: Consider adding a note that the criteria are derived from initial evidence in one repo and may be refined as more data accumulates.

4. Low — Reviewer count ambiguity

File: docs/problems/applied/konflux-ci/README.md

The text names 5 human reviewers (rcerven, johnbieren, cit1zen, ralphbean, arewm) and later refers to "Human reviewer mmorhun" who dismissed a domain-incorrect finding. mmorhun is called a "reviewer" but is not included in the count of "5 human reviewers." If the distinction is between formal approvers and commenters, the text could clarify mmorhun's role (e.g., "commenter" or "participant").

5. Low — Validation plan does not account for clean PRs

File: docs/problems/autonomy-spectrum.md

The success criteria require "the agent catches mechanical consistency issues that no human reviewer flags on at least 3 of 5 PRs." If some of the 5 tracked PRs have zero findable mechanical issues, they cannot count toward the 3-of-5 threshold. In a well-maintained doc repo, this could make the criteria unachievable through no fault of the agent.

Remediation: Consider adding an escape clause (e.g., "If fewer than 3 of 5 PRs contain findable mechanical issues, extend the sample").

---

Checks performed

Dimension	Result
Correctness	✅ Cross-references valid; evidence claims internally consistent between docs; validation plan logic is sound
Security	✅ No secrets, injection patterns, permission changes, or steganographic content
Intent & coherence	⚠ Scope matches #138; prescriptive framing noted (medium)
Style & conventions	✅ Follows established patterns; heading structure, table format, and link style are consistent
Docs currency	⚠ Glossary entry missing (medium)
Cross-repo contracts	⏭ Skipped — no exported interfaces modified

Previous run

Review — comment

Docs-only PR adding evidence-driven autonomy classes to autonomy-spectrum.md, org-specific tracking data to applied/konflux-ci/README.md, and a cross-reference in code-review.md. No code changes, no security surface. The content is well-structured and correctly separates general concepts (core doc) from org-specific tracking data (applied doc).

Two medium-severity findings worth noting — neither blocking, but both worth addressing.

Medium

1. Validation table row count mismatch (docs/problems/applied/konflux-ci/README.md)

The validation plan text says "Track the next 5 ADR PRs merged" but the tracking table has only 4 empty rows after the #367 baseline row. Either the table needs 5 empty rows to match the plan, or the text should say "next 4" if #367 is intended to count as one of the 5.

2. Prescriptive framing violates AGENTS.md (docs/problems/autonomy-spectrum.md)

AGENTS.md states: "Problem documents should present multiple options with trade-offs, not prescribe single solutions." The Proposed autonomy change subsection presents a single preferred outcome (CHANGES_REQUESTED instead of COMMENT for medium-severity mechanical findings) without exploring alternatives — e.g., a higher COMMENT tier, auto-filing tracking issues for unresolved findings, requiring a second agent pass, or human escalation. Reframing this as an options analysis with trade-offs would align with project conventions.

Low

3. Terminology mismatch with existing content (docs/problems/code-review.md)

Line 53 of code-review.md already uses "mechanical-matching dimensions" to describe sonnet-class sub-agents. The PR introduces the parallel term "mechanical-consistency" as the autonomy class name. These two phrases appear in adjacent docs without cross-reference, which could confuse readers about whether they name the same concept. Consider aligning terminology or explicitly relating them.

4. Dangling cross-reference (docs/problems/autonomy-spectrum.md)

The "CI-config-only changes" bullet says "See applied docs for tracking" but no CI-config tracking data exists anywhere in the applied docs directory. This is a forward reference to content that doesn't exist yet. Consider removing the link or noting it as future work.

5. Org-specific evidence in core doc (docs/problems/autonomy-spectrum.md)

AGENTS.md requires core problem documents to be organization-agnostic, with org-specific details in docs/problems/applied/<org-name>/. The Evidence field in the core autonomy-spectrum.md directly references a specific konflux-ci PR (#367) as the primary evidence. The applied doc already tracks this evidence in detail. The core doc's evidence field could instead use a generic description with a pointer to the applied directory.

6. Framework heading inconsistency (docs/problems/autonomy-spectrum.md)

The autonomy class framework definition lists component #5 as "Boundary conditions" but the mechanical consistency section uses "Boundary" as the heading — minor naming inconsistency between the framework definition and its first instantiation.

7. Scope beyond linked issue (docs/problems/autonomy-spectrum.md)

The "Other potential autonomy classes" subsection introduces CI-config-only changes and bot dependency bumps, neither of which is mentioned in issue #138 (scoped to mechanical consistency in documentation repos). These additions are low-risk but extend scope. Each new class would benefit from its own evidence-tracking issue.

8. H4 heading introduces new pattern (docs/problems/applied/konflux-ci/README.md)

The #### Mechanical consistency in the architecture repo heading introduces H4 depth, which doesn't exist elsewhere in this file. Existing subsections under H3 use bold inline labels (e.g., **Platform security** —). Consider using the established bold-label pattern for consistency.

9. Open question style (docs/problems/code-review.md)

The new open question bullet ends with trailing context after the question mark, making it read as a statement rather than a question. All 8 existing open questions in the same list are self-contained questions ending with ?. The trailing evidence reference could move to a parenthetical or footnote.

---

Labels: PR adds only documentation content across problem docs and applied docs

[guyoron1]

/fs-fix

[fullsend-ai-coder]

🔧 Fix agent — iteration 1 (human-triggered)

Addressed 7 of 9 review findings across 3 files. Fixed: table row count mismatch, single-outcome presentation, terminology alignment, dangling reference, org-specific content in core doc, heading naming inconsistency, and open question style. Disagreed with 2: scope-creep finding (extensibility framing is essential) and H4 heading finding (structurally correct hierarchy).

Fixed (7):

1. Validation plan says 5 PRs but table has only 4 empty rows (docs/problems/applied/konflux-ci/README.md): Added 5th empty row to tracking table to match the '5 PRs' validation plan text
2. Proposed autonomy change presents single outcome without alternatives (docs/problems/autonomy-spectrum.md): Expanded to present 3 options (CHANGES_REQUESTED, auto-file issues, acknowledgment prompt) with trade-offs per AGENTS.md convention
3. Terminology mismatch: mechanical-matching vs mechanical-consistency (docs/problems/code-review.md): Shortened open question to use 'mechanical-consistency findings' with inline cross-reference link to autonomy-spectrum.md
4. Dangling reference to CI-config tracking data in applied docs (docs/problems/autonomy-spectrum.md): Removed 'See applied docs for tracking' from CI-config-only changes bullet since no such tracking data exists
5. Org-specific PR reference in core doc (docs/problems/autonomy-spectrum.md): Generalized evidence section to remove specific konflux-ci/architecture PR [ARCHIVED] Benchmark issue 367 #367 reference, deferred to applied docs for specifics
6. Boundary vs Boundary conditions naming inconsistency (docs/problems/autonomy-spectrum.md): Renamed 'Boundary' heading to 'Boundary conditions' to match the template definition in the autonomy class framework
7. Open question style does not match existing self-contained format (docs/problems/code-review.md): Shortened the open question to a single concise line with an inline link, matching the self-contained style of other open questions in the file

Disagreed (2):

1. Other potential autonomy classes section is scope creep beyond issue Track autonomy readiness: review agent outperformed human review on mechanical consistency in documentation repo #138: The section establishes that the autonomy class framework is extensible with minimal content (3 lines). Removing it would leave the framework appearing closed-ended. Issue Track autonomy readiness: review agent outperformed human review on mechanical consistency in documentation repo #138 asks to track the mechanical-consistency class; showing that other classes exist but need their own evidence is essential framing, not scope creep.
2. H4 heading introduces new pattern not used elsewhere in file: H4 is structurally correct as a subsection of the H3 'Autonomy readiness evidence' section. The pattern anticipates future autonomy classes that would be siblings under the same H3. The existing file already uses H3 for problem areas; H4 for sub-topics within them is standard markdown hierarchy.

Tests: passed

_{Updated by fullsend fix agent}