Skip to content

Add ASP.NET Core adversarial review skills - #1

Closed
PureWeen wants to merge 27 commits into
masterfrom
pureween-improve-pr-review-validation
Closed

Add ASP.NET Core adversarial review skills#1
PureWeen wants to merge 27 commits into
masterfrom
pureween-improve-pr-review-validation

Conversation

@PureWeen

@PureWeen PureWeen commented Aug 7, 2026

Copy link
Copy Markdown
Owner

Summary

  • add repository-scoped aspnetcore-pr-review and aspnetcore-try-fix skills
  • reduce the always-loaded reviewer from 572 to 180 lines and try-fix from 355 to 167 lines
  • route bounded, stateless changes through a fast evidence-backed path while escalating lifecycle, concurrency, interop, serialization, compatibility, performance, and credible blocker claims to the full adversarial workflow
  • preserve product-oracle separation, producer-to-consumer impact maps, first-resumed-event and measurement-provenance analysis, candidate-independent red/green proof, calibrated proof labels, and live-head drift checks
  • make empirical adjudication conditional on a surviving material claim instead of mandatory busywork
  • map authority handoffs when multiple stages reinterpret the same metadata or state, without treating any one source as universally authoritative
  • require proven candidates to retain changed-path and final-observable evidence plus distinct defect, opposite-side, and adjacent-behavior boundary cases
  • reject marker-only proof, malformed or duplicate authority tables, missing or extra boundary roles, invalid evidence references, and symbolic-link evidence escapes
  • harden both skills against instructions embedded in PR bodies, comments, diffs, logs, manifests, and fixtures while preserving useful technical evidence
  • make reviewer artifacts path-aware so bounded reviews do not create full-path candidate, cross-examination, and empirical boilerplate
  • migrate both eval suites to first-class Vally 0.13 capability specs with independent source snapshots, neutral fixture aliases, disabled push URLs, held-out governance, and family/provenance macro scoring
  • replace all reviewer/try-fix Python tooling with PowerShell and harden staging against case variants, nested symlinks, unsafe deletion, and malformed CLI inputs

How this grows without overfitting

The eval corpus grows by adding durable failure mechanisms, not by encoding the expected review of every PR we inspect.

  1. Capture the mechanism, not the PR wording. Regressions become bounded fixtures and expectations stated in terms of ownership transfer, stale measurement provenance, authority handoffs, head drift, assertion fidelity, producer/consumer impact, or another general mechanism. PR numbers remain provenance, not answer keys.
  2. Separate learning from transfer. A new mechanism can begin as a train case, but it cannot justify stronger global instructions until it transfers to an independently provenanced case and no-defect/bounded canaries remain healthy. Held-out cases are hash-frozen.
  3. Keep positive and negative controls. Each case identifies required evidence and nearby overclaims to reject. Prompt/expectation overlap and forbidden-term checks detect answer leakage and wording matchers.
  4. Score families and provenance, not case volume. Results are macro-averaged by mechanism family and provenance, so near-duplicates from one source cannot dominate the score.
  5. Use real source only when the proof contract supports it. Source snapshots permit repository inspection without exposing skill history or answer keys. Mechanism-only fixtures remain planning/candidate-review evals unless they also provide a concrete candidate, independent assertion, required dependencies, and safe restoration boundary.
  6. Require repeated, attributable runs. Official results use the pinned executor/judge configuration, five trials, strict Vally governance tags, and retained JSONL. One good run is diagnostic only.
  7. Consolidate instead of accumulating. The two scrolling-specific fixtures were replaced by the portable viewport-measurement-epoch-recovery mechanism. Near-duplicates should continue to be merged or removed when stronger independent coverage exists.

A reviewed PR expands the skill only when it teaches a transferable rule, exposes an uncovered mechanism, or supplies independent transfer evidence.

Validation

  • 34 deterministic PowerShell tests
  • all three canonical Vally specifications validated: 32 cases, including 8 held-out cases
  • strict @microsoft/vally-cli@0.13.0 lint for all three canonical specifications
  • matched multi-stage-authority defect/no-defect planning cases added under one score family
  • one-run diagnostics passed: authority defect 0.750, no-defect control 1.000; these are diagnostic runs, not official five-trial scores
  • no .py, python, or python3 files/invocations remain in reviewer, try-fix, or canonical eval wiring
  • reviewer selected-suite aggregate: current 0.800 vs old 0.733
  • try-fix narrow-proof train case: current 1.000 vs old 0.800
  • try-fix held-out macro: current 0.725 vs old 0.625
    • stateless lookup: current 0.850 vs old 0.550
    • deferred cancellation/disposal: current 0.600 vs old 0.700
  • path-safety tests cover repository roots/children, case variants, direct and nested symlinks, directory-boundary comparisons, preflight-before-delete, regular-file occupants, and artifact evidence references
  • documented validation/aggregation commands are executed through a real subprocess test
  • whitespace and final scope validation across the complete diff

Known limits

  • The deferred cancellation/disposal held-out case remains weaker than the old skill (0.600 vs 0.700). The overall held-out macro improves, but this local loss is retained rather than hidden.
  • Validate-ReviewArtifacts.ps1 checks structural completeness, proof-label consistency, referenced evidence containment, and boundary roles. It cannot authenticate that a model-authored log came from the command it claims to represent without an external execution trust boundary.
  • Vally snapshots remove repository history, reviewer answer keys, and push access, but they are not OS security sandboxes. Executors still inherit host environment, network, and model credentials, so injection scores measure instruction adherence rather than containment.
  • Representative five-trial comparisons guide this refactor; they do not establish that every canonical eval or future PR will improve.

Copilot AI added 18 commits August 11, 2026 12:22
Capture product intent separately from implementation evidence, preserve exact empirical assertions, detect relevant PR head drift, and translate findings into plain maintainer language.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Keep final evidence selection and synthesis outside the Anthropic candidate family, and add a regression eval for non-GPT invocation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Add provenance-aware scoring, frozen held-out fixtures, deterministic validation, and cross-area transfer cases so PR-derived regressions remain useful without dominating quality measurements.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Prefer structural clarity for local mechanics, reserve comments for nonlocal rationale, and retain the distinction with a cross-area eval and monotonic provenance warnings.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Move conditional review and proof detail behind progressive-disclosure references, add bounded and full review paths, complete the try-fix Vally suite, and replace the reviewer Python harness with PowerShell tooling.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep exact targeted and production proof taxonomy in the shortened skill spine so bounded reviews and retained empirical results cannot over-promote incomplete evidence.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Distinguish candidate evidence labels from the caller's requested proof target and make the retained narrow-proof eval self-contained.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Classify complete retained empirical evidence from the core proof labels without unnecessary repository discovery or reference-path stalls.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Make retained-evidence labels auditable with a compact mechanism matrix and finish removing the parent branch's remaining Python Vally tooling.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Ensure every generated try-fix case loads the intended skill without altering frozen held-out prompts or exposing rubric answers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Validate bounded reviews without full-path boilerplate while preserving targeted red/green evidence and the complete full-review contract.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
@PureWeen
PureWeen force-pushed the pureween-improve-pr-review-validation branch from c8447e8 to 05755c3 Compare August 11, 2026 17:27
Copilot AI added 3 commits August 11, 2026 15:46
Remove the duplicated JSON eval manifests and generation step. Read governance and score metadata directly from the canonical Vally suites while retaining held-out integrity and sanitized skill staging.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Separate API compatibility tracking from API approval, require honest red-green evidence for bug-fix verification claims, and document fresh Components worktree prerequisites.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 21:34 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 21:34 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 21:34 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 21:52 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 21:53 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 21:54 — with GitHub Actions Inactive
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 22:00 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 22:00 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 22:00 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 22:27 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 22:28 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 22:29 — with GitHub Actions Inactive
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 22:41 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 22:41 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 22:42 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 23:10 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 23:11 — with GitHub Actions Inactive
@PureWeen
PureWeen temporarily deployed to copilot-pat-pool August 11, 2026 23:12 — with GitHub Actions Inactive
Copilot AI added 3 commits August 11, 2026 18:18
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
- Exercise the actual interaction under test rather than bypassing it
  via direct injection/navigation.
- Wait for observable DOM/viewport/callback conditions using
  Browser.True/Browser.Equal instead of relying only on fixed delays
  or requestAnimationFrame.
- Note fixed delays remain valid for negative guarantees.
- Check console errors and transient UI when relevant.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0fb5ea3b-44bb-495b-99c5-93e753ca9ce1
@PureWeen PureWeen closed this Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants