Fix skill evaluation fixtures and failure reporting - #36954

Merged

kubaflo merged 89 commits into

mainfrom

pureween-fix-skill-eval-failures

Aug 15, 2026

Commits on Aug 7, 2026

Fix skill evaluation fixtures and reporting
Copilot
committed
eb228ac
Include preflight fixture in static validation
Copilot
committed
93ed065
Checkout the preflight fixture directory
Copilot
committed
128de8a
Make the auth eval fixture self-contained
Copilot
committed
d87267c
Enforce skill guidance in capability evals
Copilot
committed
9349f10
Make try-fix cleanup constraint explicit
Copilot
committed
38d41fa
Give blocked try-fix eval the operational budget
Copilot
committed
d89b074
Use Vally aggregate verdict for skill gates
Copilot
committed
f2b6fbf
Make full-matrix evals load their target skills
Copilot
committed
a431086
Make CI-fix ownership decisions terminal
Copilot
committed
8e6081d
Review public PRs without GitHub authentication
Copilot
committed
c0a3f8d
Distinguish failed runs from incomplete runs
Copilot
committed
bf39ab6
Correct parent fallback eval targets
Copilot
committed
d9fda1e
Harden skill eval evidence and reporting
Copilot
committed
188c7fb
Make auth guidance read grader executor-neutral
Copilot
committed
889b4f3
Harden skill evaluation assertions

Vally Fixture
authored and
Copilot
committed
9e7c317
Make try-fix restoration a hard gate

Vally Fixture
authored and
Copilot
committed
6ea184c
Harden skill evaluation protocol gates

Vally Fixture
authored and
Copilot
committed
bb2a2c2
Harden Vally fixture provisioning

Vally Fixture
and
Copilot
committed
98742d0
Harden eval subprocess credential isolation

Vally Fixture
authored and
Copilot
committed
5e7fe6c
Harden trusted skill fixture bootstrap

Vally Fixture
authored and
Copilot
committed
4d97370
Block VCS metadata fixture destinations

Vally Fixture
authored and
Copilot
committed
d3f37ed
Harden eval overlay paths and semantic prompts
Copilot
committed
3b32520
Guard against persistent fixture identities
Copilot
committed
f63233f
Harden try-fix cleanup matchers
Copilot
committed
a142196
Require iOS navigation registration evidence
Copilot
committed
e4e71ae
Harden skill eval activation and interpolation
Copilot
committed
ba8641b
Fix JUnit testcase parsing for invocation vetoes
Copilot
committed
026967e
Reject multiline persistent Git identity writes
Copilot
committed
edb25ab
Reject multiline persistent Git identity commands
Copilot
committed
73a8c28
Disable YAML aliases in untrusted eval specs
Copilot
committed
e862b14
Keep trusted hermeticity spec out of checkout
Copilot
committed
9095fb7
Harden skill evaluation completeness gates
Copilot
committed
87427b2
Harden eval overlay and JUnit parsing
Copilot
committed
ab3eeed
Fail closed on missing eval signals
Copilot
committed
260ce09
Fix remaining skill evaluation failures
Copilot
committed
c6ce6dd
Guard Vally skills root containment
Copilot
committed
893621f
Make skill eval fixtures durable
Copilot
committed
69e55a7
Synthetic full test verification fixture

Vally Fixture
authored and
Copilot
committed
1da0975
Harden remaining skill evaluation gates
Copilot
committed
912ad5f
Make try-fix eval probes deterministic
Copilot
committed
a30bea7
Harden invocation veto XML parsing
Copilot
committed
fe0db0b
Close remaining skill evaluation bypasses
Copilot
committed
e32c704
Make verification protocols individually mandatory
Copilot
committed
003d907
Harden JUnit suite property parsing
Copilot
committed
38b4c18
Preserve skill validation setup after rebase
Copilot
committed
b251b0b
Align new skill eval suites with SOL
Copilot
committed
0ac4e7d
Stabilize skill evaluation gates
Copilot
committed
d9e7091
Harden credentialed eval fixtures and scenario gates
Copilot
committed
c4a5b14
Harden live skill evaluation boundaries
Copilot
committed
324a648
Isolate skill eval runtime ownership
Copilot
committed
a207343
Sandbox skill evaluation tools
Copilot
committed
7186be6
Add reachable try-fix evaluation fixture
Copilot
committed
451bd73
Add reachable full-verification fixture
Copilot
committed
643d58e
Add reachable failure-only fixture
Copilot
committed
ae7f6de
Repair reachable skill evaluation fixtures
Copilot
committed
eb289d1
Add isolated failure-only evaluation fixture
Copilot
committed
7d5e02a
Isolate failure-only evaluation fixture
Copilot
committed
b32b554
Complete trim and protocol evaluation evidence
Copilot
committed
8aa6a08
Make positive Trim AOT fixture invariant
Copilot
committed
3ae5395
Wait for background verification results
Copilot
committed
4bd6e16
Skip exhausted Copilot evaluation tokens
Copilot
committed
7d0a45c
Isolate evaluator Git metadata
Copilot
committed
5a41db3
Wait for try-fix shell completion
Copilot
committed
d006668
Contain try-fix attempt artifacts
Copilot
committed
d85c1ff
Keep materialized reviews self-contained
Copilot
committed
9028354
Enforce verification script ownership
Copilot
committed
cd98054
Serialize skill evaluation jobs
Copilot
committed
21726ee
Repin rebased evaluation fixtures
Copilot
committed
f76da61
Restrict evaluator workspace permissions
Copilot
committed
9a21e5f
Stabilize verification result evaluation
Copilot
committed
f00773b

Commits on Aug 9, 2026

Harden skill evaluation policy and reporting
Copilot
committed
78daf5a

Commits on Aug 10, 2026

Close remaining skill evaluation trust gaps
Copilot
committed
f6d4942
Harden Vally repository control isolation
Copilot
committed
2860e7b
Harden Copilot evaluator runtime state
Copilot
committed
7b4a180
Reject deleted evaluator controls
Copilot
committed
c673406
Probe Copilot model support before skill evals
Copilot
committed
f323547
Isolate evaluator tests from workflow refs
Copilot
committed
3fb6fa7
Keep XAML unit verification host-only
Copilot
committed
5952993

Commits on Aug 11, 2026

Commits on Aug 12, 2026

Require configured evaluation stimuli
Copilot
committed
c4b2e73
Use PR metadata for branch-based verification
Copilot
committed
abd7643
Bind evaluation fixtures to owning stimuli
Copilot
committed
e395815

Commits on Aug 13, 2026

Reject duplicate Vally YAML keys
Copilot
committed
51bc028
Reject Vally YAML merge keys
Copilot
committed
4bce54e

Commits on Aug 14, 2026

Reject explicit Vally YAML merge tags
Copilot
committed
fd8f3d5