code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric - #806
Conversation
…ntoso rubric
Workspace-integrity scenario was NOT ACTIVATED in plugin mode: the prompt
("Generate unit tests for its core module") was terse and small-scoped, so
the runtime did not route to the code-testing-agent skill. Reframe it with
high-level test-generation language ("comprehensive pytest test suite",
"scaffold", "thorough unit tests") that matches the skill description,
while preserving the guardrail anchors: it still points at the on-disk module
without naming it and never implies restoring the gutted tree.
ContosoUniversity rubric item #5 (find-untested-sources) was conditional on
that skill being loaded — only true in plugin mode. In isolated runs the
agent cannot satisfy it, so the judge penalized it asymmetrically, injecting
isolated-vs-plugin variance. Make the conditional deterministic: explicitly
N/A when the skill is not loaded, without lowering the bar when it is.
Applied to both eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the dotnet-test / code-testing-agent evaluation definitions to (1) improve plugin-mode activation for the workspace-integrity Python scenario and (2) reduce isolated-vs-plugin scoring variance in the ContosoUniversity rubric.
Changes:
- Reframes the workspace integrity scenario prompt to better match code-testing-agent activation signals while preserving the “don’t restore the gutted workspace” guardrail.
- Adjusts the ContosoUniversity rubric text to treat the
find-untested-sourcescriterion as N/A when the skill is not loaded (in botheval.yamlandeval.vally.yaml).
Show a summary per file
| File | Description |
|---|---|
| tests/dotnet-test/code-testing-agent/eval.yaml | Updates Contoso rubric text and rewrites the workspace-integrity scenario prompt. |
| tests/dotnet-test/code-testing-agent/eval.vally.yaml | Mirrors the same rubric and prompt changes for the Vally eval. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 2/2 changed files
- Comments generated: 2
…tato # Conflicts: # tests/dotnet-test/code-testing-agent/eval.vally.yaml # tests/dotnet-test/code-testing-agent/eval.yaml
…session timeline Address review feedback: the 'treat as satisfied (N/A) when the skill is not loaded' clause is not verifiable from the judge's inputs (which show which tools were called, not which were available). Rewrite the criterion to be decidable from the session timeline by requiring a source-to-test pairing map recorded in .testagent/research.md that either cites find-untested-sources output or documents the equivalent manual approach. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/evaluate |
Skill Validation Results
[1] Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
What
Follow-up to #802. Fixes the two remaining flagged code-testing-agent eval rows from the PR's evaluation, in both
eval.yamlandeval.vally.yaml.1.⚠️ NOT ACTIVATED (plugin)
Does not revert a gutted-looking workspace (workspace integrity)—The prompt ("Generate unit tests for its core module…") was terse and small-scoped, so in plugin mode (many skills competing) the runtime did not route to the code-testing-agent skill —
detected: [], confidence 0.23.Reframed to a realistic, high-level ask that matches the skill description's activation triggers — "comprehensive pytest test suite", "scaffold", "thorough unit tests" — mirroring the sibling ContosoUniversity / Flask prompts. Crucially, the guardrail semantics are preserved:
synthstrdecoy) without naming it,core/·io/tree,pythonpath·testpaths·pytest expectations.Assertions and rubric are unchanged — the agent must still discover and test the delivered module without reverting the tree.
2.
Generate tests for ContosoUniversity ASP.NET Core MVC app— quality wobble (3.3 → 3.0, conf 0.23)This scenario is untouched by #802; the deterministic graders still pass. The only legitimate, non-gaming robustness lever is rubric item #5 (
find-untested-sources), which was conditional on that skill being loaded — only true in plugin mode. In isolated runs the agent literally cannot satisfy it, so the judge penalized it asymmetrically, injecting isolated-vs-plugin variance (InvestigatingResults pattern #6).Made the conditional evaluate deterministically: explicitly N/A when the skill is not loaded, without lowering the bar where it is loaded. The coverage floor and all other rubric items are unchanged (no rubric softening).
Verification
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com