code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric - #2
Closed
Evangelink wants to merge 1 commit into
Closed
code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric#2Evangelink wants to merge 1 commit into
Evangelink wants to merge 1 commit into
Conversation
…ntoso rubric
Workspace-integrity scenario was NOT ACTIVATED in plugin mode: the prompt
("Generate unit tests for its core module") was terse and small-scoped, so
the runtime did not route to the code-testing-agent skill. Reframe it with
high-level test-generation language ("comprehensive pytest test suite",
"scaffold", "thorough unit tests") that matches the skill description,
while preserving the guardrail anchors: it still points at the on-disk module
without naming it and never implies restoring the gutted tree.
ContosoUniversity rubric item #5 (find-untested-sources) was conditional on
that skill being loaded — only true in plugin mode. In isolated runs the
agent cannot satisfy it, so the judge penalized it asymmetrically, injecting
isolated-vs-plugin variance. Make the conditional deterministic: explicitly
N/A when the skill is not loaded, without lowering the bar when it is.
Applied to both eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Skill Coverage Report
|
Owner
Author
|
/evaluate |
|
❌ Evaluation did not complete (upstream job failed or was skipped). View workflow run |
1 similar comment
|
❌ Evaluation did not complete (upstream job failed or was skipped). View workflow run |
Owner
Author
|
Closing — this should target dotnet/skills, not my fork. Reopened as dotnet#806. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to dotnet#802. Fixes the two remaining flagged code-testing-agent eval rows from the PR's evaluation, in both
eval.yamlandeval.vally.yaml.1.⚠️ NOT ACTIVATED (plugin)
Does not revert a gutted-looking workspace (workspace integrity)—The prompt ("Generate unit tests for its core module…") was terse and small-scoped, so in plugin mode (many skills competing) the runtime did not route to the code-testing-agent skill —
detected: [], confidence 0.23.Reframed to a realistic, high-level ask that matches the skill description's activation triggers — "comprehensive pytest test suite", "scaffold", "thorough unit tests" — mirroring the sibling ContosoUniversity / Flask prompts. Crucially, the guardrail semantics are preserved:
synthstrdecoy) without naming it,core/·io/tree,pythonpath·testpaths·pytest expectations.Assertions and rubric are unchanged — the agent must still discover and test the delivered module without reverting the tree.
2.
Generate tests for ContosoUniversity ASP.NET Core MVC app— quality wobble (3.3 → 3.0, conf 0.23)This scenario is untouched by dotnet#802; the deterministic graders still pass. The only legitimate, non-gaming robustness lever is rubric item #5 (
find-untested-sources), which was conditional on that skill being loaded — only true in plugin mode. In isolated runs the agent literally cannot satisfy it, so the judge penalized it asymmetrically, injecting isolated-vs-plugin variance (InvestigatingResults pattern dotnet#6).Made the conditional evaluate deterministically: explicitly N/A when the skill is not loaded, without lowering the bar where it is loaded. The coverage floor and all other rubric items are unchanged (no rubric softening).
Verification
Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com