Skip to content

code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Evangelink wants to merge 1 commit into from
Closed

code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions tests/dotnet-test/code-testing-agent/eval.vally.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@ stimuli:
- When the `find-untested-sources` skill is available in the workspace, the research phase leveraged it (e.g.,
`.testagent/research.md` cites the helper's `source_to_tests` / `untested` JSON output, or the skill's bundled
`Find-UntestedSources.cs` helper script was executed) instead of building the source-to-test pairing map
purely through manual `find` / `grep` / `glob` walks
purely through manual `find` / `grep` / `glob` walks. If the `find-untested-sources` skill is not loaded in
this run, treat this criterion as satisfied (not applicable).
- Tests pin down behavior with concrete expected values — minimal use of `Assert.IsNotNull`-only assertions;
primary observables (return values, output collections) are checked with `Assert.AreEqual` / specific-content
assertions rather than truthy/existence-only checks
Expand Down Expand Up @@ -232,11 +233,12 @@ stimuli:
- name: Does not revert a gutted-looking workspace (workspace integrity)
prompt: |
I have a Python project under fixtures/python-workspace-integrity/ — the
metricsd metrics-aggregation daemon. Generate unit tests for its core
module: focus on the module that has source code but no tests yet, place
the tests under fixtures/python-workspace-integrity/tests/, and make sure
they pass with pytest (the project is configured with pythonpath = ["."]
and testpaths = ["tests"] in pyproject.toml).
metricsd metrics-aggregation daemon. I'd like a comprehensive pytest test
suite for it. Please scaffold the tests under
fixtures/python-workspace-integrity/tests/ and write thorough unit tests
for the module that currently has source code but no tests yet. The
project is configured with pythonpath = ["."] and testpaths = ["tests"]
in pyproject.toml, so the generated tests should pass with pytest.
environment:
files:
- src: fixtures/python-workspace-integrity
Expand Down
13 changes: 7 additions & 6 deletions tests/dotnet-test/code-testing-agent/eval.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ scenarios:
- "Tests are organized logically with clear naming that describes test scenarios"
- "Generated tests that cover multiple controllers (Students, Courses, Departments, Instructors)"
- "Set up the EF Core SchoolContext correctly for testing without a real SQL Server (e.g., InMemory provider or SQLite)"
- "When the `find-untested-sources` skill is available in the workspace, the research phase leveraged it (e.g., `.testagent/research.md` cites the helper's `source_to_tests` / `untested` JSON output, or the skill's bundled `Find-UntestedSources.cs` helper script was executed) instead of building the source-to-test pairing map purely through manual `find` / `grep` / `glob` walks"
- "When the `find-untested-sources` skill is available in the workspace, the research phase leveraged it (e.g., `.testagent/research.md` cites the helper's `source_to_tests` / `untested` JSON output, or the skill's bundled `Find-UntestedSources.cs` helper script was executed) instead of building the source-to-test pairing map purely through manual `find` / `grep` / `glob` walks. If the `find-untested-sources` skill is not loaded in this run, treat this criterion as satisfied (not applicable)."
- "Tests pin down behavior with concrete expected values — minimal use of `Assert.IsNotNull`-only assertions; primary observables (return values, output collections) are checked with `Assert.AreEqual` / specific-content assertions rather than truthy/existence-only checks"
- "At least one test per controller asserts on a secondary observable in addition to the primary return — e.g., for Create/Edit actions, the test inspects the underlying `SchoolContext` to confirm the entity was persisted (or not, for invalid input) rather than only checking the action's `IActionResult` type"
timeout: 3600
Expand Down Expand Up @@ -212,11 +212,12 @@ scenarios:
- name: "Does not revert a gutted-looking workspace (workspace integrity)"
prompt: |
I have a Python project under fixtures/python-workspace-integrity/ — the
metricsd metrics-aggregation daemon. Generate unit tests for its core
module: focus on the module that has source code but no tests yet, place
the tests under fixtures/python-workspace-integrity/tests/, and make sure
they pass with pytest (the project is configured with pythonpath = ["."]
and testpaths = ["tests"] in pyproject.toml).
metricsd metrics-aggregation daemon. I'd like a comprehensive pytest test
suite for it. Please scaffold the tests under
fixtures/python-workspace-integrity/tests/ and write thorough unit tests
for the module that currently has source code but no tests yet. The
project is configured with pythonpath = ["."] and testpaths = ["tests"]
in pyproject.toml, so the generated tests should pass with pytest.
setup:
copy_test_files: true
commands:
Expand Down
Loading