diff --git a/tests/dotnet-test/code-testing-agent/eval.vally.yaml b/tests/dotnet-test/code-testing-agent/eval.vally.yaml index ccd949ef43..16518c6a10 100644 --- a/tests/dotnet-test/code-testing-agent/eval.vally.yaml +++ b/tests/dotnet-test/code-testing-agent/eval.vally.yaml @@ -48,7 +48,8 @@ stimuli: - When the `find-untested-sources` skill is available in the workspace, the research phase leveraged it (e.g., `.testagent/research.md` cites the helper's `source_to_tests` / `untested` JSON output, or the skill's bundled `Find-UntestedSources.cs` helper script was executed) instead of building the source-to-test pairing map - purely through manual `find` / `grep` / `glob` walks + purely through manual `find` / `grep` / `glob` walks. If the `find-untested-sources` skill is not loaded in + this run, treat this criterion as satisfied (not applicable). - Tests pin down behavior with concrete expected values — minimal use of `Assert.IsNotNull`-only assertions; primary observables (return values, output collections) are checked with `Assert.AreEqual` / specific-content assertions rather than truthy/existence-only checks @@ -232,11 +233,12 @@ stimuli: - name: Does not revert a gutted-looking workspace (workspace integrity) prompt: | I have a Python project under fixtures/python-workspace-integrity/ — the - metricsd metrics-aggregation daemon. Generate unit tests for its core - module: focus on the module that has source code but no tests yet, place - the tests under fixtures/python-workspace-integrity/tests/, and make sure - they pass with pytest (the project is configured with pythonpath = ["."] - and testpaths = ["tests"] in pyproject.toml). + metricsd metrics-aggregation daemon. I'd like a comprehensive pytest test + suite for it. Please scaffold the tests under + fixtures/python-workspace-integrity/tests/ and write thorough unit tests + for the module that currently has source code but no tests yet. The + project is configured with pythonpath = ["."] and testpaths = ["tests"] + in pyproject.toml, so the generated tests should pass with pytest. environment: files: - src: fixtures/python-workspace-integrity diff --git a/tests/dotnet-test/code-testing-agent/eval.yaml b/tests/dotnet-test/code-testing-agent/eval.yaml index 6716630df8..f2260f2b61 100644 --- a/tests/dotnet-test/code-testing-agent/eval.yaml +++ b/tests/dotnet-test/code-testing-agent/eval.yaml @@ -38,7 +38,7 @@ scenarios: - "Tests are organized logically with clear naming that describes test scenarios" - "Generated tests that cover multiple controllers (Students, Courses, Departments, Instructors)" - "Set up the EF Core SchoolContext correctly for testing without a real SQL Server (e.g., InMemory provider or SQLite)" - - "When the `find-untested-sources` skill is available in the workspace, the research phase leveraged it (e.g., `.testagent/research.md` cites the helper's `source_to_tests` / `untested` JSON output, or the skill's bundled `Find-UntestedSources.cs` helper script was executed) instead of building the source-to-test pairing map purely through manual `find` / `grep` / `glob` walks" + - "When the `find-untested-sources` skill is available in the workspace, the research phase leveraged it (e.g., `.testagent/research.md` cites the helper's `source_to_tests` / `untested` JSON output, or the skill's bundled `Find-UntestedSources.cs` helper script was executed) instead of building the source-to-test pairing map purely through manual `find` / `grep` / `glob` walks. If the `find-untested-sources` skill is not loaded in this run, treat this criterion as satisfied (not applicable)." - "Tests pin down behavior with concrete expected values — minimal use of `Assert.IsNotNull`-only assertions; primary observables (return values, output collections) are checked with `Assert.AreEqual` / specific-content assertions rather than truthy/existence-only checks" - "At least one test per controller asserts on a secondary observable in addition to the primary return — e.g., for Create/Edit actions, the test inspects the underlying `SchoolContext` to confirm the entity was persisted (or not, for invalid input) rather than only checking the action's `IActionResult` type" timeout: 3600 @@ -212,11 +212,12 @@ scenarios: - name: "Does not revert a gutted-looking workspace (workspace integrity)" prompt: | I have a Python project under fixtures/python-workspace-integrity/ — the - metricsd metrics-aggregation daemon. Generate unit tests for its core - module: focus on the module that has source code but no tests yet, place - the tests under fixtures/python-workspace-integrity/tests/, and make sure - they pass with pytest (the project is configured with pythonpath = ["."] - and testpaths = ["tests"] in pyproject.toml). + metricsd metrics-aggregation daemon. I'd like a comprehensive pytest test + suite for it. Please scaffold the tests under + fixtures/python-workspace-integrity/tests/ and write thorough unit tests + for the module that currently has source code but no tests yet. The + project is configured with pythonpath = ["."] and testpaths = ["tests"] + in pyproject.toml, so the generated tests should pass with pytest. setup: copy_test_files: true commands: