From 96deb8bd5d5c03e787d099fc1b7a191390baae8a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Mon, 22 Jun 2026 17:25:13 +0200 Subject: [PATCH 1/2] Fix code-testing-agent activation for the Flask pytest scenario The 'Generate pytest tests for the Flask tasks API' scenario failed to activate code-testing-agent in BOTH isolated and plugin mode: its prompt enumerated, file by file, exactly what to mock/inject/test (TaskService with repo mocked + clock injected, queries.apply_query over fixed lists, both repositories, the blueprint via test_client), acting as an answer key that let the base agent generate tests directly with edit tools instead of routing to the skill's research-plan-implement pipeline. Rewrite the prompt to a realistic, high-level ask (mirroring the ContosoUniversity scenario that does activate): describe the app at a layer level, keep the 'no tests yet', project-wide multi-file framing and the 80% coverage floor, and drop the per-module test checklist. Assertions, rubric and timeout are unchanged. Verified locally that the skill now activates in both isolated and plugin mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .../code-testing-agent/eval.vally.yaml | 41 +++++++------------ .../dotnet-test/code-testing-agent/eval.yaml | 41 +++++++------------ 2 files changed, 28 insertions(+), 54 deletions(-) diff --git a/tests/dotnet-test/code-testing-agent/eval.vally.yaml b/tests/dotnet-test/code-testing-agent/eval.vally.yaml index c06693fc5e..80c51225b0 100644 --- a/tests/dotnet-test/code-testing-agent/eval.vally.yaml +++ b/tests/dotnet-test/code-testing-agent/eval.vally.yaml @@ -64,35 +64,22 @@ stimuli: - name: Generate pytest tests for the Flask tasks API (Python polyglot) prompt: | I have a Python Flask web application under fixtures/python-flask-tasks/ - — a tasks REST API with a TaskService layer (with an injected clock and - configurable max title length), a TaskRepository Protocol with two - backends (InMemoryTaskRepository, SqliteTaskRepository), a separate - queries module (TaskQuery / TaskPage / apply_query) for - filter/sort/pagination, and a Flask blueprint exposing /tasks CRUD, - tag endpoints, and listing with status / tag / q / overdue / sort / - order / limit / offset query parameters (see - fixtures/python-flask-tasks/README.md for the layout). There are no - test files yet — the tests/ directory contains only a .gitkeep - marker. This is a project-wide, multi-file test generation task - across the service layer, the query layer, both repositories, and - the Flask blueprint. + — a tasks REST API with a service layer, a repository abstraction with + in-memory and SQLite backends, a query/filter/pagination module, and a + Flask blueprint exposing the /tasks CRUD, tag, and listing endpoints + (see fixtures/python-flask-tasks/README.md for the layout). This is a + project-wide, multi-file test generation task. There are no tests yet — + the tests/ directory contains only a .gitkeep marker. Please scaffold a comprehensive pytest test suite under - fixtures/python-flask-tasks/tests/, then write comprehensive unit - and integration tests across the tasks_api package — covering - TaskService (with the repository mocked and the clock injected), - the queries.apply_query function over fixed Task lists, - InMemoryTaskRepository, SqliteTaskRepository against an in-memory - connection, and the Flask blueprint via Flask's test_client. - - Coverage tooling (pytest-cov) is pre-configured in pyproject.toml - with a hard floor of 80% line + branch coverage on the tasks_api - package. The generated tests should pass with: `python -m pip - install -e ".[test]"` then `python -m pytest` from the - fixtures/python-flask-tasks/ directory — the run fails (and the - coverage XML report under fixtures/python-flask-tasks/coverage.xml - is not produced) if your tests do not clear the 80% floor across - every module. + fixtures/python-flask-tasks/tests/ and write thorough unit and + integration tests across the tasks_api package. Achieve high coverage: + pytest-cov is pre-configured in pyproject.toml with a hard floor of 80% + line + branch coverage on the tasks_api package. The generated tests + should pass with `python -m pip install -e ".[test]"` then + `python -m pytest` from the fixtures/python-flask-tasks/ directory, + producing the coverage XML report under + fixtures/python-flask-tasks/coverage.xml. environment: files: - src: . diff --git a/tests/dotnet-test/code-testing-agent/eval.yaml b/tests/dotnet-test/code-testing-agent/eval.yaml index 831a2072ca..6beb526965 100644 --- a/tests/dotnet-test/code-testing-agent/eval.yaml +++ b/tests/dotnet-test/code-testing-agent/eval.yaml @@ -59,35 +59,22 @@ scenarios: - name: "Generate pytest tests for the Flask tasks API (Python polyglot)" prompt: | I have a Python Flask web application under fixtures/python-flask-tasks/ - — a tasks REST API with a TaskService layer (with an injected clock and - configurable max title length), a TaskRepository Protocol with two - backends (InMemoryTaskRepository, SqliteTaskRepository), a separate - queries module (TaskQuery / TaskPage / apply_query) for - filter/sort/pagination, and a Flask blueprint exposing /tasks CRUD, - tag endpoints, and listing with status / tag / q / overdue / sort / - order / limit / offset query parameters (see - fixtures/python-flask-tasks/README.md for the layout). There are no - test files yet — the tests/ directory contains only a .gitkeep - marker. This is a project-wide, multi-file test generation task - across the service layer, the query layer, both repositories, and - the Flask blueprint. + — a tasks REST API with a service layer, a repository abstraction with + in-memory and SQLite backends, a query/filter/pagination module, and a + Flask blueprint exposing the /tasks CRUD, tag, and listing endpoints + (see fixtures/python-flask-tasks/README.md for the layout). This is a + project-wide, multi-file test generation task. There are no tests yet — + the tests/ directory contains only a .gitkeep marker. Please scaffold a comprehensive pytest test suite under - fixtures/python-flask-tasks/tests/, then write comprehensive unit - and integration tests across the tasks_api package — covering - TaskService (with the repository mocked and the clock injected), - the queries.apply_query function over fixed Task lists, - InMemoryTaskRepository, SqliteTaskRepository against an in-memory - connection, and the Flask blueprint via Flask's test_client. - - Coverage tooling (pytest-cov) is pre-configured in pyproject.toml - with a hard floor of 80% line + branch coverage on the tasks_api - package. The generated tests should pass with: `python -m pip - install -e ".[test]"` then `python -m pytest` from the - fixtures/python-flask-tasks/ directory — the run fails (and the - coverage XML report under fixtures/python-flask-tasks/coverage.xml - is not produced) if your tests do not clear the 80% floor across - every module. + fixtures/python-flask-tasks/tests/ and write thorough unit and + integration tests across the tasks_api package. Achieve high coverage: + pytest-cov is pre-configured in pyproject.toml with a hard floor of 80% + line + branch coverage on the tasks_api package. The generated tests + should pass with `python -m pip install -e ".[test]"` then + `python -m pytest` from the fixtures/python-flask-tasks/ directory, + producing the coverage XML report under + fixtures/python-flask-tasks/coverage.xml. setup: copy_test_files: true assertions: From 038fc9253af951f0c24095a60d19622c4f92b491 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Amaury=20Lev=C3=A9?= Date: Mon, 22 Jun 2026 20:14:37 +0200 Subject: [PATCH 2/2] Use python3 in Flask prompt to match the grader Review feedback: the prompt told the agent to run \python -m ...\ but the grader (and the vally command) invoke \python3\. On Linux runners that may lack a \python\ shim the agent could hit command-not-found. Align the prompt to \python3\ in both eval.yaml and eval.vally.yaml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- tests/dotnet-test/code-testing-agent/eval.vally.yaml | 4 ++-- tests/dotnet-test/code-testing-agent/eval.yaml | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/dotnet-test/code-testing-agent/eval.vally.yaml b/tests/dotnet-test/code-testing-agent/eval.vally.yaml index 80c51225b0..a84d8b0d6c 100644 --- a/tests/dotnet-test/code-testing-agent/eval.vally.yaml +++ b/tests/dotnet-test/code-testing-agent/eval.vally.yaml @@ -76,8 +76,8 @@ stimuli: integration tests across the tasks_api package. Achieve high coverage: pytest-cov is pre-configured in pyproject.toml with a hard floor of 80% line + branch coverage on the tasks_api package. The generated tests - should pass with `python -m pip install -e ".[test]"` then - `python -m pytest` from the fixtures/python-flask-tasks/ directory, + should pass with `python3 -m pip install -e ".[test]"` then + `python3 -m pytest` from the fixtures/python-flask-tasks/ directory, producing the coverage XML report under fixtures/python-flask-tasks/coverage.xml. environment: diff --git a/tests/dotnet-test/code-testing-agent/eval.yaml b/tests/dotnet-test/code-testing-agent/eval.yaml index 6beb526965..1a26149a82 100644 --- a/tests/dotnet-test/code-testing-agent/eval.yaml +++ b/tests/dotnet-test/code-testing-agent/eval.yaml @@ -71,8 +71,8 @@ scenarios: integration tests across the tasks_api package. Achieve high coverage: pytest-cov is pre-configured in pyproject.toml with a hard floor of 80% line + branch coverage on the tasks_api package. The generated tests - should pass with `python -m pip install -e ".[test]"` then - `python -m pytest` from the fixtures/python-flask-tasks/ directory, + should pass with `python3 -m pip install -e ".[test]"` then + `python3 -m pytest` from the fixtures/python-flask-tasks/ directory, producing the coverage XML report under fixtures/python-flask-tasks/coverage.xml. setup: