Skip to content

Reduce project scope declaration misses - #1990

Merged
Aaronontheweb merged 1 commit into
devfrom
fix/issue-1892-seeded-project-eval
Aug 18, 2026
Merged

Reduce project scope declaration misses#1990
Aaronontheweb merged 1 commit into
devfrom
fix/issue-1892-seeded-project-eval

Conversation

@Aaronontheweb

@Aaronontheweb Aaronontheweb commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1892.

This adds a seeded Git-project eval and strengthens generic project-scope guidance so agents declare the task-named path before probing it. If that declaration fails, the user-provided fallback is declared before project tools continue. The same contract is aligned across always-loaded guidance, subagents, the tool schema, and bundled project/operations skills.

It also aligns generic scratch guidance: disposable writable work outside a project uses session_dir, not platform temporary storage, unless the task explicitly requires a platform temporary path. Shell guidance asks for one operation per call and permits a pipeline only when the requested result requires it.

This changes guidance and eval coverage only. It does not change shell authority or add executable-specific parsing.

Final direct-provider evals on the exact local image:

  • project declaration before project tools: 5/5
  • unrelated negative control: 5/5
  • rejected path then valid fallback: 5/5

These trials used the direct DeepSeek API. Spark2 was unavailable and is not part of the evidence.

Validation:

  • Release build: 0 warnings, 0 errors
  • full tests: 7,550 passed, 15 expected skips
  • focused guidance/schema/skill tests: 59/59
  • strict OpenSpec, Bash syntax, headers, changed-file formatting, diff check, and changed-file Slopwatch passed
  • raw eval archives remain local and are not committed

@Aaronontheweb
Aaronontheweb enabled auto-merge (squash) August 18, 2026 16:26
@Aaronontheweb

Copy link
Copy Markdown
Collaborator Author

Sanitized post-swap evidence supports this guidance change. In one frozen fresh-session workflow, all 20 approval prompts with complete command evidence were compound shell calls; 18 used inline directory changes, 15 targeted platform temporary storage, and 19 were primarily diagnostic or read operations. A later structured file edit correctly failed because the disposable checkout was outside configured file roots. No raw session identifiers, paths, repository names, commands, or timestamps are included here. This is agent scope/tool-selection debt, not evidence for widening policy authority.

@Aaronontheweb
Aaronontheweb force-pushed the fix/issue-1892-seeded-project-eval branch from e8ae7ee to 719e4ac Compare August 18, 2026 17:07
@Aaronontheweb
Aaronontheweb merged commit 334cb4c into dev Aug 18, 2026
32 of 33 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/issue-1892-seeded-project-eval branch August 18, 2026 17:37
@Aaronontheweb Aaronontheweb mentioned this pull request Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix invalid project root in working-directory eval

1 participant