Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion .agents/scripts/commands/full-loop.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ The AI will iterate on the task until outputting:

When running as a headless worker (dispatched by the supervisor via `opencode run` or `Claude -p`), the `--headless` flag is passed automatically. The full-loop-helper.sh script enforces these rules:

1. **NEVER prompt for user input** - There is no human at the terminal. If you encounter ambiguity, make a reasonable decision and document it in a commit message. If truly blocked, exit cleanly so the supervisor can evaluate and retry.
1. **NEVER prompt for user input** - There is no human at the terminal. Use the uncertainty decision framework (rule 7) to decide whether to proceed or exit.

2. **Do NOT edit TODO.md** - Put notes in commit messages or PR body instead. See `workflows/plans.md` "Worker TODO.md Restriction".

Expand All @@ -184,6 +184,26 @@ When running as a headless worker (dispatched by the supervisor via `opencode ru

6. **git pull --rebase before push** (t174) - The PR create phase automatically runs `git pull --rebase` to sync with any remote changes before pushing, avoiding push rejections.

7. **Uncertainty decision framework** (t176) - When facing ambiguity, use this decision tree:

**PROCEED autonomously** (document decision in commit message):
- Multiple valid approaches exist but all achieve the goal — pick the simplest
- Style/naming choices are ambiguous — follow existing codebase conventions
- Task description is slightly vague but intent is clear from context
- Choosing between equivalent libraries/patterns — match project precedent
- Minor scope questions (e.g., fix adjacent issue?) — stay focused on assigned task

**EXIT cleanly** (include clear explanation in output):
- Task description contradicts what you find in the codebase
- Completing the task requires breaking changes to public APIs or shared interfaces
- The task is already done or obsolete
- Required dependencies, credentials, or services are missing and cannot be inferred
- The task requires architectural decisions that affect other tasks
- Unsure whether to create vs modify a file, and getting it wrong risks data loss

When proceeding, document the choice: `feat: add retry logic (chose exponential backoff — matches existing patterns)`
When exiting, be specific: `BLOCKED: Task says 'update auth endpoint' but 3 exist (JWT, OAuth, API key). Need clarification.`

Comment on lines +187 to +206
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# Check the full-loop.md file to understand context
wc -l .agents/scripts/commands/full-loop.md

Repository: marcusquinn/aidevops

Length of output: 105


🏁 Script executed:

# Run the suggested search script to check for duplicates
rg -i "proceed autonomously|exit cleanly|uncertainty|decision framework" .agents/ \
  -g '*.md' -C 2

# Check for similar proceed/exit criteria
rg -i "multiple valid approaches|style.*naming.*ambiguous|breaking.*api.*change|architectural decision" .agents/ \
  -g '*.md' -C 2

# Check for similar headless worker guidance
rg -i "headless.*worker.*guidance|worker.*proceed|worker.*exit" .agents/ \
  -g '*.md' -C 2

Repository: marcusquinn/aidevops

Length of output: 14501


🏁 Script executed:

# Check AGENTS.md specifically for decision-making or uncertainty guidance
cat -n .agents/AGENTS.md | head -150

Repository: marcusquinn/aidevops

Length of output: 8918


🏁 Script executed:

# List all .md files in .agents/ to understand structure
find .agents -name "*.md" -type f | head -20

Repository: marcusquinn/aidevops

Length of output: 704


Consolidate framework reference — duplicates existing content in headless-dispatch.md

This uncertainty decision framework is already fully documented as "Worker Uncertainty Framework" in .agents/tools/ai-assistants/headless-dispatch.md, including the decision tree, proceed/exit criteria, and supervisor integration loop. Repeating the entire framework here violates the single source of truth principle.

Replace the duplicate with a cross-reference:

7. **Uncertainty decision framework** (t176) - When facing ambiguous decisions, consult the [Worker Uncertainty Framework](/.agents/tools/ai-assistants/headless-dispatch.md#worker-uncertainty-framework) in headless-dispatch.md. Document your decision choice or exit reason in the commit message or output log.

This maintains the guidance while eliminating duplication and directing readers to the authoritative, detailed source.

🧰 Tools
🪛 LanguageTool

[style] ~194-~194: Consider using a different verb for a more formal wording.
Context: ...edent - Minor scope questions (e.g., fix adjacent issue?) — stay focused on assi...

(FIX_RESOLVE)

🤖 Prompt for AI Agents
In @.agents/scripts/commands/full-loop.md around lines 187 - 206, Replace the
duplicated "Uncertainty decision framework (t176)" block with a single-line
cross-reference to the canonical "Worker Uncertainty Framework" in
headless-dispatch.md; specifically remove the full decision-tree content and
instead add the suggested sentence referencing the Worker Uncertainty Framework
(headless-dispatch.md#worker-uncertainty-framework) and instruct readers to
document their decision or exit reason in the commit message or output log, so
the section titled "Uncertainty decision framework (t176)" points to the
authoritative source rather than duplicating it.

**README gate (MANDATORY - do NOT skip):**

Before emitting `TASK_COMPLETE`, answer this decision tree:
Expand Down
27 changes: 26 additions & 1 deletion .agents/scripts/supervisor-helper.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2581,13 +2581,38 @@ build_dispatch_cmd() {
fi

# t173: Explicit worker restriction — prevents TODO.md race condition
# t176: Uncertainty decision framework for headless workers
prompt="$prompt

## MANDATORY Worker Restrictions (t173)
- Do NOT edit, commit, or push TODO.md — the supervisor owns all TODO.md updates.
- Do NOT edit todo/PLANS.md or todo/tasks/* — these are supervisor-managed.
- Report status via exit code, log output, and PR creation only.
- Put task notes in commit messages or PR body, never in TODO.md."
- Put task notes in commit messages or PR body, never in TODO.md.

## Uncertainty Decision Framework (t176)
You are a headless worker with no human at the terminal. Use this framework when uncertain:

**PROCEED autonomously when:**
- Multiple valid approaches exist but all achieve the goal (pick the simplest)
- Style/naming choices are ambiguous (follow existing conventions in the codebase)
- Task description is slightly vague but intent is clear from context
- You need to choose between equivalent libraries/patterns (match project precedent)
- Minor scope questions (e.g., should I also fix this adjacent issue?) — stay focused on the assigned task

**FLAG uncertainty and exit cleanly when:**
- The task description contradicts what you find in the codebase
- Completing the task would require breaking changes to public APIs or shared interfaces
- You discover the task is already done or obsolete
- Required dependencies, credentials, or services are missing and cannot be inferred
- The task requires decisions that would significantly affect architecture or other tasks
- You are unsure whether a file should be created vs modified, and getting it wrong would cause data loss

**When you proceed autonomously**, document your decision in the commit message:
\`feat: add retry logic (chose exponential backoff over linear — matches existing patterns in src/utils/retry.ts)\`

**When you exit due to uncertainty**, include a clear explanation in your final output:
\`BLOCKED: Task says 'update the auth endpoint' but there are 3 auth endpoints (JWT, OAuth, API key). Need clarification on which one.\`"
Comment on lines 2583 to +2615
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add a deterministic “blocked” signal to avoid retry loops

The prompt instructs workers to output BLOCKED: and “exit cleanly,” but the supervisor’s evaluator doesn’t appear to deterministically consume that marker. If it’s not parsed, uncertain exits can be misclassified as clean retries. Please verify parsing and add a structured marker/exit code that the evaluator explicitly handles.

🔧 Suggested prompt tweak (paired with evaluator handling)
-**When you exit due to uncertainty**, include a clear explanation in your final output:
-`BLOCKED: Task says 'update the auth endpoint' but there are 3 auth endpoints (JWT, OAuth, API key). Need clarification on which one.`
+**When you exit due to uncertainty**, emit a machine-readable marker (and exit non-zero) so the supervisor can block deterministically:
+`TASK_BLOCKED: Task says 'update the auth endpoint' but there are 3 auth endpoints (JWT, OAuth, API key). Need clarification on which one.`
#!/bin/bash
# Verify whether any component parses BLOCKED/TASK_BLOCKED or similar markers.
rg -n "BLOCKED:|TASK_BLOCKED|uncertainty decision framework|Uncertainty Decision Framework" 
rg -n "FULL_LOOP_COMPLETE|TASK_COMPLETE|EXIT:" .agents/scripts
rg -n "evaluate_worker|extract_log_metadata" .agents/scripts

As per coding guidelines, Automation scripts - focus on: Reliability and robustness; Proper exit codes.

🤖 Prompt for AI Agents
In @.agents/scripts/supervisor-helper.sh around lines 2583 - 2615, The
supervisor prompt currently asks workers to print "BLOCKED:" but that marker
isn't deterministically parsed by the evaluator; update the prompt string
(variable prompt) to require a structured blocker token (e.g.,
TASK_BLOCKED:<JSON with reason, file, suggested_action>) and require the worker
to exit with a distinct non-zero code (e.g., exit 75) so the evaluator can
reliably detect blocked runs; then update the evaluator code paths that parse
worker output (functions like evaluate_worker and extract_log_metadata) to
explicitly look for TASK_BLOCKED and the chosen exit code and treat those runs
as "blocked" (not retriable) so retries won't be triggered.

Comment on lines 2585 to +2615

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For better readability and maintainability, consider using a heredoc to manage this large block of text. This approach is cleaner than a multi-line string assignment, as it avoids the need to escape special characters like backticks (). Using +=for string concatenation is also a more modern and readable pattern thanvar="$var..."`.

    prompt+=$(cat <<-'EOF'

## MANDATORY Worker Restrictions (t173)
- Do NOT edit, commit, or push TODO.md — the supervisor owns all TODO.md updates.
- Do NOT edit todo/PLANS.md or todo/tasks/* — these are supervisor-managed.
- Report status via exit code, log output, and PR creation only.
- Put task notes in commit messages or PR body, never in TODO.md.

## Uncertainty Decision Framework (t176)
You are a headless worker with no human at the terminal. Use this framework when uncertain:

**PROCEED autonomously when:**
- Multiple valid approaches exist but all achieve the goal (pick the simplest)
- Style/naming choices are ambiguous (follow existing conventions in the codebase)
- Task description is slightly vague but intent is clear from context
- You need to choose between equivalent libraries/patterns (match project precedent)
- Minor scope questions (e.g., should I also fix this adjacent issue?) — stay focused on the assigned task

**FLAG uncertainty and exit cleanly when:**
- The task description contradicts what you find in the codebase
- Completing the task would require breaking changes to public APIs or shared interfaces
- You discover the task is already done or obsolete
- Required dependencies, credentials, or services are missing and cannot be inferred
- The task requires decisions that would significantly affect architecture or other tasks
- You are unsure whether a file should be created vs modified, and getting it wrong would cause data loss

**When you proceed autonomously**, document your decision in the commit message:
`feat: add retry logic (chose exponential backoff over linear — matches existing patterns in src/utils/retry.ts)`

**When you exit due to uncertainty**, include a clear explanation in your final output:
`BLOCKED: Task says 'update the auth endpoint' but there are 3 auth endpoints (JWT, OAuth, API key). Need clarification on which one.`
EOF
    )


if [[ -n "$memory_context" ]]; then
prompt="$prompt
Expand Down
72 changes: 71 additions & 1 deletion .agents/tools/ai-assistants/headless-dispatch.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ tools:
**When NOT to use**:

- Interactive development (use TUI directly)
- Tasks requiring human-in-the-loop decisions mid-execution
- Tasks requiring frequent human-in-the-loop decisions (see [Worker Uncertainty Framework](#worker-uncertainty-framework) for what workers can handle autonomously)
- Single quick questions (just use `opencode run` without server overhead)

**Draft agents for reusable context**: When parallel workers share domain-specific instructions, create a draft agent in `~/.aidevops/agents/draft/` instead of duplicating prompts. Subsequent dispatches can reference the draft. See `tools/build-agent/build-agent.md` "Agent Lifecycle Tiers" for details.
Expand Down Expand Up @@ -457,6 +457,76 @@ export OPENAI_API_KEY="sk-..."
OPENCODE_PERMISSION='{"*":"allow"}' opencode run "Fix the failing tests"
```

## Worker Uncertainty Framework

Headless workers have no human to ask when they encounter ambiguity. This framework defines when workers should make autonomous decisions vs flag uncertainty and exit.

### Decision Tree

```text
Encounter ambiguity
├── Can I infer intent from context + codebase conventions?
│ ├── YES → Proceed, document decision in commit message
│ └── NO ↓
├── Would getting this wrong cause irreversible damage?
│ ├── YES → Exit cleanly with specific explanation
│ └── NO ↓
├── Does this affect only my task scope?
│ ├── YES → Proceed with simplest valid approach
│ └── NO → Exit (cross-task architectural decisions need human input)
```

### Proceed Autonomously

Workers should make their own call and keep going when:

| Situation | Action |
|-----------|--------|
| Multiple valid approaches, all achieve the goal | Pick the simplest |
| Style/naming ambiguity | Follow existing codebase conventions |
| Slightly vague task description, clear intent | Interpret reasonably, document in commit |
| Choosing between equivalent patterns/libraries | Match project precedent |
| Minor adjacent issue discovered | Stay focused on assigned task, note in PR body |
| Unclear test coverage expectations | Match coverage level of neighboring files |

**Always document**: Include the decision rationale in the commit message so the supervisor and reviewers understand why.

```text
feat: add retry logic (chose exponential backoff over linear — matches existing patterns in src/utils/retry.ts)
```

### Flag Uncertainty and Exit

Workers should exit cleanly (allowing supervisor evaluation and retry) when:

| Situation | Why exit |
|-----------|----------|
| Task contradicts codebase state | May be stale or misdirected |
| Requires breaking public API changes | Cross-cutting impact needs human judgment |
| Task appears already done or obsolete | Avoid duplicate/conflicting work |
| Missing dependencies, credentials, or services | Cannot be inferred safely |
| Architectural decisions affecting other tasks | Supervisor coordinates cross-task concerns |
| Create vs modify ambiguity with data loss risk | Irreversible — needs confirmation |
| Multiple interpretations with very different outcomes | Wrong guess wastes compute and creates cleanup work |

**Always explain**: Include a specific, actionable description of the blocker so the supervisor can resolve it.

```text
BLOCKED: Task says 'update the auth endpoint' but there are 3 auth endpoints
(JWT in src/auth/jwt.ts, OAuth in src/auth/oauth.ts, API key in src/auth/apikey.ts).
Need clarification on which one(s) to update.
```

### Integration with Supervisor

The supervisor uses worker exit behavior to drive the self-improvement loop:

- **Worker proceeds + documents** → Supervisor reviews PR normally
- **Worker exits with BLOCKED** → Supervisor reads explanation, either clarifies and retries, or creates a prerequisite task
- **Worker exits with unclear error** → Supervisor dispatches a diagnostic worker (`-diag-N` suffix)

This framework reduces wasted retries by giving workers clear criteria for when to attempt vs when to bail. Over time, task descriptions improve because the supervisor learns which ambiguities cause exits.

## CI/CD Integration

### GitHub Actions
Expand Down
2 changes: 1 addition & 1 deletion TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ Tasks with no open blockers - ready to work on. Use `/ready` to refresh this lis
- Notes: PR #642 merged. Added `--headless` flag to full-loop for autonomous worker operation.

- [ ] t175 Fix `ambiguous_skipped_ai` evaluation — add better heuristic signals #bug #supervisor ~1h (ai:40m) ref:GH#644 assignee:marcusquinn started:2026-02-08T19:38:50Z logged:2026-02-08
- Notes: Recurring evaluation outcome across batches. Evaluator can't determine success/failure, skips AI eval, defaults to retry. Add heuristics: check for commits on branch, check for uncommitted changes in worktree.
- Notes: Recurring evaluation outcome across batches. Evaluator can't determine success/failure, skips AI eval, defaults to retry. Add heuristics: check for commits on branch, check for uncommitted changes in worktree. BLOCKED: Re-prompt dispatch failed: ambiguous_skipped_ai

- [ ] t176 Add uncertainty guidance to worker dispatch prompt #feature #supervisor ~30m (ai:20m) ref:GH#645 assignee:marcusquinn started:2026-02-08T19:38:55Z logged:2026-02-08
- Notes: Workers don't know when to make autonomous decisions vs flag uncertainty. Add decision framework to dispatch prompt for headless workers.
Expand Down
Loading