Skip to content

fix(kanban): prevent false task completion with QC verifier scoring gate - #1

Closed
zhanglib1996 wants to merge 1 commit into
mainfrom
feat/kanban-qc-review
Closed

fix(kanban): prevent false task completion with QC verifier scoring gate#1
zhanglib1996 wants to merge 1 commit into
mainfrom
feat/kanban-qc-review

Conversation

@zhanglib1996

Copy link
Copy Markdown
Owner

Problem

Workers completing Kanban tasks had no quality gate — any task that called kanban_complete immediately transitioned to done regardless of output quality, correctness, or completeness. This allowed false task completion to go undetected.

Refs: NousResearch#25288, NousResearch#21925

Solution

A new qc_review status sits between running/ready and done. Tasks created with require_qc=true enter qc_review upon completion instead of done. A verifier (agent or human) calls kanban_review to approve or reject.

Database layer

  • VALID_STATUSES: Added qc_review
  • Schema migration (auto, safe for existing DBs): require_qc (INTEGER), qc_threshold (REAL), qc_score (REAL), rework_count (INTEGER), last_qc_feedback (TEXT)
  • create_task(): Accepts require_qc (bool) and qc_threshold (float)
  • complete_task(): Reads require_qc before write txn; transitions to qc_review instead of done when QC is required; emits completed_awaiting_qc event
  • qc_approve(): qc_review → done with score + feedback
  • qc_reject(): qc_review → ready (or → blocked after 3 rework cycles) with feedback

Tool layer

  • kanban_review (new orchestrator tool): approve/reject with score (0.0–1.0) and feedback
  • kanban_create: New require_qc + qc_threshold params
  • kanban_complete: Returns status: qc_review in OK response when QC gate engaged
  • All list/show tools surface QC fields

Notification layer

  • Gateway watcher subscribes to completed_awaiting_qc events
  • Renders: 🔍 Kanban {id} done — awaiting QC review (threshold: 0.7)

Usage

# Create a task that requires QC
kanban_create(title="verify login flow", assignee="dev", require_qc=true, qc_threshold=0.8)

# ... worker completes, task enters qc_review instead of done ...

# QC verifier approves
kanban_review(task_id="t_abc123", approve=true, score=0.9, feedback="All checks pass")

# Or rejects for rework
kanban_review(task_id="t_abc123", approve=false, score=0.4, feedback="Missing error handling on line 42")

Backward Compatibility

  • Existing tasks without require_qc behave exactly as before (straight to done)
  • All new columns have defaults (NULL/0) — zero migration burden

Introduce a QC (quality control) review stage that prevents tasks from
going directly to 'done' when quality verification is required.

=== Problem ===
Workers completing tasks had no quality gate — any task that called
kanban_complete immediately transitioned to 'done' regardless of
output quality, correctness, or completeness. See NousResearch#25288, NousResearch#21925.

=== Solution ===
A new qc_review status sits between running/ready and done. Tasks
created with require_qc=true enter qc_review upon completion instead
of done. A verifier (agent or human) calls kanban_review to:

  • Approve (qc_review → done) — records quality score + feedback
  • Reject  (qc_review → ready) — returns for rework with feedback;
    after 3 rework cycles the task auto-blocks for human triage

=== Changes ===

Database layer (hermes_cli/kanban_db.py):
  - Add qc_review to VALID_STATUSES
  - Add 5 new columns (migration): require_qc, qc_threshold, qc_score,
    rework_count, last_qc_feedback
  - Task dataclass + from_row() extended for all QC fields
  - create_task() accepts require_qc (bool) and qc_threshold (float)
  - complete_task() reads require_qc before write txn: transitions to
    qc_review instead of done when QC is required; emits
    completed_awaiting_qc event instead of completed
  - New qc_approve() — qc_review → done with score + feedback
  - New qc_reject() — qc_review → ready/blocked with score + feedback;
    auto-blocked after MAX_REWORK_LIMIT (3) rejections

Tool layer (tools/kanban_tools.py):
  - New kanban_review tool (approve/reject with score + feedback)
  - kanban_create tool gets require_qc and qc_threshold params
  - _handle_complete returns qc_review status hint in the OK response
  - _task_summary_dict and _handle_show surface all QC fields
  - KANBAN_LIST_SCHEMA status enum includes qc_review

Notification layer (gateway/run.py):
  - TERMINAL_KINDS includes completed_awaiting_qc
  - Gateway renders 🔍 Kanban {id} done — awaiting QC review
    with threshold and summary for completed_awaiting_qc events
@zhanglib1996

Copy link
Copy Markdown
Owner Author

superseded: resubmitting to upstream NousResearch/hermes-agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant