fix(kanban): prevent false task completion with QC verifier scoring gate - #1
Closed
zhanglib1996 wants to merge 1 commit into
Closed
fix(kanban): prevent false task completion with QC verifier scoring gate#1zhanglib1996 wants to merge 1 commit into
zhanglib1996 wants to merge 1 commit into
Conversation
Introduce a QC (quality control) review stage that prevents tasks from going directly to 'done' when quality verification is required. === Problem === Workers completing tasks had no quality gate — any task that called kanban_complete immediately transitioned to 'done' regardless of output quality, correctness, or completeness. See NousResearch#25288, NousResearch#21925. === Solution === A new qc_review status sits between running/ready and done. Tasks created with require_qc=true enter qc_review upon completion instead of done. A verifier (agent or human) calls kanban_review to: • Approve (qc_review → done) — records quality score + feedback • Reject (qc_review → ready) — returns for rework with feedback; after 3 rework cycles the task auto-blocks for human triage === Changes === Database layer (hermes_cli/kanban_db.py): - Add qc_review to VALID_STATUSES - Add 5 new columns (migration): require_qc, qc_threshold, qc_score, rework_count, last_qc_feedback - Task dataclass + from_row() extended for all QC fields - create_task() accepts require_qc (bool) and qc_threshold (float) - complete_task() reads require_qc before write txn: transitions to qc_review instead of done when QC is required; emits completed_awaiting_qc event instead of completed - New qc_approve() — qc_review → done with score + feedback - New qc_reject() — qc_review → ready/blocked with score + feedback; auto-blocked after MAX_REWORK_LIMIT (3) rejections Tool layer (tools/kanban_tools.py): - New kanban_review tool (approve/reject with score + feedback) - kanban_create tool gets require_qc and qc_threshold params - _handle_complete returns qc_review status hint in the OK response - _task_summary_dict and _handle_show surface all QC fields - KANBAN_LIST_SCHEMA status enum includes qc_review Notification layer (gateway/run.py): - TERMINAL_KINDS includes completed_awaiting_qc - Gateway renders 🔍 Kanban {id} done — awaiting QC review with threshold and summary for completed_awaiting_qc events
Owner
Author
|
superseded: resubmitting to upstream NousResearch/hermes-agent |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Workers completing Kanban tasks had no quality gate — any task that called
kanban_completeimmediately transitioned todoneregardless of output quality, correctness, or completeness. This allowed false task completion to go undetected.Refs: NousResearch#25288, NousResearch#21925
Solution
A new
qc_reviewstatus sits betweenrunning/readyanddone. Tasks created withrequire_qc=trueenterqc_reviewupon completion instead ofdone. A verifier (agent or human) callskanban_reviewto approve or reject.Database layer
qc_reviewrequire_qc(INTEGER),qc_threshold(REAL),qc_score(REAL),rework_count(INTEGER),last_qc_feedback(TEXT)create_task(): Acceptsrequire_qc(bool) andqc_threshold(float)complete_task(): Readsrequire_qcbefore write txn; transitions toqc_reviewinstead ofdonewhen QC is required; emitscompleted_awaiting_qceventqc_approve():qc_review → donewith score + feedbackqc_reject():qc_review → ready(or→ blockedafter 3 rework cycles) with feedbackTool layer
kanban_review(new orchestrator tool): approve/reject with score (0.0–1.0) and feedbackkanban_create: Newrequire_qc+qc_thresholdparamskanban_complete: Returnsstatus: qc_reviewin OK response when QC gate engagedNotification layer
completed_awaiting_qceventsUsage
Backward Compatibility
require_qcbehave exactly as before (straight todone)