Skip to content

fix(kanban): verify created artifacts before marking task done (#25288) - #25328

Closed
xxxigm wants to merge 4 commits into
NousResearch:mainfrom
xxxigm:fix/kanban-verify-artifact-completion-25288
Closed

fix(kanban): verify created artifacts before marking task done (#25288)#25328
xxxigm wants to merge 4 commits into
NousResearch:mainfrom
xxxigm:fix/kanban-verify-artifact-completion-25288

Conversation

@xxxigm

@xxxigm xxxigm commented May 14, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Stops Kanban workers from marking tasks as done while the cron job (or other artifact) they claim to have created does not actually exist. Fixes the silent-success path described in #25288.

Three small additions wired together:

  1. Kernel gate (hermes_cli/kanban_db.py) — complete_task gains a created_artifacts=[{kind, id}] parameter parallel to the existing created_cards gate. kind="cron" is verified via cron.jobs.get_job(id); phantom entries raise HallucinatedArtifactsError + audit event, task stays running so the worker can fix the claim and retry. Unknown kinds land on an advisory bucket so plugins can ship new kinds ahead of their verifier.

  2. Cron idempotency (cron/jobs.py) — create_job gains an optional idempotency_key; same key returns the existing job with reused=true instead of duplicating. New find_jobs() lookup helper used by the kernel verifier.

  3. Tool surface + agent teachingkanban_complete schema adds created_artifacts with a structured retry-friendly tool_error on rejection. cronjob schema adds idempotency_key. KANBAN_GUIDANCE rules 5a + 5b (compressed, +530 chars over baseline) and the kanban-worker SKILL pick up the new pattern with GOOD/BAD examples.

Related Issue

Fixes #25288

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • cron/jobs.pyidempotency_key on create_job + new find_jobs(idempotency_key=…, name=…) helper.
  • hermes_cli/kanban_db.py — new HallucinatedArtifactsError, ARTIFACT_VERIFIERS registry (today: cron), _normalize_artifacts, _verify_created_artifacts, plus created_artifacts=… on complete_task.
  • tools/kanban_tools.pykanban_complete schema + handler thread created_artifacts through and translate the new error into a structured tool_error (still in-flight, retry hint, three-option decision tree).
  • tools/cronjob_tools.pycronjob action='create' threads idempotency_key through; response now includes reused: bool.
  • agent/prompt_builder.pyKANBAN_GUIDANCE rules 5a/5b + Do-NOT entry; compressed to fit cached-prompt budget.
  • tests/tools/test_kanban_artifact_gate.py33 new tests covering the gate truth-table, complete_task end-to-end (incl. exact [BUG] Agent marks Kanban task as DONE but cron was never created #25288 reproduction), tool error shape, cron idempotency, and prompt teaching anchors.
  • tests/tools/test_kanban_tools.py — bumped KANBAN_GUIDANCE size guard from 4096 → 5120 with an explanatory docstring.
  • skills/devops/kanban-worker/SKILL.md — two new sections mirroring the existing created_cards pattern.

Backwards compatible: every new field defaults to None, every existing call site keeps working, no schema migration.

How to Test

# New gate suite (33 tests, ~1.3s, no real Chromium / cron daemon)
./scripts/run_tests.sh tests/tools/test_kanban_artifact_gate.py

# Full regression across kanban + cron + tools (excluding two pre-existing
# sandbox-env failures unrelated to this PR — confirmed unchanged on main)
./scripts/run_tests.sh tests/tools/test_kanban_artifact_gate.py \
    tests/tools/test_kanban_tools.py tests/tools/test_cronjob_tools.py \
    tests/cron/test_jobs.py -k "not test_loads_cursor_rules_mdc"
# expected: 192 passed

End-to-end behaviour after the fix:

> agent: kanban_complete(summary="Created cron ghost123 to monitor h13b",
                         created_artifacts=[{"kind": "cron", "id": "ghost123"}])
< tool_error: kanban_complete blocked: created_artifacts could not be verified:
              cron=ghost123 (no cron job with this id exists). Your task is still
              in-flight (no state change). Either (a) actually create the missing
              artifact and retry with the real id, (b) drop the bogus entry, or
              (c) call kanban_block. Do NOT mark the task done while the artifact
              is missing — that's the failure mode reported in #25288.

The agent's correct response (now taught by the prompt + SKILL):

> agent: cronjob(action="create", prompt="...", schedule="every 30m",
                 idempotency_key=$HERMES_KANBAN_TASK)
< {"success": true, "job_id": "abc123def456", "reused": false, ...}
> agent: kanban_complete(summary="Created cron abc123def456 to monitor h13b",
                         created_artifacts=[{"kind": "cron", "id": "abc123def456"}])
< {"ok": true, ...}   # task → done; completed event records verified_artifacts

A retry of the cron creation with the same idempotency_key returns reused: true instead of duplicating the job — fixes the second half of the post-mortem.

Checklist

  • Conventional Commits (feat(cron):, fix(kanban):, docs(kanban):, test(kanban):)
  • 4 focused commits, single author
  • 33 new tests pass; 192 existing kanban/cron/tool tests pass; 2 unrelated pre-existing failures confirmed unchanged on main
  • Tested on macOS 15.6 (darwin 24.6.0)
  • Updated KANBAN_GUIDANCE, kanban-worker SKILL, and tool schema descriptions
  • No new config keys, no architecture change, no platform-specific calls

xxxigm added 4 commits May 14, 2026 08:04
First call with the key mints a job; subsequent calls return the
existing job unchanged with response.reused=true. Prevents the
duplicate-cron half of NousResearch#25288 (agent retries created N copies).

Also adds find_jobs() helper used by the kanban verification gate
in the next commit.
…esearch#25288)

complete_task gains a created_artifacts gate parallel to the
existing created_cards gate. Each entry is {kind, id}; kind="cron"
is verified via cron.jobs.get_job(id). Phantom entries raise
HallucinatedArtifactsError + audit event; task stays in-flight so
the worker can fix the claim and retry.

ARTIFACT_VERIFIERS registry keeps the gate forward-compatible —
unknown kinds land on an advisory bucket and never block.

kanban_complete tool gets the matching schema field + a structured
retry-friendly tool_error following the NousResearch#22923 contract.
…h#25288)

KANBAN_GUIDANCE rules 5a (verify artifacts) + 5b (idempotency_key)
plus a Do-NOT entry. SKILL adds GOOD/BAD examples mirroring the
existing created_cards section.

Kept compact (KANBAN_GUIDANCE 4389 chars) so cached prompts pay
minimal extra tokens.
…#25288)

33 new tests cover the gate truth-table, complete_task end-to-end,
the kanban_complete tool error shape, the cronjob idempotency
plumbing, and the KANBAN_GUIDANCE teaching anchors.

Bumps the prompt-size guard from 4096 to 5120 to fit the new rules
(pre-fix prompt was already at ~3.85 KB).
@xxxigm
xxxigm force-pushed the fix/kanban-verify-artifact-completion-25288 branch from af128f4 to b180cd3 Compare May 14, 2026 01:05
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cron Cron scheduler and job management comp/tools Tool registry, model_tools, toolsets labels May 14, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks @xxxigm — closing this one. The artifact-verification gate + cron idempotency_key (1141 LOC) is substantive policy territory — similar in spirit to #21925 (verifier evidence) and #25356 (QC review), both of which I also closed for design-discussion reasons. Three different proposed designs for the same concern (gating completion on quality) competing in our PR queue is a signal that the design hasn't been chosen yet. If you want to revisit, please open an issue first comparing the three approaches. Appreciate the careful work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/tools Tool registry, model_tools, toolsets P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Agent marks Kanban task as DONE but cron was never created

3 participants