fix(kanban): gate task.skills on resolvability to prevent worker crash loops - #30025
fix(kanban): gate task.skills on resolvability to prevent worker crash loops#30025noestelar wants to merge 1 commit into
Conversation
…h loops The dispatcher already gates the built-in --skills kanban-worker injection on resolvability under the worker's HERMES_HOME — a missing skill is fatal at CLI startup (ValueError: Unknown skill(s): <name>), aborting the worker before the agent loop runs. Per-task skills from task.skills were passed through unconditionally, so a task whose skill name only exists in the global skills root — not the profile-scoped one the worker actually loads — crash-loops the worker until the watchdog auto-blocks it. Observed in the wild: a task with skills=['mlflow-eval-datasets-for-llm-pipelines'] on a profile-scoped HERMES_HOME respawned 1032 times before being blocked manually, because the skill lives in ~/.hermes/skills/mlops/ but not in ~/.hermes/profiles/work/skills/mlops/. Changes: - Generalize _kanban_worker_skill_available() into _skill_available_for_home(skill_name, hermes_home); the legacy name is kept as a back-compat shim that delegates. - Apply the same gate to task.skills entries in _default_spawn. Skipped skills emit a stderr warning naming the task id so operators see why the worker did not pick up the requested skill; the task still proceeds rather than crash-looping. - Existing tests stub the new helper to keep synthetic skill names resolvable; a new regression test exercises the mixed-resolvability path and asserts the warning surfaces the task id. 56/56 spawn+skill tests in test_kanban_core_functionality.py and test_kanban_db.py pass.
|
Thanks for tackling this crash-loop class. I agree with the core direction: the dispatcher should not pass One semantics question: should unresolved I think Kanban should treat current
A future extension could add an explicit optional skill channel, for example Suggested behavior:
Tests I would expect:
Separately, I opened a related feature request for task-scoped read-only skill overlays plus a lightweight orchestrator skill catalog: #33245. That feels bigger than this PR and probably should not be mixed into the minimal crash-loop fix. |
|
I did a related-issue/PR scan to place this PR in the existing Kanban/skills work. The short version: this PR is the right place for the minimal crash-loop fix, while broader task-scoped skill availability probably belongs in #33245. Relevant nearby work:
For the broader feature side:
My recommendation after the scan:
So the design split I’d suggest is:
|
|
Follow-up from live Kanban dogfood today: the fail-closed preflight semantics are still the right default, but there is one missing middle step for installs that have an explicit profile-skill sync policy. Observed workflow:
So I think the safest contract is slightly more nuanced than just “block missing skill” or “drop missing skill”: This preserves the fail-closed behavior I argued for earlier, but avoids human intervention for known-good shared skills where the local policy already says the profile may receive them. It also avoids the dangerous version of this fix: blindly copying arbitrary missing skills into every profile. The sync step must be policy-gated. I tested a local patch with these semantics:
Regression coverage I added locally:
Verification: This may be too local-policy-specific to land directly in this PR as-is, but it is important for the design discussion: “missing required skill” should not necessarily mean “block immediately” if the system has an explicit, allowlisted way to make that exact skill available to that exact profile before spawn. It should mean “resolve/sync through approved mechanisms, then block if still unavailable.” |
|
Related follow-up opened from It keeps this PR’s core safety direction (do not pass unknown This is meant as an operational refinement for profile-scoped Hermes/Kanban installs, not a broad implicit mutation path. |
|
Follow-up from dogfooding: the preflight block prevents worker startup crashes, but repeated local Kanban runs showed the next root fix should be sync-before-block for allowlisted forced skills. I could not push directly to this fork branch (GitHub returned 403 for noestelar/hermes-agent), so I published a clean one-commit branch here:
What it adds:
Verification:
=== Summary: 1 files, 166 tests passed, 0 failed (100% complete) in 7.0s (40 workers) ===
This keeps the original safety guarantee from this PR while avoiding the manual “sync profile skill, unblock, redispatch” loop for known shared skills like / . |
teknium1
left a comment
There was a problem hiding this comment.
Thanks for isolating the startup crash path. The premise remains present on current main: _default_spawn() still passes every task.skills entry to --skills (hermes_cli/kanban_db.py:8036-8039), while CLI startup raises if none load (cli.py:15931-15947).
Problems
- The new probe at
hermes_cli/kanban_db.py:5231only scans<HERMES_HOME>/skills/<name>/SKILL.md. That is narrower than the worker resolver:tools/skills_tool.py:1066-1087includes configuredskills.external_dirs,tools/skills_tool.py:1155-1170accepts frontmattername:aliases, andtools/skills_tool.py:1182-1204rejects ambiguous candidates. The proposed filter would therefore drop valid task skills before the CLI can load them. - The added tests mock the probe, so they do not exercise those resolver cases.
Suggested changes
- Rework the guard to match the worker's effective resolver and add real filesystem tests for external directories, frontmatter aliases, and ambiguity, alongside the missing-skill regression.
Automated hermes-sweeper review.
| # trees (a few dozen entries); short-circuits on first match. | ||
| try: | ||
| for skill_md in skills_root.rglob("kanban-worker/SKILL.md"): | ||
| for skill_md in skills_root.rglob(f"{skill_name}/SKILL.md"): |
There was a problem hiding this comment.
This probe is narrower than the worker resolver: skill_view searches configured skills.external_dirs, accepts frontmatter name: aliases, and rejects ambiguous matches (tools/skills_tool.py:1066-1087, 1155-1204). A valid external or aliased task skill would be dropped here; please use equivalent resolution semantics.
Problem
The dispatcher already gates the built-in
--skills kanban-workerinjection on resolvability under the worker'sHERMES_HOME(see_kanban_worker_skill_availableathermes_cli/kanban_db.py:5202). The reason is documented in that function's docstring: preloading a missing skill is fatal at CLI startup (ValueError: Unknown skill(s): <name>), aborting the worker before the agent loop runs.But per-task
task.skillsentries (hermes_cli/kanban_db.py:5387-5389before this patch) were passed through unconditionally:So a task whose skill name only exists in the global skills root — not the profile-scoped one the worker actually loads — crash-loops the worker until the dispatcher's watchdog eventually auto-blocks it.
Observed impact
A task with
skills=['mlflow-eval-datasets-for-llm-pipelines']running under a profile-scopedHERMES_HOMErespawned 1032 times before being blocked manually, because the skill lives in~/.hermes/skills/mlops/but not in~/.hermes/profiles/<name>/skills/mlops/. The CLI fails inside_apply_skills_overridesbefore any tool can callblock, so the lifecycle's normal error path never runs.Watchdog auto-block fires far too late (hundreds of respawns) when the failure happens before the agent loop.
Fix
_kanban_worker_skill_available()into_skill_available_for_home(skill_name, hermes_home). The legacy name is kept as a back-compat shim that delegates, so external callers (if any) aren't broken.task.skillsentry in_default_spawn. Skipped skills emit a single-line stderr warning naming the task id and the missing skill so operators triaging logs see why the worker did not pick up the requested skill. The task still proceeds — running without the supplementary skill context is strictly better than crash-looping until the watchdog gives up.Contract change
This is a deliberate fail-fast → silent-skip-with-warning shift for
task.skills. Trade-off:If a stricter posture is preferred — e.g.
task.skillsis treated as a hard contract and should fail the task immediately — I'm happy to flip this tokanban_db.block_task(...)instead of a warning. The current behavior matches what_kanban_worker_skill_availablealready does for the built-in skill.Tests
test_default_spawn_appends_per_task_skills,test_default_spawn_dedupes_kanban_worker_from_task_skills) updated to stub the new helper so synthetic skill names (translation,github-code-review) still resolve.test_default_spawn_drops_unresolvable_task_skillsexercises the mixed-resolvability path: one resolvable skill + one missing skill, asserts only the resolvable one reaches argv, and asserts the stderr warning contains both the task id and the missing skill name.(
spawn or skillfilter acrosstests/hermes_cli/test_kanban_core_functionality.py+tests/hermes_cli/test_kanban_db.py.)Files
hermes_cli/kanban_db.py— generalize helper + gatetask.skillstests/hermes_cli/test_kanban_core_functionality.py— stub new helper in existing tests + add regression test