WS-AUTO-002 — AFK Work Loop : worker autonome + heartbeats dynamiques - #6
Merged
Conversation
added 6 commits
May 3, 2026 18:42
While AH is in afk_manual or afk_auto, a new background thread (agent/afk_worker.py) ticks every 30 min, picks a tagged in_progress task from configured status files, routes it to a per-type provider/ model (Sonnet via OpenRouter for code, MiniMax for research/doc/ summary), and delegates execution to a leaf sub-agent. Each cycle is appended to ~/AFK_LOG.md (markdown) and ~/.hermes/afk_log.jsonl (machine-readable). MAX_DEPTH=50 finally has a chain to consume. Heartbeats (HB1/HB2/HB3) now embed a deterministic recap built from afk_log.jsonl — counts per status, top 3 completed tasks, current quota, cooldown level — replacing the aspirational "to be filled by AH later" wording. Safety: - §9.1 §9.3.2 §9.3.4: all parameters via ~/.hermes/config.yaml afk_worker.* (interval, status_files, threshold, model_routing, toolsets, log paths, picked-index path) - §2.4 SECCOMP: thread daemon in hermes-agent.service, no fork+exec - §2.5 budget: MiniMax quota gate at 80% (5h OR weekly) - Tags: [priority:P0|P1|P2], [afk:research|code_review|...], [afk:no], [blocked:*], [afk:decision_structurelle] (auto Discord block + skip) - Picked-index dedupe (~/.hermes/afk_worker_picked.json) avoids re-picking completed tasks; auto-purged when AH exits AFK - Cooldown: 3 errors -> cooldown_level++ (skip 2/4 ticks; halt at 3) - delegate_task gains 4 optional params (model_override, provider_override, base_url_override, api_key_override) - non- breaking, lets the worker route per-call without mutating config Tests: 33 new + 53 existing AFK tests all green (no regression).
Per Khéri review of Phase 4 Étape 2: the worker was silent during
delegation cycles (only AFK_LOG.md/jsonl tracked activity). Added
2 Discord posts per delegated cycle so Khéri can observe the worker
working in real-time:
- Pre-delegation: "🔄 AFK cycle starting" with task title, source
file/section/line, AFK type, priority, model, toolsets, hint
that the 600s timeout applies.
- Post-delegation: "✅/❌ AFK cycle done" with status, api_calls,
duration, error if any, sub-agent summary (≤500 chars), current
MiniMax quota, pointer to ~/AFK_LOG.md.
Skipped statuses (skipped_quota, skipped_no_task, skipped_cooldown)
remain silent to avoid spam — only successful/errored cycles post.
The existing decision_blocked + cooldown_halt posts are unchanged.
Behavior is gated by afk_worker.verbose in config.yaml. Default is
True (the new default — matches Khéri's preference). Set false for
silent operation if needed later.
6 new tests cover the verbose flag (default true, explicit on/off,
2 posts on completed cycle, 0 posts when verbose=false, 0 posts on
skipped_no_task regardless). Total: 39 tests, all green.
Étape 3 testing on VPS exposed an OQ1 design gap: the worker assumed runner._running_agents would always have at least one entry to reuse as parent_agent for delegate_task. In practice the gateway is idle when the worker ticks (no Discord conversation underway), so the registry is empty and every cycle errored: delegation: error, duration: 0s, error: "no parent_agent available at runtime" Fix: _get_afk_parent_agent now follows the cron/scheduler.py pattern. If no session agent exists, build a transient AIAgent dedicated to the worker (quiet_mode, skip_context_files, skip_memory, platform= "afk_worker") and cache it for subsequent ticks. Credentials come from _resolve_runtime_agent_kwargs() — same chain the gateway uses. The session-agent path stays as the cheap fast-path when an active conversation is running (free reuse). Transient build only happens when truly idle.
Overnight run produced 3 real completions via MiniMax + 3 false
"completed" cycles where Sonnet/OpenRouter exhausted retries:
status=completed
summary="API call failed after 3 retries: "
Bug 2 (fixed here): delegate_task hands back status=completed even
when the child's LLM call never returned. The worker took it at face
value and marked tasks as done in the picked-index, blocking retry.
Fix: in _run_delegation, detect summaries starting with "API call
failed" and promote status to error. The picked-index then schedules
a 3-tick cooldown and retries instead of locking the task as done.
+1 test simulating the prod failure pattern.
Bug 1 (still open — debug logs added): direct curl from VPS with
the same OPENROUTER_API_KEY → HTTP 200, Sonnet replies. But
delegate_task → child agent → openrouter call → 3 retries failed.
Override propagation likely loses the api_key somewhere between the
worker call and the child runtime. Added INFO logs at three points:
1. afk_worker._run_delegation: provider/model/base_url/api_key_len
before calling delegate_task
2. delegate_task: same after applying the 4 overrides to creds
3. _build_child_agent: effective creds vs parent creds for diff
All length-only on api_key — no value ever logged (§1).
Once we see the chain in journalctl, we'll know exactly where the
override falls back to parent_api_key (which is MiniMax for the
transient AFK parent, hence the 401-equivalent).
Suspect the conditional 'if any(...)' debug log isn't firing for some reason (override seems to propagate based on auxiliary_client log, but no INFO from tools.delegate_tool appears in journal). Add an entry log without any condition to confirm code path is executed and override kwargs are received.
These were added to trace where the override propagation lost the
api_key for OpenRouter calls. Bug 1 became moot once we switched
the AFK matrix away from openrouter to MiniMax (sub-agents) under
Codex (main). Keeping the override-application code (4 if-blocks)
since the wiring is correct and useful for any future per-call
provider override.
The afk_worker side log ('afk-worker delegation overrides: ...')
stays in place — it's useful in steady-state to see what the worker
picks per tick, length-only on api_key.
Tests: 40/40 still green.
4 tasks
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Résumé
Pendant que AH est en
afk_manualouafk_auto, un thread daemon tick toutes les 30 min, pick une tâche taguée dans les status files, route vers un provider/modèle adapté, délègue à un sub-agent leaf, et logue le cycle. Les heartbeats (J3/J7/J14) embarquent maintenant un récap déterministe des cycles exécutés au lieu du wording aspirationnel.Spec complète :
~/Documents/Projects/ACOS-HERMES/.workspace/WS_AFK_WORK_LOOP_SPEC.md(~830 lignes, 15 sections).Fichiers
agent/afk_worker.pyagent/afk_heartbeat.py_build_dynamic_recap()+ refonte HB1/HB2/HB3agent/afk_scheduler.pytools/delegate_tool.pygateway/run.pyafk-workerau startup, transient parent_agent buildtests/agent/test_afk_worker.pyDécisions de design (Q1-Q10 + B1-B3 + OQ1-OQ7)
~/SMCP_STATUS.md, etc. configurables)[priority:Pn], fallback FIFO ; exclusion[afk:no]/[blocked:*][afk:type];decision_structurelle→ block + post Discordleafstrictstand_by(silent skip)~/AFK_LOG.md) + JSONL (~/.hermes/afk_log.jsonl) en parallèledelegate_taskest synchrone via ThreadPoolExecutor)AIAgentdédié (patterncron/scheduler.py)process_user_messagequand mode flip → normalSécurité — §9.1, §9.3.2, §9.3.4, §2.4
~/.hermes/config.yaml afk_worker.*subprocess.run/fork+exec : thread Python pur (SECCOMP-safe)OPENROUTER_API_KEYvalue never appears in any log filecooldown_level += 1(skip 2 / 4 ticks ; halt + Discord notif au niveau 3)Test plan
skipped_no_taskconsécutifs, format MD+JSONL OKerrorpour retrydelegation.*Notes
afk_worker.enabled: true+ tagger les tâches du backlog. Le picked-index dedupe garantit qu'on ne re-pick pas une tâche complete tant qu'elle reste in_progress dans le status file.