chore(template): fire evolution crons every 10min (from hourly) — 6x team activity - #158
Closed
HongmingWang-Rabbit wants to merge 1 commit into
Closed
chore(template): fire evolution crons every 10min (from hourly) — 6x team activity#158HongmingWang-Rabbit wants to merge 1 commit into
HongmingWang-Rabbit wants to merge 1 commit into
Conversation
Observed team utilization: ~0.5% (agents idle 99.5% of the time, per maintenance cron reports). CEO north-star is 24/7 iteration, current cadence delivers 6 fires/hour across 6 evolution crons = 6 activity windows per hour. This bumps all 6 hourly crons to fire every 10 minutes with staggered offsets so they don't stampede the scheduler: | Cron | Old | New | |-------------------|-----------|-------------------------------| | Security audit | 17 * * * | 1,11,21,31,41,51 * * * | | Ecosystem watch | 8 * * * | 2,12,22,32,42,52 * * * | | Plugin curation | 22 * * * | 4,14,24,34,44,54 * * * | | UI/UX audit | 11 * * * | 5,15,25,35,45,55 * * * | | Channel expansion | 47 * * * | 7,17,27,37,47,57 * * * | | Template fitness | 15 * * * | 8,18,28,38,48,58 * * * | Each offset is unique so within any 10-min window at most one evolution cron fires. 6× the activity, 1 concurrent fire worst case. Cost: at ~5k tokens avg × 36 fires/hour × 24h = ~4.3M tokens/day/org. Roughly $25/day/org at Sonnet rates. Up from ~$4/day/org. Band-aid only — the real fix per the research is reflection-on-completion + event-driven idle loop at the workspace level (Hermes/Letta pattern), not tighter crons. That's a separate PR coming next.
HongmingWang-Rabbit
added a commit
that referenced
this pull request
Apr 15, 2026
… workers stay reactive Supersedes #158 (10-min uniform bump). That PR was too blunt — it treated research/audit/orchestration crons the same when they have fundamentally different cost/value/cadence profiles. ## The split Three layers, three cadences, grounded in the survey of Hermes/Letta/ Trigger.dev/Inngest/AG2/Rivet/n8n/Composio/SWE-agent done this session. Nobody in that survey runs while(true) per agent — they all combine event-driven reactivity with short orchestration pulses on a coordinator. This PR implements that split for our 12-workspace template. | Layer | Roles | Cadence | Purpose | |---|---|---|---| | Orchestration | PM, Dev Lead, Research Lead | every 5 min | Check backlog, dispatch work, review completed tasks | | Audit | Security Auditor | every 10 min | Focused security audit | | Audit | UI/UX Designer | every 15 min | Vision-heavy, dial back from 10 | | Deep-work | Research Lead (eco-watch) | every 30 min (8,38) | Was hourly | | Deep-work | Dev Lead (template fitness) | every 30 min (15,45) | Was hourly | | Deep-work | Technical Researcher (plugins) | hourly (unchanged) | Research-heavy, slow | | Deep-work | DevOps (channels) | hourly (unchanged) | Research-heavy, slow | | Reactive | BE, FE, DevOps, Docs | no cron | Execute A2A delegations | ## Orchestration pulse prompts The three new schedules each carry a detailed orchestration_prompt: - **PM** (5-min): scan all 12 workspaces, scan GH PRs/issues backlog (external), scan memory backlog (internal), dispatch up to 3 tasks per pulse, review completed work, write pulse summary to memory. Hard rules: under 90s wall-clock, never dispatch to busy agents, write "orchestrator-clean" and stop if genuinely nothing to do. - **Dev Lead** (5-min, offset +1 from PM): same shape, scoped to engineering team. Reviews open PRs from direct reports, matches idle engineers to labeled GH issues (security/bug/feature), dispatches with "fix/issue-N-slug" branch convention. Skips pulse if own template fitness audit is in flight (:15, :45). - **Research Lead** (5-min, offset +2 from PM): same shape, scoped to research team. Matches Market Analyst / Technical Researcher / Competitive Intelligence to research-labeled issues or memory-stashed questions. Max 2 A2A per pulse (research is slow). Skips pulse if own eco-watch is in flight (:8, :38). ## Cadence offset table No two crons fire in the same minute: :01,:11,:21,:31,:41,:51 — Security audit (Security Auditor) :02,:07,:12,:17,:22,:27,:32,:37,:42,:47,:52,:57 — Dev Lead orchestrator :04,:09,:14,:19,:24,:29,:34,:39,:44,:49,:54,:59 — Research Lead orchestrator :01,:06,:11,:16,:21,:26,:31,:36,:41,:46,:51,:56 — PM orchestrator :05,:20,:35,:50 — UI/UX audit (UIUX Designer) :08,:38 — Ecosystem watch deep-work (Research Lead) :15,:45 — Template fitness deep-work (Dev Lead) :22 — Plugin curation (Technical Researcher) :47 — Channel expansion (DevOps Engineer) Note PM and Security Auditor share :01 — this is fine because they target different workspaces so scheduler concurrency handles it. ## Cost estimate - PM pulse: 12/hour × 24 × ~3k tokens = 864k tokens/day/org ~ $5/day - Dev Lead pulse: same ~ $5/day - Research Lead pulse: same ~ $5/day - Audits (security 10min, UIUX 15min): ~$8/day/org combined - Deep-work crons (unchanged from original): ~$4/day/org **Total ~$27/day/org**. Comparable to #158's $25 but MUCH higher utility because orchestration produces dispatches that keep workers busy, whereas #158 just fired more audits against the same team. Closes #158 (superseded — will close that PR with a pointer to this one). ## Related research See docs/ecosystem-watch.md `### Hermes Agent` and today's research agent output: event-driven + reflection-on-completion + short orchestration pulses on leaders is the shape that delivers 24/7 activity without runaway cost. This is the concrete implementation.
5 tasks
Contributor
Author
|
Superseded by #159 — that PR implements the orchestrator/worker split CEO asked for (leaders poll every 5min, workers stay reactive). This uniform 10-min bump was too blunt. |
Closed
7 tasks
HongmingWang-Rabbit
added a commit
that referenced
this pull request
Apr 15, 2026
chore(template): orchestrator/worker split — leaders pulse every 5min, workers stay reactive (supersedes #158)
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
… workers stay reactive Supersedes #158 (10-min uniform bump). That PR was too blunt — it treated research/audit/orchestration crons the same when they have fundamentally different cost/value/cadence profiles. ## The split Three layers, three cadences, grounded in the survey of Hermes/Letta/ Trigger.dev/Inngest/AG2/Rivet/n8n/Composio/SWE-agent done this session. Nobody in that survey runs while(true) per agent — they all combine event-driven reactivity with short orchestration pulses on a coordinator. This PR implements that split for our 12-workspace template. | Layer | Roles | Cadence | Purpose | |---|---|---|---| | Orchestration | PM, Dev Lead, Research Lead | every 5 min | Check backlog, dispatch work, review completed tasks | | Audit | Security Auditor | every 10 min | Focused security audit | | Audit | UI/UX Designer | every 15 min | Vision-heavy, dial back from 10 | | Deep-work | Research Lead (eco-watch) | every 30 min (8,38) | Was hourly | | Deep-work | Dev Lead (template fitness) | every 30 min (15,45) | Was hourly | | Deep-work | Technical Researcher (plugins) | hourly (unchanged) | Research-heavy, slow | | Deep-work | DevOps (channels) | hourly (unchanged) | Research-heavy, slow | | Reactive | BE, FE, DevOps, Docs | no cron | Execute A2A delegations | ## Orchestration pulse prompts The three new schedules each carry a detailed orchestration_prompt: - **PM** (5-min): scan all 12 workspaces, scan GH PRs/issues backlog (external), scan memory backlog (internal), dispatch up to 3 tasks per pulse, review completed work, write pulse summary to memory. Hard rules: under 90s wall-clock, never dispatch to busy agents, write "orchestrator-clean" and stop if genuinely nothing to do. - **Dev Lead** (5-min, offset +1 from PM): same shape, scoped to engineering team. Reviews open PRs from direct reports, matches idle engineers to labeled GH issues (security/bug/feature), dispatches with "fix/issue-N-slug" branch convention. Skips pulse if own template fitness audit is in flight (:15, :45). - **Research Lead** (5-min, offset +2 from PM): same shape, scoped to research team. Matches Market Analyst / Technical Researcher / Competitive Intelligence to research-labeled issues or memory-stashed questions. Max 2 A2A per pulse (research is slow). Skips pulse if own eco-watch is in flight (:8, :38). ## Cadence offset table No two crons fire in the same minute: :01,:11,:21,:31,:41,:51 — Security audit (Security Auditor) :02,:07,:12,:17,:22,:27,:32,:37,:42,:47,:52,:57 — Dev Lead orchestrator :04,:09,:14,:19,:24,:29,:34,:39,:44,:49,:54,:59 — Research Lead orchestrator :01,:06,:11,:16,:21,:26,:31,:36,:41,:46,:51,:56 — PM orchestrator :05,:20,:35,:50 — UI/UX audit (UIUX Designer) :08,:38 — Ecosystem watch deep-work (Research Lead) :15,:45 — Template fitness deep-work (Dev Lead) :22 — Plugin curation (Technical Researcher) :47 — Channel expansion (DevOps Engineer) Note PM and Security Auditor share :01 — this is fine because they target different workspaces so scheduler concurrency handles it. ## Cost estimate - PM pulse: 12/hour × 24 × ~3k tokens = 864k tokens/day/org ~ $5/day - Dev Lead pulse: same ~ $5/day - Research Lead pulse: same ~ $5/day - Audits (security 10min, UIUX 15min): ~$8/day/org combined - Deep-work crons (unchanged from original): ~$4/day/org **Total ~$27/day/org**. Comparable to #158's $25 but MUCH higher utility because orchestration produces dispatches that keep workers busy, whereas #158 just fired more audits against the same team. Closes #158 (superseded — will close that PR with a pointer to this one). ## Related research See docs/ecosystem-watch.md `### Hermes Agent` and today's research agent output: event-driven + reflection-on-completion + short orchestration pulses on leaders is the shape that delivers 24/7 activity without runaway cost. This is the concrete implementation.
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
chore(template): orchestrator/worker split — leaders pulse every 5min, workers stay reactive (supersedes #158)
This was referenced May 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Observed team utilization during maintenance cycles: ~0.5% — agents are idle 99.5% of the time because the 6 hourly evolution crons are the only initiating signal between user-typed tasks. CEO's north-star: team runs 24/7. Current cadence falls short.
What
Bump all 6 hourly crons to fire every 10 min with staggered offsets:
17 * * * *1,11,21,31,41,51 * * * *8 * * * *2,12,22,32,42,52 * * * *22 * * * *4,14,24,34,44,54 * * * *11 * * * *5,15,25,35,45,55 * * * *47 * * * *7,17,27,37,47,57 * * * *15 * * * *8,18,28,38,48,58 * * * *Each offset differs by ≥1 min so at most one evolution cron fires concurrently. No scheduler changes required — the existing cron parser already handles comma-separated minute lists.
Cost
Acceptable for now as we measure throughput impact. Can dial back per-cron if any one is producing noise instead of signal.
This is a band-aid
The proper fix per today's research (survey of Hermes/Letta/Trigger.dev/AG2/Inngest/n8n/Rivet/Composio/SWE-agent): reflection-on-completion + event-driven idle loop at the workspace level, not tighter crons. Hermes's pattern specifically — when an agent finishes a task, it reflects and either files a follow-up, commits a skill improvement, or hibernates. Cost scales with useful work, not wall-clock. Estimated ~$0.45/day/org vs ~$25/day here.
That fix is ~40 LOC in
workspace-template/main.py+ a newidle_promptdefault inorg.yamland is the next PR in flight. Ship this cron bump first for immediate throughput, then layer the idle loop on top.Test plan
activity_logsentries, compare to baselineRelated
docs/ecosystem-watch.md→ Hermes Agent entry (research baseline)