chore(template): orchestrator/worker split — leaders pulse every 5min, workers stay reactive (supersedes #158) - #159
Merged
Conversation
… workers stay reactive Supersedes #158 (10-min uniform bump). That PR was too blunt — it treated research/audit/orchestration crons the same when they have fundamentally different cost/value/cadence profiles. ## The split Three layers, three cadences, grounded in the survey of Hermes/Letta/ Trigger.dev/Inngest/AG2/Rivet/n8n/Composio/SWE-agent done this session. Nobody in that survey runs while(true) per agent — they all combine event-driven reactivity with short orchestration pulses on a coordinator. This PR implements that split for our 12-workspace template. | Layer | Roles | Cadence | Purpose | |---|---|---|---| | Orchestration | PM, Dev Lead, Research Lead | every 5 min | Check backlog, dispatch work, review completed tasks | | Audit | Security Auditor | every 10 min | Focused security audit | | Audit | UI/UX Designer | every 15 min | Vision-heavy, dial back from 10 | | Deep-work | Research Lead (eco-watch) | every 30 min (8,38) | Was hourly | | Deep-work | Dev Lead (template fitness) | every 30 min (15,45) | Was hourly | | Deep-work | Technical Researcher (plugins) | hourly (unchanged) | Research-heavy, slow | | Deep-work | DevOps (channels) | hourly (unchanged) | Research-heavy, slow | | Reactive | BE, FE, DevOps, Docs | no cron | Execute A2A delegations | ## Orchestration pulse prompts The three new schedules each carry a detailed orchestration_prompt: - **PM** (5-min): scan all 12 workspaces, scan GH PRs/issues backlog (external), scan memory backlog (internal), dispatch up to 3 tasks per pulse, review completed work, write pulse summary to memory. Hard rules: under 90s wall-clock, never dispatch to busy agents, write "orchestrator-clean" and stop if genuinely nothing to do. - **Dev Lead** (5-min, offset +1 from PM): same shape, scoped to engineering team. Reviews open PRs from direct reports, matches idle engineers to labeled GH issues (security/bug/feature), dispatches with "fix/issue-N-slug" branch convention. Skips pulse if own template fitness audit is in flight (:15, :45). - **Research Lead** (5-min, offset +2 from PM): same shape, scoped to research team. Matches Market Analyst / Technical Researcher / Competitive Intelligence to research-labeled issues or memory-stashed questions. Max 2 A2A per pulse (research is slow). Skips pulse if own eco-watch is in flight (:8, :38). ## Cadence offset table No two crons fire in the same minute: :01,:11,:21,:31,:41,:51 — Security audit (Security Auditor) :02,:07,:12,:17,:22,:27,:32,:37,:42,:47,:52,:57 — Dev Lead orchestrator :04,:09,:14,:19,:24,:29,:34,:39,:44,:49,:54,:59 — Research Lead orchestrator :01,:06,:11,:16,:21,:26,:31,:36,:41,:46,:51,:56 — PM orchestrator :05,:20,:35,:50 — UI/UX audit (UIUX Designer) :08,:38 — Ecosystem watch deep-work (Research Lead) :15,:45 — Template fitness deep-work (Dev Lead) :22 — Plugin curation (Technical Researcher) :47 — Channel expansion (DevOps Engineer) Note PM and Security Auditor share :01 — this is fine because they target different workspaces so scheduler concurrency handles it. ## Cost estimate - PM pulse: 12/hour × 24 × ~3k tokens = 864k tokens/day/org ~ $5/day - Dev Lead pulse: same ~ $5/day - Research Lead pulse: same ~ $5/day - Audits (security 10min, UIUX 15min): ~$8/day/org combined - Deep-work crons (unchanged from original): ~$4/day/org **Total ~$27/day/org**. Comparable to #158's $25 but MUCH higher utility because orchestration produces dispatches that keep workers busy, whereas #158 just fired more audits against the same team. Closes #158 (superseded — will close that PR with a pointer to this one). ## Related research See docs/ecosystem-watch.md `### Hermes Agent` and today's research agent output: event-driven + reflection-on-completion + short orchestration pulses on leaders is the shape that delivers 24/7 activity without runaway cost. This is the concrete implementation.
This was referenced Apr 15, 2026
HongmingWang-Rabbit
left a comment
Contributor
Author
There was a problem hiding this comment.
PM Orchestration Pulse Review — 16:42 UTC
Code review: ✅ LGTM
The orchestrator/worker split is well-designed:
- Offset cron expressions (PM at :01/:06/:11…, Dev Lead at :02/:07/:12…, Research Lead at :04/:09/:14…) prevent leader collisions
- Eco-watch correctly promoted from hourly → 30min cadence
- Workers (BE/FE/DevOps) correctly have no cron — they stay reactive
One design note for the team: Security Auditor changing from hourly to every-10-min (6×/hour) may worsen the 36% collision rate flagged in #115. Worth watching post-merge.
CI failures are billing-only — all 6 jobs fail with "account payments have failed" before any code runs. This is tracked in #136. Code is merge-ready pending billing fix.
🔴 Blocking team context (issue #160): Claude Max OAuth token exhausted — all agents offline until 2026-04-17 23:00 UTC. Full A2A mesh unavailable this pulse.
This was referenced Apr 15, 2026
HongmingWang-Rabbit
added a commit
that referenced
this pull request
Apr 15, 2026
Every-10-min cadence introduced in PR #159 increased Security Auditor from 2 runs/day to 144 runs/day (144x). Combined with PM, Research Lead, Dev Lead, and other hourly evolution-lever crons, this is the likely root cause of the P0 OAuth quota exhaustion (#160, resets Apr 17 23:00 UTC). Restored: cron_expr 7 6,18 * * * (twice daily, 12-hour interval) Schedule name updated to match new cadence. Audit prompt content (DAST teardown, PM routing, PM deliverable) retained.
HongmingWang-Rabbit
added a commit
that referenced
this pull request
Apr 15, 2026
Today's multi-framework research (Hermes, Letta, Trigger.dev, Inngest, AG2,
Rivet, n8n, Composio, SWE-agent — see docs/ecosystem-watch.md) confirmed
that nobody runs while(true) per agent. The working patterns are:
(a) event-driven + hibernation (Hermes, Letta, Trigger.dev, Inngest)
(b) cron/user-triggered ephemeral runs (AG2, Rivet, n8n, SWE-agent)
Molecule AI is currently 100% in category (b). Observed team utilization:
~0.5% — agents idle 99.5% of the time because cron fires and CEO-typed
A2A are the only initiating signals. CEO's north-star is 24/7 iteration,
current cadence falls short.
This PR closes the gap by adding an in-workspace idle loop that wakes the
agent periodically ONLY when it has no active task. The shape is the
Hermes reflection-on-completion pattern combined with the Letta backlog-pull
pattern, collapsed into a ~60 LOC change in the workspace-template. Zero
new Go code. Zero new DB tables. Zero new API endpoints.
## How it works
1. `config.py` gets two new fields on WorkspaceConfig:
- `idle_prompt: str = ""` — the prompt to self-send when idle
- `idle_interval_seconds: int = 600` — how often to check (default 10 min)
Both support inline or file ref (matching the initial_prompt pattern).
2. `main.py` spawns an `_run_idle_loop()` asyncio task alongside the
existing initial_prompt task (same lifecycle hooks — cancelled in the
`finally:` of the server.serve() block).
3. The loop body:
a. Sleep interval
b. Check `heartbeat.active_tasks == 0` LOCALLY (no LLM call, no HTTP)
c. If idle → self-POST the idle_prompt via the existing /workspaces/{id}/a2a proxy
d. Loop
The agent's own concurrency control rejects the post if it becomes busy
between the check and the POST — that's the safety valve.
4. Gated on `config.idle_prompt` being non-empty. Default = "" = no loop.
Existing workspaces upgrade silently as no-ops until someone explicitly
opts in by setting idle_prompt in org.yaml (either defaults: or
per-workspace:).
## Cost analysis (from the research report)
- while(true) pattern: ~$93/day/org (12 agents × 12 thinks/hour × $0.027). Unshippable.
- Hermes reflection-on-completion: ~$0.45/day/org. Cost ∝ useful work.
- This PR's idle loop at 10-min cadence: upper bound 12 × 6/hour × 24h
× ~3k tokens × Sonnet rate ≈ $5/day/org PER ROLE, only if they're
genuinely idle every check. In practice far less because busy periods
skip the LLM call entirely (the active_tasks check is local).
## Rollout plan
Research report recommended rolling to ONE workspace first (Technical
Researcher) and measuring 24h of activity_logs before enabling for
all 12. This PR enables the mechanism; it does NOT add any default
idle_prompt to org-templates/molecule-dev/org.yaml. That's a follow-up
PR after this one lands and one workspace has been manually opted in
for measurement.
## Not touched in this PR
- No Go code (no new platform endpoint, no new DB columns)
- No org.yaml changes (zero-impact until someone opts in)
- No scheduler changes (the idle loop is a workspace concern, not a
scheduler concern — matches the research report's layering)
## Test plan
- [x] Python syntax check (ast.parse) on main.py + config.py
- [ ] Unit test: WorkspaceConfig parses idle_prompt / idle_interval_seconds from yaml
- [ ] Integration test: set idle_prompt on Technical Researcher, measure that
an A2A message is received every ~10 min while idle, and NOT received
while busy with a delegation
- [ ] Dogfood: enable on Technical Researcher for 24h, count activity_logs
delta vs baseline, confirm cost stays within model
## Related
- Today's research report (conversation output, summarized in commit trailer)
- docs/ecosystem-watch.md → `### Hermes Agent` (the canonical reflection-on-completion example)
- #159 orchestrator/worker split — complementary: leaders pulse for dispatch,
workers idle-loop for pull. Together: leaders push work, workers pull work,
no role ever sits idle with a cold queue.
This was referenced Apr 15, 2026
HongmingWang-Rabbit
added a commit
that referenced
this pull request
Apr 16, 2026
…lligence Rollout wave 2. Technical Researcher (wave 1, #216) has been the lone pilot since 2026-04-15. After 2+ days of debugging the auth chain (#285 builtin_tools, #304 a2a_tools — two different Python paths to the same endpoint, both needed the fix), the pilot fired a successful end-to-end dispatch at 2026-04-16 02:40 UTC: "Backlog item dispatched and cleared. Done in one A2A send. Step 1: Memory search found one..." Followed by correct idle-clean behavior at 03:00 and 03:10 once the seeded backlog was cleared: "Idle-clean. Backlog is empty, no new follow-ups to file." Pilot is validated. This PR widens the reflection-on-completion pattern to the other two researchers on Research Lead's team (same hierarchy as TR). ## Role-tuned idle_prompts Each of the 3 researchers gets a prompt that: - Searches its own `research-backlog:<role>` memory namespace - If found, delegates to Research Lead with a role-shaped spec (Market Analyst → TAM/pricing comparables; Competitive Intel → feature-parity/competitor track; Technical Researcher → framework eval) - If empty, reflects on last memory entry for role-specific follow-ups (filed as GH issues, then added to backlog for next tick) - If neither, writes "ma-idle" / "ci-idle" / "tr-idle" clean entry All three share: - 10-minute cadence (idle_interval_seconds: 600) - Max 1 A2A send per tick - Skip step 1 if Research Lead busy - Under 60s wall-clock budget ## Rollout so far - Wave 1 (#216, merged): Technical Researcher — pilot validated 2026-04-16 - **Wave 2 (this PR): Market Analyst + Competitive Intelligence** - Wave 3 (future): orchestrators / auditors if wave 2 looks good Not including leaders yet (PM, Dev Lead, Research Lead already have 5-min Orchestrator pulses from #159). Not including engineers (BE/FE/DevOps) — they're reactive-only per the orchestrator/worker split design. ## Related - #205 idle-loop mechanism (merged) - #216 Technical Researcher pilot (merged) - #285 + #304 auth chain fixes that unblocked the pilot - `project_north_star_24_7.md` memory — this is the "team runs 24/7" tick
HongmingWang-Rabbit
added a commit
that referenced
this pull request
Apr 16, 2026
…directive) CEO directive verbatim: *"devs should pick up issues and declare that its assigned to them, PM and leaders regularly check in. dont just rely on outside reviewer"*. Adds `idle_prompt` + `idle_interval_seconds: 600` to Frontend Engineer, Backend Engineer, and DevOps Engineer. Each engineer now polls open GH issues matching its specialty, claims unassigned ones via `gh issue edit --add-assignee @me`, leaves a public comment declaring the pickup, and commits memory to prevent double-pickup on the next tick. Previously engineers were reactive-only per the #159 orchestrator/worker split. The CEO is correcting that: devs should be a true self-organizing unit, not a work-queue that only advances when an outside reviewer dispatches. ## Per-role specialty filters | Role | Labels it claims | |---|---| | Frontend Engineer | canvas, a11y, ux, typescript, frontend, bug, security | | Backend Engineer | security, platform, go, database, bug | | DevOps Engineer | docker, ci, deployment, infra, devops, bug | Priority order within each role: security > bug > feature. ## Self-review gates Each engineer's idle_prompt includes the self-review chain: - Frontend: molecule-skill-code-review + molecule-skill-llm-judge - Backend: molecule-skill-code-review + molecule-security-scan + molecule-skill-llm-judge - DevOps: molecule-skill-code-review + molecule-freeze-scope + molecule-hitl for risky ops These plugins were wired into engineer roles by #280, #303, #310, #322 — the idle_prompt makes them the PRIMARY quality gate instead of a nice-to- have before PR. Matches the "team self-regulates, don't rely on outside reviewer" spirit. ## Hard rules (same shape as researcher idle_prompts from #216/#321) - Max 1 claim per tick (1 `gh issue edit --add-assignee` call) - Never take someone else's assigned issue - Under 90 seconds wall-clock for the claim + plan step - Don't double-pick: check `task-assigned:<role>` memory first - No busy-work fabrication: write "<role>-idle HH:MM — no work" if nothing matches ## What this does NOT change - Leaders' orchestrator pulses still dispatch (#159) — this is the TAIL pickup, not the primary dispatch path. Dev Lead still prioritizes via its own pulse. - PR merging still goes through reviewer per `feedback_never_merge_prs.md`. This directive is about the QUALITY GATE (team self-review, peer review via Dev Lead's pulse) not about bypassing merge approval. - Destructive/irreversible ops still need explicit human ack via molecule-hitl's @requires_approval decorator. ## Rollout plan - Ship template change (this PR) - After merge: rebuild workspace-template:claude-code, re-provision BE + FE + DevOps via apply_template=true, re-inject idle_prompt (platform doesn't auto-propagate org.yaml to live configs — tracked separately) - Measure: 24h of activity_logs. Should see `a2a_receive` events every 10 min per engineer, response bodies mentioning claim decisions or idle-clean states, and `gh issue edit` events showing up as assignees. ## Related - `feedback_devs_pick_up_issues_leaders_check_in.md` — memory saved last cycle - #159 orchestrator/worker split (leaders dispatch) - #216 / #321 researcher idle_prompts (same pattern applied to researchers) - `project_north_star_24_7.md` — team self-regulation is the north-star
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
chore(template): orchestrator/worker split — leaders pulse every 5min, workers stay reactive (supersedes #158)
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
Every-10-min cadence introduced in PR #159 increased Security Auditor from 2 runs/day to 144 runs/day (144x). Combined with PM, Research Lead, Dev Lead, and other hourly evolution-lever crons, this is the likely root cause of the P0 OAuth quota exhaustion (#160, resets Apr 17 23:00 UTC). Restored: cron_expr 7 6,18 * * * (twice daily, 12-hour interval) Schedule name updated to match new cadence. Audit prompt content (DAST teardown, PM routing, PM deliverable) retained.
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
Today's multi-framework research (Hermes, Letta, Trigger.dev, Inngest, AG2,
Rivet, n8n, Composio, SWE-agent — see docs/ecosystem-watch.md) confirmed
that nobody runs while(true) per agent. The working patterns are:
(a) event-driven + hibernation (Hermes, Letta, Trigger.dev, Inngest)
(b) cron/user-triggered ephemeral runs (AG2, Rivet, n8n, SWE-agent)
Molecule AI is currently 100% in category (b). Observed team utilization:
~0.5% — agents idle 99.5% of the time because cron fires and CEO-typed
A2A are the only initiating signals. CEO's north-star is 24/7 iteration,
current cadence falls short.
This PR closes the gap by adding an in-workspace idle loop that wakes the
agent periodically ONLY when it has no active task. The shape is the
Hermes reflection-on-completion pattern combined with the Letta backlog-pull
pattern, collapsed into a ~60 LOC change in the workspace-template. Zero
new Go code. Zero new DB tables. Zero new API endpoints.
## How it works
1. `config.py` gets two new fields on WorkspaceConfig:
- `idle_prompt: str = ""` — the prompt to self-send when idle
- `idle_interval_seconds: int = 600` — how often to check (default 10 min)
Both support inline or file ref (matching the initial_prompt pattern).
2. `main.py` spawns an `_run_idle_loop()` asyncio task alongside the
existing initial_prompt task (same lifecycle hooks — cancelled in the
`finally:` of the server.serve() block).
3. The loop body:
a. Sleep interval
b. Check `heartbeat.active_tasks == 0` LOCALLY (no LLM call, no HTTP)
c. If idle → self-POST the idle_prompt via the existing /workspaces/{id}/a2a proxy
d. Loop
The agent's own concurrency control rejects the post if it becomes busy
between the check and the POST — that's the safety valve.
4. Gated on `config.idle_prompt` being non-empty. Default = "" = no loop.
Existing workspaces upgrade silently as no-ops until someone explicitly
opts in by setting idle_prompt in org.yaml (either defaults: or
per-workspace:).
## Cost analysis (from the research report)
- while(true) pattern: ~$93/day/org (12 agents × 12 thinks/hour × $0.027). Unshippable.
- Hermes reflection-on-completion: ~$0.45/day/org. Cost ∝ useful work.
- This PR's idle loop at 10-min cadence: upper bound 12 × 6/hour × 24h
× ~3k tokens × Sonnet rate ≈ $5/day/org PER ROLE, only if they're
genuinely idle every check. In practice far less because busy periods
skip the LLM call entirely (the active_tasks check is local).
## Rollout plan
Research report recommended rolling to ONE workspace first (Technical
Researcher) and measuring 24h of activity_logs before enabling for
all 12. This PR enables the mechanism; it does NOT add any default
idle_prompt to org-templates/molecule-dev/org.yaml. That's a follow-up
PR after this one lands and one workspace has been manually opted in
for measurement.
## Not touched in this PR
- No Go code (no new platform endpoint, no new DB columns)
- No org.yaml changes (zero-impact until someone opts in)
- No scheduler changes (the idle loop is a workspace concern, not a
scheduler concern — matches the research report's layering)
## Test plan
- [x] Python syntax check (ast.parse) on main.py + config.py
- [ ] Unit test: WorkspaceConfig parses idle_prompt / idle_interval_seconds from yaml
- [ ] Integration test: set idle_prompt on Technical Researcher, measure that
an A2A message is received every ~10 min while idle, and NOT received
while busy with a delegation
- [ ] Dogfood: enable on Technical Researcher for 24h, count activity_logs
delta vs baseline, confirm cost stays within model
## Related
- Today's research report (conversation output, summarized in commit trailer)
- docs/ecosystem-watch.md → `### Hermes Agent` (the canonical reflection-on-completion example)
- #159 orchestrator/worker split — complementary: leaders pulse for dispatch,
workers idle-loop for pull. Together: leaders push work, workers pull work,
no role ever sits idle with a cold queue.
molecule-ai Bot
pushed a commit
that referenced
this pull request
Apr 21, 2026
…directive) CEO directive verbatim: *"devs should pick up issues and declare that its assigned to them, PM and leaders regularly check in. dont just rely on outside reviewer"*. Adds `idle_prompt` + `idle_interval_seconds: 600` to Frontend Engineer, Backend Engineer, and DevOps Engineer. Each engineer now polls open GH issues matching its specialty, claims unassigned ones via `gh issue edit --add-assignee @me`, leaves a public comment declaring the pickup, and commits memory to prevent double-pickup on the next tick. Previously engineers were reactive-only per the #159 orchestrator/worker split. The CEO is correcting that: devs should be a true self-organizing unit, not a work-queue that only advances when an outside reviewer dispatches. ## Per-role specialty filters | Role | Labels it claims | |---|---| | Frontend Engineer | canvas, a11y, ux, typescript, frontend, bug, security | | Backend Engineer | security, platform, go, database, bug | | DevOps Engineer | docker, ci, deployment, infra, devops, bug | Priority order within each role: security > bug > feature. ## Self-review gates Each engineer's idle_prompt includes the self-review chain: - Frontend: molecule-skill-code-review + molecule-skill-llm-judge - Backend: molecule-skill-code-review + molecule-security-scan + molecule-skill-llm-judge - DevOps: molecule-skill-code-review + molecule-freeze-scope + molecule-hitl for risky ops These plugins were wired into engineer roles by #280, #303, #310, #322 — the idle_prompt makes them the PRIMARY quality gate instead of a nice-to- have before PR. Matches the "team self-regulates, don't rely on outside reviewer" spirit. ## Hard rules (same shape as researcher idle_prompts from #216/#321) - Max 1 claim per tick (1 `gh issue edit --add-assignee` call) - Never take someone else's assigned issue - Under 90 seconds wall-clock for the claim + plan step - Don't double-pick: check `task-assigned:<role>` memory first - No busy-work fabrication: write "<role>-idle HH:MM — no work" if nothing matches ## What this does NOT change - Leaders' orchestrator pulses still dispatch (#159) — this is the TAIL pickup, not the primary dispatch path. Dev Lead still prioritizes via its own pulse. - PR merging still goes through reviewer per `feedback_never_merge_prs.md`. This directive is about the QUALITY GATE (team self-review, peer review via Dev Lead's pulse) not about bypassing merge approval. - Destructive/irreversible ops still need explicit human ack via molecule-hitl's @requires_approval decorator. ## Rollout plan - Ship template change (this PR) - After merge: rebuild workspace-template:claude-code, re-provision BE + FE + DevOps via apply_template=true, re-inject idle_prompt (platform doesn't auto-propagate org.yaml to live configs — tracked separately) - Measure: 24h of activity_logs. Should see `a2a_receive` events every 10 min per engineer, response bodies mentioning claim decisions or idle-clean states, and `gh issue edit` events showing up as assignees. ## Related - `feedback_devs_pick_up_issues_leaders_check_in.md` — memory saved last cycle - #159 orchestrator/worker split (leaders dispatch) - #216 / #321 researcher idle_prompts (same pattern applied to researchers) - `project_north_star_24_7.md` — team self-regulation is the north-star
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Supersedes #158. That PR was too blunt (uniform 10-min bump on all crons). This PR implements the orchestrator/worker split CEO asked for: leaders poll every 5 min to dispatch work, workers stay reactive (no cron interrupts), audit crons stay short, deep-work crons go back to slower.
Three-layer cadence
Key design decisions
The orchestration prompts
Each of the 3 new `Orchestrator pulse` schedules carries a detailed prompt (~40 lines each) that follows the same pattern:
Hard rules in every pulse: under 60-90s wall-clock, max 2-3 A2A sends, skip pulse if your own deep-work cron is in flight. This bounds cost and prevents runaway fan-out.
Cadence collision audit
No two crons share the same minute except PM (:01,...) and Security Auditor (:01,...) — those target different workspaces so scheduler concurrency handles it cleanly.
Cost
~$27/day/org total:
Comparable to #158's $25/day but much higher utility because orchestration pulses PRODUCE dispatches that keep reactive workers busy, whereas #158 just fired more audits against the same team without new dispatching.
Test plan
Related