From c6c6bb662997c4689ff857db2d1dce72ccc491ea Mon Sep 17 00:00:00 2001 From: namastex888 Date: Fri, 10 Jul 2026 04:18:54 -0300 Subject: [PATCH 1/2] =?UTF-8?q?docs(wishes):=20overnight=20bookkeeping=20?= =?UTF-8?q?=E2=80=94=20reviews,=20QA=20evidence,=20status=20flips?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../HANDOFF-20260710.md | 130 ++++++++++++++++++ .genie/wishes/agent-sync/COORDINATION.md | 59 ++++++++ .genie/wishes/council-workflow/WISH.md | 2 +- .../qa/overnight-observations-20260710.md | 28 ++++ .../wishes/rolling-pr-auth-hardening/WISH.md | 12 +- .genie/wishes/routing-matrix/WISH.md | 2 +- .../qa/routing-pin-qa-20260710.md | 121 ++++++++++++++++ .genie/wishes/skills-fable5-revamp/WISH.md | 2 +- .../reports/execution-review-20260710.md | 96 +++++++++++++ 9 files changed, 443 insertions(+), 9 deletions(-) create mode 100644 .genie/brainstorms/genie-token-efficiency-program/HANDOFF-20260710.md create mode 100644 .genie/wishes/agent-sync/COORDINATION.md create mode 100644 .genie/wishes/council-workflow/qa/overnight-observations-20260710.md create mode 100644 .genie/wishes/routing-matrix/qa/routing-pin-qa-20260710.md create mode 100644 .genie/wishes/skills-fable5-revamp/reports/execution-review-20260710.md diff --git a/.genie/brainstorms/genie-token-efficiency-program/HANDOFF-20260710.md b/.genie/brainstorms/genie-token-efficiency-program/HANDOFF-20260710.md new file mode 100644 index 000000000..354da40b6 --- /dev/null +++ b/.genie/brainstorms/genie-token-efficiency-program/HANDOFF-20260710.md @@ -0,0 +1,130 @@ +# HANDOFF — 2026-07-10 (session 8050dd5c, Fable 5) + +**For the fresh session.** Predecessor: [HANDOFF-20260709.md](HANDOFF-20260709.md) (full program background). +This file is intentionally UNTRACKED — the shared checkout sits on another session's branch +(`wish/agent-sync`); commit this to dev only after that branch resolves (§2). + +--- + +## 1. What shipped today (2026-07-10) — the new capabilities you should leverage + +| Delivery | State | Leverage | +|---|---|---| +| **routing-matrix** | Merged (#2535), released stable v5.260710.2, LIVE | 7 pinned role agents ship in the plugin (`engineer-trivial/standard/complex` opus·low/high/xhigh, `fixer` opus·medium, `reviewer` opus·xhigh, `final-gate` fable·high, `scout` haiku·low) — a fresh session should see them as subagent types. Route ALL engineering to the opus ladder; Fable only at gates. First dogfooded wish ran at ~11% Fable tokens. | +| **council-workflow** | Merged (#2538), released v5.260710.2; `/council` stamped on this machine | `/council` = native saved workflow (deliberate/audit) + 7 specialist lane skills + 6 lens cards. Live QA still pending (§3). | +| **hook-injection-hardening** | Released v5.260710.1 | — | +| **plugin-resource-shipping** | Merged to dev (#2540), NOT yet in a stable release | Skills self-contained via `${CLAUDE_SKILL_DIR}`; resource-shipping lint + fresh-install CI smoke now guard every PR. | +| **Token-efficiency program docs** | Committed (3d40966c + council PR) | Umbrella DESIGN (WRS 100) + 9 track DRAFTs + 2 executed wishes. | + +Versions: CLI v5.260710.2 (stable channel), plugin cache 5.260710.2, dev at ≥ `bedd2062` (v5.260710.3 bump). + +## 2. IN FLIGHT — agent-sync (highest priority, do this first) + +**The wish:** `.genie/wishes/agent-sync/WISH.md` (on branch `wish/agent-sync`, commit 63f63c95) — +`genie update` becomes the single canonical updater converging EVERY detected agent (Claude Code +skills + `/council` stamp, Codex `.curated` skills, Hermes symlink) on every invocation; SessionStart +hook demoted to trigger. This is Felipe's "deliver lazy" ruling made real. + +**State at handoff (04:55Z):** +- G1 (engine, `src/lib/agent-sync*.ts` + `genie-home.ts`) DONE, execution-reviewed SHIP — files + UNTRACKED in the shared checkout `~/workspace/genie`. +- G2 was being written LIVE at handoff time (uncommitted modifications to `src/genie-commands/update.ts`, + `install.ts`, `plugins/genie/scripts/{council-stamp.cjs,smart-install.js}`, `scripts/smart-install.js` + deleted, `skills/{review,brainstorm}/SKILL.md` anchors, tests) by the owning session. +- **Felipe directive on record:** PR to dev REQUIRED as soon as gates pass — see + `.genie/wishes/agent-sync/COORDINATION.md` (untracked; also contains the reset+rebase commands and + the #2540 intersection facts). +- **Branch hazard:** `wish/agent-sync` still carries 3 duplicate commits (805afcd4/0584639d/422dc6dd — + already merged via #2540 as ecbb67fc/203c97df/bbd6439e). MUST `git reset --hard 63f63c95` + + `git rebase origin/dev` before pushing, or the PR drags duplicates. +- A 3h PR watcher was polling in the dying session — IT IS DEAD NOW. Fresh session: check + `gh pr list --state open` for an agent-sync PR; when it appears, CI-watch and (with Felipe's + green-CI authorization restated) merge on green. §19: dev-target merges only. + +**After merge:** trigger the stable promotion (Felipe must name/run "Rolling PR Maintenance" — the +permission classifier and §19 both want the human in that loop; main-target PR merge is human-via-UI +only). Then the delivery caveat (wish Decision 9): the release introducing self-sync needs +`genie update` run twice, ONCE ever; afterwards every update self-syncs. + +## 3. Pending QA gates (user-ritual or data-dependent) + +1. **Council live QA** — Felipe's ritual tail: run `/council "revisar tudo"`, then write the + council-workflow wish's final execution review (INDEX says g5-gate parks in `qa/` until then). + Known QA observation to record: stamp fired only on SessionStart → mid-session `/plugin update` + left a gap (manually verified working via direct `smart-install.js` run); agent-sync supersedes. +2. **routing-matrix live pin QA** — today (07-10) is the first full day of traces under the pins. + Pull LangWatch: model split should show Fable's trace share collapsing on execution work + (baseline: Fable 71% of traces, $14.1k/7d). Endpoint langwatch.khal.ai, creds in CC settings env; + working query recipes archived in `.genie/brainstorms/genie-spend/DRAFT.md` (§KNOWN). Beware: + analytics effort-filter silently broken; groupBy metadata.model + cache-read series hits a + ClickHouse bug (worth filing upstream). +3. **plugin-resource-shipping QA criterion 1** — live installed-plugin `/wish` scaffold on a bare + repo AFTER the next stable release ships it. +4. **Follow-ups noted in the shipped wish (LOW):** fresh-install-smoke temp-dir cleanup bypassed by + `process.exit` on phase-b failures; G2 same-line guard is substring-based ("package.json" anywhere + on the line passes); G1's validation sweep not replay-safe post-G2 (trips on the lint rule's own + negative fixtures). + +## 4. The jar — planned work not yet executed + +**Program tracks (all under `.genie/brainstorms//DRAFT.md`, ordered by readiness):** + +| Track | % | Blocker / next move | +|---|---|---| +| intent-to-wish-compiler | 92 | Breaker contract RATIFIED ("cut breadth/attempts, never proof; only humans cut payout"; autonomous flex cuts; human-only partial-ship via omni). Program-scale: pours as 4 child wishes (lane classifier + breaker state machine · intake compiler · roadmap surface · spend calibration). Needs control-plane-contract + brainstorm-domain-map convergence first. | +| brainstorm-domain-map | 80 | Executable-spec compiler decided (requirement IDs, oracle classes machine/model/human, proof packets). Open: who owns irreducibly subjective truth (oracle-ownership boundary) — Felipe ruling. | +| genie-spend | ~70 | CLOSEST TO WISHABLE. 4 Felipe gaps: key/endpoint source, cadence, consumers, doctor burn threshold. Today's pin-QA data (§3.2) is its natural opener. It is also the calibration organ the ratified breaker needs (per-lane p50/p90). | +| cross-agent-delegate | ~65 | Felipe gaps: Codex install/auth reality (NOTE: agent-sync's codex adapter now ships genie skills to `~/.codex/skills/.curated/` — overlaps, reconcile), Hermes `-z` alias canonicalization, prompt-method walkthrough for refine style cards, collision with v5-completion's "Codex launch target". | +| skill-absorbs | ~60 | council disposition DONE (council-workflow shipped). pm-absorb materially unblocked by intent-compiler + domain-map. Remaining: trace→fix, wizard→genie router, report→LangWatch. | +| always-on-genie | ~60 | NEW EVIDENCE: shared-checkout collision happened TWICE today (see §5) — G10 worktree-isolation policy now has its proof. SessionStart identity/state inject design decided. | +| control-plane-contract | ~55 | skills-fable5-revamp sequencing call still open; dispatch-contract single-source design decided. | +| dream-replatform | ~50 | Substrate choice is Felipe's (local cron vs cloud agents vs hybrid). | + +**Pre-program Ready wishes (INDEX):** v5-completion (reconcile w/ cross-agent-delegate + agent-sync), +dispatch-inproc-default (re-verify: hooks demonstrably ARE armed now — §19 and git-safety both fired +today; the wish may be partially obsolete), omni-approval-ux, omni-runner-port, warp-integration, +genie-mcp, v5-housekeeping, taxonomy-rehoming. + +**Recommended order for the fresh session:** +1. Close agent-sync (merge → promote → `genie update` ×2 → verify lazy delivery incl. `/council`). +2. Council live QA + final execution review (closes council-workflow). +3. Pull pin-QA numbers → close genie-spend gaps conversationally → `/wish genie-spend` (small, high-leverage). +4. Brainstorm refinements needing Felipe: brainstorm-domain-map oracle boundary → unlocks + intent-to-wish-compiler pour; cross-agent-delegate gaps. + +## 5. Safety learnings for autonomous operation (hard-won today) + +- **Shared-checkout collision is THE failure mode.** Two incidents: (a) another session switched + `~/workspace/genie` to `wish/agent-sync` mid-workflow — three commits landed on the wrong branch, + cost a full branch surgery; (b) narrowly avoided grabbing live-uncommitted G2 files. RULE: never + execute wish work in the shared checkout; always `git worktree add` an isolated tree per wish + (this is always-on-genie G10 — until it ships, do it manually). The branch-surgery recipe that + worked: isolated worktree off origin/dev, `git mv` + `cherry-pick --no-commit` to fold, plain + cherry-picks for the rest, `git update-ref` to point branches (git-safety false-positives on + `branch -f`). +- **Hooks that fired correctly today (trust them):** §19 (agents merge dev-target only; main = + human via GitHub UI); git-safety (no `--no-verify`, no `--force`; pre-push runs full + `bun run check`); commitlint ≤100-char headers; permission classifier requires Felipe to NAME + workflow dispatches ("Rolling PR Maintenance"). +- **The merge pattern that works:** push → `gh pr create --base dev` → background + `gh pr checks N --watch --fail-fast` → merge on green. Watchers DIE with the session. +- **Validation from script files only** — inline multi-line bash false-passes (proven again today); + the G1 engineer/reviewer handled ugrep-vs-BSD-grep and binary-file leaks in sweeps: prefer + `git grep -In` with pathspec excludes. +- **Ultracode is per-turn opt-in** (keyword); Felipe's green-CI merge authorization was per-goal — + have him restate both in the fresh session for autonomous operation. +- **Cegonha (10.114.1.121) DOWN all day** — Hermes counter-reads fail-open ×3 gates, logged each time. + +## 6. Machine/tree state at handoff + +- Shared checkout `~/workspace/genie`: branch `wish/agent-sync` (other session's; duplicates + live + G2 edits). Untracked: G1 files, `COORDINATION.md`, this handoff, `.codex/`, `AGENTS.md`, + `.genie/repo-profile.md`, `.genie/wishes/omni-branch-drift-sync/`, `perr.txt` (empty — deletable). +- Codex worktree exists: `~/.codex/worktrees/8a2e/genie` (detached). +- Stale gitignored `dist/{darwin-arm64,linux-*}` trees were moved to the DYING session's scratchpad + (auto-cleaned eventually) — regenerable via `npm run build-and-sync`; sweeps now see a clean tree. +- `~/.claude/workflows/council.js` stamped and live. +- INDEX.md on dev is current through #2540 (`bedd2062`); the shared checkout's INDEX is the + agent-sync branch variant (+1 line). +- Memory: `~/.claude/projects/-Users-feliperosa-workspace-genie/memory/` (review-dispatch rule, + board daily-driver, rtk grep unreliability, one-subject-at-a-time) — all still valid. diff --git a/.genie/wishes/agent-sync/COORDINATION.md b/.genie/wishes/agent-sync/COORDINATION.md new file mode 100644 index 000000000..4173e5d0f --- /dev/null +++ b/.genie/wishes/agent-sync/COORDINATION.md @@ -0,0 +1,59 @@ +# Coordination note for G2 dispatch (from the plugin-resource-shipping session, 2026-07-10) + +> **FELIPE DIRECTIVE (04:50Z): a PR to automagik-dev/genie is REQUIRED — local delivery is not +> sufficient.** Push the branch and open the PR (base `dev`) as soon as G2 gates pass; do not sit on +> local commits. G3 can ride the same PR or a fast follow. Do the §1 reset BEFORE pushing, or the PR +> drags three already-merged duplicate commits. The orchestrating session is watching for the PR and +> will CI-gate + merge it on green (Felipe's standing authorization, dev-target only). + +Felipe ruled: this session owns G2+G3. Facts you need before dispatching G2: + +## 1. Branch surgery required first +This branch (`wish/agent-sync`) carries three foreign commits stacked on your 63f63c95 — +`805afcd4` (G1), `0584639d` (G3), `422dc6dd` (G2) of wish **plugin-resource-shipping**. They were +committed here by accident (shared checkout; your session switched the branch mid-flight of our +workflow). They are now **merged into dev via PR #2540** (rebuilt as `ecbb67fc`/`203c97df`/`bbd6439e`), +so the copies here are pure duplicates: + +```bash +git reset --hard 63f63c95 # shed duplicates (your docs commit becomes tip; G1 work is in untracked src/lib/*) +git rebase origin/dev # pick up #2540 + council merge + version bumps (dev tip ≥ bedd2062) +``` + +## 2. What #2540 changed that intersects your G2 +- `skills/brainstorm/SKILL.md:91` — the wishes-lint sentence was REWORDED (paraphrase rule). Your + lens-root anchor sentence lands in this file; rebase textually, don't restore old wording. +- `skills/wish/SKILL.md`, `skills/README.md`, `tests/e2e/v5-lifecycle.sh` — template now lives at + `skills/wish/templates/wish-template.md`, addressed via `${CLAUDE_SKILL_DIR}`. +- **New lint you must pass**: `scripts/skills-lint.ts` now has a resource-shipping rule scanning + bash fences AND inline-code spans in skill files. Imperative repo-root resource refs fail + (`cp templates/…`, unguarded `bun run wishes:lint`, imperative `scripts/*.ts` runs). Your anchor + sentence is prose/descriptive so it should pass — but run `bun run skills:lint` before committing. +- **New CI step**: `scripts/fresh-install-smoke.ts` (unit job) asserts every `${CLAUDE_SKILL_DIR}` + ref in any SKILL.md resolves inside that skill's dir, and scaffolds a wish in a bare repo with no + genie on PATH. If your lens-root anchor introduces `${CLAUDE_SKILL_DIR}` refs, they must resolve. + +## 3. Live-QA datapoint for your G2 stamp logic +The shipped stamp mechanism was verified working on this machine today: `smart-install.js` (run +manually, plugin cache 5.260710.2) stamped `~/.claude/workflows/council.js` successfully — the +SessionStart-only trigger gap Felipe hit is exactly your wish's premise. Note for your +adopt-with-backup tests: this machine now has a PRE-EXISTING stamped `council.js`, so your first +live `genie update` sync exercises the adopt/managed-manifest path, not fresh-create. + +## 4. Stale dist trees +Old gitignored `dist/{darwin-arm64,linux-*}/` trees (5.260702.1 vintage, containing pre-Fable +SKILL.md copies) were relocated to this session's scratchpad to unpollute path sweeps. Regenerable +via `npm run build-and-sync`. If your gates sweep dist/, they're gone — that's intentional. + +## 5. Note from the overnight orchestrator (session d0554818, ~03:30Z 2026-07-10) + +I am the successor of the orchestrating session. Standing watch is ACTIVE: the moment your PR to +dev appears I `gh pr checks --watch --fail-fast` and merge on green (Felipe restated the +authorization tonight, dev-target only). I will NOT touch this checkout — worktree isolation is +law; my overnight work (resource-shipping LOW follow-ups, docs) runs in separate worktrees off +origin/dev and my merges are serialized AFTER yours (you have priority; I rebase on moved dev). +Thanks for upstreaming the council.js string-args coercion (d0d61211) — I hit that live tonight; +my QA notes credit it. Reminder from your own §1: shed the three duplicate commits before pushing +(now that G1/G2/fix commits are stacked, that's `git rebase --onto origin/dev 422cdd6c`-style — +i.e. rebase onto origin/dev dropping 805afcd4/0584639d/422dc6dd; plain `git rebase origin/dev` +should also auto-skip them as already-applied patches). diff --git a/.genie/wishes/council-workflow/WISH.md b/.genie/wishes/council-workflow/WISH.md index c1a5f7274..1ddc3b555 100644 --- a/.genie/wishes/council-workflow/WISH.md +++ b/.genie/wishes/council-workflow/WISH.md @@ -2,7 +2,7 @@ | Field | Value | |-------|-------| -| **Status** | DRAFT — design review SHIP + plan review SHIP (2026-07-09, same independent reviewer, 1 fix pass each); `/work` user-gated | +| **Status** | EXECUTED (2026-07-10) — G1–G5 all SHIP (see INDEX poured entry); design + plan reviews SHIP (2026-07-09, same independent reviewer, 1 fix pass each). **Live-QA ritual (Felipe: `/council "revisar tudo"`) and the final execution review remain pending post-stable-release** — the g5-gate parks in `qa/` until then. First real dogfood tonight surfaced and fixed a string-args defect (`ec68cd8f`): see [qa/overnight-observations-20260710.md](qa/overnight-observations-20260710.md) | | **Slug** | `council-workflow` | | **Date** | 2026-07-09 | | **Author** | Felipe (planned with Fable 5) | diff --git a/.genie/wishes/council-workflow/qa/overnight-observations-20260710.md b/.genie/wishes/council-workflow/qa/overnight-observations-20260710.md new file mode 100644 index 000000000..812887a00 --- /dev/null +++ b/.genie/wishes/council-workflow/qa/overnight-observations-20260710.md @@ -0,0 +1,28 @@ +# /council — Overnight QA Observations (2026-07-10) + +**Context:** first real `/council` dogfood of the overnight run — a lens council deliberation (3 lenses, 2 rounds) plus the live-QA ritual against the stamped `~/.claude/workflows/council.js`. These are the observations to fold into the council-workflow **final execution review**, which stays pending Felipe's live-QA tail (`/council "revisar tudo"`) after the next stable release. + +--- + +## Defect found and fixed: stamped `council.js` rejected string args + +**Symptom.** Invoking the stamped `/council` with a string argument dead-ended three times in a row with `"No input received"` — the workflow never entered its Resolve phase. + +**Root cause.** The Claude Code **Workflow runtime stringifies** saved-workflow input before handing it to the script. The stamped `council.js` demanded an **object**-shaped argument and treated the incoming string as empty, so it fell through to the no-input branch. This is a runtime-contract mismatch, not a logic bug in the deliberation flow: the script's own arg-parsing assumed a shape the runtime does not deliver for saved workflows. + +**Fix.** Coerce a string argument into the expected shape at the workflow entry point (accept both a raw string topic and the object form). Validated **live** on a patched copy of the script first (the string invocation reached Resolve → Round 1), then upstreamed by the owning session and merged to dev as **`ec68cd8f`** (`fix(workflows): coerce string args in /council — runtime stringifies saved-workflow input`). Verified present on `origin/dev` in tonight's run. + +## Secondary observation: skill-text vs script-contract mismatch + +The `/council` **skill instruction text** tells the caller to pass arguments **as a string**, while the **pre-fix script** demanded an **object**. Before `ec68cd8f` these two contracts disagreed, which is exactly the surface the defect lived on. **Post-fix both work** — the coercion accepts the string form the skill text advertises and the object form the earlier script expected. No doc change is required now that the script tolerates both, but the final execution review should record that the skill text and the script contract were briefly out of sync and are now reconciled by coercion rather than by narrowing either side. + +## Related note carried from the wish record + +The stamp mechanism fired only on **SessionStart**, so a mid-session `/plugin update` left a stamping gap (worked around by running `smart-install.js` manually, plugin cache 5.260710.2). That SessionStart-only trigger gap is the premise **agent-sync** was built to close (agent-sync merged to dev 2026-07-10 via PR #2541); re-confirm the stamp path after the next stable release + `genie update` ×2. + +--- + +## Still pending (USER-GATED, post-stable-release) + +1. **Live-QA ritual** — Felipe runs `/council "revisar tudo"` against the released, self-synced `council.js`. This exercises the string-args path end-to-end on the shipped artifact (not a patched copy). +2. **Final execution review** — written after the ritual; the g5-gate parks in this `qa/` directory until then. It should incorporate this defect story and the deliberation-quality observations from the first 3-lens / 2-round dogfood. diff --git a/.genie/wishes/rolling-pr-auth-hardening/WISH.md b/.genie/wishes/rolling-pr-auth-hardening/WISH.md index 0689663e1..1ff8fb97f 100644 --- a/.genie/wishes/rolling-pr-auth-hardening/WISH.md +++ b/.genie/wishes/rolling-pr-auth-hardening/WISH.md @@ -2,7 +2,7 @@ | Field | Value | |-------|-------| -| **Status** | DRAFT | +| **Status** | DONE — implemented on dev **prior to execution of this wish**: `c4fdb32b` (fail-fast on dead/absent PAT + read/create token split) and `422caaa2` (dev==main healthy no-op) together satisfy the acceptance criteria; both commits are also promoted to `origin/main`. Discovered already-shipped by a council freshness probe (2026-07-10) — no new engineering dispatched. Minting/refreshing the PAT itself remains the open human action for Felipe | | **Slug** | `rolling-pr-auth-hardening` | | **Date** | 2026-07-05 | | **Author** | Felipe (dogfooding the revamped v5 lifecycle) | @@ -37,10 +37,10 @@ Harden the workflow: fail fast with an actionable `::error` when the secret is a ## Success Criteria -- [ ] With the secret absent or dead (current reality: present, 401): `workflow_dispatch` run fails in seconds with the actionable `::error` text, not a raw gh auth message. -- [ ] `gh pr list` step authenticates via `github.token` (no PAT dependency on the read path). -- [ ] YAML parses; workflow diff touches only `rolling-pr.yml`. -- [ ] Once Felipe refreshes the PAT: the hourly run creates/confirms the rolling PR again (post-merge QA item). +- [x] With the secret absent or dead (current reality: present, 401): `workflow_dispatch` run fails in seconds with the actionable `::error` text, not a raw gh auth message. — satisfied by `c4fdb32b`. +- [x] `gh pr list` step authenticates via `github.token` (no PAT dependency on the read path). — satisfied by `c4fdb32b` (read/create token split). +- [x] YAML parses; workflow diff touches only `rolling-pr.yml`. — verified in the shipped commits (`c4fdb32b`, `422caaa2`). +- [ ] Once Felipe refreshes the PAT: the hourly run creates/confirms the rolling PR again (post-merge QA item — **open human action**, blocked on the PAT mint). ## Execution Strategy @@ -92,7 +92,7 @@ cd "$(git rev-parse --show-toplevel)" && python3 -c "import yaml,sys; yaml.safe_ ## Review Results -_Populated by `/review`._ +**Resolution (2026-07-10): already implemented on dev, no dispatch needed.** A council freshness probe found the hardening had landed independently: `c4fdb32b` adds the fail-fast guard on an absent/dead PAT and splits the read path (`gh pr list` on `github.token`) from the create path (PAT), and `422caaa2` makes the workflow a healthy no-op when dev==main. `git branch -r --contains` confirms both commits are on `origin/dev` and `origin/main`. Acceptance criteria 1–3 are satisfied by the shipped code; criterion 4 (hourly run creates the rolling PR again) stays blocked on Felipe minting/refreshing `RELEASE_PLEASE_TOKEN` with `contents:read` + `pull-requests:write`. No engineering was dispatched for this wish. --- diff --git a/.genie/wishes/routing-matrix/WISH.md b/.genie/wishes/routing-matrix/WISH.md index 8f93cf5da..759347f5c 100644 --- a/.genie/wishes/routing-matrix/WISH.md +++ b/.genie/wishes/routing-matrix/WISH.md @@ -2,7 +2,7 @@ | Field | Value | |-------|-------| -| **Status** | EXECUTED — execution review SHIP (2026-07-09; live LangWatch QA pending) | +| **Status** | EXECUTED — execution review SHIP (2026-07-09); day-1 live pin QA recorded 2026-07-10 — **inconclusive by delivery gap** (Fable share rose on every measure, but the 7 pinned role agents did NOT appear as subagent types under plugin cache 5.260710.2, so pins were applied by hand, not mechanically; properly-pinned wish ran ~11% Fable). Re-test after next stable release carrying agent-sync + `genie update` ×2 — see [qa/routing-pin-qa-20260710.md](qa/routing-pin-qa-20260710.md) | | **Slug** | `routing-matrix` | | **Date** | 2026-07-09 | | **Author** | Felipe (planned with Fable 5 + Hermes counter-read) | diff --git a/.genie/wishes/routing-matrix/qa/routing-pin-qa-20260710.md b/.genie/wishes/routing-matrix/qa/routing-pin-qa-20260710.md new file mode 100644 index 000000000..18194c9ac --- /dev/null +++ b/.genie/wishes/routing-matrix/qa/routing-pin-qa-20260710.md @@ -0,0 +1,121 @@ +# Routing-Matrix Pin — Day-1 Live QA (2026-07-10) + +**Destination:** `.genie/wishes/routing-matrix/qa/` +**Analyst run:** 2026-07-10 ~06:51Z · source: LangWatch (`https://langwatch.khal.ai`) via `langwatch` CLI +**Pins under test:** routing-matrix (PR #2535, merged 2026-07-09 ~21:51Z, released v5.260710.2) — Fable only at gates, Opus ladder for engineering, Haiku scouts. + +--- + +## Verdict — pins are NOT collapsing Fable on execution work yet (inconclusive as a design test) + +On the first partial day under the pins, **Fable's share rose on every measure** — the opposite of the designed collapse — while **Opus per-trace engagement fell from 84% to 48%** and the **Haiku scout lane went nearly silent (10 → 4 traces)**. + +This is **not** evidence the pinning *design* is wrong. It is evidence the pins **were not mechanically in effect**: the pinned role agents were not available as subagent types, so the overnight run defaulted to a Fable orchestrator with hand-applied model overrides. The measured rise reflects an atypically gate/review/orchestration-heavy dogfood night on a Fable default, sampled over only ~7 hours — not a refutation of the ladder. Treat day-1 as **inconclusive for the design, conclusive that the expected collapse has not occurred**, and re-verify once pins are actually delivered as subagent types. + +> **Observed delivery gap (verbatim, for the record):** +> Pinned role agents (engineer-trivial/standard/complex, fixer, reviewer, final-gate, scout) did NOT appear as available subagent types in a fresh Claude Code session on 2026-07-10 03:06 (plugin cache 5.260710.2) — the session applied the routing ladder manually via model overrides. This is the lazy-delivery gap agent-sync closes; agent-sync merged to dev 2026-07-10 06:47Z (PR #2541) — re-verify after the next stable release + genie update ×2. + +--- + +## The numbers + +Comparison is **2026-07-10 00:00→06:51Z (partial, ~6.85h, post-pin but pins not mechanically active)** vs **2026-07-09 full day (pre-pin baseline — pins only merged at 21:51Z that night)**. Shares are ratios, so they remain comparable across a partial vs full day even though absolute volumes do not. + +### Model share moved the wrong way on all three measures + +| Measure | Model | 07-09 (pre-pin) | 07-10 (post-pin, ~7h) | Direction | Design intent | +|---|---|---:|---:|---|---| +| **Billable cost** | Fable | 46% ($562) | **57% ($956)** | ▲ +11pt | ▼ down (gates only) | +| | Opus | 41% ($501) | 34% ($562) | ▼ | ▲ up (engineering) | +| | Haiku | 14% ($167) | 9% ($159) | ▼ | ▲ up (scouts) | +| **Token volume** (span-level, total_tokens) | Fable | 33% | **50%** | ▲ +17pt | ▼ down | +| | Opus | 43% | 36% | ▼ | ▲ up | +| | Haiku | 24% | 14% | ▼ | ▲ up | +| **Trace touch-rate** (traces touching model ÷ distinct) | Fable | 60% | **86%** | ▲ +26pt | ▼ down | +| | Opus | 84% | 48% | ▼ −36pt | ▲ up | +| | Haiku | 20% | 5% | ▼ | ▲ up | + +Totals: 07-09 billable $1,230 / 3.67M tokens / 50 distinct traces (82 model-touches). 07-10 (partial) billable $1,677 / 3.83M tokens / 84 distinct traces (116 model-touches). + +**Read:** every lens agrees. Fable is *up*, Opus and Haiku are *down*. The single clearest signal is the flip in dominant model — on 07-09 Opus was the most-engaged model (84% of traces touched it); on 07-10 Fable overtook it (86%) and Opus engagement collapsed to 48%. That is engineering work **not** being routed to the Opus ladder. + +### Effort distribution — high-effort throughout, scout lane silent + +Effort is a per-trace field (`gen_ai.request.reasoning_effort`), read directly from trace search (the analytics effort-filter is known to be silently broken, so it was not used). + +| Effort | 07-09 (n=50) | 07-10 (n=84) | +|---|---:|---:| +| xhigh | 27 (54%) | 37 (44%) | +| high | 6 (12%) | 27 (32%) | +| max | 17 (34%) | 20 (24%) | +| medium / low | 0 | 0 | + +**Read:** the population is entirely xhigh/high/max on both days — **zero low/medium-effort traces**. Haiku scouts, which would run cheap/low-effort, essentially did not fire (only 4 Haiku-touching traces on 07-10). The one favorable movement is a modest max→high shift (34%→24% max), consistent with *some* deep-reasoning work sliding down a tier, but it is well within noise for an n=84 sample. + +### Top-5 sessions by cost (07-10, by thread) + +| # | Thread (trunc) | Cost (trace-level) | Character | +|---|---|---:|---| +| 1 | `ae132245…` | $428.82 | Team-lead orchestrator, **max** effort, Fable — merge/coordination thread ("watch #2542, then genie update ×2") | +| 2 | `4f125d00…` | $210.95 | Long-running work thread | +| 3 | `8050dd5c…` | $108.48 | Carryover thread (was #1 on 07-09 at $442) | +| 4 | `52675102…` | $60.48 | — | +| 5 | `d0554818…` | $56.78 | *This QA analysis session itself* | + +**Read:** the cost is concentrated in **orchestration/coordination threads**, led by the team-lead planner running at max effort on Fable. That kind of thread is *legitimately* Fable-heavy under the design (planning/gates). The problem is not that these threads used Fable — it is that we cannot observe the counterbalancing Opus-engineering collapse, because engineering wasn't dispatched to pinned Opus agents. The "first dogfooded wish under pins ran at ~11% Fable tokens" data point shows the ladder *does* work where pins are applied; the aggregate day is simply swamped by unpinned orchestration/review threads. + +--- + +## Why day-1 is inconclusive — caveats that bound every number above + +1. **Pins were not mechanically enforced.** The pinned role agents did not exist as subagent types in the fresh session (verbatim finding above); the ladder was applied by hand. So 07-10 does **not** actually exercise the pinning mechanism — it measures an unpinned Fable-default run. +2. **Partial window.** 07-10 is only ~6.85h (00:00–06:51Z) vs a full 07-09. Shares are ratio-based and comparable; absolute volumes and run-rates are not. +3. **Atypical workload.** This window is the genie overnight dogfood — an orchestration/review/council/gate-heavy multi-agent run. That mix is Fable-heavy *by design*. A normal execution-heavy day would look different. +4. **07-09 is a clean-ish pre-pin baseline** (pins merged 21:51Z that night, so <2h of 07-09 was post-pin) — a fair "before". +5. **Trace-count-by-model double-counts.** 116 model-touches across 84 distinct traces → multi-model orchestration traces are counted under every model they touch. **Token share is the cleaner signal**; trace touch-rate is directional only. +6. **Trace-level cost ($1,049) < span-level analytics cost ($1,613) for 07-10** — a Claude Code trace bundles multiple model spans and the trace-level `total_cost` metric under-captures sub-span cost. **Analytics groupBy `metadata.model` is authoritative for model attribution**; trace-search costs are used only for per-trace effort/percentile work. + +**Re-verification trigger:** after the next *stable* release carrying agent-sync + `genie update` ×2 on a dogfood host, confirm (a) the seven pinned role agents appear as subagent types in a fresh session, then (b) re-pull this exact comparison. Expect Fable token share to fall toward gate-only levels (~the ~11% seen on the one properly-pinned wish) and Opus engineering share to rise. + +--- + +## Methodology & working query recipes (auth redacted) + +**Access resolution (the earlier 403).** Direct REST `POST /api/trace/search` and `/api/analytics` returned HTTP 403 earlier tonight under both `Authorization: Bearer` and `X-Auth-Token`. **Root cause: payload/route shape, not the key.** The same OTLP-ingest key works fine through the official `langwatch` CLI, which wraps those endpoints. Recommendation: **drive LangWatch through the CLI, not hand-rolled REST.** + +Auth is supplied to the CLI via two env vars (key read at runtime from the `OTEL_EXPORTER_OTLP_HEADERS` bearer token in `~/.claude/settings.json` — **never printed, never stored in any artifact**): + +``` +LANGWATCH_API_KEY= # the sk-… bearer token from OTEL_EXPORTER_OTLP_HEADERS +LANGWATCH_ENDPOINT=https://langwatch.khal.ai +``` + +Verify auth: `npx -y langwatch status` → returns project resource counts (200 = key valid). + +**Cost / tokens / trace-count by model** (authoritative for model share): +``` +npx -y langwatch analytics query \ + --metric performance.total_cost \ # or performance.cost_billed | performance.total_tokens + --aggregation sum \ # trace-count uses: --metric trace-count --aggregation cardinality + --group-by metadata.model \ + --start-date 2026-07-10T00:00:00Z --end-date 2026-07-10T23:59:59Z \ + --time-scale full --format json +``` +Returns `{ currentPeriod, previousPeriod }` — `previousPeriod` is the auto-computed prior equal-length window (a single-day query yields the prior day free; the two reconcile across queries). + +Valid metric enum (discovered via a bad-metric error): `performance.total_cost`, `.cost_billed`, `.cost_non_billed`, `.prompt_tokens`, `.completion_tokens`, `.cache_read_tokens`, `.cache_write_tokens`, `.reasoning_tokens`, `.total_processed_tokens`, `.total_tokens`, `.tokens_per_second`; plus `metadata.trace_id|thread_id|user_id|span_type`. + +**Top sessions by cost:** same as above with `--group-by metadata.thread_id`. + +**Effort distribution + per-trace percentiles** (bypass the broken analytics effort-filter): +``` +npx -y langwatch trace search \ + --start-date 2026-07-10T00:00:00Z --end-date 2026-07-10T23:59:59Z \ + --limit 2000 --format json +``` +Each trace carries `metadata.gen_ai.request.reasoning_effort` (per-trace) and `metrics.total_cost` — histogram + p50/p90 computed client-side. `pagination.totalHits` gives the true distinct-trace count. + +**Traps confirmed tonight:** +- `analytics query` exposes **no `--filter` flag** in the CLI — effort/metadata filtering is not available there. Use trace search for anything effort-scoped. (The underlying analytics effort-filter is also reported silently broken.) +- **Do not** combine `--group-by metadata.model` with a cache-read metric — known ClickHouse bug (avoided; not attempted). +- **Per-trace model is not available from trace search** (spans return empty; trace metadata has no model field). Per-model cost *distributions* (p50/p90) are therefore not derivable via the CLI — only per-model means (analytics cost ÷ trace-count) and per-effort percentiles. diff --git a/.genie/wishes/skills-fable5-revamp/WISH.md b/.genie/wishes/skills-fable5-revamp/WISH.md index e254a9874..270496e59 100644 --- a/.genie/wishes/skills-fable5-revamp/WISH.md +++ b/.genie/wishes/skills-fable5-revamp/WISH.md @@ -2,7 +2,7 @@ | Field | Value | |-------|-------| -| **Status** | DRAFT — plan review SHIP on full scope G1–G8 (2026-07-04, 2 review rounds); `/work` Wave 1 committed (G1–G6, G8), G7 verification complete (`verification.md`) — awaiting execution review | +| **Status** | EXECUTED — execution review **SHIP** (2026-07-10, HIGH 0 / MEDIUM 0 / 3 LOW observations; genie side independently re-verified, omni G5/G6 attested via `verification.md`) — [reports/execution-review-20260710.md](reports/execution-review-20260710.md). Merged PR #2518 (merge `1308e4c6`); plan review SHIP on full scope G1–G8 (2026-07-04, 2 rounds); G7 verification complete (`verification.md`). LOW follow-ups: omni-side confirm of G5/G6 numbers, `verification.md` diff-count one commit stale (cosmetic), and a "superseded by later wishes" note for `learn`/`council` (both since removed by other wishes) | | **Slug** | `skills-fable5-revamp` | | **Date** | 2026-07-04 | | **Author** | Felipe (planned with Fable 5) | diff --git a/.genie/wishes/skills-fable5-revamp/reports/execution-review-20260710.md b/.genie/wishes/skills-fable5-revamp/reports/execution-review-20260710.md new file mode 100644 index 000000000..ac6783289 --- /dev/null +++ b/.genie/wishes/skills-fable5-revamp/reports/execution-review-20260710.md @@ -0,0 +1,96 @@ +# Execution Review — `skills-fable5-revamp` + +## Verdict: **SHIP** + +The wish delivered all ten Success Criteria at merge (PR #2518, merge commit `1308e4c6`, fable5 tip `a4f089a5`), and its acceptance intent survives in the current `origin/dev` (`b1f07913`, v5.260710.5). Every difference between what the wish predicted and what `origin/dev` shows today is attributable to **later merged wishes** (council-workflow, token-efficiency, routing-pin, plugin-resource-shipping, agent-sync), not to a gap in this wish. I independently re-ran 8 of `verification.md`'s checks against the delivered commit and current dev; all corroborate. No HIGH or MEDIUM gaps on the genie surface. The only limitation is environmental: the omni half (G5/G6) lives in the separate `automagik-dev/omni` repo and cannot be re-run from this checkout — those two groups rest on `verification.md`'s attestation. + +**Gap count:** HIGH 0 · MEDIUM 0 · LOW 3 (all observations, none SHIP-blocking). + +--- + +## What I verified independently (not taken on faith) + +Reference points: **delivered** = `a4f089a5` (2nd parent of merge #2518, includes `verification.md` + the lint fix); **current** = `origin/dev` @ `b1f07913`. Checks run with `git grep` / `git show` (bare `grep` is unreliable here) from script files in the scratchpad. + +| # | Check (SC) | My result | verification.md claim | Match | +|---|-----------|-----------|-----------------------|-------| +| 1 | Dead-namespace grep on `skills/**` (SC4/SC10) | **0 hits** at both `a4f089a5` and `origin/dev` | 0 (was 118) | ✅ | +| 2 | Reasoning-extraction grep on `skills/**` (SC6) | **0 hits** at both refs | 0 | ✅ | +| 3 | 17-skill genie total, delivered (SC3) | **1,385 lines** (summed per-file) | 1,385 (−66.9%) | ✅ exact | +| 4 | Budgets — every `SKILL.md` ≤ 200 (SC2) | Max delivered = 125 (brainstorm); max current = 148 (review) | all ≤ 200 | ✅ | +| 5 | Frontmatter byte-0 + `name`=dir, 17 skills (SC1) | **17/17 OK** | 17/17 | ✅ | +| 6 | Diff shape vs merge-base `63015670` (SC8) | **13 A / 28 M, 0 D, 0 R**, all in-scope | 12 A / 27 M, 0 D/R | ✅ (see LOW-2) | +| 7 | G1 `refine/prompts/optimizer.md` present | **721 lines**, both refs | present | ✅ | +| 8 | G8 shared manifest / marker-gating / backup / uninstall consumption | read module; conforms (below) | PASS | ✅ | + +The SC8 merge-base I computed (`63015670`) is the exact commit `verification.md` cites — the diff-shape check is anchored to the same baseline. + +--- + +## Per-group findings + +| Group | Scope | Verdict | Evidence / notes | +|-------|-------|---------|------------------| +| **G1** genie extraction-heavy (`refine`, `genie-hacks`) | delivered | **SHIP** | `refine` 803→48, `genie-hacks` 626→47; `refine/prompts/optimizer.md` = 721 lines and the SKILL Reads it at dispatch; dead-ref grep clean. Catalog moved to `genie-hacks/references/{catalog,contributing}.md` (in SC8 A-list). | +| **G2** lifecycle core (`brainstorm`,`wish`,`work`,`review`,`fix`,`trace`) | delivered | **SHIP** | All six ≤ 200 at delivery; all 8 frozen-contract families present (template `cp`, `wishes:lint` gate, `genie task` linkage, DRAFT/SHIP/FIX-FIRST/BLOCKED, reviewer≠engineer, ≤2 fix loops, orchestrator-only `task done`, session-close outcome words). Later growth (work 103→123, fix 75→99, review 114→148, brainstorm 125→133, wish 71→73) is from routing-pin `1430def4`, council-workflow lens-panels `2af7ebd9`, template-shipping `ecbb67fc`, agent-sync `ff497ae4` — **not this wish**; all still ≤ 200. | +| **G3** routing & onboarding (`genie`,`wizard`,`learn`,`docs`,`omni`) | delivered | **SHIP** | All five delivered and lint-clean. `learn` (61 lines) shipped correctly, then **removed later** by the token-efficiency wish (`3d40966c`) — absence in current dev is attributable drift, not a G3 gap. Router route table preserved. | +| **G4** PM & multi-agent (`pm`,`dream`,`council`,`report`) | delivered | **SHIP** | All four delivered (pm 108, dream 103, council 107, report 101); grounded-progress clause present. `council` shipped correctly, then **retired later** by council-workflow G3 (`22a7ed50`, "/council is the native workflow now") — attributable drift, not a G4 gap. | +| **G5** omni skill tier (`omni`,`omni-agent`,`omni-setup`,`omni-ops`) | **not re-runnable here** | SHIP *(attested)* | Lives in `automagik-dev/omni`, absent from this genie checkout (`git ls-tree origin/dev plugins/` → only `plugins/genie`, `plugins/hermes-genie`). `verification.md` records `omni-ops` 484→41 and SC5 `G5-OK`. See LOW-1. | +| **G6** omni commands/agents/rules (15 files) | **not re-runnable here** | SHIP *(attested)* | Same repo boundary. `verification.md` SC5 `G6-OK`; `rules/omni-agent.md` no-frontmatter documented as a pre-existing baseline format, not a regression. See LOW-1. | +| **G7** cross-repo verification | delivered | **SHIP** | `verification.md` present, thorough, 10/10 SC PASS. I re-derived its genie-side numbers exactly (1,385 total; 0 dead refs; 0 reasoning-extraction; 0 D/R). Its two in-scope fixes (conventions `install` add; Status annotation) are consistent with the shipped tree. | +| **G8** v4 legacy audit & cleanup | delivered + current | **SHIP** | `legacy-v4.ts` present both refs. Conservative by construction; details below. Foundation was **reused** by `doctor.ts` in current dev (`--fix` calls `cleanupV4`) — intent expanded, not eroded. | + +### G8 detail (the one destructive surface — reviewed closely) + +`src/genie-commands/legacy-v4.ts` (read at `a4f089a5`) satisfies every G8 acceptance criterion: + +- **One shared manifest** — `V4_LEGACY_MANIFEST` is the single source of truth. `uninstall.ts:16` imports `orchestrationRulesPath` from `./legacy-v4.js` and derives `ORCHESTRATION_RULES_PATH` from it (`uninstall.ts:19`); **no literal path restated** (grep for `genie-orchestration.md` in uninstall.ts → none). AC4 met: zero duplicated path literals. +- **Marker-gating** — rules file removed only when content contains `genie spawn` or `genie team create`; unreadable/unmarked → `user-modified` → kept with warning (`cleanupRulesFile`). +- **Cache scoping** — only `4.*`-prefixed dirs carrying `.orphaned_at` are removed; `isDirectory()` excludes symlinks ("never follow a link out of the cache"). +- **Backup-first** — `backupFile` before `unlinkSync`; cache dirs back up a manifest listing (not the re-downloadable payload); log to `/logs/v4-cleanup.log`. +- **Idempotent** — clean machine → `noOp: true`, nothing printed/written. +- **install.sh** is flag-only: forwards `--skip-v4-cleanup`, hands off via a plain (non-exec) `genie install` call; the sole "legacy path" occurrence in `install.sh` is a **comment** (`install.sh:366`) pointing at the TS module, not bash logic. `bash -n install.sh` clean (SC9). +- **Adjudicated skills-lint expansion** (2nd OUT-scope expansion) confirmed: `scripts/skills-lint.ts` warns to stderr and skips omni checks when the omni CLI is absent, strict mode preserved behind `SKILLS_LINT_REQUIRE_OMNI=1` (exit 2). This is the +1 `M` explaining my 28-vs-27 count. + +--- + +## Success-criteria checklist (against merged state) + +| SC | Criterion | Status | Evidence | +|----|-----------|--------|----------| +| SC1 | 36 surfaces conform (FM byte-0, name=dir, trigger desc) | ✅ genie 17/17 re-verified; omni attested | script `verify-final.sh` ALL-OK; omni per verification.md (1 documented rules-file FM exception) | +| SC2 | Budgets (SKILL ≤200, cmd/agent ≤40, rule ≤30) | ✅ | delivered max 125; current dev max 148; all ≤ 200 | +| SC3 | Total always-loaded ≤ 3,300 (≥40% cut) | ✅ | genie 1,385 re-derived exactly; combined 2,016 (−63.4%) per verification.md | +| SC4 | `skills:lint` + `wishes:lint` exit 0 | ✅ | dead-ref grep 0 (I ran it); both scripts wired into `check`/`check:fast` (`package.json:24-25,29-30`) | +| SC5 | Per-file ceilings not treated as targets | ✅ | most files near ~120; binding ≤3,300 met with headroom | +| SC6 | Omni structural check exits 0 | ✅ attested | verification.md SC5 `G5-OK`/`G6-OK` (omni repo) | +| SC7 | Zero reasoning-extraction language | ✅ | grep 0 hits both refs (I ran it) | +| SC8 | Frozen contracts intact | ✅ | template `cp`, lint gates, `genie task` linkage, status vocab, three-tier omni routing, `allowed-tools` — verified at delivery; template later re-homed to `${CLAUDE_SKILL_DIR}` by plugin-resource-shipping `ecbb67fc` (intent preserved) | +| SC9 | Diff shape M+A only, no D/R | ✅ | 13 A / 28 M / 0 D / 0 R, all in-scope (I ran it) | +| SC10 | v4 rules verdict implemented, one shared manifest | ✅ | `legacy-v4.ts` + `uninstall.ts` consumption; colocated pgserve-free tests | + +*(SC numbering above follows the WISH "Success Criteria" list; verification.md's SC1–SC10 map 1:1.)* + +--- + +## Gaps (ranked) + +**HIGH — none. MEDIUM — none.** + +**LOW-1 — Omni half (G5/G6) is not independently verifiable from this checkout.** The omni plugin lives in `automagik-dev/omni`, which is not present in the genie repo (`plugins/` holds only `genie` and `hermes-genie`). My SHIP for G5/G6 rests entirely on `verification.md`'s pasted `G5-OK`/`G6-OK` and the recorded per-group execution reviews. *Recommendation:* have the reviewer with the omni checkout (or the live-QA pass) re-confirm the omni 1,329→631 numbers and the omni-ops routing-table→references resolution before treating the omni PR as reviewed. This is a review-environment limitation, not evidence of a defect. + +**LOW-2 — `verification.md` diff-shape count is one commit stale.** It records genie "12 A / 27 M"; the final delivered branch (`a4f089a5`) is "13 A / 28 M". The extra `M` is `scripts/skills-lint.ts`, modified by the lint-fix commit `a4f089a5` that landed *after* verification.md was authored at `af81783b`. Cosmetic staleness in the record; the load-bearing invariant (zero D, zero R, all files in-scope) holds. No action required beyond awareness. + +**LOW-3 — Two delivered skills (`learn`, `council`) no longer exist in current dev.** Both shipped correctly under this wish (G3 `learn`, G4 `council`) and were later removed by other wishes (`3d40966c` token-efficiency; `22a7ed50` council-workflow). A reader diffing `verification.md`'s per-file table against today's `origin/dev` will see the mismatch. *Optional:* a one-line "superseded by later wishes" note in the wish record would prevent future confusion. Not a defect in this wish. + +*(QA Criteria in the WISH — fresh-session routing, lifecycle dry-run, `/refine` runtime dispatch, omni verbs, live v4-cleanup on a v4 box — are runtime/live-QA checks outside a static read-only review's scope; they belong to the post-merge QA pass, not this execution review.)* + +--- + +## Bottom line + +The genie side of `skills-fable5-revamp` is a clean, well-evidenced SHIP: the 118→0 dead-CLI re-grounding, the ≥40% line reduction (1,385 genie / 2,016 combined), the byte-0 frontmatter conformance, the zero-deletion diff shape, the extracted `optimizer.md`, and a genuinely conservative G8 cleanup engine all verify independently. `verification.md` is accurate — every genie-side number I re-derived matched, and where it differs from today's tree the cause is a later merged wish, correctly outside this wish's ownership. Nothing here blocks flipping the wish Status to reflect an execution-review SHIP. + +--- + +*Reviewed by subagent (session d0554818 overnight run), 2026-07-10.* From 686ffb36645c445457d8dd5696fd08ef5248eafe Mon Sep 17 00:00:00 2001 From: namastex888 Date: Fri, 10 Jul 2026 04:19:04 -0300 Subject: [PATCH 2/2] docs(genie): morning brief + DRAFT advancement + INDEX hygiene --- .genie/INDEX.md | 22 ++++---- .genie/brainstorms/always-on-genie/DRAFT.md | 6 +++ .../brainstorms/cross-agent-delegate/DRAFT.md | 6 ++- .genie/brainstorms/genie-spend/DRAFT.md | 24 ++++++++- .../genie-spend-calibration-20260710.md | 51 +++++++++++++++++++ .../MORNING-BRIEF-20260710.md | 51 +++++++++++++++++++ .genie/brainstorms/skill-absorbs/DRAFT.md | 1 + 7 files changed, 148 insertions(+), 13 deletions(-) create mode 100644 .genie/brainstorms/genie-spend/genie-spend-calibration-20260710.md create mode 100644 .genie/brainstorms/genie-token-efficiency-program/MORNING-BRIEF-20260710.md diff --git a/.genie/INDEX.md b/.genie/INDEX.md index 4e96376de..50a697a19 100644 --- a/.genie/INDEX.md +++ b/.genie/INDEX.md @@ -17,22 +17,24 @@ - [WISH: hook-injection-hardening](wishes/hook-injection-hardening/WISH.md) — BLOCKED-clearing safety edit: `execFileSync` at 3 hook sites (audit-context, freshness×2) + hostile-filename regression tests + `core.bare` probe removal; flips panel verdict BLOCKED→FIX-FIRST — SHIPPED → PR #2536 (wish/hook-injection-hardening→main), G1+G2+whole-wish reviews SHIP, 729 pass/0 fail (2026-07-09) -- [WISH: v5-completion](wishes/v5-completion/WISH.md) — CLAUDE.md-for-v5 rewrite ∥ Codex launch target + Hermes decision ∥ distribution 5.x (drafted 2026-07-02) -- [WISH: dispatch-inproc-default](wishes/dispatch-inproc-default/WISH.md) — HIGH discovered defect: v5 hooks fall open by default (daemon deleted in demolition); re-arm branch-guard + unblock omni approvals (drafted 2026-07-02) -- [WISH: omni-approval-ux](wishes/omni-approval-ux/WISH.md) — correlated approval identity, reaction approve/deny, anti-spam feedback; grounded in the 2026-07-03 live WhatsApp QA (drafted 2026-07-03) -- [WISH: omni-runner-port](wishes/omni-runner-port/WISH.md) — umbrella Group 5: approval-capture spike, global genie.db queue, genie omni serve, inbound one-shot (drafted 2026-07-02) -- [WISH: warp-integration](wishes/warp-integration/WISH.md) — umbrella Group 3: genie init, Warp launch-config emitter, genie launch, /work multi-session opt-in (drafted 2026-07-02) +- [WISH: v5-completion](wishes/v5-completion/WISH.md) — **DONE** (all 3 groups SHIP-reviewed 2026-07-02; reconcile scope w/ cross-agent-delegate + agent-sync per HANDOFF §4): CLAUDE.md-for-v5 rewrite ∥ Codex launch target + Hermes decision ∥ distribution 5.x +- [WISH: dispatch-inproc-default](wishes/dispatch-inproc-default/WISH.md) — **DONE** (both groups SHIP-reviewed 2026-07-02; branch-guard re-armed, regression-gated — hooks demonstrably armed tonight): v5 hooks no longer fall open by default; re-arm branch-guard + unblock omni approvals +- [WISH: omni-approval-ux](wishes/omni-approval-ux/WISH.md) — **DONE** (G1–G3 SHIP-reviewed, 2 HIGHs found+fixed; one Felipe-approved live round-trip pending): correlated approval identity, reaction approve/deny, anti-spam feedback +- [WISH: omni-runner-port](wishes/omni-runner-port/WISH.md) — **DONE** (all 5 groups SHIP-reviewed 2026-07-02; live WhatsApp QA blocked-with-runbook, needs Felipe Omni instance): approval-capture spike, global genie.db queue, genie omni serve, inbound one-shot +- [WISH: warp-integration](wishes/warp-integration/WISH.md) — **DONE** (all 4 groups SHIP-reviewed 2026-07-02; pane-render checklist awaiting Felipe): genie init, Warp launch-config emitter, genie launch, /work multi-session opt-in ## Poured -- [agent-sync](wishes/agent-sync/DESIGN.md) · [WISH](wishes/agent-sync/WISH.md) — `genie update` = único verbo canônico que converge TODOS os coding agents detectados (Claude Code skills+council stamp, Codex Agent-Skills em .curated/, Hermes symlink+enable); engine interna manifest+adopt-with-backup+orphan-removal; hook CC vira trigger; mata scripts/smart-install.js divergente — EXECUTADO 2026-07-10: G1 `033b423f` (fix loop matou HIGH data-loss), G2 `b85ff792` (+hardening), G3 `e871b0fb` (+M1 constantes), ride-along `d0d61211` (coerção string-args do /council, achado no live QA); final review SHIP, 806 pass/1 skip; resta: ritual live user-gated + decisão de PR-topology (3 commits plugin-resource-shipping na linhagem, conteúdo já em dev via #2540) +- [agent-sync](wishes/agent-sync/DESIGN.md) · [WISH](wishes/agent-sync/WISH.md) · [COORDINATION](wishes/agent-sync/COORDINATION.md) — **MERGED #2541 (dev, 2026-07-10 06:47Z; dev now v5.260710.5→.6)** — `genie update` = único verbo canônico que converge TODOS os coding agents detectados (Claude Code skills+council stamp, Codex Agent-Skills em .curated/, Hermes symlink+enable); engine interna manifest+adopt-with-backup+orphan-removal; hook CC vira trigger; mata scripts/smart-install.js divergente. Na dev: G1 `346c02e1`, G2 `ff497ae4`, G3 `642c9f27` (doctor freshness + manifest-verified uninstall), final-review doc `46cc3fb7`; a coerção string-args do /council entrou separada como `ec68cd8f`. PR-topology resolvida no rebase (os 3 commits plugin-resource-shipping saíram; conteúdo já estava em dev via #2540). **Resta:** ritual live user-gated (`genie update` ×2 pós-stable → verificar os 7 role agents como subagent types) — ver [MORNING-BRIEF](brainstorms/genie-token-efficiency-program/MORNING-BRIEF-20260710.md). - [council-workflow](brainstorms/council-workflow/DESIGN.md) · [WISH](wishes/council-workflow/WISH.md) — /council vira saved workflow nativo (deliberation + audit), lens library 13 (7 persona skills renomeadas por lane + 6 cards), distribuição via install-stamp em ~/.claude/workflows; implementa a disposição council do skill-absorbs G4 — EXECUTADO 2026-07-10: G1 `0b222e17`, G2 `12270b21` (1 fix loop: unwrap p/ body-style do runtime), G3 `22a7ed50`, G4 `2af7ebd9`, G5-eng `6494253d` — todos SHIP; QA vivo USER-GATED pós-release (ritual Felipe: merge→release→plugin update→rodar /council "revisar tudo"; g5-gate para na cauda qa/ até lá); final execution review após o QA - [plugin-resource-shipping](brainstorms/plugin-resource-shipping/DRAFT.md) · [WISH](wishes/plugin-resource-shipping/WISH.md) — in-skill template via ${CLAUDE_SKILL_DIR}, probe-guarded lint refs, resource-shipping lint, fresh-install CI smoke — EXECUTED 2026-07-10 (first wish under the routing matrix: engineers opus·high, gate fable·high; 0 fix loops all groups; gate SHIP after branch surgery — concurrent session had switched the shared checkout): G1 ecbb67fc, G3 203c97df, G2 bbd6439e → PR #2540 merged, CI 10/10; QA pending: live installed-plugin scaffold on next release -- [routing-matrix](brainstorms/routing-matrix/DRAFT.md) · [WISH](wishes/routing-matrix/WISH.md) — pinned role agents (Fable gates / Opus ladder / Haiku scouts), stage pins + pane flags, complexity columns + lint, escalation caps — executed; execution review SHIP, final gate 719 pass / 1 skip / 0 fail; live LangWatch pin QA pending (2026-07-09) +- [routing-matrix](brainstorms/routing-matrix/DRAFT.md) · [WISH](wishes/routing-matrix/WISH.md) — pinned role agents (Fable gates / Opus ladder / Haiku scouts), stage pins + pane flags, complexity columns + lint, escalation caps — executed; execution review SHIP, final gate 719 pass / 1 skip / 0 fail. **Day-1 live pin QA recorded 2026-07-10 ([qa/](wishes/routing-matrix/qa/routing-pin-qa-20260710.md)) — inconclusive by delivery gap (pins not yet subagent types); re-test post-stable + `genie update` ×2** (2026-07-09) +- [skills-fable5-revamp](wishes/skills-fable5-revamp/WISH.md) · [execution review](wishes/skills-fable5-revamp/reports/execution-review-20260710.md) — Fable-5 revamp of all 36 genie/omni prompt surfaces (118→0 dead-CLI refs, ≥40% line cut, byte-0 frontmatter) + G8 v4-legacy cleanup engine — **merged PR #2518; execution review SHIP 2026-07-10 (HIGH 0 / MEDIUM 0 / 3 LOW: omni-side confirm, stale diff-count, superseded-note for `learn`/`council`)** +- [rolling-pr-auth-hardening](wishes/rolling-pr-auth-hardening/WISH.md) — fail-fast on dead/absent PAT + read/create token split in rolling-pr.yml — **DONE (already implemented on dev+main: `c4fdb32b` + `422caaa2`; found by council freshness probe 2026-07-10; PAT mint remains Felipe's open action)** - [genie-token-efficiency-program](brainstorms/genie-token-efficiency-program/DESIGN.md) — umbrella (WRS 100, independent review SHIP; 9 seed groups): genie → control-plane thesis; routing matrix (Fable gates, Opus ladder, no Sonnet, $17.9k/21d baseline); 17-skill dispositions; always-on identity; cross-agent delegate (Codex/Hermes companion sessions); genie spend; brainstorm domain-map upgrade — crystallized 2026-07-09 -- [genie-mcp](brainstorms/genie-mcp/DESIGN.md) · [WISH](wishes/genie-mcp/WISH.md) — genie MCP server: Warp/Claude Code/Codex consume genie.db state read-only; stdio, auto-registered; spike-first (2026-07-03) +- [genie-mcp](brainstorms/genie-mcp/DESIGN.md) · [WISH](wishes/genie-mcp/WISH.md) — **DONE** (G1–G3 SHIP-reviewed 2026-07-03; Warp live-UI QA awaits Felipe): genie MCP server — Warp/Claude Code/Codex consume genie.db state read-only; stdio, auto-registered; spike-first - [Genie v5 — lightweight body](brainstorms/genie-v5-lightweight-body/DESIGN.md) — skills+files+stock Warp replace the v4 harness; CC/Codex/Hermes targets; omni keeps one runner; crystallized 2026-07-01, umbrella-scale (8 seed groups) - [WISH: v5-foundation](wishes/v5-foundation/WISH.md) — umbrella Groups 1+2: genie.db engine + CLI + core skills — DONE 2026-07-02, all groups SHIP - [WISH: v5-demolition](wishes/v5-demolition/WISH.md) — umbrella Group 6 pulled forward (D8): harness deletion, bare-name cutover, v4 branch, PR #2499 — DONE 2026-07-02 - - [WISH: v5-housekeeping](wishes/v5-housekeeping/WISH.md) — true-lightweight tree cleanup (root files, .genie v4 history, metrics bot) + README replan (drafted 2026-07-02) -- [WISH: taxonomy-rehoming](wishes/taxonomy-rehoming/WISH.md) — plans migrate to genie's own taxonomy (`.genie/wishes|brainstorms`); path claims made coherent; user skills speak genie (2026-07-02) + - [WISH: v5-housekeeping](wishes/v5-housekeeping/WISH.md) — **DONE** (all 3 groups SHIP-reviewed 2026-07-02): true-lightweight tree cleanup (root files, .genie v4 history, metrics bot) + README replan +- [WISH: taxonomy-rehoming](wishes/taxonomy-rehoming/WISH.md) — **DONE** (both groups SHIP-reviewed 2026-07-02; one user action pending: PATH line in ~/.zshrc): plans migrate to genie's own taxonomy (`.genie/wishes|brainstorms`); path claims made coherent; user skills speak genie diff --git a/.genie/brainstorms/always-on-genie/DRAFT.md b/.genie/brainstorms/always-on-genie/DRAFT.md index 0924caf13..adc50b736 100644 --- a/.genie/brainstorms/always-on-genie/DRAFT.md +++ b/.genie/brainstorms/always-on-genie/DRAFT.md @@ -9,6 +9,12 @@ - 3 of 5 plugin hook scripts are silently DEAD on v5 shapes (session-context, validate-wish, validate-completion; one says "Run /forge"). rules/genie-orchestration.md (20 lines) is the one v5-correct plugin doc. - Native worktrees: `isolation:"worktree"` per-agent, `--worktree`, EnterWorktree, `.worktreeinclude`, worktree.baseRef; project plugins auto-load in worktrees (v2.1.200). +## EVIDENCE for G10 worktree isolation (2026-07-09 / 2026-07-10) + +The worktree-isolation policy now has hard-won proof, not just a thesis: +- **Shared-checkout collision happened TWICE.** (a) A concurrent session switched the shared checkout `~/workspace/genie` to another wish's branch mid-workflow — three commits landed on the wrong branch and cost a full branch-surgery recovery (isolated worktree off origin/dev + `git mv`/`cherry-pick --no-commit` fold + `git update-ref`). (b) A second session narrowly avoided grabbing another session's live-uncommitted files out of the same working tree. Both are the exact failure mode G10 exists to remove: **never execute wish work in the shared checkout; always `git worktree add` an isolated tree per wish.** Tonight's overnight run enforced this by hand (every worker + this docs finisher ran in a separate worktree off `origin/dev`, merges serialized) — that manual discipline is what G10 should make automatic. +- **Council 3-of-3 takeover de-escalation (2026-07-10).** The first real `/council` deliberation reviewed an "agent-sync takeover" clause that would have let one session seize the shared checkout, and **de-escalated it 3-of-3 to a read-only watch** — i.e. the humans-in-the-loop lenses independently converged on *observe, never take over the shared tree*. This is corroborating design pressure for G10: isolation is the mechanism that makes "read-only watch" the only needed posture, because no session ever needs write access to another's tree. + ## DECIDED (umbrella D9, D10, D14) - Two-layer always-on: (1) thin rules identity ≤40 lines (who genie is, lifecycle routing, control-plane invariants); (2) SessionStart additionalContext via in-process dispatch — live wishes+status, ready groups from genie.db, first-run wizard branch. Deep playbooks stay on-demand. - Hook contract: schema-version manifest, v4/v5 fixtures, per-hook CI smoke, unknown schema ⇒ fail closed with message; silent noop prohibited; dead scripts rewritten in-process or deleted; /forge purged. diff --git a/.genie/brainstorms/cross-agent-delegate/DRAFT.md b/.genie/brainstorms/cross-agent-delegate/DRAFT.md index 1fc78d163..c2c213ba6 100644 --- a/.genie/brainstorms/cross-agent-delegate/DRAFT.md +++ b/.genie/brainstorms/cross-agent-delegate/DRAFT.md @@ -20,8 +20,12 @@ ## Degradation policy (learned live, 2026-07-09) First real invocation of the auto plan-gate counter-read hit cegonha unreachable (network path down; host fine an hour earlier). Decided behavior to encode in the delegate skill: **counter-read fails OPEN** — the gate proceeds on the internal reviewer alone, logs "counter-read unavailable (host unreachable)" in the review record, and the next gate retries. Never block a plan gate on external-agent availability; never silently pretend the counter-read happened. +## RECONCILE (2026-07-10 — agent-sync shipped, PR #2541 merged to dev) + +**The Codex *plumbing* is now built and owned by agent-sync — this track consumes it, does not rebuild it.** `genie update` (and `genie install`) now fan the canonical `~/.genie/plugins/genie` source into every DETECTED coding agent on every invocation, including a **Codex adapter that ships genie skills to `~/.codex/skills/.curated/`** (managed manifest + adopt-with-backup + orphan removal). So the "one delegate skill + per-agent adapter references" decision is unchanged, but its Codex prerequisite — *getting genie's skills onto the Codex side* — is solved for free: the delegate skill can assume the curated genie skills are present on any machine that has run `genie update`. What remains this track's own work is the **delegation runtime** (companion `codex exec` / `hermes chat -q` sessions, JSON hand-back, background+poll, the `wish-` session persistence on the genie.db row), not the skill distribution. The Codex-install/auth GAP below narrows accordingly to *auth + which roles first*, since "are genie's skills installed on Codex" is answered by agent-sync. + ## GAPS -- [ ] Codex reality on your machines: installed? auth method (API key vs OAuth)? Which lifecycle roles do you want Codex for first — engineer on suitable groups, PR review dissent, or both? +- [ ] Codex reality on your machines: installed? auth method (API key vs OAuth)? Which lifecycle roles do you want Codex for first — engineer on suitable groups, PR review dissent, or both? (Skill distribution to `~/.codex/skills/.curated/` is now handled by agent-sync — see RECONCILE above; this GAP is now auth + role-scoping only.) - [ ] Hermes canonical vs alias: confirm `-z` is a cegonha alias (helper works today; adapter should document canonical `hermes chat -q` + fallback). - [ ] Budget/limits for external agents: max concurrent companion sessions? Hermes host load limits (cegonha is shared infra — benchmarks run there)? - [ ] Session lifecycle: when a wish ships, are its companion sessions retired (hermes sessions delete is gated on shared infra) or kept as history? diff --git a/.genie/brainstorms/genie-spend/DRAFT.md b/.genie/brainstorms/genie-spend/DRAFT.md index 3900737af..6b1245b79 100644 --- a/.genie/brainstorms/genie-spend/DRAFT.md +++ b/.genie/brainstorms/genie-spend/DRAFT.md @@ -6,7 +6,7 @@ - Baseline: $17,857 / 95.77M billable tokens / 11.79B cache-read tokens / 1097 traces in 21d (~1000 in last 4 days). Fable $13.8k · Opus $7.1k · Haiku $1.2k · Sonnet $0.5k (span-attributed shares). - Working analytics recipes (archived in parent DRAFT): tokens+cost by model (groupBy metadata.model), top sessions (groupBy metadata.thread_id), by effort (filter-loop on gen_ai.request.reasoning_effort — no effort groupBy exists), top-N traces (trace/search + client sort, scrollId pagination). - `metadata.labels` groupBy EXISTS but CC's OTel export emits no genie labels; thread_id = whole CC session (no per-subagent split); model lives on spans, not traces. -- LangWatch CLI (`npm i -g langwatch`) wraps the same endpoints, `--format json`, designed to be agent-driven. +- LangWatch CLI (`npm i -g langwatch`) wraps the same endpoints, `--format json`, designed to be agent-driven. **Access proven 2026-07-10: the existing OTLP-ingest key authenticates fine through the CLI; hand-rolled REST `POST /api/analytics` + `/api/trace/search` returned 403 (payload/route shape, not the key). Phase 1 should shell out to the `langwatch` CLI, NOT hand-roll REST.** Substrate + recipes archived in [genie-spend-calibration-20260710.md](genie-spend-calibration-20260710.md). - Hermes north star: **cost per group accepted without reopening** — measure decisions, not calls. ## 7-DAY BURN ANALYSIS (run 2026-07-09 — first real analysis pass) @@ -55,8 +55,28 @@ Totals: **$18,175 · 1,113 traces · 63 sessions.** Token buckets: prompt (fresh 3. Fewer tool calls per turn (batching, scripts over call-chains): every call re-reads full context — new guidance item for work/dispatch contract. 4. Stable prompt prefixes so cache writes stay linear (constraint on always-on-genie inject: deterministic, session-start-only). +## CALIBRATION — 2026-07-10 (day-1 pin-QA window, partial ~6.85h) — [full file](genie-spend-calibration-20260710.md) + +Measured via the `langwatch` CLI during the routing-pin day-1 QA. This window is a gate/review-heavy overnight dogfood — good for **shape/recipe** calibration, not a representative full day. Cross-ref [routing-matrix/qa/routing-pin-qa-20260710.md](../../wishes/routing-matrix/qa/routing-pin-qa-20260710.md). + +**$/day trend point (span-level `performance.total_cost`):** 07-08 full **$4,352** · 07-09 full **$1,230** (pins merged 21:51Z) · 07-10 00:00–06:51Z **$1,613** partial → **~$5,650/day run-rate** (upper-ish bound, not typical). The 21-day baseline averages ~$850/day, but recent dogfood days sit well above — the exact heavy-spend spike `genie spend` needs to surface. + +**$/model split (07-10, billable `cost_billed`):** Fable $956 (**57%**) · Opus $562 (34%) · Haiku $159 (9%) — Fable share rose on every measure this window (see the pin-QA file). + +**Per-effort cost-per-trace (SOLID — from trace search's per-trace `reasoning_effort` + `total_cost`):** this is the cleanest per-lane number available and a good basis for a `genie spend --by-effort` view / per-lane budget alerts. +| Effort | n | p50 | p90 | mean | +|---|---:|---:|---:|---:| +| xhigh | 37 | $5.39 | $26.23 | $9.74 | +| high | 27 | $5.00 | $20.81 | $7.83 | +| max | 20 | $18.76 | $43.56 | $23.85 | +| **all** | 84 | **$6.76** | **$32.33** | $12.49 | + +`max` is the expensive deep-reasoning/gate tier (~3–4× the p50 of xhigh/high). + +**Phase-1 query shapes that proved WORKABLE (all via the CLI, existing key):** cost-by-model, cost-by-day (per-day windows; each query returns the prior period free), token-volume-by-model, trace-count-by-model, top-sessions-by-cost (`--group-by metadata.thread_id`), effort histogram + per-effort cost percentiles. **BROKEN/blocked tonight:** (1) direct REST 403 → use the CLI; (2) **effort-filtered analytics** unavailable (`analytics query` has no `--filter`; underlying effort-filter silently broken) → **effort splits must be computed client-side from trace search**; (3) **per-model p50/p90 NOT derivable** — trace search returns no per-trace model (spans empty; model lives only at span level in analytics) → **parked**; per-effort percentiles are the usable substitute, or span-level export the CLI doesn't expose; (4) **cache-read + `--group-by metadata.model`** hits a known ClickHouse bug → avoid (query cache-read without the model groupBy). Net: a CLI-backed Phase-1 covering model/day/thread/effort splits is achievable now; per-model percentiles and effort-filtered analytics are the two gaps to design around. + ## GAPS -- [ ] Key/endpoint source for `genie spend`: read from CC settings env (OTEL_EXPORTER_OTLP_*), from genie config, or both with precedence? (Machine-portability: teammates' machines have the same settings?) +- [ ] Key/endpoint source for `genie spend`: read from CC settings env (OTEL_EXPORTER_OTLP_*), from genie config, or both with precedence? (Machine-portability: teammates' machines have the same settings?) — **2026-07-10 evidence: the OTLP-ingest key from `OTEL_EXPORTER_OTLP_HEADERS` in `~/.claude/settings.json` authenticates the CLI; endpoint `https://langwatch.khal.ai`.** - [ ] Cadence: on-demand only, or a scheduled snapshot (e.g. daily line into .genie/ or omni message)? You already run "live metrics" commits — integrate or keep separate? - [ ] Consumers: just you, or team/omni-channel reporting? - [ ] Should `genie doctor` warn when burn-rate exceeds a threshold (needs a threshold from you)? diff --git a/.genie/brainstorms/genie-spend/genie-spend-calibration-20260710.md b/.genie/brainstorms/genie-spend/genie-spend-calibration-20260710.md new file mode 100644 index 000000000..9b93e3f59 --- /dev/null +++ b/.genie/brainstorms/genie-spend/genie-spend-calibration-20260710.md @@ -0,0 +1,51 @@ +# `genie spend` — Calibration Evidence (2026-07-10) + +**For:** the genie-spend DRAFT (Phase-1 query design). +**Source:** LangWatch `https://langwatch.khal.ai` via `langwatch` CLI. Measured 2026-07-10 ~06:51Z. +**Scope note:** 07-10 is a **partial ~6.85h overnight window** (00:00–06:51Z) from a gate/review-heavy multi-agent dogfood run — good for *shape/recipe* calibration, not representative of a normal full day's absolute spend. + +--- + +## $/day trend point + +Span-level `performance.total_cost`, per UTC calendar day: + +| Day | Window | Total cost | Note | +|---|---|---:|---| +| 07-08 | full | **$4,352** | full pre-pin day | +| 07-09 | full | **$1,230** | pre-pin (pins merged 21:51Z) | +| 07-10 | 00:00–06:51Z (~6.85h) | **$1,613** partial → **~$5,650/day run-rate** | overnight dogfood; run-rate is an upper-ish bound, not typical | + +Context: the 21-day baseline averages ~$850/day ($17,857 / 21d), but recent dogfood days (07-08 at $4.3k, the 07-10 overnight run-rate at ~$5.6k) sit well above that average — the spend curve is in a heavy-dogfood spike, which is exactly the regime `genie spend` needs to surface. + +## $/model split (07-10, billable `cost_billed`) + +| Model | Billable cost | Share | +|---|---:|---:| +| Fable | $956 | **57%** | +| Opus | $562 | 34% | +| Haiku | $159 | 9% | +| **Total** | **$1,677** | | + +(For directional trend vs 07-09 and token/trace shares, see `routing-pin-qa-20260710.md` — Fable share rose on every measure this window.) + +## Per-lane cost calibration + +**Per-effort cost-per-trace (solid — derived from trace search, per-trace effort + cost):** + +| Effort lane | 07-10 n | p50 | p90 | mean | +|---|---:|---:|---:|---:| +| xhigh | 37 | $5.39 | $26.23 | $9.74 | +| high | 27 | $5.00 | $20.81 | $7.83 | +| max | 20 | $18.76 | $43.56 | $23.85 | +| **all** | 84 | **$6.76** | **$32.33** | $12.49 | + +`max` is the expensive deep-reasoning/gate tier (~3–4× the p50 of xhigh/high). This is the cleanest per-lane number available and is a good basis for a `genie spend --by-effort` view or per-lane budget alerts. + +**Per-model mean cost-per-trace (rough — use with caution):** Fable ≈ $12.8, Opus ≈ $13.4 per model-touch (07-10). These divide span-level cost by *overlapping* model-touch counts (a multi-model trace counts under each model), so they are means, not medians, and denominators are inflated. **Per-model p50/p90 is NOT derivable via the CLI** — trace search returns no per-trace model (spans empty; model lives only at span level inside analytics). If `genie spend` wants per-model percentiles, it needs span-level export, which the CLI does not expose. + +--- + +## Which Phase-1 queries proved workable vs broken tonight + +**Workable — all through the `langwatch` CLI (`analytics query` + `trace search`), no bespoke REST layer needed, and the existing OTLP-ingest key authenticates fine:** cost-by-model, cost-by-day (via per-day windows; each query also returns the prior period free), token-volume-by-model (`performance.total_tokens`), trace-count-by-model (`trace-count`/cardinality), top-sessions-by-cost (`--group-by metadata.thread_id`), and the effort histogram with per-effort cost percentiles (from trace search's per-trace `reasoning_effort` + `total_cost`). Any `genie spend` Phase-1 that maps to those six shapes can be built directly on the CLI. **Broken or blocked tonight:** (1) direct REST `POST /api/analytics` and `/api/trace/search` returned **403** — the CLI wraps the same endpoints and works, so Phase-1 should shell out to the CLI rather than hand-roll REST; (2) **effort-filtered analytics** is unavailable — the CLI `analytics query` has no `--filter` flag and the underlying effort-filter is reported silently broken, so any effort-scoped spend number must be computed client-side from trace search; (3) **per-model cost distributions (p50/p90)** are not derivable (no per-trace model attribution); (4) **cache-read-by-model** hits a known ClickHouse bug when combined with `--group-by metadata.model` and must be avoided (query cache-read without the model groupBy). Net: a CLI-backed Phase-1 covering model/day/thread/effort splits is achievable now; per-model percentiles and effort-filtered analytics are the two gaps to design around. diff --git a/.genie/brainstorms/genie-token-efficiency-program/MORNING-BRIEF-20260710.md b/.genie/brainstorms/genie-token-efficiency-program/MORNING-BRIEF-20260710.md new file mode 100644 index 000000000..40f519e98 --- /dev/null +++ b/.genie/brainstorms/genie-token-efficiency-program/MORNING-BRIEF-20260710.md @@ -0,0 +1,51 @@ +# MORNING BRIEF — 2026-07-10 (Felipe's decision menu) + +One page, in priority order. This is the living document; [HANDOFF-20260710.md](HANDOFF-20260710.md) is the historical record of the night. Each item says what to do and what it unblocks. + +--- + +### 1. Merge promotion PR #2542 → stable release *(human-via-UI, §19)* +[PR #2542](https://github.com/automagik-dev/genie/pull/2542) `release: agent-sync + /council native workflow` is OPEN, base `main` ← head `dev`. **Verified: it already carries the resource-shipping LOW follow-ups (`65759f53`) and dev tip `9b15140f` — its head IS the live dev branch (v5.260710.6).** Only the *title label* says "5.260710.5" (stale from when it was created before #2543 merged); the diff is current. No rolling-PR refresh needed for correctness — merge it as-is via the GitHub UI. This mints the signed CalVer stable release that carries agent-sync + /council. + +### 2. After stable: `genie update` ×2 (one-time self-sync bootstrap) +Run `genie update` **twice** on the dogfood host — once ever. The release that introduces self-sync needs the first run to install the new updater and the second to let it self-sync (wish Decision 9). Then verify the delivery actually happened: +- **7 role agents appear as subagent types** in a fresh Claude Code session (`engineer-trivial/standard/complex`, `fixer`, `reviewer`, `final-gate`, `scout`). This is the exact gap that made last night's pin QA inconclusive. +- **string-args fix is stamped** into the local council workflow: `grep -n "typeof" ~/.claude/workflows/council.js` (or grep for the string-coercion the fix added) confirms `ec68cd8f` reached the stamped script. + +### 3. Council live-QA ritual: `/council "revisar tudo"` +Run it against the released, self-synced `council.js`. This exercises the string-args path end-to-end on the shipped artifact and **unblocks the council-workflow final execution review** (the g5-gate parks in `.genie/wishes/council-workflow/qa/` until then). Defect story + secondary skill-text-vs-script mismatch are pre-written in [overnight-observations-20260710.md](../../wishes/council-workflow/qa/overnight-observations-20260710.md). + +### 4. Routing-pin re-test +Once #2 confirms the 7 agents are real subagent types, re-pull the exact LangWatch comparison. Method + working `langwatch` CLI recipes are in [routing-pin-qa-20260710.md](../../wishes/routing-matrix/qa/routing-pin-qa-20260710.md). Expect Fable token share to fall toward gate-only (~11%, the level the one properly-pinned wish hit) and Opus engineering share to rise. Last night was inconclusive *only* because pins weren't mechanically enforced — not because the design is wrong. + +### 5. genie-spend — close the 4 open gaps (closest-to-wishable track) +Tonight's calibration ([genie-spend DRAFT](../genie-spend/DRAFT.md) CALIBRATION section) answers most of these; your ruling turns them into a `/wish`: +- **Key/endpoint source** → recommend: read the OTLP-ingest key from `OTEL_EXPORTER_OTLP_HEADERS` in `~/.claude/settings.json`, endpoint `https://langwatch.khal.ai`. **Proven to authenticate the `langwatch` CLI last night.** +- **Substrate** → recommend: **shell out to the `langwatch` CLI, not hand-rolled REST** (direct REST returned 403; the CLI wraps the same endpoints and works). +- **Effort splits** → recommend: compute **client-side from trace search** (`analytics query` has no `--filter`; the effort-filter is silently broken). +- **Percentiles** → recommend: ship **per-effort p50/p90** (solid: xhigh $5.39/$26.23, high $5.00/$20.81, max $18.76/$43.56); **per-model p50/p90 is parked** — not derivable via the CLI (no per-trace model). Also decide **cadence** (on-demand vs a daily line into `.genie/`), **consumers** (just you vs team/omni), and whether **`genie doctor`** warns above a burn threshold (needs a number from you). + +### 6. Mint the rolling-pr PAT *(only open action on rolling-pr-auth-hardening)* +The workflow hardening already shipped (`c4fdb32b` + `422caaa2`, on dev AND main). The one remaining human action: repo Settings → Secrets → `RELEASE_PLEASE_TOKEN`, a PAT with `contents:read` + `pull-requests:write`. Until then the hourly rolling-PR run fails fast with the actionable error (by design). + +### 7. brainstorm-domain-map — oracle-ownership ruling +Decide **who owns irreducibly subjective truth** (the oracle-ownership boundary: machine / model / human oracle classes). This is the one open blocker; ruling it **unblocks the intent-to-wish-compiler pour** (WRS 92, ready to split into 4 child wishes once domain-map + control-plane-contract converge). + +### 8. dream-replatform — substrate choice +Your call: local cron vs cloud agents vs hybrid for the scheduler. Cron stays *trigger, never authority*; omni approval gates. This is the gate on the dream track (~50%). + +### 9. LOW leftovers (skills-fable5-revamp execution review, all optional) +- **omni-side confirm** — have a reviewer with the `automagik-dev/omni` checkout re-verify the omni G5/G6 numbers (1,329→631); last night's SHIP for the omni half rests on `verification.md` attestation (that repo isn't in the genie checkout). +- **superseded note** — add a one-line "superseded by later wishes" marker where `verification.md` lists `learn`/`council` (both shipped correctly then removed by other wishes) to prevent future diff confusion. + +--- + +### What happened tonight (2026-07-10) + +| Thread | Outcome | +|---|---| +| **agent-sync** | Merged to dev as **PR #2541** (by the owning session; final review SHIP `46cc3fb7`). Dev → v5.260710.5. | +| **resource-shipping LOW follow-ups** | Merged as **#2543** (`65759f53`) — temp-dir cleanup, precise same-line guard, replay-safe sweep. | +| **/council** | First real dogfood (3 lenses, 2 rounds); live QA caught a string-args defect, fixed + merged as **`ec68cd8f`**. | +| **routing-pin QA** | Pulled — Fable share *rose* (inconclusive: pins weren't yet subagent types). Re-test after #1+#2. | +| **rolling-pr-auth-hardening** | Found **already implemented** on dev+main (`c4fdb32b`+`422caaa2`) — no work; only the PAT mint (#6) is left. | diff --git a/.genie/brainstorms/skill-absorbs/DRAFT.md b/.genie/brainstorms/skill-absorbs/DRAFT.md index f206c31b6..756d30970 100644 --- a/.genie/brainstorms/skill-absorbs/DRAFT.md +++ b/.genie/brainstorms/skill-absorbs/DRAFT.md @@ -13,6 +13,7 @@ - pm → ABSORB: triage guidance → work reference; autopilot → dream successor (with human-gate policy). - council → lens LIBRARY (consumed by review panels + brainstorm domain-experts) + THIN /council route preserved (strategy pre-artifact, dissent preservation, appeal court for reviewer↔gate disagreements). - **2026-07-09 UPDATE — implemented via [council-workflow](../council-workflow/DESIGN.md) (poured):** /council = native saved workflow (deliberation + audit modes); lens library = 7 lane skills (renamed personas) + 6 deliberation cards; "thin route" superseded — the route IS the workflow command. GAP "/council name?" closed (name kept). Consumer rewiring (/review panels + /brainstorm domain-experts) moved INTO that wish's scope. + - **2026-07-10 — DONE.** council-workflow EXECUTED (G1–G5 all SHIP, merged to dev) and **dogfooded for the first time tonight** (a 3-lens / 2-round `/council` deliberation, plus the string-args fix `ec68cd8f` found in live QA). The council disposition of this umbrella is now delivered; only the council-workflow live-QA ritual + final execution review remain, tracked in that wish, not here. - report → REFACTOR: observability re-sourced to LangWatch + portable local markdown/JSON fallback; keep gh-issue intake. - refine → re-scoped to cross-LLM prompt adapter (OWN track: cross-agent-delegate). - learn → PARKED at .genie/attic/skills/learn/ (brain transfer pending; not this wish).