feat(blogger): live DataForSEO SEO engine + daily burn cadence

.gitignore

@@ -32,6 +32,8 @@ out
 routines/**/runs/
 routines/**/baselines/
 routines/**/state.json
+routines/**/opportunity-queue.json
+routines/**/burn-state.json
 
 # Database exports / PII dumps — never commit (5804)
 database_export/

routines/blogger/AGENT.md

@@ -0,0 +1,50 @@
+# blogger — agent mission
+
+Daily SEO-driven blog engine for knowcap.ai. Fires 07:00 Sun–Thu (open:agent, bypass).
+Read this fully, then execute. The full spec is [`ROUTINE.md`](./ROUTINE.md); the per-mode
+prompts + gates are [`_skills/write-blog-draft/SKILL.md`](../_skills/write-blog-draft/SKILL.md).
+
+## Mission
+
+Every day: pull live MENA SEO demand, refresh the ranked opportunity queue, surface a
+digest. On burn-cadence blog days: write ONE genuinely-useful, ICP-gated, Knowcap-angled
+post targeting the top fresh opportunity, open a draft PR. Never publish thin content to
+hit a quota.
+
+## Daily run (in order)
+
+1. **SEO pull** — `node routines/blogger/scripts/seo-pull.mjs` (creds in `~/.claude/secrets/blogger.md`).
+   Writes `opportunity-queue.json` (ranked, ICP-filtered, deduped vs shipped) + a digest.
+   If DataForSEO errors (auth/funds), log one line + STOP (don't fabricate keywords).
+2. **Digest** — surface the top picks to Hassan in the run window / Claude agents sidebar.
+3. **Burn-state** — read `burn-state.json`; decide if today is a blog-gen day (see ROUTINE.md
+   "Daily flow + burn cadence"). If not → done for today (insights only).
+4. **Blog day** — pick the persona's top fresh opportunity as `target_keyword`. Assemble inputs
+   (persona section, VISION, POSITIONING from the hub; shipped slugs; queue row's volume+comp).
+5. **Mode** — Knowcap MCP Demo-org → persona project → source with ≥3 human-confirmed memories?
+   → **case-study** (cite `source_knowcap_ids`, generate the verification panel via
+   `lib/gen-verification-panel.mjs`). Else → **SEO-grounded thesis** (answer the keyword's real
+   SERP / People-Also-Ask demand with a unique Knowcap angle).
+6. **Write** via write-blog-draft SKILL → run ALL gates (banned words, 1300–1600 words, keyword
+   in title+H2+≥3, slug unique, exactly 5 FAQ, frontmatter). Fail → regenerate/skip, never ship thin.
+7. **Output** — draft → `docs/content-pipeline/drafts/<slug>.md`, open `[blog-draft]` PR to main.
+   Update `burn-state.json` (total_posts++, posts_this_week++).
+
+## Hard rules
+
+- **ICP only** — odoo-partners / mena-audit-firms / mena-agencies / regulated-verticals. Off-ICP keyword (even high-volume) → skip it.
+- **Knowcap angle mandatory** — every post ties the keyword to verified-facts / human-confirmation / audit-trail. If a keyword can't carry that angle honestly, skip it.
+- **Quality > quota** — no fresh opportunity clears the gates → skip the day. A missed burn post beats a thin one (Google scaled-content-abuse demotes the whole domain).
+- **Honesty** — case-study only on genuinely human-confirmed claims. SEO-grounded thesis must add real value, not keyword-stuff. Never claim a verification panel for unconfirmed claims.
+- **No duplicates** — dedup against last 20 shipped slugs AND in-flight drafts. The queue already flags `covered`.
+- **English posts**, MENA-targeted (mine EN + AR demand, publish EN).
+- **DataForSEO down → stop**, don't invent data. One-line error, never silent.
+
+## Files
+
+- `scripts/seo-pull.mjs` — the SEO engine (this is what makes it daily + live)
+- `opportunity-queue.json` — ranked queue (gitignored runtime state)
+- `burn-state.json` — cadence tracker (gitignored)
+- `state.json` — persona rotation cursor (gitignored)
+- `lib/gen-verification-panel.mjs` — data-driven Knowcap UI SVG (case-study posts)
+- `runs/<stamp>/` — per-run digest + draft + report (gitignored)

routines/blogger/ROUTINE.md

@@ -12,6 +12,24 @@ See [`triggers.yml`](./triggers.yml) — cron `0 6 * * MON` (Monday 06:00 UTC =
 
 Calls [`_skills/write-blog-draft/SKILL.md`](../_skills/write-blog-draft/SKILL.md) in `thesis` / `case-study` / `comparison` mode (routine picks).
 
+## Daily flow + burn cadence (SEO engine)
+
+This routine fires DAILY (07:00 Sun–Thu). Every run:
+
+1. **Pull SEO insights** — `node routines/blogger/scripts/seo-pull.mjs` → refresh `opportunity-queue.json` + digest (every day, ~$0.1–0.5).
+2. **Surface the digest** to Hassan (the open:agent run window / Claude agents sidebar — see AGENT.md).
+3. **Decide if today is a blog-gen day** from `routines/blogger/burn-state.json`:
+   - **burn phase**: generate a post on 3 days/week (Sun/Tue/Thu) until `total_posts >= 24` (~8 weeks), then flip to steady.
+   - **steady phase**: generate 1 post/week (Sun).
+   - Not a blog-gen day → stop after the digest (insights only, no post).
+4. On a blog-gen day → mode selection + write, using the queue's top fresh opportunity as `target_keyword`.
+
+`burn-state.json` (gitignored runtime state):
+```json
+{ "phase": "burn", "started": "2026-06-15", "total_posts": 0, "week_start": "2026-06-15", "posts_this_week": 0, "blog_days": ["SUN","TUE","THU"] }
+```
+After each shipped draft PR: `total_posts++`, `posts_this_week++`. Reset `posts_this_week` on week rollover. Flip `phase→steady` + `blog_days→["SUN"]` when `total_posts >= 24`. If the file is absent, seed burn from today. **Quality bar:** never publish to hit a quota — if no fresh opportunity clears the gates, skip the day (a missed burn post beats a thin one).
+
 ## Mode selection (runtime)
 
 ```
@@ -59,8 +77,8 @@ Each run reads `cursor`, picks `personas[cursor]`, then advances `cursor = (curs
 2. **Read `../claude-knowcap/knowledge/people/PRODUCT-PERSONAS.md`** → pick the persona's section (study segment names map to slugs: "Odoo implementation partners" → `odoo-partners`, "Audit / accounting firms" → `mena-audit-firms`, etc.)
 3. **Read `../claude-knowcap/knowledge/strategies/VISION.md`** → voice + anti-positioning
 4. **Read `../claude-knowcap/knowledge/strategies/POSITIONING.md`** → three sentences + anti-positioning
-5. **Read most recent `../claude-knowcap/knowledge/topics/research/audits/SEO-AUDIT-*.md`** → top-3 keyword opportunities (currently: scan for keyword candidates manually; eventually: parse a `keyword_opportunities_by_persona` table)
-6. **Query Google Trends via pytrends** (urllib3<2.0 required) → validate 5-year MENA interest, pick highest recent growth
+5. **Run `node routines/blogger/scripts/seo-pull.mjs`** (live SEO engine, replaces the old static SEO-audit scan AND the dead Google Trends step). Pulls DataForSEO Google-Ads keyword demand for MENA (KSA + Egypt + UAE) in EN + AR, expands persona seeds into real related keywords with **search volume + competition**, filters to Knowcap ICP intent, ranks by `volume × competition-weight`, dedups against shipped posts, and writes `routines/blogger/opportunity-queue.json` + a digest. **`target_keyword` = the persona's top fresh (uncovered) opportunity** from the queue. Auth: DataForSEO creds in `~/.claude/secrets/blogger.md`.
+6. **(Google Trends removed.)** DataForSEO volume + competition from step 5 is the demand signal — Trends was near-zero for this B2B ICP. Record the chosen keyword's `search_volume` + `competition` in frontmatter.
 7. **Try `case-study` mode:**
    - Query Knowcap MCP `mcp__knowcap__list_sources` → Demo org, persona project
    - For each source, `mcp__knowcap__list_memories` filtered to source via `metadata.source_id`

routines/blogger/scripts/seo-pull.mjs

@@ -0,0 +1,193 @@
+#!/usr/bin/env node
+/**
+ * seo-pull.mjs — daily SEO opportunity pull for the Knowcap blogger routine.
+ *
+ * Replaces the static keyword-opportunities.md + the dead Google Trends step.
+ * Pulls LIVE MENA keyword demand from DataForSEO (Google Ads keywords_for_keywords:
+ * expands persona seed terms into related keywords with real search volume +
+ * competition), EN + AR across KSA / Egypt / UAE, filters to Knowcap's ICP intent,
+ * ranks, dedups against already-shipped posts, and writes a ranked opportunity queue
+ * + a human digest.
+ *
+ * NO browser. Auth = HTTP Basic from ~/.claude/secrets/blogger.md.
+ *
+ * Usage:
+ *   node seo-pull.mjs                         # all personas, all geos, EN+AR
+ *   node seo-pull.mjs --persona odoo-partners # one persona
+ *   node seo-pull.mjs --locations "Saudi Arabia" --langs en   # scope to conserve balance
+ *   node seo-pull.mjs --max 25                # cap queue size per persona
+ *
+ * Outputs (gitignored — runtime state):
+ *   routines/blogger/opportunity-queue.json
+ *   routines/blogger/runs/<stamp>/seo-digest.md   (caller passes --stamp; else prints)
+ */
+
+import fs from 'fs'
+import path from 'path'
+import os from 'os'
+import { fileURLToPath } from 'url'
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url))
+const BLOGGER_DIR = path.resolve(__dirname, '..')            // routines/blogger
+const REPO_ROOT = path.resolve(BLOGGER_DIR, '..', '..')      // knowcap-website
+const BLOG_DIR = path.join(REPO_ROOT, 'app', 'content', 'blog')
+const SECRETS = path.join(os.homedir(), '.claude', 'secrets', 'blogger.md')
+
+// ---- args ----
+const argv = process.argv.slice(2)
+const getArg = (name, def) => {
+  const i = argv.indexOf('--' + name)
+  return i >= 0 && argv[i + 1] ? argv[i + 1] : def
+}
+const ONLY_PERSONA = getArg('persona', null)
+const LOCATIONS = getArg('locations', 'Saudi Arabia,Egypt,United Arab Emirates').split(',').map(s => s.trim())
+const LANGS = getArg('langs', 'en,ar').split(',').map(s => s.trim())
+const MAX = parseInt(getArg('max', '25'), 10)
+const STAMP = getArg('stamp', null)
+
+// ---- creds ----
+function readCreds() {
+  const txt = fs.readFileSync(SECRETS, 'utf8')
+  const login = (txt.match(/^DATAFORSEO_LOGIN=(.+)$/m) || [])[1]?.trim()
+  const pass = (txt.match(/^DATAFORSEO_PASSWORD=(.+)$/m) || [])[1]?.trim()
+  if (!login || !pass || login.startsWith('<')) {
+    console.error('FATAL: DataForSEO creds missing in ' + SECRETS)
+    process.exit(2)
+  }
+  return 'Basic ' + Buffer.from(`${login}:${pass}`).toString('base64')
+}
+const AUTH = readCreds()
+
+// ---- personas: seed terms + ICP intent filter ----
+// Seeds are deliberately broad so keywords_for_keywords discovers the real demand.
+const PERSONAS = {
+  'odoo-partners': {
+    seeds_en: ['odoo implementation', 'odoo partner', 'erp implementation', 'odoo project'],
+    seeds_ar: ['تطبيق اودو', 'اودو'],
+  },
+  'mena-audit-firms': {
+    seeds_en: ['audit documentation', 'audit firm software', 'pdpl compliance', 'engagement letter'],
+    seeds_ar: ['تدقيق', 'محضر اجتماع'],
+  },
+  'mena-agencies': {
+    seeds_en: ['meeting notes software', 'client meeting notes', 'ai meeting notes'],
+    seeds_ar: ['محضر اجتماع', 'تفريغ اجتماع'],
+  },
+  'regulated-verticals': {
+    seeds_en: ['compliance meeting records', 'ai governance', 'audit trail software'],
+    seeds_ar: ['حوكمة الذكاء الاصطناعي', 'الامتثال'],
+  },
+}
+// Keep only keywords whose intent Knowcap can credibly own.
+const ICP_INTENT = /(meeting|minute|note|transcri|record|audit|complian|pdpl|govern|odoo|erp|scope|verif|attest|sign[- ]?off|محضر|اجتماع|تدقيق|اودو|امتثال|حوكمة|توثيق)/i
+const COMP_WEIGHT = { LOW: 1.0, MEDIUM: 0.6, HIGH: 0.35 }
+
+// ---- shipped-slug dedup ----
+function shippedSlugs() {
+  if (!fs.existsSync(BLOG_DIR)) return []
+  return fs.readdirSync(BLOG_DIR).filter(f => f.endsWith('.md')).map(f => f.replace(/\.md$/, ''))
+}
+const slugify = s => String(s).toLowerCase().replace(/[^a-z0-9؀-ۿ]+/g, '-').replace(/^-|-$/g, '')
+function isCovered(keyword, slugs) {
+  const ks = new Set(slugify(keyword).split('-').filter(w => w.length > 3))
+  if (!ks.size) return false
+  for (const slug of slugs) {
+    const ss = new Set(slug.split('-'))
+    let hit = 0
+    for (const w of ks) if (ss.has(w)) hit++
+    if (hit / ks.size >= 0.6) return true
+  }
+  return false
+}
+
+// ---- DataForSEO call ----
+async function keywordsForKeywords(seeds, locationName, languageCode) {
+  const body = [{ keywords: seeds, location_name: locationName, language_code: languageCode }]
+  const res = await fetch('https://api.dataforseo.com/v3/keywords_data/google_ads/keywords_for_keywords/live', {
+    method: 'POST',
+    headers: { Authorization: AUTH, 'Content-Type': 'application/json' },
+    body: JSON.stringify(body),
+  })
+  const json = await res.json()
+  if (json.status_code !== 20000) throw new Error(`API ${json.status_code}: ${json.status_message}`)
+  const task = (json.tasks || [])[0] || {}
+  if (task.status_code !== 20000) throw new Error(`task ${task.status_code}: ${task.status_message}`)
+  return { items: task.result || [], cost: json.cost || 0 }
+}
+
+// ---- main ----
+const personas = ONLY_PERSONA ? [ONLY_PERSONA] : Object.keys(PERSONAS)
+const slugs = shippedSlugs()
+const out = { generated_at: new Date().toISOString(), source: 'dataforseo:google_ads/keywords_for_keywords', locations: LOCATIONS, langs: LANGS, cost: 0, personas: {}, top: [] }
+const seen = new Set()
+
+for (const persona of personas) {
+  const cfg = PERSONAS[persona]
+  if (!cfg) { console.error('unknown persona: ' + persona); continue }
+  const bucket = []
+  for (const loc of LOCATIONS) {
+    for (const lang of LANGS) {
+      const seeds = lang === 'ar' ? cfg.seeds_ar : cfg.seeds_en
+      if (!seeds?.length) continue
+      try {
+        const { items, cost } = await keywordsForKeywords(seeds, loc, lang)
+        out.cost += cost
+        for (const it of items) {
+          const kw = it.keyword
+          const vol = it.search_volume || 0
+          if (!kw || vol <= 0) continue
+          if (!ICP_INTENT.test(kw)) continue
+          const comp = (it.competition || '').toUpperCase()
+          const score = Math.round(vol * (COMP_WEIGHT[comp] ?? 0.5))
+          bucket.push({ keyword: kw, search_volume: vol, competition: comp || null, cpc: it.cpc ?? null, score, location: loc, lang, covered: isCovered(kw, slugs) })
+        }
+        console.error(`  ${persona} | ${loc} | ${lang}: +${items.length} (cost $${cost})`)
+      } catch (e) {
+        console.error(`  ERROR ${persona} | ${loc} | ${lang}: ${e.message}`)
+      }
+    }
+  }
+  // dedupe by keyword (keep highest score), sort, cap
+  const byKw = new Map()
+  for (const r of bucket) {
+    const k = r.keyword.toLowerCase()
+    if (!byKw.has(k) || byKw.get(k).score < r.score) byKw.set(k, r)
+  }
+  const ranked = [...byKw.values()].sort((a, b) => b.score - a.score).slice(0, MAX)
+  out.personas[persona] = ranked
+  for (const r of ranked) {
+    if (r.covered) continue
+    const key = r.keyword.toLowerCase()
+    if (seen.has(key)) continue
+    seen.add(key)
+    out.top.push({ ...r, persona })
+  }
+}
+out.top.sort((a, b) => b.score - a.score)
+out.top = out.top.slice(0, 20)
+
+// ---- write queue ----
+const QUEUE = path.join(BLOGGER_DIR, 'opportunity-queue.json')
+fs.writeFileSync(QUEUE, JSON.stringify(out, null, 2))
+
+// ---- digest ----
+const lines = []
+lines.push(`# SEO opportunity digest — ${out.generated_at.slice(0, 10)}`)
+lines.push('')
+lines.push(`Source: DataForSEO (MENA ${LOCATIONS.join(', ')} · ${LANGS.join('/')}) · API cost this run: $${out.cost.toFixed(3)}`)
+lines.push(`Fresh opportunities (not yet covered by a shipped post): **${out.top.length}**`)
+lines.push('')
+lines.push('## Top picks (write next)')
+lines.push('')
+lines.push('| # | keyword | persona | vol | comp | score |')
+lines.push('|---|---|---|---|---|---|')
+out.top.slice(0, 10).forEach((r, i) => lines.push(`| ${i + 1} | ${r.keyword} | ${r.persona} | ${r.search_volume} | ${r.competition || '-'} | ${r.score} |`))
+const digest = lines.join('\n')
+
+if (STAMP) {
+  const runDir = path.join(BLOGGER_DIR, 'runs', STAMP)
+  fs.mkdirSync(runDir, { recursive: true })
+  fs.writeFileSync(path.join(runDir, 'seo-digest.md'), digest)
+}
+console.log(digest)
+console.error(`\nwrote ${QUEUE} — ${out.top.length} fresh opportunities, total API cost $${out.cost.toFixed(3)}`)