diff --git a/.gitignore b/.gitignore index 8826227..2e27755 100644 --- a/.gitignore +++ b/.gitignore @@ -32,6 +32,8 @@ out routines/**/runs/ routines/**/baselines/ routines/**/state.json +routines/**/opportunity-queue.json +routines/**/burn-state.json # Database exports / PII dumps — never commit (5804) database_export/ diff --git a/routines/blogger/AGENT.md b/routines/blogger/AGENT.md new file mode 100644 index 0000000..cf9bc4e --- /dev/null +++ b/routines/blogger/AGENT.md @@ -0,0 +1,50 @@ +# blogger — agent mission + +Daily SEO-driven blog engine for knowcap.ai. Fires 07:00 Sun–Thu (open:agent, bypass). +Read this fully, then execute. The full spec is [`ROUTINE.md`](./ROUTINE.md); the per-mode +prompts + gates are [`_skills/write-blog-draft/SKILL.md`](../_skills/write-blog-draft/SKILL.md). + +## Mission + +Every day: pull live MENA SEO demand, refresh the ranked opportunity queue, surface a +digest. On burn-cadence blog days: write ONE genuinely-useful, ICP-gated, Knowcap-angled +post targeting the top fresh opportunity, open a draft PR. Never publish thin content to +hit a quota. + +## Daily run (in order) + +1. **SEO pull** — `node routines/blogger/scripts/seo-pull.mjs` (creds in `~/.claude/secrets/blogger.md`). + Writes `opportunity-queue.json` (ranked, ICP-filtered, deduped vs shipped) + a digest. + If DataForSEO errors (auth/funds), log one line + STOP (don't fabricate keywords). +2. **Digest** — surface the top picks to Hassan in the run window / Claude agents sidebar. +3. **Burn-state** — read `burn-state.json`; decide if today is a blog-gen day (see ROUTINE.md + "Daily flow + burn cadence"). If not → done for today (insights only). +4. **Blog day** — pick the persona's top fresh opportunity as `target_keyword`. Assemble inputs + (persona section, VISION, POSITIONING from the hub; shipped slugs; queue row's volume+comp). +5. **Mode** — Knowcap MCP Demo-org → persona project → source with ≥3 human-confirmed memories? + → **case-study** (cite `source_knowcap_ids`, generate the verification panel via + `lib/gen-verification-panel.mjs`). Else → **SEO-grounded thesis** (answer the keyword's real + SERP / People-Also-Ask demand with a unique Knowcap angle). +6. **Write** via write-blog-draft SKILL → run ALL gates (banned words, 1300–1600 words, keyword + in title+H2+≥3, slug unique, exactly 5 FAQ, frontmatter). Fail → regenerate/skip, never ship thin. +7. **Output** — draft → `docs/content-pipeline/drafts/.md`, open `[blog-draft]` PR to main. + Update `burn-state.json` (total_posts++, posts_this_week++). + +## Hard rules + +- **ICP only** — odoo-partners / mena-audit-firms / mena-agencies / regulated-verticals. Off-ICP keyword (even high-volume) → skip it. +- **Knowcap angle mandatory** — every post ties the keyword to verified-facts / human-confirmation / audit-trail. If a keyword can't carry that angle honestly, skip it. +- **Quality > quota** — no fresh opportunity clears the gates → skip the day. A missed burn post beats a thin one (Google scaled-content-abuse demotes the whole domain). +- **Honesty** — case-study only on genuinely human-confirmed claims. SEO-grounded thesis must add real value, not keyword-stuff. Never claim a verification panel for unconfirmed claims. +- **No duplicates** — dedup against last 20 shipped slugs AND in-flight drafts. The queue already flags `covered`. +- **English posts**, MENA-targeted (mine EN + AR demand, publish EN). +- **DataForSEO down → stop**, don't invent data. One-line error, never silent. + +## Files + +- `scripts/seo-pull.mjs` — the SEO engine (this is what makes it daily + live) +- `opportunity-queue.json` — ranked queue (gitignored runtime state) +- `burn-state.json` — cadence tracker (gitignored) +- `state.json` — persona rotation cursor (gitignored) +- `lib/gen-verification-panel.mjs` — data-driven Knowcap UI SVG (case-study posts) +- `runs//` — per-run digest + draft + report (gitignored) diff --git a/routines/blogger/ROUTINE.md b/routines/blogger/ROUTINE.md index 19a0f46..663d850 100644 --- a/routines/blogger/ROUTINE.md +++ b/routines/blogger/ROUTINE.md @@ -12,6 +12,24 @@ See [`triggers.yml`](./triggers.yml) — cron `0 6 * * MON` (Monday 06:00 UTC = Calls [`_skills/write-blog-draft/SKILL.md`](../_skills/write-blog-draft/SKILL.md) in `thesis` / `case-study` / `comparison` mode (routine picks). +## Daily flow + burn cadence (SEO engine) + +This routine fires DAILY (07:00 Sun–Thu). Every run: + +1. **Pull SEO insights** — `node routines/blogger/scripts/seo-pull.mjs` → refresh `opportunity-queue.json` + digest (every day, ~$0.1–0.5). +2. **Surface the digest** to Hassan (the open:agent run window / Claude agents sidebar — see AGENT.md). +3. **Decide if today is a blog-gen day** from `routines/blogger/burn-state.json`: + - **burn phase**: generate a post on 3 days/week (Sun/Tue/Thu) until `total_posts >= 24` (~8 weeks), then flip to steady. + - **steady phase**: generate 1 post/week (Sun). + - Not a blog-gen day → stop after the digest (insights only, no post). +4. On a blog-gen day → mode selection + write, using the queue's top fresh opportunity as `target_keyword`. + +`burn-state.json` (gitignored runtime state): +```json +{ "phase": "burn", "started": "2026-06-15", "total_posts": 0, "week_start": "2026-06-15", "posts_this_week": 0, "blog_days": ["SUN","TUE","THU"] } +``` +After each shipped draft PR: `total_posts++`, `posts_this_week++`. Reset `posts_this_week` on week rollover. Flip `phase→steady` + `blog_days→["SUN"]` when `total_posts >= 24`. If the file is absent, seed burn from today. **Quality bar:** never publish to hit a quota — if no fresh opportunity clears the gates, skip the day (a missed burn post beats a thin one). + ## Mode selection (runtime) ``` @@ -59,8 +77,8 @@ Each run reads `cursor`, picks `personas[cursor]`, then advances `cursor = (curs 2. **Read `../claude-knowcap/knowledge/people/PRODUCT-PERSONAS.md`** → pick the persona's section (study segment names map to slugs: "Odoo implementation partners" → `odoo-partners`, "Audit / accounting firms" → `mena-audit-firms`, etc.) 3. **Read `../claude-knowcap/knowledge/strategies/VISION.md`** → voice + anti-positioning 4. **Read `../claude-knowcap/knowledge/strategies/POSITIONING.md`** → three sentences + anti-positioning -5. **Read most recent `../claude-knowcap/knowledge/topics/research/audits/SEO-AUDIT-*.md`** → top-3 keyword opportunities (currently: scan for keyword candidates manually; eventually: parse a `keyword_opportunities_by_persona` table) -6. **Query Google Trends via pytrends** (urllib3<2.0 required) → validate 5-year MENA interest, pick highest recent growth +5. **Run `node routines/blogger/scripts/seo-pull.mjs`** (live SEO engine, replaces the old static SEO-audit scan AND the dead Google Trends step). Pulls DataForSEO Google-Ads keyword demand for MENA (KSA + Egypt + UAE) in EN + AR, expands persona seeds into real related keywords with **search volume + competition**, filters to Knowcap ICP intent, ranks by `volume × competition-weight`, dedups against shipped posts, and writes `routines/blogger/opportunity-queue.json` + a digest. **`target_keyword` = the persona's top fresh (uncovered) opportunity** from the queue. Auth: DataForSEO creds in `~/.claude/secrets/blogger.md`. +6. **(Google Trends removed.)** DataForSEO volume + competition from step 5 is the demand signal — Trends was near-zero for this B2B ICP. Record the chosen keyword's `search_volume` + `competition` in frontmatter. 7. **Try `case-study` mode:** - Query Knowcap MCP `mcp__knowcap__list_sources` → Demo org, persona project - For each source, `mcp__knowcap__list_memories` filtered to source via `metadata.source_id` diff --git a/routines/blogger/scripts/seo-pull.mjs b/routines/blogger/scripts/seo-pull.mjs new file mode 100644 index 0000000..8c1ea45 --- /dev/null +++ b/routines/blogger/scripts/seo-pull.mjs @@ -0,0 +1,193 @@ +#!/usr/bin/env node +/** + * seo-pull.mjs — daily SEO opportunity pull for the Knowcap blogger routine. + * + * Replaces the static keyword-opportunities.md + the dead Google Trends step. + * Pulls LIVE MENA keyword demand from DataForSEO (Google Ads keywords_for_keywords: + * expands persona seed terms into related keywords with real search volume + + * competition), EN + AR across KSA / Egypt / UAE, filters to Knowcap's ICP intent, + * ranks, dedups against already-shipped posts, and writes a ranked opportunity queue + * + a human digest. + * + * NO browser. Auth = HTTP Basic from ~/.claude/secrets/blogger.md. + * + * Usage: + * node seo-pull.mjs # all personas, all geos, EN+AR + * node seo-pull.mjs --persona odoo-partners # one persona + * node seo-pull.mjs --locations "Saudi Arabia" --langs en # scope to conserve balance + * node seo-pull.mjs --max 25 # cap queue size per persona + * + * Outputs (gitignored — runtime state): + * routines/blogger/opportunity-queue.json + * routines/blogger/runs//seo-digest.md (caller passes --stamp; else prints) + */ + +import fs from 'fs' +import path from 'path' +import os from 'os' +import { fileURLToPath } from 'url' + +const __dirname = path.dirname(fileURLToPath(import.meta.url)) +const BLOGGER_DIR = path.resolve(__dirname, '..') // routines/blogger +const REPO_ROOT = path.resolve(BLOGGER_DIR, '..', '..') // knowcap-website +const BLOG_DIR = path.join(REPO_ROOT, 'app', 'content', 'blog') +const SECRETS = path.join(os.homedir(), '.claude', 'secrets', 'blogger.md') + +// ---- args ---- +const argv = process.argv.slice(2) +const getArg = (name, def) => { + const i = argv.indexOf('--' + name) + return i >= 0 && argv[i + 1] ? argv[i + 1] : def +} +const ONLY_PERSONA = getArg('persona', null) +const LOCATIONS = getArg('locations', 'Saudi Arabia,Egypt,United Arab Emirates').split(',').map(s => s.trim()) +const LANGS = getArg('langs', 'en,ar').split(',').map(s => s.trim()) +const MAX = parseInt(getArg('max', '25'), 10) +const STAMP = getArg('stamp', null) + +// ---- creds ---- +function readCreds() { + const txt = fs.readFileSync(SECRETS, 'utf8') + const login = (txt.match(/^DATAFORSEO_LOGIN=(.+)$/m) || [])[1]?.trim() + const pass = (txt.match(/^DATAFORSEO_PASSWORD=(.+)$/m) || [])[1]?.trim() + if (!login || !pass || login.startsWith('<')) { + console.error('FATAL: DataForSEO creds missing in ' + SECRETS) + process.exit(2) + } + return 'Basic ' + Buffer.from(`${login}:${pass}`).toString('base64') +} +const AUTH = readCreds() + +// ---- personas: seed terms + ICP intent filter ---- +// Seeds are deliberately broad so keywords_for_keywords discovers the real demand. +const PERSONAS = { + 'odoo-partners': { + seeds_en: ['odoo implementation', 'odoo partner', 'erp implementation', 'odoo project'], + seeds_ar: ['تطبيق اودو', 'اودو'], + }, + 'mena-audit-firms': { + seeds_en: ['audit documentation', 'audit firm software', 'pdpl compliance', 'engagement letter'], + seeds_ar: ['تدقيق', 'محضر اجتماع'], + }, + 'mena-agencies': { + seeds_en: ['meeting notes software', 'client meeting notes', 'ai meeting notes'], + seeds_ar: ['محضر اجتماع', 'تفريغ اجتماع'], + }, + 'regulated-verticals': { + seeds_en: ['compliance meeting records', 'ai governance', 'audit trail software'], + seeds_ar: ['حوكمة الذكاء الاصطناعي', 'الامتثال'], + }, +} +// Keep only keywords whose intent Knowcap can credibly own. +const ICP_INTENT = /(meeting|minute|note|transcri|record|audit|complian|pdpl|govern|odoo|erp|scope|verif|attest|sign[- ]?off|محضر|اجتماع|تدقيق|اودو|امتثال|حوكمة|توثيق)/i +const COMP_WEIGHT = { LOW: 1.0, MEDIUM: 0.6, HIGH: 0.35 } + +// ---- shipped-slug dedup ---- +function shippedSlugs() { + if (!fs.existsSync(BLOG_DIR)) return [] + return fs.readdirSync(BLOG_DIR).filter(f => f.endsWith('.md')).map(f => f.replace(/\.md$/, '')) +} +const slugify = s => String(s).toLowerCase().replace(/[^a-z0-9؀-ۿ]+/g, '-').replace(/^-|-$/g, '') +function isCovered(keyword, slugs) { + const ks = new Set(slugify(keyword).split('-').filter(w => w.length > 3)) + if (!ks.size) return false + for (const slug of slugs) { + const ss = new Set(slug.split('-')) + let hit = 0 + for (const w of ks) if (ss.has(w)) hit++ + if (hit / ks.size >= 0.6) return true + } + return false +} + +// ---- DataForSEO call ---- +async function keywordsForKeywords(seeds, locationName, languageCode) { + const body = [{ keywords: seeds, location_name: locationName, language_code: languageCode }] + const res = await fetch('https://api.dataforseo.com/v3/keywords_data/google_ads/keywords_for_keywords/live', { + method: 'POST', + headers: { Authorization: AUTH, 'Content-Type': 'application/json' }, + body: JSON.stringify(body), + }) + const json = await res.json() + if (json.status_code !== 20000) throw new Error(`API ${json.status_code}: ${json.status_message}`) + const task = (json.tasks || [])[0] || {} + if (task.status_code !== 20000) throw new Error(`task ${task.status_code}: ${task.status_message}`) + return { items: task.result || [], cost: json.cost || 0 } +} + +// ---- main ---- +const personas = ONLY_PERSONA ? [ONLY_PERSONA] : Object.keys(PERSONAS) +const slugs = shippedSlugs() +const out = { generated_at: new Date().toISOString(), source: 'dataforseo:google_ads/keywords_for_keywords', locations: LOCATIONS, langs: LANGS, cost: 0, personas: {}, top: [] } +const seen = new Set() + +for (const persona of personas) { + const cfg = PERSONAS[persona] + if (!cfg) { console.error('unknown persona: ' + persona); continue } + const bucket = [] + for (const loc of LOCATIONS) { + for (const lang of LANGS) { + const seeds = lang === 'ar' ? cfg.seeds_ar : cfg.seeds_en + if (!seeds?.length) continue + try { + const { items, cost } = await keywordsForKeywords(seeds, loc, lang) + out.cost += cost + for (const it of items) { + const kw = it.keyword + const vol = it.search_volume || 0 + if (!kw || vol <= 0) continue + if (!ICP_INTENT.test(kw)) continue + const comp = (it.competition || '').toUpperCase() + const score = Math.round(vol * (COMP_WEIGHT[comp] ?? 0.5)) + bucket.push({ keyword: kw, search_volume: vol, competition: comp || null, cpc: it.cpc ?? null, score, location: loc, lang, covered: isCovered(kw, slugs) }) + } + console.error(` ${persona} | ${loc} | ${lang}: +${items.length} (cost $${cost})`) + } catch (e) { + console.error(` ERROR ${persona} | ${loc} | ${lang}: ${e.message}`) + } + } + } + // dedupe by keyword (keep highest score), sort, cap + const byKw = new Map() + for (const r of bucket) { + const k = r.keyword.toLowerCase() + if (!byKw.has(k) || byKw.get(k).score < r.score) byKw.set(k, r) + } + const ranked = [...byKw.values()].sort((a, b) => b.score - a.score).slice(0, MAX) + out.personas[persona] = ranked + for (const r of ranked) { + if (r.covered) continue + const key = r.keyword.toLowerCase() + if (seen.has(key)) continue + seen.add(key) + out.top.push({ ...r, persona }) + } +} +out.top.sort((a, b) => b.score - a.score) +out.top = out.top.slice(0, 20) + +// ---- write queue ---- +const QUEUE = path.join(BLOGGER_DIR, 'opportunity-queue.json') +fs.writeFileSync(QUEUE, JSON.stringify(out, null, 2)) + +// ---- digest ---- +const lines = [] +lines.push(`# SEO opportunity digest — ${out.generated_at.slice(0, 10)}`) +lines.push('') +lines.push(`Source: DataForSEO (MENA ${LOCATIONS.join(', ')} · ${LANGS.join('/')}) · API cost this run: $${out.cost.toFixed(3)}`) +lines.push(`Fresh opportunities (not yet covered by a shipped post): **${out.top.length}**`) +lines.push('') +lines.push('## Top picks (write next)') +lines.push('') +lines.push('| # | keyword | persona | vol | comp | score |') +lines.push('|---|---|---|---|---|---|') +out.top.slice(0, 10).forEach((r, i) => lines.push(`| ${i + 1} | ${r.keyword} | ${r.persona} | ${r.search_volume} | ${r.competition || '-'} | ${r.score} |`)) +const digest = lines.join('\n') + +if (STAMP) { + const runDir = path.join(BLOGGER_DIR, 'runs', STAMP) + fs.mkdirSync(runDir, { recursive: true }) + fs.writeFileSync(path.join(runDir, 'seo-digest.md'), digest) +} +console.log(digest) +console.error(`\nwrote ${QUEUE} — ${out.top.length} fresh opportunities, total API cost $${out.cost.toFixed(3)}`)