feat(blogger): live DataForSEO SEO engine + daily burn cadence - #61
Conversation
…static keywords + Trends) Turns the blogger from static-keyword + dead-Google-Trends into a daily, live SEO-driven engine. - scripts/seo-pull.mjs: zero-dep DataForSEO pull (Google-Ads keywords_for_keywords). Expands persona seeds into real related keywords with search volume + competition, MENA (KSA+EG+UAE) EN+AR, filters to Knowcap ICP intent, ranks by volume x competition-weight, dedups vs shipped posts -> opportunity-queue.json + digest. Auth: HTTP Basic from ~/.claude/secrets/blogger.md (never committed). Verified live: 369 keywords -> 13 fresh ranked opportunities, $0.075/run. - AGENT.md: the brain mission (email-manager style) — daily flow, burn cadence, hard rules (ICP-only, Knowcap-angle, quality>quota, honesty, dedup, EN posts). - ROUTINE.md: input #5 now runs seo-pull (was static SEO-audit scan); #6 Google Trends removed (near-zero for B2B); new "Daily flow + burn cadence" section (3 posts/wk for ~8wk -> 1/wk, never publish thin to hit quota). - .gitignore: opportunity-queue.json + burn-state.json (runtime state). Brain-first half of the knowcap/blogger routine rebuild (routine-creator). The thin schedule shell + Windows task land in claude-knowcap next.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughAdds a new ChangesDataForSEO SEO Pull and Blogger Routine Integration
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@routines/blogger/ROUTINE.md`:
- Around line 80-82: The ROUTINE.md file marks Google Trends as removed in step
6, but other sections in the same document still reference Google Trends in the
Tool/MCP table, PR body summary, and Failure modes section. Search the file for
all remaining mentions of Google Trends and update them to reference
DataForSEO-based signals (search volume and competition) instead, ensuring
consistency with the workflow change described in steps 5 and 6 where DataForSEO
replaces Trends as the demand signal source.
- Around line 15-19: The routine documentation states it fires DAILY at 07:00
Sun–Thu, but the Trigger section in this file currently defines a Monday-only
cron schedule. Update the Trigger section to use a cron expression that matches
the documented daily cadence (07:00 every Sunday through Thursday), ensuring the
cron definition and the documented cadence contract are consistent so operators
execute the routine on the correct schedule.
In `@routines/blogger/scripts/seo-pull.mjs`:
- Around line 145-147: The catch block at lines 145-147 currently logs the error
message and allows execution to continue, which results in partial data being
written to the opportunity-queue.json file when DataForSEO API calls fail. To
fix this, replace the current error handling that only logs the error with a
mechanism that stops execution immediately (such as throwing the error, calling
process.exit(1), or rethrowing the exception). This ensures that when DataForSEO
authentication or funding errors occur, the script terminates cleanly without
emitting incomplete or fabricated keyword data, preventing downstream issues
with partial data being persisted.
- Around line 49-56: The readCreds function does not handle errors from
fs.readFileSync when the SECRETS file is missing or unreadable, causing an
unstructured crash. Wrap the fs.readFileSync call in a try-catch block and
handle any thrown errors by logging a fatal error message and calling
process.exit(2), matching the same controlled error path used for the existing
credential validation checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 9caf2c7a-74f1-42f0-a87d-692d746275e4
📒 Files selected for processing (4)
.gitignoreroutines/blogger/AGENT.mdroutines/blogger/ROUTINE.mdroutines/blogger/scripts/seo-pull.mjs
| ## Daily flow + burn cadence (SEO engine) | ||
|
|
||
| This routine fires DAILY (07:00 Sun–Thu). Every run: | ||
|
|
||
| 1. **Pull SEO insights** — `node routines/blogger/scripts/seo-pull.mjs` → refresh `opportunity-queue.json` + digest (every day, ~$0.1–0.5). |
There was a problem hiding this comment.
Cadence contract is inconsistent with the routine trigger definition.
This section says the routine fires daily, but the existing Trigger section still points to Monday-only cron. Please unify these so operators don’t run the wrong schedule.
🧰 Tools
🪛 LanguageTool
[grammar] ~19-~19: Ensure spelling is correct
Context: ... Every run: 1. Pull SEO insights — node routines/blogger/scripts/seo-pull.mjs → refresh `opport...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@routines/blogger/ROUTINE.md` around lines 15 - 19, The routine documentation
states it fires DAILY at 07:00 Sun–Thu, but the Trigger section in this file
currently defines a Monday-only cron schedule. Update the Trigger section to use
a cron expression that matches the documented daily cadence (07:00 every Sunday
through Thursday), ensuring the cron definition and the documented cadence
contract are consistent so operators execute the routine on the correct
schedule.
| 5. **Run `node routines/blogger/scripts/seo-pull.mjs`** (live SEO engine, replaces the old static SEO-audit scan AND the dead Google Trends step). Pulls DataForSEO Google-Ads keyword demand for MENA (KSA + Egypt + UAE) in EN + AR, expands persona seeds into real related keywords with **search volume + competition**, filters to Knowcap ICP intent, ranks by `volume × competition-weight`, dedups against shipped posts, and writes `routines/blogger/opportunity-queue.json` + a digest. **`target_keyword` = the persona's top fresh (uncovered) opportunity** from the queue. Auth: DataForSEO creds in `~/.claude/secrets/blogger.md`. | ||
| 6. **(Google Trends removed.)** DataForSEO volume + competition from step 5 is the demand signal — Trends was near-zero for this B2B ICP. Record the chosen keyword's `search_volume` + `competition` in frontmatter. | ||
| 7. **Try `case-study` mode:** |
There was a problem hiding this comment.
Google Trends is marked removed here, but later sections still require/report it.
After this change, the same file still references Trends in the Tool/MCP table, PR body summary, and Failure modes. Those should be updated to DataForSEO-based signals to keep the runbook executable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@routines/blogger/ROUTINE.md` around lines 80 - 82, The ROUTINE.md file marks
Google Trends as removed in step 6, but other sections in the same document
still reference Google Trends in the Tool/MCP table, PR body summary, and
Failure modes section. Search the file for all remaining mentions of Google
Trends and update them to reference DataForSEO-based signals (search volume and
competition) instead, ensuring consistency with the workflow change described in
steps 5 and 6 where DataForSEO replaces Trends as the demand signal source.
| function readCreds() { | ||
| const txt = fs.readFileSync(SECRETS, 'utf8') | ||
| const login = (txt.match(/^DATAFORSEO_LOGIN=(.+)$/m) || [])[1]?.trim() | ||
| const pass = (txt.match(/^DATAFORSEO_PASSWORD=(.+)$/m) || [])[1]?.trim() | ||
| if (!login || !pass || login.startsWith('<')) { | ||
| console.error('FATAL: DataForSEO creds missing in ' + SECRETS) | ||
| process.exit(2) | ||
| } |
There was a problem hiding this comment.
Handle missing/unreadable secrets file with the same controlled fatal path.
If ~/.claude/secrets/blogger.md is missing/unreadable, readFileSync throws before your explicit credential checks, producing an unstructured crash path.
Based on learnings: “If DataForSEO errors (auth/funds), log one line and STOP; never fabricate keywords.”
Suggested fix
function readCreds() {
- const txt = fs.readFileSync(SECRETS, 'utf8')
+ let txt
+ try {
+ txt = fs.readFileSync(SECRETS, 'utf8')
+ } catch {
+ console.error('FATAL: DataForSEO creds missing in ' + SECRETS)
+ process.exit(2)
+ }
const login = (txt.match(/^DATAFORSEO_LOGIN=(.+)$/m) || [])[1]?.trim()
const pass = (txt.match(/^DATAFORSEO_PASSWORD=(.+)$/m) || [])[1]?.trim()
if (!login || !pass || login.startsWith('<')) {🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@routines/blogger/scripts/seo-pull.mjs` around lines 49 - 56, The readCreds
function does not handle errors from fs.readFileSync when the SECRETS file is
missing or unreadable, causing an unstructured crash. Wrap the fs.readFileSync
call in a try-catch block and handle any thrown errors by logging a fatal error
message and calling process.exit(2), matching the same controlled error path
used for the existing credential validation checks.
Source: Learnings
| } catch (e) { | ||
| console.error(` ERROR ${persona} | ${loc} | ${lang}: ${e.message}`) | ||
| } |
There was a problem hiding this comment.
Fail fast on DataForSEO call failures instead of continuing with partial data.
The catch currently logs and continues, which allows a partial opportunity-queue.json to be emitted after an upstream SEO failure. This should stop the run immediately.
Based on learnings: “If DataForSEO errors (auth/funds), log one line and STOP; never fabricate keywords.”
Suggested fix
- } catch (e) {
- console.error(` ERROR ${persona} | ${loc} | ${lang}: ${e.message}`)
- }
+ } catch (e) {
+ console.error(`ERROR DataForSEO ${persona} | ${loc} | ${lang}: ${e.message}`)
+ process.exit(1)
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| } catch (e) { | |
| console.error(` ERROR ${persona} | ${loc} | ${lang}: ${e.message}`) | |
| } | |
| } catch (e) { | |
| console.error(`ERROR DataForSEO ${persona} | ${loc} | ${lang}: ${e.message}`) | |
| process.exit(1) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@routines/blogger/scripts/seo-pull.mjs` around lines 145 - 147, The catch
block at lines 145-147 currently logs the error message and allows execution to
continue, which results in partial data being written to the
opportunity-queue.json file when DataForSEO API calls fail. To fix this, replace
the current error handling that only logs the error with a mechanism that stops
execution immediately (such as throwing the error, calling process.exit(1), or
rethrowing the exception). This ensures that when DataForSEO authentication or
funding errors occur, the script terminates cleanly without emitting incomplete
or fabricated keyword data, preventing downstream issues with partial data being
persisted.
Source: Learnings
1 participant
Blogger → live daily SEO engine (brain half of the routine rebuild)
Replaces the static keyword table + dead Google Trends with a live DataForSEO engine.
What's here
routines/blogger/scripts/seo-pull.mjs— zero-dep DataForSEO pull. Expands persona seeds into real related keywords (Google-Ads keywords_for_keywords) with search volume + competition, MENA (KSA+EG+UAE) EN + AR, ICP-filtered, ranked byvolume × competition-weight, deduped vs shipped posts →opportunity-queue.json+ digest. Verified live: 369 keywords → 13 fresh ranked opportunities, $0.075/run.routines/blogger/AGENT.md— the brain mission (email-manager style): daily flow, burn cadence, hard rules (ICP-only, Knowcap-angle, quality>quota, honesty, dedup, EN posts).routines/blogger/ROUTINE.md— input fix(landing): redirect /privacy → /policy (unblock CWS extension publish) #5 now runsseo-pull.mjs; [landing-redesign] Dark theme + app auth links #6 (Google Trends) removed; new Daily flow + burn cadence section (3 posts/wk for ~8 wk → 1/wk; never publish thin to hit quota)..gitignore—opportunity-queue.json+burn-state.json(runtime state).Decisions (from the routine-creator interview)
DataForSEO source · MENA EN+AR data → EN posts · daily pull · burn 3/wk×8wk→1/wk · SEO-grounded thesis allowed when no verified Demo source · case-study when ≥3 confirmed memories exist.
Creds
DataForSEO login/password live in
~/.claude/secrets/blogger.md(never committed). Account verified; runs on current balance.Next
The thin schedule shell + Windows task (07:00 Sun–Thu, open:agent) land in
claude-knowcap/Routines/blogger/(separate PR, per routine-creator).🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Documentation