diff --git a/README.md b/README.md index 67d7033..aecacc6 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,15 @@ no-ranking promise the README makes, that `/llms.txt`, `/sitemap.xml`, `/robots. npm run audit:live-promise ``` +To preview (no network writes) or run the credential-free IndexNow submission +that tells Bing — and Bing-derived DuckDuckGo — to crawl every public URL +(the key file must be live on production first; the script refuses otherwise): + +```bash +npm run submit:indexnow -- --dry-run # preview URL set + endpoints +npm run submit:indexnow # verify key file, then submit to Bing/IndexNow +``` + For the repeatable live walk of the private-beta funnel (the backlog item "Live-surface walk of the private-beta funnel"): a real-browser (Playwright Chromium) walk of the funnel stops home → `/demo` → `/packages`, with the diff --git a/docs/growth/search-index-coverage-2026-08-11.md b/docs/growth/search-index-coverage-2026-08-11.md new file mode 100644 index 0000000..6c18eb4 --- /dev/null +++ b/docs/growth/search-index-coverage-2026-08-11.md @@ -0,0 +1,135 @@ +# SEO Fix Kit — full search-index coverage (Google + Bing/DuckDuckGo) + +Packet for the lane-1 item: **Establish full search-index coverage: all 7 public +pages indexed on Google plus a Bing/DuckDuckGo presence** +(backlog owner: +`/home/nish/workspaces/agent-state/seo-fix-kit-improvement-loop/backlog.md`, +[scout 2026-08-09, rank: 2, risk: green]). + +Status: **agent-completable half shipped, engine half needs owner credentials + +time.** This packet (2026-08-11) re-verified the gap live, shipped the +credential-free IndexNow submission path (key file routes in the Worker, +submission script, tests, wrangler routing) that gets Bing — and therefore +DuckDuckGo — crawling all public URLs the moment the release lands, and +documented the two owner-only legs (Google Search Console request-indexing, +Bing Webmaster ownership) as an exact manual kit. Search-engine indexing is +externally owned and time-dependent; the acceptance outcome ("all 7 pages on +Google, seofixkit.com first on Bing/DDG") cannot be produced from a lane +without credentials, but every lever that does not need credentials is now +built and one command away. + +## Fresh live evidence (2026-08-11, ~10:40 IST) + +- **Google: homepage only.** Startpage (a live Google-index proxy) for + `site:seofixkit.com` returns exactly one organic result: + `https://seofixkit.com` "SEO Fix Kit - Proof-Backed SEO Repair Beta". Per-path + probes `site:seofixkit.com/{terms,demo,methodology,packages,check,privacy,support}` + all return zero results. Direct Google is CAPTCHA-walled from this VPS + (`/sorry/index`, screenshot `/tmp/serde2-google_gbv1_site.png`), so Startpage + is the reproducible Google-index receipt. Backlog evidence (2026-08-09, real + anti-detection browser) showed `/` + `/terms`; the `/terms` result has since + dropped out or been consolidated — either way, 7 of 8 public routes remain + unindexed on Google. +- **Bing: zero presence.** Real-browser `bing.com/search?q=seofixkit.com` + returns ~82,200 results, none of them seofixkit.com (all unrelated + "Inn at the Market" hotel pages — Bing is fuzzy-matching the query text). + Screenshot `/tmp/seo-serde-bing_domain.png`. +- **DuckDuckGo: blocked this run, zero by construction.** DDG html/lite/main + endpoints all bot-walled this IP (418/error page; screenshots + `/tmp/serde2-duckduckgo_main.png`). DDG's web index is Bing-derived, and + prior real-browser evidence (2026-08-09) recorded a zero-results SERP for + `seofixkit.com`. No change in DDG coverage is possible before Bing crawls. +- **Site surfaces are crawl-ready:** `https://seofixkit.com/robots.txt` HTTP 200 + (`Allow: /`, `Sitemap: https://seofixkit.com/sitemap.xml`); + `https://seofixkit.com/sitemap.xml` HTTP 200 listing 8 public URLs + (`/`, `/demo`, `/check`, `/methodology`, `/packages`, `/privacy`, `/support`, + `/terms` — the item said 7; `/check` was added to the sitemap after the item + was filed, so the target set is now 8). Every page has unique title/meta/OG/ + canonical; homepage carries Organization/WebSite/SoftwareApplication/FAQPage + schema; `/check` carries WebPage/FAQ JSON-LD. +- **No ownership artifacts observable:** DNS TXT on seofixkit.com carries only + SPF (`v=spf1 include:_spf.porkbun.com include:_spf.mx.cloudflare.net ~all`); + no `google-site-verification` meta, no `bing-site-verification` meta, no + BingSiteAuth.xml, no IndexNow key file live (both candidate paths return the + SPA fallback HTML with body mismatch — the exact bug `run_worker_first` + routing fixes in this packet). + +## What this packet ships + +- `shared/index-now.js` — committed IndexNow key (`3219d564f9f914772e178f33ae543e60`), + key file paths, payload builder, endpoints (`api.indexnow.org`, + `www.bing.com/indexnow`). Key is world-readable by spec, not a credential. +- `shared/audit-engine.js` — hoisted the canonical route list to + `ROOT_PUBLIC_PATHS`, shared by `rootSitemap()` and IndexNow so the sitemap + and the submission set can never drift. +- `worker/index.js` — serves `GET /{key}.txt` and `GET /.well-known/{key}.txt` + (text/plain, `x-robots-tag: noindex`), apex-only like every other public + surface; www requests keep 301ing to apex. +- `wrangler.jsonc` — both key paths added to `run_worker_first` so the SPA + asset fallback cannot shadow them (verified live: without this, both paths + currently return the SPA HTML with HTTP 200). +- `server/index.js` — local dev-server parity for both key paths. +- `scripts/submit-indexnow.mjs` + `scripts/submit-indexnow.test.mjs` — + `node scripts/submit-indexnow.mjs` verifies the key file is live at both + locations (refuses otherwise, exit 2), parses the live sitemap locs, POSTs + the payload to both endpoints, and reports per-endpoint accept/reject + (exit 0/3). `--dry-run` previews with no network writes. +- `package.json` — `submit:indexnow` script + `test:indexnow` wired into the + canonical `check` gate. +- `worker/index.test.mjs` — apex key file 200 with exact key body + noindex, + www 301 to apex, for both paths. + +## Owner manual legs (credentials required — cannot be done by an agent) + +Both are one-time, copy-paste steps that materially accelerate the outcome: + +1. **Google Search Console — request indexing (accelerates Google re-crawl).** + - Add property `https://seofixkit.com` (URL-prefix) at + https://search.google.com/search-console (any Google account). + - Verify: DNS TXT record `google-site-verification=...` at Porkbun (or the + HTML meta tag; DNS is preferred and survives redeploys). + - Submit `https://seofixkit.com/sitemap.xml` in Sitemaps. + - Open URL Inspection for each of the 8 sitemap URLs and click + "Request indexing" (spread over a few days; Google throttles). + - Expected outcome: `site:seofixkit.com` on Google returns all 8 pages. + Without GSC, Google re-crawls on its own schedule via the sitemap and + the homepage's internal links (already shipped: `/check` is linked from + `/`, `/demo`, `/packages`, `/methodology`, `/support`, `/terms`, + `/privacy`). +2. **Bing Webmaster Tools — ownership + sitemap (long-term Bing health).** + - Add site at https://www.bing.com/webmasters (Microsoft account). + - Verify: DNS TXT `ms=...` at Porkbun. + - Submit `https://seofixkit.com/sitemap.xml`. + - IndexNow (shipped here) covers the crawl trigger; Webmaster Tools adds + the dashboard, crawl logs, and index coverage reports. + +## Resume path (agent-completable, after this PR is merged AND released) + +1. Confirm release landed: `curl -s https://seofixkit.com/{key}.txt` returns + the key text (not the SPA HTML). +2. `node scripts/submit-indexnow.mjs` — verify 2x "key matches", 8 URLs listed, + ACCEPTED on both endpoints. Bing crawls on its schedule (typically minutes + to hours); DDG follows from Bing's index. +3. Re-run the verification queries from "Fresh live evidence" above; record + receipts in this file under a "Re-verified" section. Google's leg moves + only via owner step 1 or its own re-crawl cadence. + +## Acceptance / verification mapping + +- Bing/DDG first-result presence: owned by IndexNow submission (this packet) + + crawler schedule; externally observable via + `bing.com/search?q=seofixkit.com` and `duckduckgo.com/?q=seofixkit.com`. +- All public pages on Google: owned by Search Console request-indexing (owner) + + sitemap re-crawl; externally observable via `site:seofixkit.com`. +- No index bloat / duplicate-host junk: already enforced — www 301s to apex, + every emitted URL (canonical, og:url, sitemap, robots, key file) is + apex-only; submission set is exactly the sitemap set with no query strings. +- Rollback: remove the worker key-file routes + `run_worker_first` entries and + delete `shared/index-now.js`; no product surface or public copy changes. + +## Files changed in this packet + +- `shared/index-now.js` (new), `scripts/submit-indexnow.mjs` (new), + `scripts/submit-indexnow.test.mjs` (new), `docs/growth/search-index-coverage-2026-08-11.md` (this file) +- `shared/audit-engine.js`, `worker/index.js`, `worker/index.test.mjs`, + `server/index.js`, `wrangler.jsonc`, `package.json` diff --git a/package.json b/package.json index 0256e5f..b01693a 100644 --- a/package.json +++ b/package.json @@ -14,6 +14,7 @@ "cf:dry-run": "npm run build && node scripts/wrangler-dry-run.mjs", "ops:audit-owned": "node scripts/run-live-audit-batch.mjs", "audit:live-promise": "node scripts/live-promise-spot-check.mjs", + "submit:indexnow": "node scripts/submit-indexnow.mjs", "test:billing": "node server/dodo-payment-smoke-test.js", "test:billing-route": "node --test worker/routes/billing.test.mjs", "test:product-truth": "node server/product-truth-smoke-test.js", @@ -41,9 +42,10 @@ "test:promise-audit": "node --test shared/promise-audit.test.mjs", "test:live-promise-spot-check": "node --test scripts/live-promise-spot-check.test.mjs", "test:app-contract": "node --test src/app-contract.test.mjs", + "test:indexnow": "node --test scripts/submit-indexnow.test.mjs", "test:canary-dry-run": "node --test scripts/wrangler-dry-run.test.mjs", "test:check-inventory": "node --test scripts/check-chain-inventory.test.mjs", - "check": "npm run test:billing && npm run test:billing-route && npm run test:product-truth && npm run test:audit && npm run test:large-crawl-security && npm run test:report-retention-security && npm run test:local-developer-api-security && npm run test:worker-dispatch && npm run test:worker-email && npm run test:public-pages && npm run test:public-check && npm run test:audit-engine && npm run test:account && npm run test:ai-answer-readiness && npm run test:growth-opportunities && npm run test:repair-queue && npm run test:repair-proof-receipt && npm run test:repair-implementation-pack && npm run test:repair-agent && npm run test:developer-api && npm run test:remediation-brief && npm run test:audit-batch-runner && npm run test:webhooks && npm run test:app-contract && npm run test:promise-audit && npm run test:live-promise-spot-check && npm run test:funnel-walk && npm run test:large-crawl && npm run test:canary-dry-run && npm run test:check-inventory && npm run build", + "check": "npm run test:billing && npm run test:billing-route && npm run test:product-truth && npm run test:audit && npm run test:large-crawl-security && npm run test:report-retention-security && npm run test:local-developer-api-security && npm run test:worker-dispatch && npm run test:worker-email && npm run test:public-pages && npm run test:public-check && npm run test:audit-engine && npm run test:account && npm run test:ai-answer-readiness && npm run test:growth-opportunities && npm run test:repair-queue && npm run test:repair-proof-receipt && npm run test:repair-implementation-pack && npm run test:repair-agent && npm run test:developer-api && npm run test:remediation-brief && npm run test:audit-batch-runner && npm run test:webhooks && npm run test:app-contract && npm run test:promise-audit && npm run test:live-promise-spot-check && npm run test:indexnow && npm run test:funnel-walk && npm run test:large-crawl && npm run test:canary-dry-run && npm run test:check-inventory && npm run build", "audit:funnel-walk": "node scripts/run-private-beta-funnel-walk.mjs", "test:funnel-walk": "node --test scripts/run-private-beta-funnel-walk.test.mjs" }, diff --git a/scripts/submit-indexnow.mjs b/scripts/submit-indexnow.mjs new file mode 100644 index 0000000..b1dcd64 --- /dev/null +++ b/scripts/submit-indexnow.mjs @@ -0,0 +1,115 @@ +// Submit the live sitemap URL set to IndexNow (Bing/Naver/Seznam/Yandex). +// +// Usage: +// node scripts/submit-indexnow.mjs # verify key file live, then submit +// node scripts/submit-indexnow.mjs --dry-run # print what would happen, no network writes +// +// Requirements: the IndexNow key file must already be live on the production +// host (the worker route in worker/index.js serves it once the repo change is +// released). The script refuses to submit until both key-file locations return +// the exact key text, because a submission with an unreachable key is silently +// discarded by the engines. +// +// Exit codes: 0 = accepted by every endpoint, 2 = key file not live yet, +// 3 = one or more endpoints rejected the submission. +import { INDEX_NOW_ENDPOINTS, INDEX_NOW_HOST, indexNowKeyFileBody, indexNowKeyFilePaths, buildIndexNowPayload } from "../shared/index-now.js"; + +const ORIGIN = `https://${INDEX_NOW_HOST}`; + +export function parseSitemapLocs(xml) { + const locs = []; + const re = /([^<]+)<\/loc>/g; + let m; + while ((m = re.exec(xml)) !== null) locs.push(m[1]); + return locs; +} + +export async function fetchSitemapUrls({ fetchImpl = globalThis.fetch } = {}) { + const res = await fetchImpl(`${ORIGIN}/sitemap.xml`); + if (!res.ok) throw new Error(`sitemap fetch failed: HTTP ${res.status}`); + const locs = parseSitemapLocs(await res.text()); + if (locs.length === 0) throw new Error("sitemap returned zero entries"); + return locs; +} + +export async function verifyKeyFileLive({ fetchImpl = globalThis.fetch } = {}) { + const expected = indexNowKeyFileBody().trim(); + const results = []; + for (const path of indexNowKeyFilePaths()) { + const url = `${ORIGIN}${path}`; + let ok = false; + let detail = ""; + try { + const res = await fetchImpl(url); + const body = (await res.text()).trim(); + ok = res.status === 200 && body === expected; + detail = ok ? "key matches" : `HTTP ${res.status}, body mismatch`; + } catch (err) { + detail = `fetch failed: ${err.message}`; + } + results.push({ url, ok, detail }); + } + return results; +} + +export async function submitUrlList(urlList, { endpoints = INDEX_NOW_ENDPOINTS, fetchImpl = globalThis.fetch } = {}) { + const payload = buildIndexNowPayload(urlList); + const outcomes = []; + for (const endpoint of endpoints) { + try { + const res = await fetchImpl(endpoint, { + method: "POST", + headers: { "content-type": "application/json; charset=utf-8" }, + body: JSON.stringify(payload) + }); + outcomes.push({ endpoint, status: res.status, ok: res.status === 200 || res.status === 202 }); + } catch (err) { + outcomes.push({ endpoint, status: 0, ok: false, error: err.message }); + } + } + return outcomes; +} + +export async function main({ dryRun = false, fetchImpl = globalThis.fetch } = {}) { + const keyChecks = await verifyKeyFileLive({ fetchImpl }); + const keyLive = keyChecks.every((c) => c.ok); + console.log(`IndexNow key file check (${ORIGIN}):`); + for (const c of keyChecks) console.log(` ${c.ok ? "OK " : "MISS"} ${c.url} (${c.detail})`); + + if (!keyLive && !dryRun) { + console.error( + "\nKey file is not live yet. Merge + release the worker change that serves the key file, then re-run this script." + ); + return { ok: false, code: 2, keyChecks }; + } + + const urls = await fetchSitemapUrls({ fetchImpl }); + console.log(`\nSitemap URL set (${urls.length} URLs):`); + for (const u of urls) console.log(` ${u}`); + + if (dryRun) { + console.log("\nDry run: would submit the above URL set to:"); + for (const e of INDEX_NOW_ENDPOINTS) console.log(` POST ${e}`); + return { ok: true, code: 0, keyChecks, urls, dryRun: true }; + } + + const outcomes = await submitUrlList(urls, { fetchImpl }); + console.log("\nSubmission results:"); + let allOk = true; + for (const o of outcomes) { + allOk = allOk && o.ok; + console.log(` ${o.ok ? "ACCEPTED" : "REJECTED"} ${o.endpoint} (HTTP ${o.status}${o.error ? `, ${o.error}` : ""})`); + } + return { ok: allOk, code: allOk ? 0 : 3, keyChecks, urls, outcomes }; +} + +if (import.meta.url === `file://${process.argv[1]}`) { + const dryRun = process.argv.includes("--dry-run"); + try { + const result = await main({ dryRun }); + process.exitCode = result.code; + } catch (err) { + console.error(`submit-indexnow failed: ${err.message}`); + process.exitCode = 1; + } +} diff --git a/scripts/submit-indexnow.test.mjs b/scripts/submit-indexnow.test.mjs new file mode 100644 index 0000000..4c04bc3 --- /dev/null +++ b/scripts/submit-indexnow.test.mjs @@ -0,0 +1,116 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { + INDEX_NOW_ENDPOINTS, + INDEX_NOW_HOST, + INDEX_NOW_KEY, + buildIndexNowPayload, + indexNowKeyFilePaths +} from "../shared/index-now.js"; +import { ROOT_PUBLIC_PATHS } from "../shared/audit-engine.js"; +import { + fetchSitemapUrls, + main, + parseSitemapLocs, + submitUrlList, + verifyKeyFileLive +} from "./submit-indexnow.mjs"; + +function xmlSitemap(locs) { + return `${locs + .map((l) => `${l}`) + .join("")}`; +} + +test("IndexNow key constants are coherent", () => { + assert.match(INDEX_NOW_KEY, /^[0-9a-f]{32}$/); + assert.equal(INDEX_NOW_HOST, "seofixkit.com"); + assert.deepEqual(indexNowKeyFilePaths(), [ + `/${INDEX_NOW_KEY}.txt`, + `/.well-known/${INDEX_NOW_KEY}.txt` + ]); + assert.ok(INDEX_NOW_ENDPOINTS.includes("https://api.indexnow.org/indexnow")); + assert.ok(INDEX_NOW_ENDPOINTS.includes("https://www.bing.com/indexnow")); +}); + +test("buildIndexNowPayload mirrors the full sitemap route set, apex-only", () => { + const urls = ROOT_PUBLIC_PATHS.map((p) => `https://${INDEX_NOW_HOST}${p}`); + const payload = buildIndexNowPayload(urls); + assert.equal(payload.host, INDEX_NOW_HOST); + assert.equal(payload.key, INDEX_NOW_KEY); + assert.equal(payload.keyLocation, `https://${INDEX_NOW_HOST}/${INDEX_NOW_KEY}.txt`); + assert.deepEqual(payload.urlList, urls); + for (const u of payload.urlList) assert.doesNotMatch(u, /www\./); +}); + +test("parseSitemapLocs extracts every loc", () => { + const locs = parseSitemapLocs(xmlSitemap(["/", "/demo", "/check"])); + assert.deepEqual(locs, ["/", "/demo", "/check"]); + assert.deepEqual(parseSitemapLocs(""), []); +}); + +test("fetchSitemapUrls reads the live-style sitemap and rejects empty sets", async () => { + const locs = ROOT_PUBLIC_PATHS.map((p) => `https://${INDEX_NOW_HOST}${p}`); + const fetchImpl = async (url) => + new Response(xmlSitemap(locs), { status: 200, headers: { "content-type": "application/xml" } }); + const got = await fetchSitemapUrls({ fetchImpl }); + assert.deepEqual(got, locs); + await assert.rejects(fetchSitemapUrls({ fetchImpl: async () => new Response("", { status: 200 }) })); + await assert.rejects(fetchSitemapUrls({ fetchImpl: async () => new Response("", { status: 500 }) })); +}); + +test("verifyKeyFileLive requires the exact key text at both locations", async () => { + const good = async () => new Response(`${INDEX_NOW_KEY}\n`, { status: 200 }); + const results = await verifyKeyFileLive({ fetchImpl: good }); + assert.equal(results.length, 2); + assert.ok(results.every((r) => r.ok)); + assert.ok(results.every((r) => r.detail === "key matches")); + + const mismatch = async (url) => + url === `https://${INDEX_NOW_HOST}${indexNowKeyFilePaths()[0]}` + ? new Response("wrong-key\n", { status: 200 }) + : new Response(`${INDEX_NOW_KEY}\n`, { status: 200 }); + const bad = await verifyKeyFileLive({ fetchImpl: mismatch }); + assert.equal(bad.filter((r) => r.ok).length, 1); +}); + +test("submitUrlList posts the payload and treats 200/202 as accepted", async () => { + const seen = []; + const fetchImpl = async (url, opts) => { + seen.push({ url, body: JSON.parse(opts.body) }); + return new Response("", { status: url.includes("bing.com") ? 202 : 200 }); + }; + const urls = [`https://${INDEX_NOW_HOST}/`, `https://${INDEX_NOW_HOST}/check`]; + const outcomes = await submitUrlList(urls, { fetchImpl }); + assert.equal(seen.length, INDEX_NOW_ENDPOINTS.length); + assert.ok(outcomes.every((o) => o.ok)); + assert.ok(seen.every((s) => s.body.urlList.length === 2 && s.body.key === INDEX_NOW_KEY)); + + const rejecting = await submitUrlList(urls, { + fetchImpl: async () => new Response("", { status: 403 }) + }); + assert.ok(rejecting.every((o) => !o.ok)); +}); + +test("main refuses to submit when the key file is not live yet (exit code 2)", async () => { + const fetchImpl = async () => new Response("SPA fallback", { status: 200 }); + const result = await main({ fetchImpl }); + assert.equal(result.ok, false); + assert.equal(result.code, 2); + assert.ok(result.keyChecks.every((c) => !c.ok)); +}); + +test("main dry-run submits nothing and stays green without a live key file", async () => { + let networkWrites = 0; + const locs = ROOT_PUBLIC_PATHS.map((p) => `https://${INDEX_NOW_HOST}${p}`); + const fetchImpl = async (url, opts) => { + if (opts) networkWrites += 1; + if (url.endsWith("/sitemap.xml")) return new Response(xmlSitemap(locs), { status: 200 }); + return new Response("SPA fallback", { status: 200 }); + }; + const result = await main({ dryRun: true, fetchImpl }); + assert.equal(result.code, 0); + assert.equal(networkWrites, 0); + assert.equal(result.dryRun, true); + assert.equal(result.urls.length, ROOT_PUBLIC_PATHS.length); +}); diff --git a/server/index.js b/server/index.js index 4fba189..69d8ece 100644 --- a/server/index.js +++ b/server/index.js @@ -6,6 +6,7 @@ import { fileURLToPath } from "node:url"; import { chromium } from "playwright"; import { auditUrl } from "./audit/engine.js"; import { rootSitemap } from "../shared/audit-engine.js"; +import { indexNowKeyFileBody, indexNowKeyFilePaths } from "../shared/index-now.js"; import { buildWhiteLabelReportHtml, defaultBranding, @@ -2529,6 +2530,11 @@ app.get("/robots.txt", (req, res) => { res.type("text").send(`User-agent: *\nAllow: /\n\nSitemap: ${origin}/sitemap.xml\n`); }); +// IndexNow key file parity with the Worker (production serves both paths). +app.get([...indexNowKeyFilePaths()], (req, res) => { + res.set("x-robots-tag", "noindex, nofollow").type("text").send(indexNowKeyFileBody()); +}); + app.get("/sitemap.xml", (req, res) => { const origin = `http://${req.get("host")}`; res.type("xml").send(rootSitemap(origin)); diff --git a/shared/audit-engine.js b/shared/audit-engine.js index fbb7eaa..c14958d 100644 --- a/shared/audit-engine.js +++ b/shared/audit-engine.js @@ -3021,9 +3021,13 @@ function wait(ms) { return new Promise((resolve) => setTimeout(resolve, ms)); } +// The canonical public route set for the apex host, shared by the sitemap, +// IndexNow submissions, and any future search-submission surface. Keep in sync +// with the public page routes in worker/index.js and worker/routes/pages.js. +export const ROOT_PUBLIC_PATHS = ["/", "/demo", "/check", "/methodology", "/packages", "/small-business-seo-audit", "/rendered-vs-static-seo-audit", "/ai-answer-readiness", "/privacy", "/proof", "/support", "/terms"]; + export function rootSitemap(origin) { - const urls = ["/", "/demo", "/check", "/methodology", "/packages", "/small-business-seo-audit", "/rendered-vs-static-seo-audit", "/ai-answer-readiness", "/privacy", "/proof", "/support", "/terms"]; - return `\n${urls - .map((path) => `${origin}${path}`) - .join("")}`; + return `\n${ROOT_PUBLIC_PATHS.map( + (path) => `${origin}${path}` + ).join("")}`; } diff --git a/shared/index-now.js b/shared/index-now.js new file mode 100644 index 0000000..4c40a53 --- /dev/null +++ b/shared/index-now.js @@ -0,0 +1,46 @@ +// IndexNow (Bing/Naver/Seznam/Yandex instant indexing) support. +// +// IndexNow needs no account or credentials: a site hosts a key file at +// `/{key}.txt` (root, or `/.well-known/{key}.txt`), and anyone can then submit +// the site's URLs to the IndexNow endpoints. Bing picks those URLs up for +// crawling on its schedule, and DuckDuckGo's index is Bing-derived, so this is +// the credential-free leg of the search-index coverage item. Google does not +// participate in IndexNow; the Google leg is Search Console request-indexing +// (owner credentials) plus ordinary sitemap re-crawling. +// +// The key is intentionally a committed, world-readable value: the IndexNow +// spec requires the key file to be fetchable by anyone, and the key itself is +// not a security credential (https://www.indexnow.org/documentation). It only +// has to be stable and unique to this host. +export const INDEX_NOW_KEY = "3219d564f9f914772e178f33ae543e60"; + +export const INDEX_NOW_ENDPOINTS = [ + "https://api.indexnow.org/indexnow", + "https://www.bing.com/indexnow" +]; + +export const INDEX_NOW_HOST = "seofixkit.com"; + +export function indexNowKeyFilePaths() { + return [`/${INDEX_NOW_KEY}.txt`, `/.well-known/${INDEX_NOW_KEY}.txt`]; +} + +export function indexNowKeyFileBody() { + return `${INDEX_NOW_KEY}\n`; +} + +export function indexNowKeyLocation() { + return `https://${INDEX_NOW_HOST}/${INDEX_NOW_KEY}.txt`; +} + +// The URL set submitted to IndexNow mirrors the live sitemap: every public +// route, apex-only, no query strings, no www (www 301s to apex so submitting +// it would be duplicate-URL noise). +export function buildIndexNowPayload(urls, { host = INDEX_NOW_HOST } = {}) { + return { + host, + key: INDEX_NOW_KEY, + keyLocation: indexNowKeyLocation(), + urlList: [...urls] + }; +} diff --git a/worker/index.js b/worker/index.js index 307c4e5..8d95d69 100644 --- a/worker/index.js +++ b/worker/index.js @@ -1,5 +1,6 @@ import { isEmailConfigured } from "../shared/fulfillment.js"; import { VERSION, rootSitemap } from "../shared/audit-engine.js"; +import { indexNowKeyFileBody, indexNowKeyFilePaths } from "../shared/index-now.js"; import { createAdminBetaSession, createInvite, @@ -614,6 +615,21 @@ export default { }); } + // IndexNow key file: served at both the spec root and the .well-known + // alias so Bing/Naver/Seznam/Yandex can validate the submission key + // without any account credentials. Both paths are in wrangler.jsonc + // run_worker_first so the SPA asset fallback never shadows them. + for (const keyPath of indexNowKeyFilePaths()) { + if (url.pathname === keyPath) { + return new Response(indexNowKeyFileBody(), { + headers: secureHeaders({ + "content-type": "text/plain; charset=utf-8", + "x-robots-tag": "noindex, nofollow" + }) + }); + } + } + if (url.pathname === "/sitemap.xml") { return new Response(rootSitemap(origin), { headers: secureHeaders({ "content-type": "application/xml; charset=utf-8" }) diff --git a/worker/index.test.mjs b/worker/index.test.mjs index 1188358..fdc5f03 100644 --- a/worker/index.test.mjs +++ b/worker/index.test.mjs @@ -262,6 +262,22 @@ test("Worker dispatch serves the real before/after repair proof receipt", async assert.match(await proofDotMd.text(), /^# SEO Fix Kit .* Repair proof receipt/m); }); +test("Worker dispatch serves the IndexNow key file at root and .well-known, apex-only", async () => { + const env = await fakeWorkerEnv(); + const { INDEX_NOW_KEY, indexNowKeyFilePaths } = await import("../shared/index-now.js"); + + for (const keyPath of indexNowKeyFilePaths()) { + const apex = await worker.fetch(new Request(`https://seofixkit.com${keyPath}`), env, fakeCtx()); + assert.equal(apex.status, 200); + assert.equal((await apex.text()).trim(), INDEX_NOW_KEY); + assert.match(apex.headers.get("x-robots-tag") || "", /noindex/); + + const www = await worker.fetch(new Request(`https://www.seofixkit.com${keyPath}`), env, fakeCtx()); + assert.equal(www.status, 301); + assert.equal(www.headers.get("location"), `https://seofixkit.com${keyPath}`); + } +}); + test("Worker dispatch exposes public-safe deep health readiness", async () => { const env = await fakeWorkerEnv(); const response = await worker.fetch(new Request("https://seofixkit.test/api/deep-health"), env, fakeCtx());