feat: ship credential-free IndexNow submission path for Bing/DDG search-index coverage #99
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| # SEO Fix Kit — full search-index coverage (Google + Bing/DuckDuckGo) | ||
|
|
||
| Packet for the lane-1 item: **Establish full search-index coverage: all 7 public | ||
| pages indexed on Google plus a Bing/DuckDuckGo presence** | ||
| (backlog owner: | ||
| `/home/nish/workspaces/agent-state/seo-fix-kit-improvement-loop/backlog.md`, | ||
| [scout 2026-08-09, rank: 2, risk: green]). | ||
|
Comment on lines
+5
to
+7
There was a problem hiding this comment. 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win Remove the local workstation path. This public document exposes a developer identifier and local directory layout. Replace it with a repository-relative backlog reference or an issue identifier. 🤖 Prompt for AI Agents |
||
|
|
||
| Status: **agent-completable half shipped, engine half needs owner credentials + | ||
| time.** This packet (2026-08-11) re-verified the gap live, shipped the | ||
| credential-free IndexNow submission path (key file routes in the Worker, | ||
| submission script, tests, wrangler routing) that gets Bing — and therefore | ||
| DuckDuckGo — crawling all public URLs the moment the release lands, and | ||
|
Comment on lines
+12
to
+13
There was a problem hiding this comment.
When the release is deployed but AGENTS.md reference: AGENTS.md:L7-L7 Useful? React with 👍 / 👎. |
||
| documented the two owner-only legs (Google Search Console request-indexing, | ||
| Bing Webmaster ownership) as an exact manual kit. Search-engine indexing is | ||
| externally owned and time-dependent; the acceptance outcome ("all 7 pages on | ||
| Google, seofixkit.com first on Bing/DDG") cannot be produced from a lane | ||
| without credentials, but every lever that does not need credentials is now | ||
| built and one command away. | ||
|
Comment on lines
+9
to
+19
There was a problem hiding this comment. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🧩 Analysis chain🌐 Web query:
💡 Result: The official IndexNow documentation and FAQ explicitly state that a successful submission does not guarantee crawling or indexing [1][2]. When a URL is submitted, search engines receive an HTTP 200 response, which only confirms that the search engine has received the notification [3][4]. After receiving a submission, a search engine evaluates the URL based on its own criteria, including crawl quota, scheduling logic, and content quality signals [1][2]. IndexNow serves to alert search engines that content has changed, which increases the likelihood of faster discovery and prioritization compared to pages not submitted through the protocol, but final decisions on whether to crawl or index remain at the discretion of the search engine's algorithms [1][2]. Regarding DuckDuckGo, the official IndexNow documentation does not list it as a direct participant [5]. However, because DuckDuckGo incorporates results from Bing’s index, it benefits indirectly from IndexNow submissions made to Bing [6][7][8]. Therefore, while DuckDuckGo is not a participant that independently receives and processes IndexNow pings, updates submitted to IndexNow-participating engines like Bing can still be reflected in DuckDuckGo search results [9][7]. Citations:
🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- target document ---'
sed -n '1,150p' docs/growth/search-index-coverage-2026-08-11.md
printf '%s\n' '--- related README wording ---'
rg -n -i -C 3 'IndexNow|DuckDuckGo|Bing.*crawl|crawl.*Bing|all 7 pages|seofixkit' --glob 'README*' --glob '*.md' .Repository: nish3451/seo-fix-kit Length of output: 50377 Qualify IndexNow and DuckDuckGo outcomes IndexNow acceptance confirms receipt of a notification. It requests discovery but does not guarantee crawling or indexing. Remove claims that IndexNow gets Bing or DuckDuckGo crawling, that DuckDuckGo follows from Bing, or that IndexNow owns first-result presence. State that Bing and DuckDuckGo independently decide whether to crawl and index URLs. Apply the same correction to any README copy. 🤖 Prompt for AI Agents |
||
|
|
||
| ## Fresh live evidence (2026-08-11, ~10:40 IST) | ||
|
|
||
| - **Google: homepage only.** Startpage (a live Google-index proxy) for | ||
| `site:seofixkit.com` returns exactly one organic result: | ||
| `https://seofixkit.com` "SEO Fix Kit - Proof-Backed SEO Repair Beta". Per-path | ||
| probes `site:seofixkit.com/{terms,demo,methodology,packages,check,privacy,support}` | ||
| all return zero results. Direct Google is CAPTCHA-walled from this VPS | ||
| (`/sorry/index`, screenshot `/tmp/serde2-google_gbv1_site.png`), so Startpage | ||
| is the reproducible Google-index receipt. Backlog evidence (2026-08-09, real | ||
| anti-detection browser) showed `/` + `/terms`; the `/terms` result has since | ||
| dropped out or been consolidated — either way, 7 of 8 public routes remain | ||
| unindexed on Google. | ||
| - **Bing: zero presence.** Real-browser `bing.com/search?q=seofixkit.com` | ||
| returns ~82,200 results, none of them seofixkit.com (all unrelated | ||
| "Inn at the Market" hotel pages — Bing is fuzzy-matching the query text). | ||
| Screenshot `/tmp/seo-serde-bing_domain.png`. | ||
| - **DuckDuckGo: blocked this run, zero by construction.** DDG html/lite/main | ||
| endpoints all bot-walled this IP (418/error page; screenshots | ||
| `/tmp/serde2-duckduckgo_main.png`). DDG's web index is Bing-derived, and | ||
| prior real-browser evidence (2026-08-09) recorded a zero-results SERP for | ||
| `seofixkit.com`. No change in DDG coverage is possible before Bing crawls. | ||
| - **Site surfaces are crawl-ready:** `https://seofixkit.com/robots.txt` HTTP 200 | ||
| (`Allow: /`, `Sitemap: https://seofixkit.com/sitemap.xml`); | ||
| `https://seofixkit.com/sitemap.xml` HTTP 200 listing 8 public URLs | ||
| (`/`, `/demo`, `/check`, `/methodology`, `/packages`, `/privacy`, `/support`, | ||
| `/terms` — the item said 7; `/check` was added to the sitemap after the item | ||
| was filed, so the target set is now 8). Every page has unique title/meta/OG/ | ||
| canonical; homepage carries Organization/WebSite/SoftwareApplication/FAQPage | ||
| schema; `/check` carries WebPage/FAQ JSON-LD. | ||
| - **No ownership artifacts observable:** DNS TXT on seofixkit.com carries only | ||
| SPF (`v=spf1 include:_spf.porkbun.com include:_spf.mx.cloudflare.net ~all`); | ||
| no `google-site-verification` meta, no `bing-site-verification` meta, no | ||
| BingSiteAuth.xml, no IndexNow key file live (both candidate paths return the | ||
| SPA fallback HTML with body mismatch — the exact bug `run_worker_first` | ||
| routing fixes in this packet). | ||
|
|
||
| ## What this packet ships | ||
|
|
||
| - `shared/index-now.js` — committed IndexNow key (`3219d564f9f914772e178f33ae543e60`), | ||
| key file paths, payload builder, endpoints (`api.indexnow.org`, | ||
| `www.bing.com/indexnow`). Key is world-readable by spec, not a credential. | ||
| - `shared/audit-engine.js` — hoisted the canonical route list to | ||
| `ROOT_PUBLIC_PATHS`, shared by `rootSitemap()` and IndexNow so the sitemap | ||
| and the submission set can never drift. | ||
| - `worker/index.js` — serves `GET /{key}.txt` and `GET /.well-known/{key}.txt` | ||
| (text/plain, `x-robots-tag: noindex`), apex-only like every other public | ||
| surface; www requests keep 301ing to apex. | ||
| - `wrangler.jsonc` — both key paths added to `run_worker_first` so the SPA | ||
| asset fallback cannot shadow them (verified live: without this, both paths | ||
| currently return the SPA HTML with HTTP 200). | ||
| - `server/index.js` — local dev-server parity for both key paths. | ||
| - `scripts/submit-indexnow.mjs` + `scripts/submit-indexnow.test.mjs` — | ||
| `node scripts/submit-indexnow.mjs` verifies the key file is live at both | ||
| locations (refuses otherwise, exit 2), parses the live sitemap locs, POSTs | ||
| the payload to both endpoints, and reports per-endpoint accept/reject | ||
| (exit 0/3). `--dry-run` previews with no network writes. | ||
| - `package.json` — `submit:indexnow` script + `test:indexnow` wired into the | ||
| canonical `check` gate. | ||
| - `worker/index.test.mjs` — apex key file 200 with exact key body + noindex, | ||
| www 301 to apex, for both paths. | ||
|
|
||
| ## Owner manual legs (credentials required — cannot be done by an agent) | ||
|
|
||
| Both are one-time, copy-paste steps that materially accelerate the outcome: | ||
|
|
||
| 1. **Google Search Console — request indexing (accelerates Google re-crawl).** | ||
| - Add property `https://seofixkit.com` (URL-prefix) at | ||
| https://search.google.com/search-console (any Google account). | ||
| - Verify: DNS TXT record `google-site-verification=...` at Porkbun (or the | ||
| HTML meta tag; DNS is preferred and survives redeploys). | ||
| - Submit `https://seofixkit.com/sitemap.xml` in Sitemaps. | ||
| - Open URL Inspection for each of the 8 sitemap URLs and click | ||
| "Request indexing" (spread over a few days; Google throttles). | ||
| - Expected outcome: `site:seofixkit.com` on Google returns all 8 pages. | ||
| Without GSC, Google re-crawls on its own schedule via the sitemap and | ||
| the homepage's internal links (already shipped: `/check` is linked from | ||
| `/`, `/demo`, `/packages`, `/methodology`, `/support`, `/terms`, | ||
| `/privacy`). | ||
| 2. **Bing Webmaster Tools — ownership + sitemap (long-term Bing health).** | ||
| - Add site at https://www.bing.com/webmasters (Microsoft account). | ||
| - Verify: DNS TXT `ms=...` at Porkbun. | ||
| - Submit `https://seofixkit.com/sitemap.xml`. | ||
| - IndexNow (shipped here) covers the crawl trigger; Webmaster Tools adds | ||
| the dashboard, crawl logs, and index coverage reports. | ||
|
|
||
| ## Resume path (agent-completable, after this PR is merged AND released) | ||
|
|
||
| 1. Confirm release landed: `curl -s https://seofixkit.com/{key}.txt` returns | ||
| the key text (not the SPA HTML). | ||
| 2. `node scripts/submit-indexnow.mjs` — verify 2x "key matches", 8 URLs listed, | ||
| ACCEPTED on both endpoints. Bing crawls on its schedule (typically minutes | ||
| to hours); DDG follows from Bing's index. | ||
| 3. Re-run the verification queries from "Fresh live evidence" above; record | ||
| receipts in this file under a "Re-verified" section. Google's leg moves | ||
| only via owner step 1 or its own re-crawl cadence. | ||
|
|
||
| ## Acceptance / verification mapping | ||
|
|
||
| - Bing/DDG first-result presence: owned by IndexNow submission (this packet) | ||
| + crawler schedule; externally observable via | ||
| `bing.com/search?q=seofixkit.com` and `duckduckgo.com/?q=seofixkit.com`. | ||
| - All public pages on Google: owned by Search Console request-indexing (owner) | ||
| + sitemap re-crawl; externally observable via `site:seofixkit.com`. | ||
| - No index bloat / duplicate-host junk: already enforced — www 301s to apex, | ||
| every emitted URL (canonical, og:url, sitemap, robots, key file) is | ||
| apex-only; submission set is exactly the sitemap set with no query strings. | ||
| - Rollback: remove the worker key-file routes + `run_worker_first` entries and | ||
| delete `shared/index-now.js`; no product surface or public copy changes. | ||
|
|
||
| ## Files changed in this packet | ||
|
|
||
| - `shared/index-now.js` (new), `scripts/submit-indexnow.mjs` (new), | ||
| `scripts/submit-indexnow.test.mjs` (new), `docs/growth/search-index-coverage-2026-08-11.md` (this file) | ||
| - `shared/audit-engine.js`, `worker/index.js`, `worker/index.test.mjs`, | ||
| `server/index.js`, `wrangler.jsonc`, `package.json` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| // Submit the live sitemap URL set to IndexNow (Bing/Naver/Seznam/Yandex). | ||
| // | ||
| // Usage: | ||
| // node scripts/submit-indexnow.mjs # verify key file live, then submit | ||
| // node scripts/submit-indexnow.mjs --dry-run # print what would happen, no network writes | ||
| // | ||
| // Requirements: the IndexNow key file must already be live on the production | ||
| // host (the worker route in worker/index.js serves it once the repo change is | ||
| // released). The script refuses to submit until both key-file locations return | ||
| // the exact key text, because a submission with an unreachable key is silently | ||
| // discarded by the engines. | ||
| // | ||
| // Exit codes: 0 = accepted by every endpoint, 2 = key file not live yet, | ||
| // 3 = one or more endpoints rejected the submission. | ||
| import { INDEX_NOW_ENDPOINTS, INDEX_NOW_HOST, indexNowKeyFileBody, indexNowKeyFilePaths, buildIndexNowPayload } from "../shared/index-now.js"; | ||
|
|
||
| const ORIGIN = `https://${INDEX_NOW_HOST}`; | ||
|
|
||
| export function parseSitemapLocs(xml) { | ||
| const locs = []; | ||
| const re = /<loc>([^<]+)<\/loc>/g; | ||
| let m; | ||
| while ((m = re.exec(xml)) !== null) locs.push(m[1]); | ||
| return locs; | ||
| } | ||
|
|
||
| export async function fetchSitemapUrls({ fetchImpl = globalThis.fetch } = {}) { | ||
| const res = await fetchImpl(`${ORIGIN}/sitemap.xml`); | ||
| if (!res.ok) throw new Error(`sitemap fetch failed: HTTP ${res.status}`); | ||
| const locs = parseSitemapLocs(await res.text()); | ||
| if (locs.length === 0) throw new Error("sitemap returned zero <loc> entries"); | ||
| return locs; | ||
| } | ||
|
|
||
| export async function verifyKeyFileLive({ fetchImpl = globalThis.fetch } = {}) { | ||
| const expected = indexNowKeyFileBody().trim(); | ||
| const results = []; | ||
| for (const path of indexNowKeyFilePaths()) { | ||
| const url = `${ORIGIN}${path}`; | ||
| let ok = false; | ||
| let detail = ""; | ||
| try { | ||
| const res = await fetchImpl(url); | ||
| const body = (await res.text()).trim(); | ||
| ok = res.status === 200 && body === expected; | ||
| detail = ok ? "key matches" : `HTTP ${res.status}, body mismatch`; | ||
| } catch (err) { | ||
| detail = `fetch failed: ${err.message}`; | ||
| } | ||
| results.push({ url, ok, detail }); | ||
| } | ||
| return results; | ||
| } | ||
|
|
||
| export async function submitUrlList(urlList, { endpoints = INDEX_NOW_ENDPOINTS, fetchImpl = globalThis.fetch } = {}) { | ||
| const payload = buildIndexNowPayload(urlList); | ||
| const outcomes = []; | ||
| for (const endpoint of endpoints) { | ||
| try { | ||
| const res = await fetchImpl(endpoint, { | ||
| method: "POST", | ||
| headers: { "content-type": "application/json; charset=utf-8" }, | ||
| body: JSON.stringify(payload) | ||
| }); | ||
| outcomes.push({ endpoint, status: res.status, ok: res.status === 200 || res.status === 202 }); | ||
| } catch (err) { | ||
| outcomes.push({ endpoint, status: 0, ok: false, error: err.message }); | ||
| } | ||
| } | ||
| return outcomes; | ||
|
Comment on lines
+55
to
+70
There was a problem hiding this comment. 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win 🧩 Analysis chain🌐 Web query:
💡 Result: According to the official IndexNow FAQ, you should submit your URL-list payload to only one of the participating IndexNow endpoints [1][2]. The protocol is designed such that all search engines adopting IndexNow have agreed that URLs submitted to any one participating endpoint will be automatically shared with all other participating search engines [3][4][5]. Therefore, submitting the same list to multiple endpoints is unnecessary [6]. The FAQ explicitly states: "You may submit your request to only one of the following participating endpoints. Each endpoint sends your submission directly to its respective search engine, and your submission will be shared across all IndexNow-enabled search engines" [1][2]. Using the global IndexNow endpoint (api.indexnow.org) is a common and efficient way to reach all participating engines with a single request [7][8][9]. Citations:
🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- target file ---'
cat -n scripts/submit-indexnow.mjs | sed -n '1,140p'
printf '%s\n' '--- related references ---'
rg -n -S --glob '!node_modules' 'INDEX_NOW_ENDPOINTS|submitUrlList|IndexNow|api\.indexnow\.org|www\.bing\.com/indexnow|status.*3' .Repository: nish3451/seo-fix-kit Length of output: 18101 🏁 Script executed: #!/bin/bash
set -eu
printf '%s\n' '--- shared IndexNow contract ---'
cat -n shared/index-now.js | sed -n '1,100p'
printf '%s\n' '--- submit-indexnow tests ---'
cat -n scripts/submit-indexnow.test.mjs | sed -n '1,130p'
printf '%s\n' '--- operational documentation ---'
cat -n README.md | sed -n '80,105p'
cat -n docs/growth/search-index-coverage-2026-08-11.md | sed -n '48,70p;95,125p'Repository: nish3451/seo-fix-kit Length of output: 13128 🏁 Script executed: #!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re
source = Path("scripts/submit-indexnow.mjs").read_text()
shared = Path("shared/index-now.js").read_text()
endpoints = re.findall(r'"(https://[^"]+/indexnow)"', shared)
loop = re.search(r'for \(const endpoint of endpoints\)', source)
accepted = re.search(r'res\.status === 200 \|\| res\.status === 202', source)
exit_logic = re.search(r'code: allOk \? 0 : 3', source)
print("configured_endpoints:", endpoints)
print("default_endpoint_count:", len(endpoints))
print("submitUrlList_iterates_each_endpoint:", bool(loop))
print("200_or_202_are_accepted:", bool(accepted))
print("main_returns_3_when_not_all_outcomes_are_ok:", bool(exit_logic))
# Model the source-level aggregation for one accepted response and one rejection.
statuses = [200, 403]
outcomes = [{"status": s, "ok": s in (200, 202)} for s in statuses]
all_ok = all(item["ok"] for item in outcomes)
print("mixed_statuses:", statuses)
print("mixed_outcomes:", outcomes)
print("modeled_exit_code:", 0 if all_ok else 3)
PYRepository: nish3451/seo-fix-kit Length of output: 524 Submit through one IndexNow endpoint. IndexNow shares submissions across participating search engines. Configure one endpoint and do not require every endpoint to accept the payload. Update the shared contract, tests, and operational documentation. 🤖 Prompt for AI Agents |
||
| } | ||
|
|
||
| export async function main({ dryRun = false, fetchImpl = globalThis.fetch } = {}) { | ||
| const keyChecks = await verifyKeyFileLive({ fetchImpl }); | ||
| const keyLive = keyChecks.every((c) => c.ok); | ||
| console.log(`IndexNow key file check (${ORIGIN}):`); | ||
| for (const c of keyChecks) console.log(` ${c.ok ? "OK " : "MISS"} ${c.url} (${c.detail})`); | ||
|
|
||
| if (!keyLive && !dryRun) { | ||
| console.error( | ||
| "\nKey file is not live yet. Merge + release the worker change that serves the key file, then re-run this script." | ||
| ); | ||
| return { ok: false, code: 2, keyChecks }; | ||
| } | ||
|
|
||
| const urls = await fetchSitemapUrls({ fetchImpl }); | ||
| console.log(`\nSitemap URL set (${urls.length} URLs):`); | ||
| for (const u of urls) console.log(` ${u}`); | ||
|
|
||
| if (dryRun) { | ||
| console.log("\nDry run: would submit the above URL set to:"); | ||
| for (const e of INDEX_NOW_ENDPOINTS) console.log(` POST ${e}`); | ||
| return { ok: true, code: 0, keyChecks, urls, dryRun: true }; | ||
| } | ||
|
|
||
| const outcomes = await submitUrlList(urls, { fetchImpl }); | ||
| console.log("\nSubmission results:"); | ||
| let allOk = true; | ||
| for (const o of outcomes) { | ||
| allOk = allOk && o.ok; | ||
| console.log(` ${o.ok ? "ACCEPTED" : "REJECTED"} ${o.endpoint} (HTTP ${o.status}${o.error ? `, ${o.error}` : ""})`); | ||
| } | ||
| return { ok: allOk, code: allOk ? 0 : 3, keyChecks, urls, outcomes }; | ||
| } | ||
|
|
||
| if (import.meta.url === `file://${process.argv[1]}`) { | ||
| const dryRun = process.argv.includes("--dry-run"); | ||
| try { | ||
| const result = await main({ dryRun }); | ||
| process.exitCode = result.code; | ||
| } catch (err) { | ||
| console.error(`submit-indexnow failed: ${err.message}`); | ||
| process.exitCode = 1; | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
State the dry-run prerequisite accurately.
main()returns code 2 for a missing key file only whendryRunis false. With--dry-run, it logs failed key checks, fetches the sitemap, and returns code 0. State that the key file must be live before a real submission, not before a preview.🤖 Prompt for AI Agents