Repository navigation
Expand search and answer-engine discovery - #8339
Merged
Merged
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe changes add localized RSS feeds and discovery links, introduce scheduled IndexNow submission for recently modified sitemap URLs, centralize message loading, update crawler permissions, and adjust agent-readable page metadata. ChangesSearch discovery
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Cron
participant IndexNowRoute
participant Sitemap
participant IndexNowHelpers
participant IndexNowEndpoint
Cron->>IndexNowRoute: Authorized GET request
IndexNowRoute->>Sitemap: Select recently modified URLs
IndexNowRoute->>IndexNowHelpers: Build and submit payload
IndexNowHelpers->>IndexNowEndpoint: POST IndexNow JSON
IndexNowEndpoint-->>IndexNowRoute: Return submission status
Possibly related PRs
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (2 errors, 2 warnings)
✅ Passed checks (21 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Greptile SummaryThis PR wires up three search-engine discovery improvements: locale-aware RSS feeds served at
Confidence Score: 5/5Safe to merge. All three new surfaces (RSS routes, IndexNow cron, robots.ts) are additive and independently covered by tests; the refactored deepMergeMessages extraction is a straight code move with no behavioral change. The RSS builder correctly escapes XML special characters in all user-provided text fields. The IndexNow selection logic is anchored to sitemap dates rather than wall-clock time, which is intentional and covered by tests including a delayed-deployment scenario. The i18n autodiscovery title is properly routed through getTranslations. The two-variable naming scheme (CRON_SECRET in Vercel, INDEXNOW_TRIGGER_SECRET in GitHub Actions) is documented explicitly in the migration notes as needing the same value, so the system functions correctly when configured as instructed. No files require special attention. The migration notes require two secrets to be configured (CRON_SECRET in Vercel and INDEXNOW_TRIGGER_SECRET in GitHub) with matching values before the post-deployment trigger works end-to-end. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant GHA as GitHub Actions
participant Cron as /api/cron/indexnow
participant Lib as indexnow.ts
participant SM as sitemap()
participant API as api.indexnow.org
GHA->>Cron: POST (Authorization header)
Cron->>Cron: validate CRON_SECRET
Cron->>SM: sitemap()
SM-->>Cron: SitemapEntry[]
Cron->>Lib: recentlyModifiedUrls(entries, now)
Lib-->>Cron: string[] (anchored to newest date minus 48h)
alt "urls.length == 0"
Cron-->>GHA: 200 ok submitted 0
else "urls.length > 0"
Cron->>Lib: submitIndexNowUrls(urls)
Lib->>API: POST /indexnow (AbortSignal 10s)
API-->>Lib: HTTP status
Lib-->>Cron: status code
Cron-->>GHA: 200 ok submitted N
end
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant GHA as GitHub Actions
participant Cron as /api/cron/indexnow
participant Lib as indexnow.ts
participant SM as sitemap()
participant API as api.indexnow.org
GHA->>Cron: POST (Authorization header)
Cron->>Cron: validate CRON_SECRET
Cron->>SM: sitemap()
SM-->>Cron: SitemapEntry[]
Cron->>Lib: recentlyModifiedUrls(entries, now)
Lib-->>Cron: string[] (anchored to newest date minus 48h)
alt "urls.length == 0"
Cron-->>GHA: 200 ok submitted 0
else "urls.length > 0"
Cron->>Lib: submitIndexNowUrls(urls)
Lib->>API: POST /indexnow (AbortSignal 10s)
API-->>Lib: HTTP status
Lib-->>Cron: status code
Cron-->>GHA: 200 ok submitted N
end
Reviews (7): Last reviewed commit: "fix(web): bound IndexNow sitemap selecti..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/app/layout.tsx`:
- Around line 38-45: Update the metadata export in the layout to define the RSS
alternate through the Metadata API’s alternates.types configuration, then remove
the manual head element and link. Preserve the existing metadata while using the
application/rss+xml key and the current feed URL.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 395b3341-5ec7-4517-bef5-fdf49cd9e3da
📒 Files selected for processing (12)
web/app/[locale]/components/blog-posts.tsweb/app/api/cron/indexnow/route.tsweb/app/feed.xml/route.tsweb/app/layout.tsxweb/app/lib/agent-page-paths.tsweb/app/lib/blog-feed.tsweb/app/lib/indexnow.tsweb/app/robots.tsweb/tests/agent-page-variants.test.tsweb/tests/search-discovery.test.tsweb/tests/seo.test.tsweb/vercel.json
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/app/layout.tsx`:
- Line 44: Update the title construction near blogTranslations("layoutTitle") to
localize the RSS suffix through the translation catalog, using a localized
rssLabel/rssSuffix message or a fully localized alternate title. Preserve the
existing layout title while ensuring non-English pages never append the
hardcoded English “RSS”.
In `@web/app/lib/blog-feed.ts`:
- Line 2: Update buildBlogRssFeed and its callers, including the feed.xml route,
so RSS titles and entries use the requested locale’s messages and catalog data
instead of always importing englishMessages. If the feed remains English-only,
remove localized-feed promotion and clearly scope the route accordingly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: b6b592a2-ecd4-4564-ab76-98ddb23cc6c6
📒 Files selected for processing (2)
web/app/layout.tsxweb/app/lib/blog-feed.ts
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/app/feed.xml/route.ts`:
- Around line 1-13: Update the GET function in the root feed route to pass
routing.defaultLocale to buildLocalizedBlogRssFeed instead of the hardcoded
"en", and import routing from the existing routing configuration so the
generated content and self-links follow the configured default locale.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 55c56101-cf77-4a31-9358-a18c958680eb
📒 Files selected for processing (8)
web/app/[locale]/feed.xml/route.tsweb/app/feed.xml/route.tsweb/app/layout.tsxweb/app/lib/blog-feed.tsweb/app/lib/localized-blog-feed.tsweb/i18n/messages.tsweb/i18n/request.tsweb/tests/search-discovery.test.ts
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@web/app/lib/indexnow.ts`:
- Around line 16-31: Update the modifiedEntries processing to avoid Math.max
spread and reduce repeated collection passes: use one pass over entries to
retain eligible entries while tracking newestModification incrementally, then a
second pass to select URLs within the computed lookback window. Preserve the
existing date validity, latest cutoff, URL conversion, and empty-collection
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 925b608e-338f-476f-9d99-98166d088db8
📒 Files selected for processing (2)
web/app/lib/indexnow.tsweb/tests/search-discovery.test.ts
Contributor
Author
|
Live Vercel preview: https://cmux-hpgh9kmse-manaflow.vercel.app Validated on the hosted deployment:
Both feeds parse as XML; English has 19 items, Japanese has 16; localized autodiscovery and canonical feed URLs are correct; crawler groups and sitemap are present; stale Base variants are absent; unauthenticated IndexNow POST returns 401. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
Why:
Tests:
Dogfood tag: aeoseo
Need help on this PR? Tag
/codesmithwith what you need. Autofix is disabled.Summary by cubic
Expand search and answer‑engine discovery with localized RSS feeds, open crawl access, and IndexNow submissions on a daily cron and after production deploys. The root feed mirrors the default locale for consistent canonical URLs and metadata; improves indexing across Bing/DuckDuckGo and Yandex.
New Features
/_next./feed.xml(default locale) and/<locale>/feed.xml; add a localized<link rel="alternate" type="application/rss+xml">in<head>with translated labels; setContent-Language; per‑locale canonical blog URLs./docs/basefromllms.txt; add an “Updates” link to the RSS feed.Migration
CRON_SECRETfor the/api/cron/indexnowendpoint in Vercel. Configure a GitHub repository secretINDEXNOW_TRIGGER_SECRETwith the same value for the workflow trigger.https://cmux.com/82cc8125a8624a4db9e07502db0b7d46.txt.Written for commit b21a24a. Summary will update on new commits.
Summary by CodeRabbit
/docs/basevariant links from agent-readable outputs.