docs: improve LLM discovery surfaces

[Logan1x]

Summary

• Add repo-root llms.txt for GitHub/raw and agent discovery
• Mirror llms.txt, llms-full.txt, robots.txt, and sitemap.xml at the site root on deploy
• Surface the root and docs LLM entry points in the docs homepage and README

Verification

• Ran npm ci and npm run build in website/ successfully
• Build completes with pre-existing docs warnings, but the site artifacts are generated

Notes

• I couldn't push directly to the upstream repo from this environment, so the change is opened from my fork for review/merge.

[alt-glitch]

This was generated by AI during triage.

Related to the llms.txt discovery lineage: merged #18276 already publishes website/static/llms.txt (#42398 restructured it), and closed #48750 previously attempted a root-level llms.txt + link-validation CI. Not a duplicate — this adds a NEW repo-root llms.txt, deploy-time mirroring of llms/robots/sitemap at the site root, a generation script, and README/docs surfacing. Also related to open #28505 (bundle llms-full.txt as a self-documentation skill — a different surface). Worth de-conflicting the root vs website/static/ copy so they don't drift.

[teknium1]

Thanks for improving discovery surfaces. Current main already publishes generated llms.txt and llms-full.txt at both the root and /docs/ via .github/workflows/deploy-site.yml:164-173, and documents those root URLs in website/docs/index.mdx:134-141.

Problems

• The new llms.txt:35 says the live site mirrors that file, but deployment copies the generated website/build/llms.txt; the generator owns that content in website/scripts/generate-llms-txt.py:197-232. This creates a second independently maintained index and makes the stated mirroring relationship incorrect. The existing member comment correctly flagged this drift risk.

Suggested changes

• Rework the repo-root and deployed indexes to use one canonical source, or make the repository-root file an explicit pointer rather than a competing curated copy.
• Salvage the robots/sitemap and README work only after reconciling with the root LLM publication already on main.

Automated hermes-sweeper review.

[teknium1]

The deploy workflow copies generated website/build/llms.txt, not this independently maintained root file (.github/workflows/deploy-site.yml:164-173). Please make one source canonical or change this statement; otherwise the two indexes can drift immediately.