[Docs] Generate llms.txt Documentation for Improved LLM Accessibility

[crypdick]

Description

As LLMs increasingly contribute to code generation, debugging, and library usage, documentation ought to be machine-readable in addition to human-readable. The current documentation is not efficiently parsable by LLMs, which degrades performance of coding co-pilots and also makes it harder for future LLMs to be trained on Ray documentation.

Proposed Solution
Adopt the llms.txt convention to generate a standardized llms.txt file alongside existing documentation. This file would:

1. Provide a succinct, structured library documentation that is easily parsable by co-pilots.
2. Back-populated for previous library versions to ensure AI agents can reference version-specific documentation.

Use case

• LLM-Centric Development: By formatting documentation for LLMs, we reduce “hallucinations” and errors in AI-generated code, improving developer productivity. By having version-accurate llms txt files, we help co-pilots reliably code for any Ray version.
• Future-Proofing: by making our technical content easy to parse by LLMs, we ensure that future models have a better understanding of the Ray library.
• Automating tutorial maintenance: adopting this standard would make it easier to create self-healing tutorials, where agents automatically submit pull requests that fix tutorials when we make breaking changes to our API.