[Doc] Upgrade docs build system

[peytondmurray]

Why are these changes needed?

This PR upgrades the build system for the docs.

Related issue number

Closes #37944.

Will contain all the changes in #39766, but will be reviewed in pieces via PRs against this branch.

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[peytondmurray]

Todo

• Move the parse_navbar_config etc. to e.g. custom_directives.py
• Resolve conflicts with master

[peytondmurray]

According to RTD support the build is timing out (??). This takes 8m locally...troubleshooting now.

[peytondmurray]

Note: I had to reduce the version of watchfiles to 0.19.0 due to the premerge check. I'd argue the constraints should not be applied to the doc build requirements, as they're separate from the rest of the ray requirements.

Once click-contrib/sphinx-click#129 is merged, we can remove this dependency from the requirements-doc.txt.

[angelinalg]

• The Ask AI button doesn't have a clear border. It should also be black in non-dark-mode. The Ask AI button is black in dark mode, so perhaps the color setting is just reversed.

• The search bar changes size/flashes when you click on the side bar. (This is not a blocker, but we should polish it up if we can in a follow up if it's too big of a lift for this PR.)

Screen.Recording.2023-11-20.at.10.17.00.AM.mov

• The dark mode switch has three modes. Two of them seem to be the same dark mode. Is this intentional?
• We lost the Welcome to Ray page in the side bar, that links to the top level landing page. Overview also got a new child page, Learn more. I think Learn more is included in the Getting started page on master. Is there a reason this PR rearranges that?

[emmyscode]

• Some inconsistencies on the main landing page with padding and a permanent light background for the code samples.
• How does the user select the Ray version?
• Footer includes Sphinx version information; do we want this?
• The AskAI button has a very faint background so it doesn't look like a clickable button; missing Ray logo
• In the rightmost column on pages, there's an "Edit on Github" button; is that something we want?

[can-anyscale]

nit: can we add a line here https://github.com/ray-project/ray/blob/master/ci/pipeline/determine_tests_to_run.py#L276 something like:

elif changed_file == ".readthedocs.yaml":
  RAY_CI_DOC_AFFECTED = 1

It will help to get CI results quicker. The entire test suite is run on this change despite this is only a doc change.

[can-anyscale]

[JoshTanke]

@peytondmurray Was doc/source/ray-overview/eco-gallery.yml supposed to be deleted? Is there a new place to add integrations now?

[peytondmurray]

@JoshTanke Yes, it was. Those integrations were being generated from that yaml file and inserted into ray-overview/ray-libraries.rst. Rather than have the extra step, just modify ray-overview/ray-libraries.rst directly.

[JoshTanke]

@JoshTanke Yes, it was. Those integrations were being generated from that yaml file and inserted into ray-overview/ray-libraries.rst. Rather than have the extra step, just modify ray-overview/ray-libraries.rst directly.

makes sense, thank you for the info!