[docs] [docs infra] Left-hand side nav loading is slow

[angelinalg]

Description

I may have complained about this several times, but I really hate how our doc navigation works. You have to click a parent page, wait for it to load, and then check out the child pages.
this is super slow and it sometimes shifts the UI up and down
other products’ docs have a way to let people expand/collapse quickly to navigate.

What we want is that the drop-down carat expands/collapses the drop-down without loading the page.

We currently need to refresh the page on-click, because we can't generate the whole nav for every page on our docs.
if you remember, introducing detailed API reference pages blew up the number of documents we have on our docs and having a full nav for every single document means that we incur docs build times of about 40-60 minutes, which is infeasible for readthedocs.com (and frankly for us, too).
we've gone through several options many times, and within the standard sphinx ecosystem there's not much we can do here, unfortunately.
what we can do for sure is to separately generate a LHS nav bar and embed it at runtime. this would not work with vanilla sphinx, but would instead be a rather invasive/custom procedure.

Link

No response

[angelinalg]

Assigning to @peytondmurray to investigate, not necessarily fix.

[peytondmurray]

I may have complained about this several times, but I really hate how our doc navigation works. You have to click a parent page, wait for it to load, and then check out the child pages.
this is super slow and it sometimes shifts the UI up and down
other products’ docs have a way to let people expand/collapse quickly to navigate.

What we want is that the drop-down carat expands/collapses the drop-down without loading the page.

Since #41115 got merged, we now have a sidebar that has more sensible navigation behavior, so I'm going to close this issue. There's a another issue here that hasn't been addressed, though:

We currently need to refresh the page on-click, because we can't generate the whole nav for every page on our docs.
if you remember, introducing detailed API reference pages blew up the number of documents we have on our docs and having a full nav for every single document means that we incur docs build times of about 40-60 minutes, which is infeasible for readthedocs.com (and frankly for us, too).
we've gone through several options many times, and within the standard sphinx ecosystem there's not much we can do here, unfortunately.

This is hugely irritating to me right now. Sphinx's read/write performance is so slow that it's IMO totally unreasonable for a project of this size. There's absolutely no reason why reading some text and generating some HTML from it should take 50 minutes unless you're parsing gigabytes of files. I'd really like to write a sphinx builder that is a compiled python extension that can do this faster. Given that the builtin HTML builder is only ~1000 lines of Python, this wouldn't be a huge project - @angelinalg let me know if you'd be interested in pursuing this.

On my local machine builds complete in just over 3 minutes (the job is run in parallel), but on RTD they take ~50 😞 (because by default RTD build jobs run serially). As a stopgap measure, I want to try to enable multi-process builds on RTD. I'll make an issue for this right now.

what we can do for sure is to separately generate a LHS nav bar and embed it at runtime. this would not work with vanilla sphinx, but would instead be a rather invasive/custom procedure.

This is in fact the approach taken in #41115, where the primary sidebar is cached the first time an HTML page is generated, then reused for every other page (we still need to update which sections automatically open up on each page, but that's a relatively "small" overhead).