Use Sphinx for pep builds and better rendering

[willingc]

TODO:

• Deploy on RTD and GH-Pages
• Determine after both are deployed when should each be used. I could see RTD for PEP docs similar to devguide and gh-pages to perhaps integrate with Python dot org

[AA-Turner]

Update:

GH-pages is live at https://python.github.io/peps

Read the Docs has a PR at python/peps#2023

A

[AA-Turner]

Could this be assigned to me please? (It is also dependent on PEP 676, but I'll have to deal with the outcome either way!)
A

[encukou]

PEPs are now built by Sphinx and hosted on GH Pages. PR previews are being build on RTD. Congratulations and thank you!

What's left to do here?

[ericholscher]

Just curious why the hosting is done on GH pages, and not RTD since you already have a configuration there?

[CAM-Gerlach]

What's left to do here?

I don't think anything, unless we want to switch the hosting over to RTD. But at least at this time, there doesn't appear to be an obvious need to, and if so it probably won't happen for quite a while, since @AA-Turner is taking a break right now to focus on school after all his hard work getting PEP 676 over the finish line.

One point of annoyance is that the RTD deploy previews take a full 3.5 minutes to generate and upload, and with very limited real-time build feedback, over twice the time of the actual GH pages builds at 1.5 minutes, Using other services like Netlify, I've never had the deploy previews take more than a few seconds longer than the GHP build, and often they're even slightly faster (plus, they give real-time build output). I'm all for using a common service, and maybe I doth protest too much, but the developer experience wasn't nearly as good as I expected (and am used to, even with services much less tightly focused on this particular use case).

[ericholscher]

@CAM-Gerlach Appreciate the candid response. We're look into our builds being slower, as that's something we need to address. We're also working to improve the build system UX overall, and appreciate your ideas. Definitely no rush on any further action on this, was just curious the thought process.

We're working to improve our platform, but I did want to note that we're a small open source project, and not someone with hundreds of millions of dollars in funding. We are hoping that getting more projects like Python to use community infrastructure will improve the quality that we're able to provide and increase contribution to our platform. The overall goal is to have additional people investing in community-controlled infra, but it's an uphill battle.

[willingc]

What's the viability of separating out where things are built for preview and where things are hosted?

Could we use gh-pages for preview and RTD for hosting (and possibly a second preview as they work toward faster builds)? We've been very satisfied for RTD for all Jupyter projects. Just my 2 cents.

[pauleveritt]

Count me as someone that's ok with this project (temporarily?) paying a 2m "tax" on build time. Integration with RTD is one of the reasons I jumped in here.

[CAM-Gerlach]

@ericholscher See @hugovk and my comments in python/peps#2429 for some additional context and specific suggestions that should help hopefully speed things up on RTD's end to close to equivalent to what we can get on other services.

What's the viability of separating out where things are built for preview and where things are hosted?

Right now that's what we're doing already with the PEPs repo, except the opposite—using RTD for previews and GHP for test and production builds. The opposite is theoretically possible, but you'd have to do some hacks to get the preview page into a new unique subdirectory, RTD would still be a long pole on the checks going green (though less important, since you could review the preview in that time), and we'd still need GHA to test our build system itself by running the build with the appropriate flags (python -bb -X dev -W error -m sphinx -n -W -E -A --keep-going) and potentially show the full, colorized output (maybe RTD does, but I couldn't immediately find it).

[ericholscher]

@ericholscher See @hugovk and my python/peps#2429 (comment) in python/peps#2429 for some additional context and specific suggestions that should help hopefully speed things up on RTD's end to close to equivalent to what we can get on other services.

Great, thanks. I'll dive into it a bit more.

I'm fine keeping things where they are for now, and diving into getting the speed issues figured out, and then discuss how to move forward. Definitely don't want to make folks do more work or hack around things, and want to ensure that the platform fits the use case.

[willingc]

PEPs have been overhauled and are much more user friendly now. I'm going to close this as completed.

[humitos]

@CAM-Gerlach

One point of annoyance is that the RTD deploy previews take a full 3.5 minutes to generate and upload, and with very limited real-time build feedback, over twice the time of the actual GH pages builds at 1.5 minutes

I'd be curious if these times continue to be like that. We've improved our builds a lot since this comment was posted. Do you have a recent build on Read the Docs and GH Pages where I can see these times and compare against? 🙏🏼

@willingc

PEPs have been overhauled and are much more user friendly now. I'm going to close this as completed.

What's the workflow you are using? GH Pages for hosting and RTD for previews as talked in this issue or something different?

[hugovk]

One point of annoyance is that the RTD deploy previews take a full 3.5 minutes to generate and upload, and with very limited real-time build feedback, over twice the time of the actual GH pages builds at 1.5 minutes

I'd be curious if these times continue to be like that. We've improved our builds a lot since this comment was posted. Do you have a recent build on Read the Docs and GH Pages where I can see these times and compare against? 🙏🏼

RTD: 2.65 min https://readthedocs.org/projects/pep-previews/builds/23490045/

GH Pages: 1.65 min https://github.com/python/peps/actions/runs/7961332204/job/21732306085

(These times aren't a problem for me.)

What's the workflow you are using? GH Pages for hosting and RTD for previews as talked in this issue or something different?

Yep, GH Pages for hosting and RTD for previews.

[hugovk]

RTD: 2.65 min readthedocs.org/projects/pep-previews/builds/23490045

GH Pages: 1.65 min python/peps/actions/runs/7961332204/job/21732306085

The largest chunks:

• RTD: 2.3 min for make dirhtml JOBS=$(nproc) BUILDDIR=_readthedocs/html

• GH Pages: 1.4 min for make dirhtml JOBS=$(nproc)

Comparing logs, that evaluates to sphinx-build with -j 2 for RTD and -j 4 for GH.

[humitos]

@hugovk thanks for the update on this.

Comparing logs, that evaluates to sphinx-build with -j 2 for RTD and -j 4 for GH.

Yeah, we are using AWS m5.large images for the builders and they have 2 CPU virtual. The following one on the table is the m5.xlarge with 4 CPU virtual. That could be something to evaluate in the future regarding costs and others.

OK, I'm happy to know that this is the only difference on those times.

(These times aren't a problem for me.)

If there is anything else you still prefer from GH Pages than from Read the Docs that's blocking the migration (in case it's still something under consideration), we are happy to receive that feedback and work in that direction if it's a good fit for our product 👍🏼 . Let us know.