make html works but readthedocs-build fails inside readthedocs/build:latest

[eine]

This works:

docker run --rm -t readthedocs/build:latest bash -c "\
cd ~ && \
git clone -b builders https://github.com/1138-4EB/ghdl && \
cd ghdl/doc && \
pip install sphinx_rtd_theme -r requirements.txt && \
make html"

But this fails:

docker run --rm -t readthedocs/build:latest bash -c "\
cd ~ && \
git clone -b builders https://github.com/1138-4EB/ghdl && \
cd ghdl && \
pip install sphinx_rtd_theme -r doc/requirements.txt readthedocs-build && \
rtd-build"

with error:

Theme error:
sphinx_rtd_theme is no longer a hard dependency since version 1.4.0. Please install it manually.(pip install sphinx_rtd_theme)

[humitos]

I'm not sure to understand your issue. From what I know by reading the README, this repository shouldn't be tied to the docker image and it should be completely independent.

[eine]

As you say, this repository does not need to be tied to any image at all, because it seems to be expected to run natively. However, I do not want to install neither python nor any dependency in the host. I want a fully portable build system.

Therefore, an alternative is to write my own image which includes all the dependencies: sphinx, latex, themes, recommonmark, etc. For example, https://microbadger.com/images/ghdl/ext:sphinx is a stripped down version of readthedocs/build:latest to support sphinx only (i.e. make html). Note that this "works": https://1138-4eb.github.io/ghdl/doc/ Well, it does work as commented in readthedocs/readthedocs-docker-images#47, i.e. without any of the features offered by RTD.

Now, I want to have the missing features. I want to reproduce exactly the same build steps I get every time I push to the repo: build html, build pdf and update links in the "versions" menu of the frontrend for all the existing verions. Ideally, if I want to have some version/release but it is not generated already, I'd like the same call to look for the tag in the git history and build that too.

I expect to require quite many additional packages to do so. Then, instead of extending my image to include texlive, graphviz, or any other dependency I may need, I'd like to use readthedocs/build:latest. I expected to have everything already installed there. However, as commented in readthedocs/readthedocs-docker-images#48, the image is not complete.

---

I don't think that adding these lines would make a difference, because all of the dependencies are already installed above. I am not using neither virtualenv nor --use-wheel explicitly, but it is supposed to be done automatically by rtd-build: https://github.com/rtfd/readthedocs-build#the-build

Supossing that I can have rtd-build properly find the dependencies. What is the way to get, at least, PDF and HTML (including missing features in readthedocs/readthedocs-docker-images#48) products? Running rtd-build inside a readthedocs/build:latest container with my freshly project cloned? Note that this is the same as rtd build all commented in #2692.

Shall this not be a supported workflow/feature now, do you think it is useful? Would you be open to supporting it? To be more explicit, I think this issue is a gray area according to open-source-philosophy. I'd word it like this:

• Usage of Read the Docs Python Code for Open Source projects
• Deploying of Read the Docs build products to hosting services different from https://readthedocs.org

What's the main motivation here? To allow users to extend RTD, using it as a stage in larger site building pipes. That is, let users enhance the base build image(s), add tools for additional pre/post-processing; have them build the site locally, and then upload the product to RTD.org. Indeed, there is no need for the last part, so you may consider the second case to be unsupported even for RTD.org. Yet, I think it'd be desirable to let people stay with RTD.org, instead of moving to GitHub pages or any other hosting service.

---

Considering the big picture here, it'd be great to have readthedocs/build:latest split to several images. E.g., one for each python version, latex separately, nodejs and js in a different one, etc. Note that this can be achieved with minimum impact in build time, by using docker multi-stage builds as proposed in readthedocs/readthedocs-docker-images#29. The first pull from dockerhub to each server will take approximately the same time. Furthermore:

• Instead of spinning a 8.5GB container for each build, smaller containers can be used, depending on the version and types of selected outputs. I.e., for GHDL only pyton 3.6 is required, sphinx and latex. There should be 2-3 containers involved: one to run sphinx, a different one to run pdflatex and a third (optional) one for orchestracion, preprocessing and/or post-processing.
  • This should have meaningful impact reducing ram usage.
  • This allows to use more compact images, say alpine based, if all the required dependencies for that task are available. Just some random examples:
    • Sphinx: https://hub.docker.com/r/florianholzapfel/docker-alpine-sphinx/tags/ (28MB compressed)
    • Sphinx: https://hub.docker.com/r/masterforweb/sphinx-alpine-docker/tags/ (11MB compressed)
    • Sphinx + platuml + tex: https://github.com/tsgkdt/sphinx-plantuml/blob/tex/Dockerfile (2GB compressed)
  • Users are required to download only the images they will need, no the full build system that covers every possible project. This, along with reduced ram usage, makes it easier to have RTD running on low-cost devices.

However, I am afraid to know that you don't actually execute multiple calls to docker run, but that you spin a single container, copy the project there and execute everything at once, as if it was a VM. Is there any resource I should check in order to know what is the 'runner sequence'? Something as: clone project, move to, pull image, spin container with bind mount, start virtualenv, install dependencies...

---

Somehow related to this, the structure we are using in the master branch branch works with RTD.org: https://github.com/ghdl/ghdl

• root/
  • .readthedocs.yml
  • doc/
    • requirements.txt
    • conf.py

However, it won't work with rtd-build:

• If rtd-build is executed from root/, an error is thrown because conf.py is not found.
• If rtd-build is executed from root/doc, an error is thrown because .readthedocs.yml is not found.

rtd-build allows to set the output path as a cli arg (--outdir). However, it does not allow to set --base. I found two workarounds:

• Add a rtd.py which sets the name, type, output_base and base options as explained in https://github.com/rtfd/readthedocs-build#library-use. Then, execute python rtd.py instead of rtd-build.
• Move .readthedocs.yml to doc/, and change the route to requirements.txt. Possible issues:
  • Will RTD.org work?
  • Will rtd-build know that it is being executed from the subdir of a clone git repo?

[humitos]

OK. I will try to explain my self as best I can and trying to answer your questions here. I had to read your message a couple of times to fully understand it and maybe I'm still confused in some parts.

As you say, this repository does not need to be tied to any image at all, because it seems to be expected to run natively

First of all, this repository it's not stable yet and I think it's the future of RTD (but those are still just ideas since the core team has not the time to work heavily on this repo).

Now, I want to have the missing features. I want to reproduce exactly the same build steps I get every time I push to the repo: build html, build pdf and update links in the "versions" menu of the frontrend for all the existing verions

This sounds like a custom use of RTD since RTD is not meant (at least at this time) to be ran/called in that way. I mean, nowadays everything fits percect in the whole RTD codebase but all of those parts are not prepared to be ran independently. As I said before, I'd say that the original authors of RTD want to go in this direction but at the moment it's not possible without making some custom mofidications.

Running rtd-build inside a readthedocs/build:latest container with my freshly project cloned? Note that this is the same as rtd build all commented in #2692.

I don't have these answers, but what I can say is that RTD codebase doesn't use this command. It calls sphinx-build directly but I think the idea in the future is to delegate the whole building to rtd build all as you said. So, probably rtd build all is completed broken right now since RTD itself is not using it :)

Usage of Read the Docs Python Code for Open Source projects

There is nothing that blocks anyone to use the code for the open source projects.

Deploying of Read the Docs build products to hosting services different from readthedocs.org

There are a couple of options here:

• use a CNAME if what you only want is a different domain
• use the readthedocs.com option, in case you want other type of features (private repos, private docs, permissions, etc)
• use your own custom RTD installation

and finally, what you are proposing here

• use just the docker image and the builder --but unfortunately these repositories are not prepared to support it yet :(

Yet, I think it'd be desirable to let people stay with RTD.org, instead of moving to GitHub pages or any other hosting service.

Not sure to follow you here. In the cases you mentioned, why people would move to GH pages? The only thing that I can think of from the top of my head is because it allows people to upload random html instead of build the docs; but in that case, using RTD to build the docs and push them to GH pages makes no sense to me.

Considering the big picture here, it'd be great to have readthedocs/build:latest split to several images. E.g., one for each python version, latex separately, nodejs and js in a different one, etc

I agree in a 100% with you here. The only limitation to move on this direction I'd say that it has been time. But definetely it's something that we want.

Is there any resource I should check in order to know what is the 'runner sequence'?

You can find this at https://github.com/rtfd/readthedocs.org/blob/3e464e2fc5cf5e239a7f7db6af065d52160c64b9/readthedocs/projects/tasks.py#L67

That's where the RTD building process starts. It's not very easy to follow, so it can take you some time to fully understand it :(

---

I'd say that most of the things/features/changes that you want are something that has been discussed some time ago (or is still in discussion) but there are other priorities for the core team to work on now. So, I'd say that we would be very happy to accept PR's to move forward on these topics if they are compatible with the RTD platform itself (I mean, I think that a PR won't be accepted if it adds a feature to this custom building process but breaks rtd.org in the middle)

Hope my comment helps you to understand where we are and where we are going.

Let me know your thoughts.

Note: all of this are my own thoughts and it's probably that I'm missing some things and I'm sure that the original authors of this project can say something complete different :) . If you found something "rude" in some way or not polite in my comments, I appologies, it's not my intention... It's just that I speak English like a monkey :)

[eine]

NOTE: although I didn't explicitly say it, in my previous comment I used RTD to refer to the codebase related to sphinx/latex builds and RTD.org to refer to the full app/stack. I keep this approach here.

First of all, this repository it's not stable yet and I think it's the future of RTD (but those are still just ideas since the core team has not the time to work heavily on this repo)

Does this mean that there is ATM no RTD builder as a library, i.e. separated from the backend and frontend, from user and project management, etc.?

This sounds like a custom use of RTD since RTD is not meant (at least at this time) to be ran/called in that way.

I'd add outside of readthedocs.org. Actually, I can check the sequence of instructions that RTD.org executes. E.g., in this build:

• Let me rename /home/docs/checkouts/readthedocs.org/user_builds/ghdl-devfork/ to my_prj.
• Then my_prj/envs/update-refs-to-org to my_env, and my_prj/checkouts/update-refs-to-org/doc/_build/ to my_prj_checkouts.

We get:

$ python3.5 -mvirtualenv --no-site-packages --no-download my_env
$ python my_env/bin/pip install --use-wheel --upgrade --cache-dir my_prj/.cache/pip Pygments==2.2.0 setuptools==37.0.0 docutils==0.13.1 mock==1.0.1 pillow==2.6.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.5.4 recommonmark==0.4.0 sphinx==1.6.5 sphinx-rtd-theme<0.3 readthedocs-sphinx-ext<0.6
$ python my_env/bin/pip install --exists-action=w --cache-dir my_prj/.cache/pip -rdoc/requirements.txt
$ cat doc/conf.py
$ python my_env/bin/sphinx-build -T -b readthedocs -d _build/doctrees-readthedocs -D language=en . _build/html
$ python my_env/bin/sphinx-build -T -b json -d _build/doctrees-json -D language=en . _build/json
$ python my_env/bin/sphinx-build -T -b readthedocssinglehtmllocalmedia -d _build/doctrees-readthedocssinglehtmllocalmedia -D language=en . _build/localmedia
$ python my_env/bin/sphinx-build -b latex -D language=en -d _build/doctrees . _build/latex
$ pdflatex -interaction=nonstopmode my_prj_checkouts/latex/GHDL.tex
$ makeindex -s python.ist GHDL.idx
$ pdflatex -interaction=nonstopmode my_prj_checkouts/latex/GHDL.tex
$ mv -f my_prj_checkouts/latex/GHDL my_prj/artifacts/update-refs-to-org/sphinx_pdf/ghdl-devfork.pdf

This partially resembles the suggestion in #48. Indeed, the error I get when tying to execute python my_env/bin/pip install... is the same as in #48.

I mean, nowadays everything fits percect in the whole RTD codebase but all of those parts are not prepared to be ran independently.

Although not meant to do so, the system is almost ready. The only missing parts are steps which are executed before and/or after the snippet above. I expect these to be, cloning the repo before, uploading products after, an optionally retrieving/sending some info to the database.

As I said before, I'd say that the original authors of RTD want to go in this direction but at the moment it's not possible without making some custom mofidications.

I have no problem with introducing custom modifications to test this. However, I'd like to avoid the codebase related to the frontend, user creation, webhooks, etc. I.e., I want to know what happens when I manually trigger 'Build version master' in the RTD.org frontend, that is related to clone and build execution. Is this content separated or is everything defined in the same source files?

I don't have these answers, but what I can say is that RTD codebase doesn't use this command. It calls sphinx-build directly but I think the idea in the future is to delegate the whole building to rtd build all as you said. So, probably rtd build all is completed broken right now since RTD itself is not using it :)

Thanks for explaining this. Indeed, it makes me think that I've been asking for a non-existent feature. That might have made it more difficult for you to understand/follow this issue. Sorry about that.

If I don't take it wrong now, rtd-build is an external library which would produce and execute the code snippet above. Then, RTD.org would behave as a caller, along with all the app related stuff. What are the technical issues that prevent this switch right now? I.e. why might rtd-build be completely broken?

There is nothing that blocks anyone to use the code for the open source projects.

No, there is not. But that is due to the license. I was suggesting to include it in the 'official support' list. As you stated above, RTD as a project is not supporting any usage outside readthedocs.org, not even reproducing the build procedure locally. Then, in practice, it is quite difficult to use RTD, and not RTD.org. To word it differently, I see little incentive to use RTD as a builder (not a hosted service) instead of sphinx. At the same time, I'd like to have that incentive, because of the main RTD.org feature: multiple versions of the documentation, related to either branches or releases.

There are a couple of options here:

• use a CNAME if what you only want is a different domain
• use the readthedocs.com option, in case you want other type of features (private repos, private docs, permissions, etc)
• use your own custom RTD installation

None of these options provide a solution to have a tarball with the RTD built site that I can upload to any static file server. Well, the custom RTD.org installation would allow me to explore the container and copy the files. But that's quite dirty, and so overkill. This is especially so because I might want to use it in a external CI service. Spinning RTD.org inside a CI and having it automatically configured is not a option. Sure, it can be configured locally and upload a snapshot of the stack. Yet, it is a workaround, not a desired solution.

and finally, what you are proposing here

• use just the docker image and the builder --but unfortunately these repositories are not prepared to support it yet :(

Well, what's missing here? Let's guess it!

Not sure to follow you here. In the cases you mentioned, why people would move to GH pages? The only thing that I can think of from the top of my head is because it allows people to upload random html instead of build the docs; but in that case, using RTD to build the docs and push them to GH pages makes no sense to me.

Say we want to build a site with a manually written landing page, hosted at mysite.io. We want to use RTD to build mysite.io/doc. Then, we have a second project in a different git repo with it's own RTD; it is to be hosted at mysite.io/gui. Each of the sections is a different RTD build, each with different configured branches/version/releases. Furthermore, we have a third project, that is a Hugo site/blog; it is to be hosted at mysite.io/blog.

Right now, we need to build doc, gui and blog, each on it's own CI system, and have a last stage to fetch the tarballs of all of them and have them merged in GitHub Pages. Note that this is just an alternative to running RTD (not RTD.org) locally, i.e. inside a CI. If so, I'd build doc, gui and blog, all of them in the same CI job, and push the site once.

The point here is that I don't know if maintainers are ok with pushing in this direction. In the sense that making it easier to use RTD resources outside of RTD.org might make it easier to integrate RTD docs in larger sites. If RTD does not officially host those larger sites, it is expected that the number of projects actually using the service might be reduced. This can be a good new, because traffic and resource requirements to keep readthedocs.org online would be reduced. I also believe that it would help spread the usage of RTD as a library/tool. However, traffic reduction might affect income due to ads. I don't know whether it'd be possible for projects built with rtd-theme to keep the ad shown in readthedocs.org, while being hosted outside of readthedocs.org.

I agree in a 100% with you here. The only limitation to move on this direction I'd say that it has been time. But definetely it's something that we want.

Is there any analysis/roadmap about the architecture of the RTD codebase and/or the required changes? I mean, there has been little time to implement it, but was any at all dedicated to it until now? Just trying to guess what the situation is. I don't want to sound critical at all.

That's where the RTD building process starts. It's not very easy to follow, so it can take you some time to fully understand it :(

Thanks a lot. Sadly, I know so little python. Do you know of any tool which provides an 'execution flow graph' in order to make it easier to see the sequence?

---

I'd say that most of the things/features/changes that you want are something that has been discussed some time ago (or is still in discussion) but there are other priorities for the core team to work on now.

Yet, we are 'spending' a few comments to know what the current status is. Shall we make profit of this conversation and gather the related content in a section such as Using RTD as a static site generator? If the content must be 'this is currently not supported and it is not a priority', let it be. But I believe we can add some meaningful references: readthedocs-docker-images#48, this repo, this thread... Can you think of any hard-to-find related conversation/discussion which I might not easily find in the issues and/or the documentation?

So, I'd say that we would be very happy to accept PR's to move forward on these topics if they are compatible with the RTD platform itself (I mean, I think that a PR won't be accepted if it adds a feature to this custom building process but breaks rtd.org in the middle)

As said, I feel not comfortable at all with Python. However, I can contribute with analyzing modularization possibilities in order to optimize the build process and hopefully make it easier to reuse individual resources outside of RTD.org. E.g. in readthedocs/readthedocs-docker-images#29 I made a very first demo of how to use docker multi-stage builds. If the suggestion is accepted, we should discuss about removing the first two commands (i.e. creation of the virtualenv and installation of base dependencies), and use multiple images instead. This should not break RTD.org, because we are just changing when and how dependencies are installed. But I'd probably need help with the syntax for the required Python modifications. Have a look at this alternative snippets:

$ docker run --name prj-sphinx \
    -tv /home/docs/checkouts/readthedocs.org/user_builds/ghdl-devfork/checkouts/update-refs-to-org:/src \
    rtd/build:3.5-sphinx sh -c "\
 pip install --exists-action=w --cache-dir my_prj/.cache/pip -rdoc/requirements.txt && \
 cat doc/conf.py && \
 sphinx-build -T -b readthedocs -d _build/doctrees-readthedocs -D language=en . _build/html && \
 sphinx-build -T -b json -d _build/doctrees-json -D language=en . _build/json && \
 sphinx-build -T -b readthedocssinglehtmllocalmedia -d _build/doctrees-readthedocssinglehtmllocalmedia -D language=en . _build/localmedia && \
 sphinx-build -b latex -D language=en -d _build/doctrees . _build/latex && \
 pdflatex -interaction=nonstopmode my_prj_checkouts/latex/GHDL.tex && \
 makeindex -s python.ist GHDL.idx && \
 pdflatex -interaction=nonstopmode my_prj_checkouts/latex/GHDL.tex && \
 mv -f my_prj_checkouts/latex/GHDL  my_prj/artifacts/update-refs-to-org/sphinx_pdf/ghdl-devfork.pdf"

$ docker run --name prj-sphinx \
    -tv /home/docs/checkouts/readthedocs.org/user_builds/ghdl-devfork/checkouts/update-refs-to-org:/src \
    rtd/build:3.5-sphinx sh -c "\
 cp -vr /src ~/prj/ && cd ~/prj && \
 pip install --exists-action=w --cache-dir .cache/pip -rdoc/requirements.txt && \
 cat doc/conf.py && \
 sphinx-build -T -b readthedocs -d _build/doctrees-readthedocs -D language=en . _build/html && \
 sphinx-build -T -b json -d _build/doctrees-json -D language=en . _build/json && \
 sphinx-build -T -b readthedocssinglehtmllocalmedia -d _build/doctrees-readthedocssinglehtmllocalmedia -D language=en . _build/localmedia && \
 sphinx-build -b latex -D language=en -d _build/doctrees . _build/latex"
$ docker cp prj-sphinx:/home/docs/prj/_build ./
$ docker rm -f prj-sphinx
$ docker run --name prj-latex \
    -tv /home/docs/checkouts/readthedocs.org/user_builds/ghdl-devfork/checkouts/update-refs-to-org/doc/_build:/src \
    rtd/build:latex sh -c "\
    cp -vr /src ~/prj/ && cd ~/prj && \
 pdflatex -interaction=nonstopmode latex/GHDL.tex && \
 makeindex -s python.ist GHDL.idx && \
 pdflatex -interaction=nonstopmode latex/GHDL.tex && \
 mv -f latex/GHDL  my_prj/artifacts/update-refs-to-org/sphinx_pdf/ghdl-devfork.pdf"
$ docker cp prj-latex:/home/docs/prj/latex/GHDL /home/docs/checkouts/readthedocs.org/user_builds/ghdl-devfork/artifacts/update-refs-to-org/sphinx_pdf/ghdl-devfork.pdf
$ docker rm -f prj-latex

$ docker volume create prj-vol
$ docker run --rm -tv prj-vol:/vol -v /home/docs/checkouts/readthedocs.org/user_builds/ghdl-devfork/checkouts/update-refs-to-org:/src rtd/build:busybox cp -vr /src /vol/
$ docker run --name prj-sphinx -tv prj-vol:/src rtd/build:3.5-sphinx sh -c "\
 pip install --exists-action=w --cache-dir .cache/pip -rdoc/requirements.txt && \
 cat doc/conf.py && \
 sphinx-build -T -b readthedocs -d _build/doctrees-readthedocs -D language=en . _build/html && \
 sphinx-build -T -b json -d _build/doctrees-json -D language=en . _build/json && \
 sphinx-build -T -b readthedocssinglehtmllocalmedia -d _build/doctrees-readthedocssinglehtmllocalmedia -D language=en . _build/localmedia && \
 sphinx-build -b latex -D language=en -d _build/doctrees . _build/latex"
$ docker run --rm -tv prj-vol:/src rtd/build:3.5-latex sh -c "\
 pdflatex -interaction=nonstopmode doc/_build/latex/GHDL.tex && \
 makeindex -s python.ist GHDL.idx && \
 pdflatex -interaction=nonstopmode doc/_build/latex/GHDL.tex && \
 mv -f latex/GHDL  my_prj/artifacts/update-refs-to-org/sphinx_pdf/ghdl-devfork.pdf"
$ docker run --name prj-tmp -dv prj-vol:/vol rtd/build:busybox sh
$ docker cp prj-tmp:/vol/_build _build/
$ docker volume rm -f prj-tmp
$ mv -f _build/latex/GHDL /home/docs/checkouts/readthedocs.org/user_builds/ghdl-devfork/artifacts/update-refs-to-org/sphinx_pdf/ghdl-devfork.pdf

---

Hope my comment helps you to understand where we are and where we are going.

They did. Thanks a lot!

Note: I think we share the same mother tongue. So, the same applies to rudeness ;) and pre-apologies.

[ericholscher]

I think we can just say at the current time, this repo is only used as a configuration parser for readthedocs.org builds, and it's unsupported for other uses. If you want to build things locally, I'd recommend using Docker, which this tool has no knowledge of.

[eine]

@ericholscher, what do you mean with 'using Docker'? All the examples here use docker containers. Precisely readthedocs/build.

[ericholscher]

I'm saying running readthedocs-build as a CLI tool is currently unsupported, and it shouldn't be used on it's own. We should update the docs/README on this, as it should be made clear we aren't actively supporting that use case.

[eine]

I created a PR to remove references to unsupported uses.

In the comment above you suggested to use docker in order to build things locally. Did you mean using some library/package in the RTD codebase on it's own, or to use the full RTD.org stack?

[ericholscher]

Either way, but note that those aren't supposed use cases either. You can read more here: http://docs.readthedocs.io/en/latest/open-source-philosophy.html#official-support -- we don't support installs or doing builds on any third party platform, as that is too much work for us.