Build and deploy TriBITS documentation with Sphinx (#386) #387

marcinwrobel1986 · 2021-07-09T06:59:08Z

Implements #386

This PR

adds CI job for creating and deploing documentation:
- generates documentatios as static html with css and js according to used template
- template used by Sphinx is Read The Docs template
- deployment is currently done to GitHub Pages
adds Sphinx configuration for generating documentation
adds index.rst for documentation landing page
adds Python requirements (Sphinx and RTD theme)
adds Python3 helper script, which edits generated rst files:
- script is looking for include:: directive in rst files
- changes paths to be relative for Sphinx source directory
- does document title formatting

The documentation is currently being deployed at:

https://marcinwrobel1986.github.io/TriBITS/

Tasks

Get Sphinx installed locally on SNL RHEL 7 machine and reproduce build ... See below [@bartlettroscoe]
Create separate sphinx documents for each of the documents "TriBITS Project Build Reference", "TriBITS Users Guide" and "TriBITS Maintainers Guide" with a non-Sphinx base landing index page that points to these (See below and #386 comment) [@marcinwrobel1986].
Restructure the building of the Sphinx documents so that none of the original source documents get changed (see below) [@marcinwrobel1986].
Decide how to deploy the Sphinx documentation and how it relates to the current https://tribits.org site:
- Decide if Sphinx documentation generation branch should be in main TriBITS repo or auxiliary GitHub repo ... It will be on on branch deploy-doc-site that can be removed later if needed
- Decide if new GitHub Pages site generated by GitHub Actions process replaces https://tribits.org or is just referenced by tribits.org. ... For now, keep https://tribits.org and add links to point to Sphinx documents in addition to DocUtils HTML files
Remove need for Python2 and only require Python3 when documentation is build when calling tribits/doc/build_docs.sh and tribits/doc/guides/generate-guide.sh (and don't require the python hack) [@bartlettroscoe]
Remove calling rst2html.py to build the flat HTML files with docutils when extracting the documentation from the CMake files to create the intermediate *.rst files with tribits/doc/build_docs.sh and tribits/doc/guides/generate-guide.sh when called when building Sphinx documentation (that is a waste in that case) [@bartlettroscoe]
Merge PR

.github/workflows/deploy_docs.yml

bartlettroscoe · 2021-07-10T04:08:00Z

@marcinwrobel1986, it seems it may be possible to get Sphinx to run a script to generate the input files so that we don't need commit them to the repo to any branch. That way, we can also use readthedocs.com to deploy the documentation. Some details are in:

Is it possible to get read the docs to run a custom python script as part of the build? readthedocs/readthedocs.org#2276
Replace current import process for Github with Actions integration readthedocs/readthedocs.org#8133

It seems like we might be able to put code in the bottom of config.py to invoke the generation of the documentation on the fly so we don't need to commit anything on a branch for each documentation deployment.

SIDE-NOTE: A funny quote from readthedocs/readthedocs.org#8133 (comment):

Python community is in such a state that if you hear that you do something that's discouraged by it, you may be certain you are doing it right.

tribits/doc/sphinx/requirements.txt

bartlettroscoe · 2021-07-12T19:40:11Z

FYI: As per the instructions at:

https://www.sphinx-doc.org/en/master/usage/installation.html

I installed the latest Anaconda3-2021.05-Linux-x86_64 on my SNL RHEL 7 machine using:

$ cd ~/junk/
$ bash ./Anaconda3-2021.05-Linux-x86_64.sh

which installed it under:

~/anaconda3/

and tried to install an updated Sphinx with that 'conda' and I got:

$ which conda
~/anaconda3/bin/conda

$ conda install sphinx
Collecting package metadata (current_repodata.json): failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.anaconda.com/pkgs/main/linux-64/current_repodata.json>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

If your current network has https://www.anaconda.com blocked, please file
a support request with your network engineering team.

'https://repo.anaconda.com/pkgs/main/linux-64'

Seems like a proxy issue or something. I could spend some time trying to debug this to at least do the downloads. Otherwise, it may be very hard for me to build this documentation on a Linux RHEL 7 machine at SNL if it requires a newer Sphinx install. (Which is one reason why I have never been able to use Sphinx in the past. I could never get it to install on SNL machines.)

Next, I could try installing everything I need on my Windows 10 machine and running under cygwin?

I just don't have the luxury of an up-to-date Linux distribution at SNL.

marcinwrobel1986 · 2021-07-13T06:55:47Z

FYI: As per the instructions at:

https://www.sphinx-doc.org/en/master/usage/installation.html

I installed the latest Anaconda3-2021.05-Linux-x86_64 on my SNL RHEL 7 machine using:
$ cd ~/junk/
$ bash ./Anaconda3-2021.05-Linux-x86_64.sh
which installed it under:
~/anaconda3/
and tried to install an updated Sphinx with that 'conda' and I got:
$ which conda
~/anaconda3/bin/conda

$ conda install sphinx
Collecting package metadata (current_repodata.json): failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.anaconda.com/pkgs/main/linux-64/current_repodata.json>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

If your current network has https://www.anaconda.com blocked, please file
a support request with your network engineering team.

'https://repo.anaconda.com/pkgs/main/linux-64'
Seems like a proxy issue or something. I could spend some time trying to debug this to at least do the downloads. Otherwise, it may be very hard for me to build this documentation on a Linux RHEL 7 machine at SNL if it requires a newer Sphinx install. (Which is one reason why I have never been able to use Sphinx in the past. I could never get it to install on SNL machines.)

Next, I could try installing everything I need on my Windows 10 machine and running under cygwin?

I just don't have the luxury of an up-to-date Linux distribution at SNL.

Hello Ross,

that's all really too much. Anaconda is a huge platform.

First some checking:

python -V                          
> Python 2.7.18
which python 
> /usr/bin/python

python3 -V                          
> Python 3.8.10
which python3
> /usr/bin/python3

In my case, it's telling me I have two Python interpreters installed in my OS. I have other built from source and installed as well, but it doesn't matter now. If one has no Python or Python3 one could install it. Easiest way would be from package manager. It could be yum or dnf for RHEL.
If one is logged as root (which one probably shouldn't be) then would be just:

yum install Python3 or dnf install Python3
With other user one needs to use sudo or login as root, whatever is preferable:
sudo yum install Python3 or sudo dnf install Python3

The above is regarding Python installation.

One needs to install python packages, so:

required packages are in: <main_repository_path>/tribits/doc/sphinx/requirements.txt
the most popular/fastest/easiest is to go with pip

some checking:

pip2 -V                          
> pip 20.3.4 from /usr/local/lib/python2.7/dist-packages/pip (python 2.7)
pip3 -V
> pip 21.1.2 from /home/marcin/.local/lib/python3.9/site-packages/pip (python 3.9)

pip could be found under pip -V instead of pip2 or pip3, it is OS dependent
let's assume one doesn't use any virtual environment, but global interpreter
one needs to install packages for PYTHON3, so pip needs to point to Python3 interpreter
if one is logged as root then would be just pip install Sphinx sphinx-rtd-theme
or sudo pip install Sphinx sphinx-rtd-theme if not logged as root
one could go to <main_repository_path>/tribits/doc/sphinx and install all requirements with pip install -r requirements.txt, same as before => as root or with sudo
in order to build a documentation with <main_repository_path>/tribits/doc/build_docs.sh Python2 interpreter needs to have docutils installed

bartlettroscoe · 2021-07-13T13:45:12Z

So I will try the following in install Sphinx.

On my Windows 10 machine off the SNL network:

$ wget https://files.pythonhosted.org/packages/7e/7c/13a7bac354edfe1c2f2165ded2eb8ad61d492aa210a2405c85552f94c408/Sphinx-4.1.0.tar.gz
$ wget https://files.pythonhosted.org/packages/38/16/5a94dfc1263cc4169e05b5f52628f26dffae9ac99892338145f26949270b/sphinx_rtd_theme-0.5.2.tar.gz

Then copy these files to RHEL7 machine and run:

$ pip3 install sphinx ~/Sphinx-4.1.0.tar.gz
$ pip3 install sphinx_rtd_theme ~/sphinx_rtd_theme-0.5.2.tar.gz

bartlettroscoe · 2021-07-13T14:16:54Z

@marcinwrobel1986

Next action items:

@bartlettroscoe:

Install Sphinx and locally reproduce build of Sphinx documentation for TriBiTS.
Remove need to have Python 2 installed or need to run docutils when building Sphinx documentation.

@marcinwrobel1986:

Break up into three different Sphinx documents with a top-level non-Sphinx landing page.

Then we will touch base again on Thursday and try to wrap up this documentation effort quickly after that.

bartlettroscoe · 2021-07-13T23:40:08Z

@marcinwrobel1986, okay, with some local help, I figured out how to get pip3 to communicate and download files on my RHEL 7 machine and install Sphinx and I was able to build the Sphinx documents locally and view them in the local browser (see detailed notes below).

The state of the repo after doing this build was:

$ git status
On branch 386-sphinx-doc
Your branch is up-to-date with 'marcinwrobel1986/386-Build-TriBITS-Documentation-with-Sphinx-and-ReadTheDocs'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   tribits/doc/guides/TribitsGuidesBody.rst
        modified:   tribits/doc/guides/TribitsToolsDocumentation.rst
        typechange: tribits/doc/guides/maintainers_guide/TribitsCoreDetailedReference.rst
        typechange: tribits/doc/guides/users_guide/TribitsCoreDetailedReference.rst

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        tribits/doc/sphinx/_build/
        tribits/doc/sphinx/build_ref.rst
        tribits/doc/sphinx/maintainers_guide.rst
        tribits/doc/sphinx/users_guide.rst

no changes added to commit (use "git add" and/or "git commit -a")

The biggest issue I see is that this process modifies the original source files:

$ git diff --name-status 
M       tribits/doc/guides/TribitsGuidesBody.rst
M       tribits/doc/guides/TribitsToolsDocumentation.rst
T       tribits/doc/guides/maintainers_guide/TribitsCoreDetailedReference.rst
T       tribits/doc/guides/users_guide/TribitsCoreDetailedReference.rst

That will make maintenance of this documentation very difficult. (A simple rule is that a build process should never change a manually maintained file.)

To avoid this, can we just put the Sphinx base *.rst files and generated Sphinx HTML _build/ directory for each document right beside the existing *.rst files like:

tribits/doc/
  build_ref/
    TribitsBuildReference.rst
    build_ref.rst  # Generated
    _build/     # Generated
  guides/
    maintainers_guide/
      TribitsMaintainersGuide.rst
      maintainers_guide.rst   # Generated
      _build/         # Generated
    users_guide/
      TribitsUsersGuide.rst
      users_guide.rst   # Generated
      _build/     # Generated

?

The document maintainers_guide.rst, for example, could just be a copy of the document TribitsMaintainersGuide.rst with minor modifications as needed for Sphinx. Then none of the paths would have to change and none of the original source documents would have to change.

I will add a "Task" section above to keep track of everything we need to do to get this merged and deployed.

Thanks again for all of your help on this!

DETAILED NOTES: (click to expand)

.

After getting the proxy set up to work with pip in my local SNL RHEL 7 machine, I was able to install Sphinx and the RTD theme with:

$ pip3 install sphinx sphinx_rtd_theme
...
Installing collected packages: sphinxcontrib-applehelp, imagesize, sphinxcontrib-serializinghtml, sphinxcontrib-jsmath, pytz, babel, Pygments, idna, certifi, charset-normalizer, urllib3, requests, docutils, MarkupSafe, Jinja2, sphinxcontrib-devhelp, pyparsing, packaging, alabaster, snowballstemmer, sphinxcontrib-qthelp, sphinxcontrib-htmlhelp, sphinx, sphinx-rtd-theme
Successfully installed Jinja2-3.0.1 MarkupSafe-2.0.1 Pygments-2.9.0 alabaster-0.7.12 babel-2.9.1 certifi-2021.5.30 charset-normalizer-2.0.1 docutils-0.17.1 idna-3.2 imagesize-1.2.0 packaging-21.0 pyparsing-2.4.7 pytz-2021.1 requests-2.26.0 snowballstemmer-2.1.0 sphinx-4.1.0 sphinx-rtd-theme-0.5.2 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 urllib3-1.26.6

Now I need to give this a try!

I checked out the branch:

$ gitdist-status --dist-repos=.
-----------------------------------------------------------------------------------------------------------------------------------
| ID | Repo Dir       | Branch         | Tracking Branch                                                              | C | M | ? |
|----|----------------|----------------|------------------------------------------------------------------------------|---|---|---|
|  0 | TriBITS (Base) | 386-sphinx-doc | marcinwrobel1986/386-Build-TriBITS-Documentation-with-Sphinx-and-ReadTheDocs |   |   |   |
-----------------------------------------------------------------------------------------------------------------------------------

$ git log-short --name-status -1
92daca4 "#386: - removed RTD attempt"
Author: Marcin Wróbel <[email protected]>
Date:   Tue Jul 13 14:56:38 2021 +0200 (10 hours ago)

D       .readthedocs.yaml
M       tribits/doc/sphinx/conf.py
M       tribits/doc/sphinx/sphinx_rst_generator.py

I then ran it on 'crf450' as:

$ cd ~/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx

$ python3 ./tribits/doc/sphinx/sphinx_rst_generator.py
...
DONE! ALL GOOD!

and it did:

$ git diff --name-status 
M       tribits/doc/guides/TribitsGuidesBody.rst
M       tribits/doc/guides/TribitsToolsDocumentation.rst
T       tribits/doc/guides/maintainers_guide/TribitsCoreDetailedReference.rst
T       tribits/doc/guides/users_guide/TribitsCoreDetailedReference.rst

This unfortunately modified the orginal *.rst files:

tribits/doc/guides/TribitsGuidesBody.rst
tribits/doc/guides/TribitsToolsDocumentation.rst

It changed the paths like:

diff --git a/tribits/doc/guides/TribitsGuidesBody.rst b/tribits/doc/guides/TribitsGuidesBody.rst
index 9df368a..7496dda 100644
--- a/tribits/doc/guides/TribitsGuidesBody.rst
+++ b/tribits/doc/guides/TribitsGuidesBody.rst
@@ -834,7 +834,7 @@ project's native repositories.  The list of repositories is defined using the
 macro `tribits_project_define_extra_repositories()`_.  For example, the extra
 repos file:
 
-.. include:: ExtraReposList.cmake
+.. include:: ../guides/ExtraReposList.cmake
    :literal:
 
 shows the specification of both TriBITS Repositories and non-TriBITS VC

That is not a good workflow. Instead, the Sphinx documents should be generated right there in the same directories as:

tribits/doc/guides/maintainers_guide/
tribits/doc/guides/users_guide/
tribits/doc/build_ref/

and not require any modifications to the original source files.

Now to build the Sphinx documentation:

$ cd ~/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/

$ export PATH=$HOME/.local/bin:$PATH

$ which sphinx-build 
~/.local/bin/sphinx-build

$ sphinx-build --version
sphinx-build 4.1.0

$ make html
Running Sphinx v4.1.0
making output directory... done
WARNING: html_static_path entry '_static' does not exist
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 4 source files that are out of date
updating environment: [new config] 4 added, 0 changed, 0 removed
reading sources... [100%] users_guide                                                                                                                                               
../guides/TribitsGuidesBody.rst:6204: WARNING: Error in "include" directive:
no content permitted.

.. include:: ../../examples/TribitsExampleProject/cmake/ctest/TribitsExProjCTestDriver.cmake
     :literal:

  and then create a set of CTest -S driver scripts that uses that file.  One
  example is the file
  ``TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake``:
../guides/TribitsGuidesBody.rst:6204: WARNING: Error in "include" directive:
no content permitted.

.. include:: ../../examples/TribitsExampleProject/cmake/ctest/TribitsExProjCTestDriver.cmake
     :literal:

  and then create a set of CTest -S driver scripts that uses that file.  One
  example is the file
  ``TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake``:
../guides/TribitsGuidesBody.rst:4064: WARNING: duplicate label <project>packagedependencies.xml, other instance in /home/rabartl/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/maintainers_guide.rst
../guides/TribitsGuidesBody.rst:4683: WARNING: duplicate label checkin-test.py, other instance in /home/rabartl/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/maintainers_guide.rst
../../README.DIRECTORY_CONTENTS.rst:2: WARNING: duplicate label tribits/tribits/, other instance in /home/rabartl/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/maintainers_guide.rst
../guides/TribitsToolsDocumentation.rst:114: WARNING: duplicate label snapshot-dir.py, other instance in /home/rabartl/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/maintainers_guide.rst
../guides/TribitsToolsDocumentation.rst:140: WARNING: duplicate label is_checkin_tested_commit.py, other instance in /home/rabartl/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/maintainers_guide.rst
../guides/TribitsToolsDocumentation.rst:152: WARNING: duplicate label get-tribits-packages-from-files-list.py, other instance in /home/rabartl/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/maintainers_guide.rst
../guides/TribitsToolsDocumentation.rst:165: WARNING: duplicate label get-tribits-packages-from-last-tests-failed.py, other instance in /home/rabartl/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/maintainers_guide.rst
../guides/TribitsToolsDocumentation.rst:178: WARNING: duplicate label filter-packages-list.py, other instance in /home/rabartl/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/maintainers_guide.rst

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] users_guide                                                                                                                                                
generating indices... genindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 11 warnings.

The HTML pages are in _build/html.

That generated:

$ ~/tree.py --depth=2 -cf _build/html/
  html/
  +-.buildinfo
  +-_sources/
  | +-build_ref.rst.txt
  | +-index.rst.txt
  | +-maintainers_guide.rst.txt
  | +-users_guide.rst.txt
  +-_static/
  | +-basic.css
  | +-css/
  | +-doctools.js
  | +-documentation_options.js
  | +-file.png
  | +-fonts/
  | +-jquery-3.5.1.js
  | +-jquery.js
  | +-js/
  | +-language_data.js
  | +-minus.png
  | +-plus.png
  | +-pygments.css
  | +-searchtools.js
  | +-underscore-1.13.1.js
  | +-underscore.js
  +-build_ref.html
  +-genindex.html
  +-index.html
  +-maintainers_guide.html
  +-objects.inv
  +-search.html
  +-searchindex.js
  +-users_guide.html

And looking at this in firefox on that machine I can see the build documentation. So that worked!

The final state of the git repo is:

$ git status
On branch 386-sphinx-doc
Your branch is up-to-date with 'marcinwrobel1986/386-Build-TriBITS-Documentation-with-Sphinx-and-ReadTheDocs'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   tribits/doc/guides/TribitsGuidesBody.rst
        modified:   tribits/doc/guides/TribitsToolsDocumentation.rst
        typechange: tribits/doc/guides/maintainers_guide/TribitsCoreDetailedReference.rst
        typechange: tribits/doc/guides/users_guide/TribitsCoreDetailedReference.rst

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        tribits/doc/sphinx/_build/
        tribits/doc/sphinx/build_ref.rst
        tribits/doc/sphinx/maintainers_guide.rst
        tribits/doc/sphinx/users_guide.rst

no changes added to commit (use "git add" and/or "git commit -a")

marcinwrobel1986 · 2021-07-14T06:31:52Z

That will make maintenance of this documentation very difficult. (A simple rule is that a build process should never change a manually maintained file.)

Ross, I would said that a simple rule is when you have hardcoded relative paths which does not work when run from other directory, you make a generic script which change them, changed files are not commited anywhere.
Just want to remind, that currently script build_docs.sh is use to build rst files which(the script) was not modified in any way. And it's output was supposed to work differently.
The idea to build documentation totally with Python3 (now combination of Python and bash scripts) was as well to get rid of any addition and make nice, clear process.

If you take a look at the problem in very generic way, you have some output and you want this output to work with other tool. Other tool expects diffent input then your output. What to do? Natural way would be to modified the output to fulfill requirements for the input of the other tool. Like adapter, right?.

This is exactly how it's done now. It has nothing to do with the maintenance of the documentation. One makes any changes one wants to the documentation, files are generated with build_docs.sh script and then adapter script changes relative paths in .rst files, so they work correctly with Sphinx, but just for documentation generation process, nothing stays there, nothing is comitted anywhere. When building locally one could then just use git reset --hard origin/<branch_name> and it's as it was.

bartlettroscoe · 2021-07-14T14:11:11Z

@marcinwrobel1986, see the list of "Tasks" above and see if this is consistent with what we discussed yesterday.

I would said that a simple rule is when you have hardcoded relative paths which does not work when run from other directory, you make a generic script which change them, changed files are not commited anywhere.

The problem is that reStructuredText does not really give you any other option that I can see. You can't use variables to specify the paths and the paths are relative the the file that includes them. It is very ridged. I was able to play some tricks with symlinks and avoid needing to edit or copy files. But if you do need to modify the files, then you copy then to a different location and then modify them. You never modify them in place. Does that make sense?

marcinwrobel1986 · 2021-07-15T09:07:01Z

@bartlettroscoe new landing page is deployed, please check whenever you have a moment

bartlettroscoe · 2021-07-15T13:59:39Z

@bartlettroscoe new landing page is deployed, please check whenever you have a moment

@marcinwrobel1986, thanks! I will give that a try right now and work to get rid of to the Python2 and DocUtils requirement (in a separate PR branch that can be merged to this PR).

marcinwrobel1986 · 2021-07-15T14:15:33Z

@bartlettroscoe new landing page is deployed, please check whenever you have a moment

@marcinwrobel1986, thanks! I will give that a try right now and work to get rid of to the Python2 and DocUtils requirement (in a separate PR branch that can be merged to this PR).

@bartlettroscoe please mind that now Sphnix is runnig from Python script, you can check the changes in the CI job

bartlettroscoe · 2021-07-15T14:18:03Z

@bartlettroscoe please mind that now Sphnix is runnig from Python script, you can check the changes in the CI job.

@marcinwrobel1986, yup, I saw what from inspecting the updated .github/workflows/deploy_docs.yml. Thanks!

bartlettroscoe · 2021-07-15T14:41:56Z

@marcinwrobel1986, I pulled the updated branch at the commit 1b87c9c:

$ git log-short --name-status -1
1b87c9c "#386: - fixed typo"
Author: Marcin Wróbel <[email protected]>
Date:   Thu Jul 15 10:22:50 2021 +0200 (6 hours ago)

M       .github/workflows/deploy_docs.yml

and built the updated documentation (see details below) and I was able to view the updated documentation with three separate Sphinx documents. Thanks for doing that!

However, it looks like the build process is still modifying manually maintained source files:

$ git diff --name-status 
M       tribits/doc/guides/TribitsGuidesBody.rst
M       tribits/doc/guides/TribitsToolsDocumentation.rst
T       tribits/doc/guides/maintainers_guide/TribitsCoreDetailedReference.rst
T       tribits/doc/guides/users_guide/TribitsCoreDetailedReference.rst

Can we meet soon to discuss how to adjust things to avoid this? A quick meeting to discuss the issues and the different options would be best.

NOTE: There are several issues with this Sphinx generated documentation including formatting issues (like not showing bulleted lists of links correctly), relative links between the documents are broken (because they assume the documents are side-by-side), the sphinx search results is only shows the "Introduction" section, the page title in the browser starts with "Introduction" and the messy TriBITS version but at least this is a good start. We can do that tweaking later. To merge this branch, we just need to not break the existing DoUtils documentation and we need to not have the build process modify any original source files.

DETAILED NOTES (click to expand)

(7/15/2021)

@marcinwrobel1986 has an updated Sphinx documentation build. I need to try this out and give feedback. Let's give it a try.

Updated branch to:

$ git log-short --name-status -1
1b87c9c "#386: - fixed typo"
Author: Marcin Wróbel <[email protected]>
Date:   Thu Jul 15 10:22:50 2021 +0200 (6 hours ago)

M       .github/workflows/deploy_docs.yml

Now to figure out how to build the updated documentation from looking at .github/workflows/deploy_docs.yml. It looks like the full generation is now done inside of the script sphinx_rst_generator.py which is run from the working directly ./tribits/doc/sphinx. So I run that:

$ cd tribits/doc/sphinx/

$ which sphinx-build 
~/.local/bin/sphinx-build

$ sphinx-build --version
sphinx-build 4.1.0

$ python3 sphinx_rst_generator.py

...

DONE! ALL GOOD!

===> Generating Sphinx documentation:

===> Generating mainteiners_guide

Running Sphinx v4.1.0
making output directory... done
WARNING: html_static_path entry '_static' does not exist
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                                                                                                                     
../../guides/TribitsGuidesBody.rst:6204: WARNING: Error in "include" directive:
no content permitted.

.. include:: ../../../examples/TribitsExampleProject/cmake/ctest/TribitsExProjCTestDriver.cmake
     :literal:

  and then create a set of CTest -S driver scripts that uses that file.  One
  example is the file
  ``TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake``:
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                                                                                                                      
generating indices... genindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 2 warnings.

The HTML pages are in _build/html.
===> Generating users_guide

Running Sphinx v4.1.0
making output directory... done
WARNING: html_static_path entry '_static' does not exist
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                                                                                                                     
../../guides/TribitsGuidesBody.rst:6204: WARNING: Error in "include" directive:
no content permitted.

.. include:: ../../../examples/TribitsExampleProject/cmake/ctest/TribitsExProjCTestDriver.cmake
     :literal:

  and then create a set of CTest -S driver scripts that uses that file.  One
  example is the file
  ``TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake``:
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                                                                                                                      
generating indices... genindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 2 warnings.

The HTML pages are in _build/html.
===> Generating build_ref

Running Sphinx v4.1.0
making output directory... done
WARNING: html_static_path entry '_static' does not exist
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 1 source files that are out of date
updating environment: [new config] 1 added, 0 changed, 0 removed
reading sources... [100%] index                                                                                                                                                               
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index                                                                                                                                                                
generating indices... genindex done
writing additional pages... search done
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 1 warning.

The HTML pages are in _build/html.

The updated repo state is:

$ git status
On branch 386-sphinx-doc
Your branch is up-to-date with 'marcinwrobel1986/386-Build-TriBITS-Documentation-with-Sphinx-and-ReadTheDocs'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   tribits/doc/guides/TribitsGuidesBody.rst
        modified:   tribits/doc/guides/TribitsToolsDocumentation.rst
        typechange: tribits/doc/guides/maintainers_guide/TribitsCoreDetailedReference.rst
        typechange: tribits/doc/guides/users_guide/TribitsCoreDetailedReference.rst

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        tribits/doc/sphinx/build_ref/_build/
        tribits/doc/sphinx/combined_docs/build_ref/
        tribits/doc/sphinx/combined_docs/maintainers_guide/
        tribits/doc/sphinx/combined_docs/users_guide/
        tribits/doc/sphinx/maintainers_guide/_build/
        tribits/doc/sphinx/users_guide/_build/

no changes added to commit (use "git add" and/or "git commit -a")

It is still modifying the original source files.

The new landing page is:

tribits/doc/sphinx/combined_docs/index.html

I was able to view all three separate documents in the local browser.

marcinwrobel1986 · 2021-07-15T14:58:31Z

However, it looks like the build process is still modifying manually maintained source files:
$ git diff --name-status 
M       tribits/doc/guides/TribitsGuidesBody.rst
M       tribits/doc/guides/TribitsToolsDocumentation.rst
T       tribits/doc/guides/maintainers_guide/TribitsCoreDetailedReference.rst
T       tribits/doc/guides/users_guide/TribitsCoreDetailedReference.rst
Can we meet soon to discuss how to adjust things to avoid this? A quick meeting to discuss the issues and the different options would be best.

Ross, this is just done when documentation is generated, it does not stay in the repository. This is needed for documentation to be correctly generated. It should not be a problem at all. The other way would be to change paths (include directive) in .rst files and adjust them for Sphinx, but I did not want to change anything in rst files. You said you got a plan next year, to split the rst files into smaller chunks. Then you could adjust paths as well.

Regarding meeting, sure I am available, but after last night storm I don't have my fiber working and I am on my mobile internet. This should be solved hopefully tonight.

NOTE: There are several issues with this Sphinx generated documentation including formatting issues (like not showing bulleted lists of links correctly), relative links between the documents are broken (because they assume the documents are side-by-side), the sphinx search results is only shows the "Introduction" section, the page title in the browser starts with "Introduction" and the messy TriBITS version but at least this is a good start. We can do that tweaking later. To merge this branch, we just need to not break the existing DoUtils documentation and we need to not have the build process modify any original source files.

As we discussed, regarding search, there is nothing that could be done as each documentation is a single html page. Regarding other issues I can't tell for now. You have to show me the current and desired behaviour and then maybe I could help.

bartlettroscoe · 2021-07-15T15:03:42Z

Ross, this is just done when documentation is generated, it does not stay in the repository.

@marcinwrobel1986, the problem is that when you are iteratively building and tweaking the documentation this becomes a major problem. I can give you some concreate development and debugging workflows where this becomes a major problem and is asking for trouble.

Why not just generate the *.rst files with the ./tribits/doc/build_docs.sh script, copy the entire tribits/doc/guides/ directory to tribits/doc/sphinx/guides/ and then this build process can make any changes it wants?

bartlettroscoe · 2021-07-15T15:07:58Z

Regarding other issues I can't tell for now. You have to show me the current and desired behavior and then maybe I could help.

@marcinwrobel1986, let's just get the basic mechanics of the build and deployment process done so we can merge this branch, deploy a process to automatically deploy updated versions of these documents, and link to them from tribits.org as alterative forms of these documents. Then we can do a more careful review of all of the issues and address them later. We then really need to get back to refactoring the guts of TriBITS to complete the bulk of epic #367 (and #63 and #299).

marcinwrobel1986 · 2021-07-15T15:15:27Z

Ross, this is just done when documentation is generated, it does not stay in the repository.

@marcinwrobel1986, the problem is that when you are iteratively building and tweaking the documentation this becomes a major problem. I can give you some concreate development and debugging workflows where this becomes a major problem and is asking for trouble.

Why not just generate the *.rst files with the ./tribits/doc/build_docs.sh script, copy the entire tribits/doc/guides/ directory to tribits/doc/sphinx/guides/ and then this build process can make any changes it wants?

Ross as I wrote:

The other way would be to change paths (include directive) in .rst files and adjust them for Sphinx, but I did not want to change anything in rst files. You said you got a plan next year, to split the rst files into smaller chunks. Then you could adjust paths as well.

Once you want to go with Sphinx, you can change .rst files in the repository and any changes during generation process will not be necessary. I did it only because it was not working with Sphinx.

bartlettroscoe · 2021-07-15T15:37:31Z

@marcinwrobel1986, I don't want to abandon the flat DocUtils-generated HTML files just yet because there are still a number of issues with the Sphinx generated documentation and I/we don't have the time right now to address all of them. At the same time, I don't want leave this great work sitting on a branch and I want people to have access to the (imperfect but still valuable) Sphinx generated documentation. Given all of that, I think the best course of action is to address the problem of modifying manually maintained *.rst files and the basic deployment issue and then I think we can merge. Then I think I will update tribits.org to point to both the flat DocUitils and the (in-progress) Sphinx generated versions of these three documents. Then as we have time later (likely in FY22 if we have the funding), resolve the remaining issues with the Sphinx-generated documentation and make the switch to only generate the Sphinx documentation and take advantage of all of the features of Sphinx (like breaking up into lots of smaller *.rst files and fixing the links between the different documents that we can't do until we switch to Sphinx only).

bartlettroscoe · 2021-07-15T16:01:23Z

Why not just generate the *.rst files with the ./tribits/doc/build_docs.sh script, copy the entire tribits/doc/guides/ directory to tribits/doc/sphinx/guides/ and then this build process can make any changes it wants?

@marcinwrobel1986, let me scope that out locally and see how that goes. Seems like that should be pretty easy to do.

…#386) This is not really the build reference for the TriBITS project per-say. It the build reference for an arbitrary TriBITS project <Project>. It does not make sense, I don't think, to produce this document for the TriBITS project itself since it only has one package. And people who are building the TriBITS package and running TriBITS test will know quite a lot about TriBITS so I think they can live with this generic document.

…st files (TriBITSPub#386) You have to actually call the tool create-project-build-ref.py to generate the final *.rst files. You just need to call it and tell it to not generate any of the other output (which is easy to do).

I just had to fix a minor defect in tribits/doc/build_ref/create-build-ref.sh and now this works fine with building the Sphinx documentation.

There is no need to cd into that directory anymore. It knows where it is and how to build the docs in-source.

I also make sure this does not run over top of the existing test build_docs (see comments in file).

bartlettroscoe · 2021-07-22T22:54:45Z

@marcinwrobel1986, I did the following:

Fix some issues and clean up commits:
- Fix titile of 'Generic' build reference [Done]
- Change to build_docs.sh --skip-final-generation [Done]
- Change to not have to cd to dir to call build_docs.sh [Done]
- Add test to check building of Sphinx documentation [Done]
- Run tests to build docutils and Sphinx documentation together [Done]
- Rebase branch on top of 'github/master' [Done]
- Examine the files that have changed and look for a chance to clean up some of the commits to remove some files that should not have been committed [Done]
- Push rebased and clenaed up commits of Marcin's [Done]
- Push my commits to my branch in my fork and create a staged PR for Marcin to review [Done]
- Push my commits to Marcins' branch in PR Build and deploy TriBITS documentation with Sphinx (#386) #387 [Done]
- Force push updated branch in PR [Done]
- Let GHA tests run ... All passed, including new TriBITS_build_sphinx_docs test [Done]

My more detailed review notes are below. The changes for my commits that I pushed are shown in marcinwrobel1986#1.

I think this is ready to merge and deploy!

DETAILED NOTES: (click to expand)

.

I pulled the updated copy of the branch 'marcinwrobel1986/386-Build-TriBITS-Documentation-with-Sphinx-and-ReadTheDocs' at the commit:

dd385c9 "#386: - only deploy when push to master"
Author: Marcin Wróbel <[email protected]>
Date:   Tue Jul 20 17:21:51 2021 +0200 (2 days ago)

M       .github/workflows/deploy_docs.yml

I went in and build the documentation locally on my RHEL 7 machine 'crf450' with:

$ cd ~/Trilinos.base2/Trilinos/TriBITS/tribits/doc/sphinx/

$ which sphinx-build
/home/rabartl/.local/bin/sphinx-build

$ which python3
/usr/bin/python3

$ python3 sphinx_rst_generator.py

After that, the state of the repo was:

$ git status
On branch 386-sphinx-doc
Your branch is up-to-date with 'marcinwrobel1986/386-Build-TriBITS-Documentation-with-Sphinx-and-ReadTheDocs'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)

        tribits/doc/sphinx/build_ref/_build/
        tribits/doc/sphinx/build_ref/index.rst
        tribits/doc/sphinx/combined_docs/build_ref/
        tribits/doc/sphinx/combined_docs/maintainers_guide/
        tribits/doc/sphinx/combined_docs/users_guide/
        tribits/doc/sphinx/copied_files/
        tribits/doc/sphinx/maintainers_guide/_build/
        tribits/doc/sphinx/maintainers_guide/index.rst
        tribits/doc/sphinx/users_guide/_build/
        tribits/doc/sphinx/users_guide/index.rst

nothing added to commit but untracked files present (use "git add" to track)

Yea, no modified source files!

Looking at the generated documentation starting at:

tribits/doc/sphinx/combined_docs/index.html

I was able to view the three updated documents.

To rebuild the documentation, I had to run:

$ git clean -xdf -- .

$ python3 sphinx_rst_generator.py

That seemed to work.

I rebased and cleaned up the commits and pushed. I created the stacked PR:

Updates to 386-Build-TriBITS-Documentation-with-Sphinx-and-ReadTheDocs (TriBITSPub/TriBITS#386) marcinwrobel1986/TriBITS#1

to show the changes and pushed all commits.

The files that were added are tiny:

$ du -sh tribits/doc/sphinx/
84K     tribits/doc/sphinx/

Very nice!

Now the tests are TriBITS GHA running ... They all passed!

You can't just run 'python3 sphinx_rst_generator.py' if you have already built the documentation locally. It will crash. But note the warning at the top about committing your work (or at least working in tracked non-ignored files).

It is important to be able to rebuild the documentation iteratively locally in a robust way. The rebuild makes this an expensive tests by TriBITS standards (over 35s on my super fast Linux machine) but this test is disabled by default and it only takes one core to run so hopefully this is not too bad.

I changed from 4-char to 2-char indents and wrapped lines to fit in 90 chars. This really should be reformatted a little closer to 85 chars or so but at least now I can do side-by-side editing on a reasonable side screen. I know the Python standard is 4 char indents but if you do that you need a lot more functions and a lot more wrapped lines to fit in 85 chars or so.

bartlettroscoe · 2021-07-23T21:57:20Z

@marcinwrobel1986, I know why the Sphinx documentation build generates the error:

../copied_files/TribitsGuidesBody.rst:6204: WARNING: Error in "include" directive:
no content permitted.

.. include:: ../copied_files/TribitsExProjCTestDriver.cmake
     :literal:

  and then create a set of CTest -S driver scripts that uses that file.  One
  example is the file
  ``TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake``:

The reason is because when script sphinx_rst_generator.py copies and replaces the include paths it takes the original includes in the file tribits/doc/guides/TribitsGuidesBody.rst:

4) Add custom CTest -S driver scripts.

  For driving different builds and tests, one needs to set up one or more
  CTest -S driver scripts.  There are various ways to do this but a simple
  approach that avoids duplication is to first create a file like
  ``TribitsExampleProject/cmake/ctest/TribitsExProjCTestDriver.cmake``:

  .. include:: ../../examples/TribitsExampleProject/cmake/ctest/TribitsExProjCTestDriver.cmake
     :literal:

  and then create a set of CTest -S driver scripts that uses that file.  One
  example is the file
  ``TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake``:

  .. include:: ../../examples/TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake
     :literal:

and it replaces it in the copied file tribits/doc/sphinx/copied_files/TribitsGuidesBody.rst with:

4) Add custom CTest -S driver scripts.

  For driving different builds and tests, one needs to set up one or more
  CTest -S driver scripts.  There are various ways to do this but a simple
  approach that avoids duplication is to first create a file like
  ``TribitsExampleProject/cmake/ctest/TribitsExProjCTestDriver.cmake``:

.. include:: ../copied_files/TribitsExProjCTestDriver.cmake
     :literal:

  and then create a set of CTest -S driver scripts that uses that file.  One
  example is the file
  ``TribitsExampleProject/cmake/ctest/general_gcc/ctest_serial_debug.cmake``:

.. include:: ../copied_files/ctest_serial_debug.cmake
     :literal:

5) Test CTest -S driver scripts

Note that it lost the indentation of two spaces before the includes. It turns out that you need that indentation.

Fixing this is likely easiest with a unit test for that code to highlight the problem and then to fix it. I will see if I can do that quickly with your code.

NOTE: I pushed the commit bafff87 to reformat that script to 90-chars wide so that I could view it in split view on a reasonable sized monitor (and with my bad eyes). Hopefully that is not an issue. I felt like I needed to do this to be able to effectively work with this code.

bartlettroscoe · 2021-07-23T22:23:45Z

@marcinwrobel1986, so I looked at writing a unit test for the function change_paths_and_get_includes() but it is embedded in the class SphinxRstGenerator with a very non-trivial constructor so you can't easily call it to unit test it. But the function change_paths_and_get_includes() only has trivial dependencies on the rest of that class. It will be trivial to factor that function out so it can be unit tested. In fact, many of the functions in that class should be free functions and don't need to be in that class. (Being in a class with a non-trivial constructor just makes them hard to unit test.)

…it tested (TriBITSPub#386) First, the function is_rst_file() no benefit from being a member of the class. This is not Java. Every function does not have to belong to a class. Second, the function change_paths_and_get_includes() has a very clear purpose and is unit testable but not when it was inside of the class. Removing it from the clas will allow it to be unit tested which will allow us to find and fix a defect quickly.

…#386) This shows the problem. Now debuging the code and fixing it and ensuring it stays fixed will be easy.

…SPub#386) This fixes a defect where loosing the indent was causing an error in the converted *.rst file.

bartlettroscoe · 2021-07-23T23:46:53Z

@marcinwrobel1986, it is fixed in commit 1d66677 with unit test to demonstrate and protect it.

Now this builds with zero errors.

…ITSPub#386) These includes got copies and pasted from another unit test driver.

…riBITSPub#386) When unit testing, you can't copy these files. I have no idea how this unit test passed locally when I ran it. In production mode, the default is to copy the files. This is actually a pretty nice algorithm. I also moved the unit test higher in the CMakeLists.txt file so this might make it more likely that this unit test runs before the actual build of the documentation.

bartlettroscoe

Seems like we could simplify this some after looking at this in some more detail after having to debug the include indentation issue. I will take a look at removing the field and see if my hunch is right.

bartlettroscoe · 2021-07-24T01:00:48Z

tribits/doc/sphinx/sphinx_rst_generator.py

+      'maintainers_guide': {
+        'src': os.path.join(doc_path, 'guides', 'maintainers_guide',
+           'TribitsMaintainersGuide.rst'),
+        'src_path': os.path.join(doc_path, 'guides', 'maintainers_guide'),


Why is this path duplicated in the 'src_path' field? This is the base path in the of the 'src' field in every case, no? Can't you just call os.path.dirname() on the src field and get this?

On second thought, there are more pressing things than to refine this code. It works. Time to merge and move on.

marcinwrobel1986 · 2021-07-24T06:39:39Z

@marcinwrobel1986, so I looked at writing a unit test for the function change_paths_and_get_includes() but it is embedded in the class SphinxRstGenerator with a very non-trivial constructor so you can't easily call it to unit test it. But the function change_paths_and_get_includes() only has trivial dependencies on the rest of that class. It will be trivial to factor that function out so it can be unit tested. In fact, many of the functions in that class should be free functions and don't need to be in that class. (Being in a class with a non-trivial constructor just makes them hard to unit test.)

@bartlettroscoe Hello Ross, this is not hard, you just need to know how to do that. There are no private methods, so you can test every method without any problems. If these functions would be use somwhere else, they would be outside a a class, but they are not. One can test change_paths_and_get_includes(), a few Mocks would be needed but that's it.

bartlettroscoe · 2021-07-24T21:02:29Z

One liner:

@marcinwrobel1986 that is essentially what I wrote on a couple of more readable/maintainable lines. After that function was isolated and run in a unit test harness (which was 90% of of the work), it was easy to fix. A good bit of that code likely could have been written faster with TDD and then there would have been unit tests left over at the end to support any needed future maintenance (win-win). All of this code does not meet the 5 criteria in when not to write automated tests.

Brings in numerous refactorings to TriBITS over the last 3 months, but there should be no breaks in backward compatibility. Almost every file in TriBITS is changed due to the lower-casing of command, macro and function names in PR TriBITSPub/TriBITS#379. But the main driver for this snapshot is to bring in the change in PR TriBITSPub/TriBITS#413 that should make it so that Kokkos INTERFACE_COMPILE_OPTIONS get propagated to downstream targets in TriBITS and therefore to external customers through installed <Package>Config.cmake files and IMPORTED targets. I should have done several snapshots in the last few months and not done a big snapshot like this (but I have been testing with Trilinos locally along the way). Overall, this merge brings in changes from a bunch of TriBITS PRs including (from most recent): * TriBITSPub/TriBITS#413: Change internal TriBITS target_link_libraries() to PUBLIC (TriBITSPub/TriBITS#299) component: core type: enhancement * TriBITSPub/TriBITS#410: Upgrade from cmake 3.21.0 to 3.21.2 (TriBITSPub/TriBITS#363, TriBITSPub/TriBITS#394) * TriBITSPub/TriBITS#394: DO NOT MERGE: Show TriBITS test failures with CMake 3.21.0 that don't occur with CMake 3.17.5 (TriBITSPub/TriBITS#363) * TriBITSPub/TriBITS#409: Add getTestDictStatusField() to handle empty 'status' field (SESW-383) component: ci_support type: enhancement * TriBITSPub/TriBITS#408: Deal with spaces in CDash url parts (SESW-383) component: ci_support type: enhancement * TriBITSPub/TriBITS#403: Spelling fixes * TriBITSPub/TriBITS#407: Move tribits_get_build_url_and_write_to_file() to stand-alone module (TriBITSPub/TriBITS#154) component: ctest_driver type: enhancement * TriBITSPub/TriBITS#388: Fixing typos (TriBITSPub/TriBITS#377) * TriBITSPub/TriBITS#406: Fix tribits_ctest_driver() package-by-package mode for CMake 3.19+ (TriBITSPub/TriBITS#363, TriBITSPub/TriBITS#394) component: ctest_driver type: bug * TriBITSPub/TriBITS#401: Improve GitHub Actions and CDash integration (TriBITSPub/TriBITS#154) type: enhancement * TriBITSPub/TriBITS#366: CI: draft action yaml * TriBITSPub/TriBITS#402: Revert some incorrect uppercase->lowercase changes * TriBITSPub/TriBITS#387: Build and deploy TriBITS documentation with Sphinx (TriBITSPub/TriBITS#386) component: documentation type: enhancement * TriBITSPub/TriBITS#398: Deal with pr null in not preprending build name (TriBITSPub/TriBITS#363) type: bug * TriBITSPub/TriBITS#396: Send PR results to 'Pull Request' CDash group and add PR ID to build names (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#397: Print the ninja path and version (TriBITSPub/TriBITS#363) * TriBITSPub/TriBITS#393: GitHub Actions based testing for TriBITS (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#389: TriBITS CI testing with GitHub Actions (TriBITSPub/TriBITS#363) type: enhancement * TriBITSPub/TriBITS#392: Fix broken tests for non-Fortran and CMake 3.21 builds (TriBITSPub/TriBITS#363) component: core * TriBITSPub/TriBITS#391: Fix up build_docs.sh for Sphinx doc generation (TriBITSPub/TriBITS#386) component: documentation type: enhancement * TriBITSPub/TriBITS#390: Add test for doc generation and fix usage of Python3 component: documentation type: bug * TriBITSPub/TriBITS#385: Replace last few references to TribitsDevelopersGuide.html (TriBITSPub/TriBITS#381) component: documentation type: enhancement * TriBITSPub/TriBITS#384: Split TribitsDevelopersGuide.* into TribitsUsersGuide.* and TribitsMaintainersGuide.* (TriBITSPub/TriBITS#381) component: documentation type: enhancement * TriBITSPub/TriBITS#383: Remove endfunction(<string>) and endmacro(<string>) (TriBITSPub/TriBITS#274, TriBITSPub/TriBITS#382) component: common_tpls type: bug * TriBITSPub/TriBITS#380: More package-arch data-structure documentation updates (TriBITSPub/TriBITS#63) component: documentation type: enhancement * TriBITSPub/TriBITS#379: Convert TriBITS to lower-case CMake command, macro, and function names (TriBITSPub/TriBITS#274) The following files were conflicting where I went with what is on the Trilinos 'develop' branch: * cmake/tribits/common_tpls/FindTPLBLAS.cmake * cmake/tribits/common_tpls/FindTPLLAPACK.cmake * cmake/tribits/common_tpls/FindTPLNetcdf.cmake (It looks like the above changes never made it back into TriBITS proper. The conflicts were due to the case changes in cmake command calls in these files due to TriBITSPub/TriBITS#379.) There was also a conflict in the file: * cmake/tribits/core/installation/TribitsProjectConfigTemplate.cmake.in I looked at that one carefully and I think that may have been due to fixes on both sides and then the case changes from TriBITSPub/TriBITS#379.

marcinwrobel1986 requested a review from bartlettroscoe July 9, 2021 06:59

marcinwrobel1986 self-assigned this Jul 9, 2021

bartlettroscoe added component: documentation type: enhancement labels Jul 9, 2021

bartlettroscoe changed the title ~~386 build tri bits documentation with sphinx and read the docs~~ Build and deploy TriBI (#386) Jul 9, 2021

bartlettroscoe changed the title ~~Build and deploy TriBI (#386)~~ Build and deploy TriBITS documentation with Sphinx and Read the Docs (#386) Jul 9, 2021

bartlettroscoe reviewed Jul 10, 2021

View reviewed changes

.github/workflows/deploy_docs.yml Outdated Show resolved Hide resolved

bartlettroscoe mentioned this pull request Jul 10, 2021

Build TriBITS Documentation with Sphinx and deploy to the web #386

Closed

4 tasks

bartlettroscoe reviewed Jul 10, 2021

View reviewed changes

tribits/doc/sphinx/requirements.txt Show resolved Hide resolved

bartlettroscoe changed the title ~~Build and deploy TriBITS documentation with Sphinx and Read the Docs (#386)~~ Build and deploy TriBITS documentation with Sphinx (#386) Jul 16, 2021

marcinwrobel1986 and others added 8 commits July 22, 2021 16:29

Changed page titles (TriBITSPub#386)

ddf4e65

Remove modification of .rst files (TriBITSPub#386)

a66427a

TriBITSPub#386: - only deploy when push to master

8d28c1b

Update to call build_docs.sh --skip-final-generation (TriBITSPub#363)

e5dd4ab

I just had to fix a minor defect in tribits/doc/build_ref/create-build-ref.sh and now this works fine with building the Sphinx documentation.

Remove need to cd into dir to call build_docs.sh (TriBITSPub#363)

07d244b

There is no need to cd into that directory anymore. It knows where it is and how to build the docs in-source.

Add test build_sphinx_docs (TriBITSPub#368)

f6a9311

I also make sure this does not run over top of the existing test build_docs (see comments in file).

bartlettroscoe force-pushed the 386-Build-TriBITS-Documentation-with-Sphinx-and-ReadTheDocs branch from dd385c9 to 8d28c1b Compare July 22, 2021 22:32

bartlettroscoe mentioned this pull request Jul 22, 2021

Updates to 386-Build-TriBITS-Documentation-with-Sphinx-and-ReadTheDocs (TriBITSPub/TriBITS#386) marcinwrobel1986/TriBITS#1

Merged

bartlettroscoe added 3 commits July 23, 2021 09:31

bartlettroscoe added 4 commits July 23, 2021 16:36

Add failing unit test for change_paths_and_get_includes() (TriBITSPub…

8e071ce

…#386) This shows the problem. Now debuging the code and fixing it and ensuring it stays fixed will be easy.

Fix bug to maintain indent for transformed include directives (TriBIT…

1d66677

…SPub#386) This fixes a defect where loosing the indent was causing an error in the converted *.rst file.

Add ctest driver for sphinx_rst_generator_UnitTests.py (TriBITSPub#386)

1708a5b

bartlettroscoe added 2 commits July 23, 2021 18:39

Remove unneeded package includes, including deprecated packages (TriB…

65396aa

…ITSPub#386) These includes got copies and pasted from another unit test driver.

bartlettroscoe reviewed Jul 24, 2021

View reviewed changes

bartlettroscoe merged commit 051ea5a into TriBITSPub:master Jul 24, 2021

bartlettroscoe mentioned this pull request Sep 4, 2021

TriBITS Snapshot update as of 2021-09-03 trilinos/Trilinos#9663

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build and deploy TriBITS documentation with Sphinx (#386) #387

Build and deploy TriBITS documentation with Sphinx (#386) #387

marcinwrobel1986 commented Jul 9, 2021 •

edited

Loading

bartlettroscoe commented Jul 10, 2021

bartlettroscoe commented Jul 12, 2021

marcinwrobel1986 commented Jul 13, 2021

bartlettroscoe commented Jul 13, 2021

bartlettroscoe commented Jul 13, 2021

bartlettroscoe commented Jul 13, 2021

marcinwrobel1986 commented Jul 14, 2021

bartlettroscoe commented Jul 14, 2021 •

edited

Loading

marcinwrobel1986 commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

marcinwrobel1986 commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

marcinwrobel1986 commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

marcinwrobel1986 commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

bartlettroscoe commented Jul 22, 2021

bartlettroscoe commented Jul 23, 2021 •

edited

Loading

bartlettroscoe commented Jul 23, 2021

bartlettroscoe commented Jul 23, 2021

bartlettroscoe left a comment •

edited

Loading

bartlettroscoe Jul 24, 2021

bartlettroscoe Jul 24, 2021

marcinwrobel1986 commented Jul 24, 2021

bartlettroscoe commented Jul 24, 2021 •

edited

Loading

Build and deploy TriBITS documentation with Sphinx (#386) #387

Build and deploy TriBITS documentation with Sphinx (#386) #387

Conversation

marcinwrobel1986 commented Jul 9, 2021 • edited Loading

This PR

Tasks

bartlettroscoe commented Jul 10, 2021

bartlettroscoe commented Jul 12, 2021

marcinwrobel1986 commented Jul 13, 2021

bartlettroscoe commented Jul 13, 2021

bartlettroscoe commented Jul 13, 2021

bartlettroscoe commented Jul 13, 2021

marcinwrobel1986 commented Jul 14, 2021

bartlettroscoe commented Jul 14, 2021 • edited Loading

marcinwrobel1986 commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

marcinwrobel1986 commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

marcinwrobel1986 commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

marcinwrobel1986 commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

bartlettroscoe commented Jul 15, 2021

bartlettroscoe commented Jul 22, 2021

bartlettroscoe commented Jul 23, 2021 • edited Loading

bartlettroscoe commented Jul 23, 2021

bartlettroscoe commented Jul 23, 2021

bartlettroscoe left a comment • edited Loading

Choose a reason for hiding this comment

bartlettroscoe Jul 24, 2021

Choose a reason for hiding this comment

bartlettroscoe Jul 24, 2021

Choose a reason for hiding this comment

marcinwrobel1986 commented Jul 24, 2021

bartlettroscoe commented Jul 24, 2021 • edited Loading

marcinwrobel1986 commented Jul 9, 2021 •

edited

Loading

bartlettroscoe commented Jul 14, 2021 •

edited

Loading

bartlettroscoe commented Jul 23, 2021 •

edited

Loading

bartlettroscoe left a comment •

edited

Loading

bartlettroscoe commented Jul 24, 2021 •

edited

Loading