New Website: New Pipeline [3/3]

[ThomasDelteil]

• New website: review from [2/3] to [3/3]: New Pipelines ThomasDelteil/incubator-mxnet#6 Review the step from [2/3] to [3/3]
• [WIP] New Website: New Docs [1/3] #15884 merged
• New Website: Remove Old Content [2/3] #15885 merged

Website available here: https://mxnet-beta.staged.apache.org/

Once merged:

• Change the mxnet-validation/website pointer to ci/jenkins/Jenkinsfile_website_full
• Turn on regular 4/day runs of the pipeline here: http://jenkins.mxnet-ci.amazon-ml.com/job/docs-pipelines/job/website-build-master/

Description

Instructions on using these updates are on the wiki: https://cwiki.apache.org/confluence/display/MXNET/Building+the+New+Website

Break out each docs package generation to separate pipelines.
✅Python - porting the mx-theme to be local to the repo to facilitate modification as part of this project's PRs; this is from the "beta site"
✅R - using the same theme as Python; also from the "beta site"; found to be not reproducible or current, so disabled for now.
✅C/CPP - currently co-mingled Doxygen microsite output
✅Julia - julia docs output to microsite
✅Scala - scaladocs output to microsite
✅Java - javadocs output to microsite
✅Clojure - lein output to microsite
✅Jekyll website for main pages

Legend:
✅Artifacts generating in CI
❌WIP

Jenkins Updates

• New docs folder pipelines to generate artifacts for each docs package - also has an MXNet binary build pipeline that can be used to share the latest binary as an archived artifact.
• Publish job - takes artifacts from the docs pipelines, collates the website and publishes it to incubator-mxnet-site.
  • using the last successful build URLs for each pipeline assures that the website gets published even if one docs package is failing
• Dockerfile updates - to speed up testing, each docs package has a custom dockerfile that only installs dependencies required for that specific package rather than installing everything (as is done now).
• New Python and R sites use Conda, so miniconda Python deps are installed with conda now.
• New support for Jekyll: Docker container and related runtime functions

Preview

The publishing job uses CI artifacts generated from my fork of mxnet and pushes to my fork of mxnet-site, and you can see the output on a dev server here:
http://54.211.76.74/

TODO when this PR is approved

• Add notifications for when docs package builds fail.
• Convert test pipelines to production website deployment flows.
• Upload old version artifacts.
• New PR triggering of a website test
• Switch to GPU builds for the Python docs, so the outputs are in the tutorials

Comments

• Stash operations vs. archiving - recommendations in the docs suggest that large artifacts should be archived; stash is super slow; archived artifacts seems to be faster and can be used between pipelines. This is helpful for the MXNet binary and for the Scala package, both of which are used by various other docs packages. However, there's an implication with the master server. Archived artifacts are stored there, so if the pipeline is related to PR validation, this would be unwieldy. If related to publishing final artifacts for specific versions, well, that's probably ok.
• It seems that efficiency in development and testing can be gained by checkpointing the docker containers after the dependencies are installed.
• A version/branch parameter would be useful for the Jenkins pipelines for generating docs artifacts from different branches.
• Publishing scripts seem to need a security refactor, or we don't bother offering stand-alone access to them; running local versus on Jenkins.
• A bug bash is needed on the new Python docs - I turned off "warnings as errors" just so I could get this thing working, but that needs to be turned back on.

[anirudh2290]

ThomasDelteil#6 (comment) this comment is not incorporated here. Is this still being used ?

[ThomasDelteil]

not it is not, deleting

[anirudh2290]

will you be adding this : ThomasDelteil#6 (comment)

[ThomasDelteil]

actually this is called from other functions, does not require the set -ex

[aaronmarkham]

SHIP IT! Assuming it passes CI of course! 📿 🙏

[ThomasDelteil]

CI passing, merging. If any issue ping @ThomasDelteil, @aaronmarkham, @sad-

[QueensGambit]

@ThomasDelteil, @aaronmarkham, @sad-

There a few dead links or missing files for the MXNet C++ package:

• https://mxnet-beta.staged.apache.org/get_started/ubuntu_setup.html

Install the MXNet Package for C++ (also missing in contents (quick links))

• https://mxnet-beta.staged.apache.org/get_started/c_plus_plus

404 Not Found

The requested URL was not found on this server.

• https://mxnet-beta.staged.apache.org/api/

C/C++ Guide ›

• https://mxnet-beta.staged.apache.org/api/cpp/
  leads to

Index of /api/cpp

• https://mxnet-beta.staged.apache.org/get_started/windows_setup.html#build-from-source
  has wrong formatting at a few cases.

[aaronmarkham]

@ThomasDelteil, @aaronmarkham, @sad-

There a few dead links or missing files for the MXNet C++ package:

• https://mxnet-beta.staged.apache.org/get_started/ubuntu_setup.html

Install the MXNet Package for C++ (also missing in contents (quick links))

• https://mxnet-beta.staged.apache.org/get_started/c_plus_plus

404 Not Found

The requested URL was not found on this server.

• https://mxnet-beta.staged.apache.org/api/

C/C++ Guide ›

• https://mxnet-beta.staged.apache.org/api/cpp/
  leads to

Index of /api/cpp

• https://mxnet-beta.staged.apache.org/get_started/windows_setup.html#build-from-source
  has wrong formatting at a few cases.

@QueensGambit Thanks for the feedback. Can you create separate issues for anything you find, so others can pick them up and work on them?