[WIP] Improving Python Docs API (#16392)

* doc fixes for gluon api sub modules only

* update to the docs

* adding the autodoc js plugin

* adding autodoc javascript, fixing some warnings

* fixing model zoo

* python api docs major refactor -> ndarray, symbol and other mxnet libraries

* restructure top level toctree navs and bubble more submodules to the top

* modify import to reference defined classes and functions directly

* fix some broken links

* add extra classes imported from c++

* fix .htaccess redirects since api links have changed again

* included license on files missing license

* Update monitor.py

docs/python_docs/README.md

@@ -21,4 +21,118 @@ Preview at https://mxnet-beta.staged.apache.org/
 
 ## [Building the Docs and Website](https://cwiki.apache.org/confluence/display/MXNET/Building+the+New+Website)
 
-## [Technical details for the building the Python microsite](python/README.md)
+## [Technical details for the building the Python microsite](python/README.md)
+
+# Python binding docs
+
+The following guide will help you build a local version of the Python API website,
+so that you may work on and test any contributions.
+
+It is recommended that you read the MXNet developer wiki's info on [building the website & docs](https://cwiki.apache.org/confluence/display/MXNET/Building+the+New+Website) as that includes info on how to test and build the site using Docker. The following information should only be used if you can't use Docker or if you're trying to run the site locally.
+
+## Setup
+
+The default configuration requires a GPU and CUDA 9.2 and expects Ubuntu.
+However, you may setup the website on macOS or Windows with or without a GPU.
+
+### Prerequisites
+
+* [Conda >= 4.6.13](https://www.anaconda.com/distribution/#download-section) (install to PATH)
+
+You can update Conda with the following:
+
+```
+conda update conda
+```
+
+### GPU setup
+To run the full build, including tests of all tutorials,
+**you will need at least two GPUs**.
+Distributed training is a key feature of MXNet,
+so multiple GPUs are required for running through every tutorial.
+* [CUDA 9.2](https://developer.nvidia.com/cuda-downloads)
+
+### CPU-only setup
+In the `environment.yml` file:
+* Change `mxnet-cu92` to `mxnet`.
+
+### macOS setup
+In the `environment.yml` file:
+* Change `mxnet-cu92` to `mxnet`. (There is no CUDA package for mac anyway.)
+* Change `pytorch-cpu` to `pytorch`.
+* Change `torchvision-cpu` to `torchvision`.
+
+### Windows Setup
+If you have a GPU and have installed CUDA 9.2 you can leave the MXNet dependency alone.
+Otherwise, in the `environment.yml` file:
+* Change `mxnet-cu92` to `mxnet`.
+
+Install recommended software:
+* [git bash](https://gitforwindows.org/)
+* Be sure to install `Conda` in `PATH`
+* Install `make` from a `git bash` terminal with Admin rights
+    - [Install chocolatey](https://chocolatey.org/install)
+    - Use `choco to install make`
+* Restart terminals after installations to make sure PATH is set.
+    - The `choco`, `make`, and `conda` commands should work in `git bash`.
+
+### Conda environment setup
+Run the following commands from the project root (`python-docs`) to setup the environment.
+
+```bash
+conda env create -f environment.yml
+source activate mxnet-docs
+```
+
+## Build the docs
+
+* Change directories to `python-docs/python`.
+
+To build without GPUs and without testing the notebooks (faster):
+
+```bash
+make EVAL=0
+```
+
+To build with testing the notebooks (requires GPU):
+
+```bash
+make
+```
+
+The build docs will be available at `build/_build/html`.
+
+Each build may take a few minutes even without evaluation. To accelerate it, we can use one of the following ways:
+
+1. open `build/conf.py`, add the folders you want to skip into `exclude_patterns`, such as `exclude_patterns = ['templates', 'api', 'develop', 'blog']`.
+2. move the files into a different folder, such as `mv api /tmp/`, and then `make clean`.
+
+## Check results
+
+To run a server to see the website:
+
+1. Start a http server: `cd build/_build/html; python -m http.server`
+2. For viewing a remote machine, ssh to your machine with port forwarding: `ssh -L8000:localhost:8000 your_machine`
+3. Open http://localhost:8000 in your local machine
+
+## Run tutorials
+
+In addition to view the built html pages, you can run the Jupyter notebook from a remote machine.
+1. Install `notedown` plugin: `pip install https://github.com/mli/notedown/tarball/master` in remote server
+2. Start Jupyter notebook `jupyter notebook --NotebookApp.contents_manager_class='notedown.NotedownContentsManager'` in remote server
+3. ssh to your machine with port forwarding: `ssh -L8888:localhost:8888 your_machine`
+4. Open http://localhost:8888 in your local machine and run the md files directly
+
+Optionally, one can run the following to launch the notedown plugin automatically when starting jupyter notebook.
+1. Generate the jupyter configure file `~/.jupyter/jupyter_notebook_config.py` if it
+is not existing by run `jupyter notebook --generate-config`
+2. Add `c.NotebookApp.contents_manager_class = 'notedown.NotedownContentsManager'` to `~/.jupyter/jupyter_notebook_config.py`
+3. Simply run `jupyter notebook`
+
+## Troubleshooting
+Dependencies and the setup steps for this website are changing often. Here are some troubleshooting tips.
+
+* You might need to update the environment for the latest modules.
+```bash
+conda env update -f environment.yml
+```

docs/python_docs/_static/autodoc.js

@@ -0,0 +1,49 @@
+/*!
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/* Customizations to the Sphinx auto module plugin output */
+function auto_index() {
+    var targets = $("dl.class>dt,dl.function>dt");
+
+    var li_node = $("li.current>span>a.current.reference.internal").parent().parent();
+    var html = "<ul id='autodoc'>";
+    if (li_node.length > 0) {
+        for (var i = 0; i < targets.length; ++i) {
+            var id = $(targets[i]).attr('id');
+            if (id) {
+                var paths = id.split('.')
+                if (paths.length >= 2) {
+                    var id_simple = paths.pop();
+                    id_simple = paths.pop() + "." + id_simple;
+                } else {
+                    var id_simple = id;
+                }
+                html += "<li><span class='link-wrapper'><a class='reference internal' href='#";
+                html += id;
+                html += "'>" + id_simple + "</a></span</li>";
+            }
+        }
+        html += "</ul>";
+        li_node.append(html);
+        li_node.prepend("<a><span id='autodoc_toggle' onclick='$(\"#autodoc\").toggle()'>[toggle]</span></a>")
+    } else {
+        setTimeout(auto_index, 500);
+    }
+}
+$(document).ready(auto_index);

docs/python_docs/_static/mxnet.css

@@ -196,4 +196,11 @@ p {
 
 .install .title {
     margin-top: 1em;
+}
+
+/* autodoc */
+#autodoc_toggle {
+    float: right;
+    margin: 4px;
+    cursor: pointer;
 }

docs/python_docs/environment.yml

@@ -36,3 +36,4 @@ dependencies:
   - breathe
   - mock
   - awscli
+  - autodocsumm

docs/python_docs/python/Makefile

@@ -25,7 +25,6 @@ IPYNB_MARKDOWN = $(shell find . -not -path "./build/*" -not -path "*.ipynb_check
 RST = $(shell find . -not -path "./build/*" -not -path "*.ipynb_checkpoints*" -name "*.rst")
 
 OBJ = $(patsubst %.rst, build/%.rst, $(RST)) \
-	$(patsubst %.md, build/%.md, $(PURE_MARKDOWN)) \
 	$(patsubst %.md, build/%.ipynb, \
 		$(filter-out $(PURE_MARKDOWN), $(IPYNB_MARKDOWN)))
 
@@ -47,7 +46,7 @@ build/%: %
 html: $(OBJ)
 	mkdir -p build
 	cp Makefile_sphinx build/Makefile
-	sphinx-autogen build/api/*.rst build/api/*/*.rst   -t build/_templates/
+	sphinx-autogen build/api/*.rst build/api/**/*.rst   -t build/_templates/
 	# make -C build linkcheck doctest html
 	make -C build html
 	sed -i.bak 's/33\,150\,243/23\,141\,201/g'  build/_build/html/_static/material-design-lite-1.3.0/material.blue-deep_orange.min.css