This repository has been archived by the owner on Nov 17, 2023. It is now read-only.
fix broken links and reorganize build from source page #12962
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,93 +1,83 @@ | ||
| # Build MXNet from Source | ||
|
|
||
| This document explains how to build MXNet from source code. | ||
|
|
||
|
|
||
| ## Overview | ||
|
|
||
| Building from source follows this general two-step flow of building the shared library, then installing your preferred language binding. Use the following links to jump to the different sections of this guide. | ||
|
|
||
| 1. Build the MXNet shared library, `libmxnet.so`. | ||
| * [Clone the repository](#clone-the-mxnet-project) | ||
| * [Prerequisites](#prerequisites) | ||
| * [Math library selection](#math-library-selection) | ||
| * [Install GPU software](#install-gpu-software) | ||
| * [Install optional software](#install-optional-software) | ||
| * [Adjust your build configuration](#build-configurations) | ||
| * [Build MXNet](#build-mxnet) | ||
| * [with NCCL](#build-mxnet-with-nccl) (optional) | ||
| * [for C++](#build-mxnet-with-c++) (optional) | ||
| * [Usage Examples](#usage-examples) | ||
| * [systems with GPUs and Intel CPUs](#recommended-for-Systems-with-NVIDIA-GPUs-and-Intel-CPUs) | ||
| * [GPUs with non-Intel CPUs](#recommended-for-Systems-with-Intel-CPUs) | ||
| * [Intel CPUs](#recommended-for-Systems-with-Intel-CPUs) | ||
| * [non-Intel CPUs](#recommended-for-Systems-with-non-Intel-CPUs) | ||
| 2. [Install the language API binding(s)](#installing-mxnet-language-bindings) you would like to use for MXNet. | ||
| MXNet's newest and most popular API is Gluon. Gluon is built into the Python binding. If Python isn't your preference, you still have more options. MXNet supports several other language APIs: | ||
| - [Python (includes Gluon)](../api/python/index.html) | ||
| - [C++](../api/c++/index.html) | ||
| - [Clojure](../api/clojure/index.html) | ||
| - Java (coming soon) | ||
| - [Julia](../api/julia/index.html) | ||
| - [Perl](../api/perl/index.html) | ||
| - [R](../api/r/index.html) | ||
| - [Scala](../api/scala/index.html) | ||
|
|
||
| <hr> | ||
|
|
||
| ## Build Instructions by Operating System | ||
|
|
||
| Detailed instructions are provided per operating system. Each of these guides also covers how to install the specific [Language Bindings](#installing-mxnet-language-bindings) you require. | ||
| You may jump to those, but it is recommended that you continue reading to understand more general "build from source" options. | ||
|
|
||
| * [Amazon Linux / CentOS / RHEL](centos_setup.html) | ||
| * [macOS](osx_setup.html) | ||
| * [Raspbian](raspian_setup.html) | ||
| * [TX2](tx2_setup.html) | ||
| * [Ubuntu](ubuntu_setup.html) | ||
| * [Windows](windows_setup.html) | ||
|
|
||
|
|
||
| <hr> | ||
|
|
||
| ## Clone the MXNet Project | ||
|
|
||
| 1. Clone or fork the MXNet project. | ||
| ```bash | ||
| git clone --recursive https://github.com/apache/incubator-mxnet mxnet | ||
| cd mxnet | ||
| ``` | ||
|
|
||
| <hr> | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| The following sections will help you decide which specific prerequisites you need to install. | ||
|
|
||
| #### Math Library Selection | ||
| It is useful to consider your math library selection prior to your other prerequisites. | ||
| MXNet relies on the | ||
| [BLAS](https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprograms) (Basic | ||
| Linear Algebra Subprograms) library for numerical computations. | ||
| Those can be extended with [LAPACK (Linear Algebra Package)](https://github.com/Reference-LAPACK/lapack), an additional set of mathematical functions. | ||
|
|
||
| MXNet supports multiple mathematical backends for computations on the CPU: | ||
| * [Apple Accelerate](https://developer.apple.com/documentation/accelerate) | ||
| * [ATLAS](http://math-atlas.sourceforge.net/) | ||
| * [MKL](https://software.intel.com/en-us/intel-mkl) (MKL, MKLML) | ||
| * [MKL-DNN](https://github.com/intel/mkl-dnn) | ||
| * [OpenBLAS](http://www.openblas.net/) | ||
|
|
||
| The default order of choice for the libraries if found follows the path from the most | ||
| (recommended) to less performant backends. | ||
| The following lists show this order by library and `cmake` switch. | ||
|
|
@@ -122,21 +112,21 @@ https://software.intel.com/en-us/mkl | |
| It has following flavors: | ||
|
|
||
| * MKL is a complete math library, containing all the functionality found in ATLAS, OpenBlas and LAPACK. It is free under | ||
| community support licensing (https://software.intel.com/en-us/articles/free-mkl), | ||
| but needs to be downloaded and installed manually. | ||
|
|
||
| * MKLML is a subset of MKL. It contains a smaller number of functions to reduce the | ||
| size of the download and reduce the number of dynamic libraries user needs. | ||
|
|
||
| <!-- [Removed until #11148 is merged.] This is the most effective option since it can be downloaded and installed automatically | ||
| by the cmake script (see cmake/DownloadMKLML.cmake).--> | ||
|
|
||
| * MKL-DNN is a separate open-source library, it can be used separately from MKL or MKLML. It is | ||
| shipped as a subrepo with MXNet source code (see 3rdparty/mkldnn or the [MKL-DNN project](https://github.com/intel/mkl-dnn)) | ||
|
|
||
| Since the full MKL library is almost always faster than any other BLAS library it's turned on by default, | ||
| however it needs to be downloaded and installed manually before doing `cmake` configuration. | ||
| Register and download on the [Intel performance libraries website](https://software.intel.com/en-us/performance-libraries). | ||
|
|
||
| Note: MKL is supported only for desktop builds and the framework itself supports the following | ||
| hardware: | ||
|
|
@@ -150,6 +140,32 @@ If you have a different processor you can still try to use MKL, but performance | |
| unpredictable. | ||
|
|
||
|
|
||
| #### Install GPU Software | ||
|
|
||
| If you want to run MXNet with GPUs, you must install [NVDIA CUDA and cuDNN](https://developer.nvidia.com/cuda-downloads). | ||
|
|
||
|
|
||
| #### Install Optional Software | ||
|
|
||
| These might be optional, but they're typically desirable as the extend or enhance MXNet's functionality. | ||
|
|
||
| * [OpenCV](http://opencv.org/) - Image Loading and Augmentation. Each operating system has different packages and build from source options for OpenCV. Refer to your OS's link in the [Build Instructions by Operating System](#build-instructions-by-operating-system) section for further instructions. | ||
| * [NCCL](https://developer.nvidia.com/nccl) - NVIDIA's Collective Communications Library. Instructions for installing NCCL are found in the following [Build MXNet with NCCL](#build-mxnet-with-nccl) section. | ||
|
|
||
| More information on turning these features on or off are found in the following [build configurations](#build-configurations) section. | ||
|
|
||
|
|
||
| <hr> | ||
|
|
||
| ## Build Configurations | ||
|
|
||
| There is a configuration file for make, | ||
| [`make/config.mk`](https://github.com/apache/incubator-mxnet/blob/master/make/config.mk), that contains all the compilation options. You can edit it and then run `make` or `cmake`. `cmake` is recommended for building MXNet (and is required to build with MKLDNN), however you may use `make` instead. | ||
|
|
||
| <hr> | ||
|
|
||
| ## Build MXNet | ||
|
|
||
| ### Build MXNet with NCCL | ||
| - Download and install the latest NCCL library from NVIDIA. | ||
| - Note the directory path in which NCCL libraries and header files are installed. | ||
|
|
@@ -183,53 +199,88 @@ nosetests --verbose tests/python/gpu/test_nccl.py | |
| **Recommendation to get the best performance out of NCCL:** | ||
| It is recommended to set environment variable NCCL_LAUNCH_MODE to PARALLEL when using NCCL version 2.1 or newer. | ||
|
|
||
| <hr> | ||
|
|
||
| ### Build MXNet with C++ | ||
|
|
||
| * To enable C++ package, just add `USE_CPP_PACKAGE=1` when you run `make` or `cmake`. | ||
|
|
||
| <hr> | ||
|
|
||
| ### Usage Examples | ||
|
|
||
| * `-j` runs multiple jobs against multi-core CPUs. | ||
|
|
||
| For example, you can specify using all cores on Linux as follows: | ||
|
|
||
| ```bash | ||
| cmake -j$(nproc) | ||
| ``` | ||
|
|
||
|
|
||
| #### Recommended for Systems with NVIDIA GPUs and Intel CPUs | ||
| * Build MXNet with `cmake` and install with MKL DNN, GPU, and OpenCV support: | ||
|
|
||
| ```bash | ||
| cmake -j USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 USE_MKLDNN=1 | ||
|
There was a problem hiding this comment. we usually give these default build options for user to use while building from source. Should we have a section describing different build options from here - https://github.com/apache/incubator-mxnet/blob/master/make/config.mk explaining what each of those do so that the user can make an informed personalized choice for himself? Provide the default options, but also add this section. Thoughts? There was a problem hiding this comment. It's interesting that we tend to promote using MKLDNN for better performance and in the CMAKE instructions, they imply that it is turned on by default. And yes, it should be mentioned what happens when you just run Looking this config.mk file actually makes things even more unclear... @anirudhacharya Can you propose the logic that is followed and the recommended flags per general situation? There was a problem hiding this comment. There is a conversation here - https://lists.apache.org/thread.html/ca3c858a712a66f4b3443dfb395d84e8fdf2c86e297bcd28a3a36ebd@%3Cdev.mxnet.apache.org%3E on including mkl-dnn in mxnet default build. I think the recommended flags seem okay for now, the way it is. As and when it is decided to include MKL-DNN in the default build, then the recommended flags can also be updated. What I was suggesting was to have a section in the documentation that describes the flags already provided in the config.mk file( each environment variable has a comment describing what it does). Please let me know if I need to help in any way here, and I would be glad to. |
||
| ``` | ||
|
|
||
| #### Recommended for Systems with NVIDIA GPUs | ||
| * Build with both OpenBLAS, GPU, and OpenCV support: | ||
|
|
||
| ```bash | ||
| cmake -j BLAS=open USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 | ||
| ``` | ||
|
|
||
| #### Recommended for Systems with Intel CPUs | ||
| * Build MXNet with `cmake` and install with MKL DNN, and OpenCV support: | ||
|
|
||
| ```bash | ||
| cmake -j USE_CUDA=0 USE_MKLDNN=1 | ||
| ``` | ||
|
|
||
| #### Recommended for Systems with non-Intel CPUs | ||
| * Build MXNet with `cmake` and install with OpenBLAS and OpenCV support: | ||
|
|
||
| ```bash | ||
| cmake -j USE_CUDA=0 BLAS=open | ||
| ``` | ||
|
|
||
| #### Other Examples | ||
|
|
||
| * Build without using OpenCV: | ||
|
|
||
| ```bash | ||
| cmake USE_OPENCV=0 | ||
| ``` | ||
|
|
||
| * Build on **macOS** with the default BLAS library (Apple Accelerate) and Clang installed with `xcode` (OPENMP is disabled because it is not supported by the Apple version of Clang): | ||
|
|
||
| ```bash | ||
| cmake -j BLAS=apple USE_OPENCV=0 USE_OPENMP=0 | ||
| ``` | ||
|
|
||
| * To use OpenMP on **macOS** you need to install the Clang compiler, `llvm` (the one provided by Apple does not support OpenMP): | ||
|
|
||
| ```bash | ||
| brew install llvm | ||
| cmake -j BLAS=apple USE_OPENMP=1 | ||
| ``` | ||
|
|
||
| <hr> | ||
|
|
||
| ## Installing MXNet Language Bindings | ||
| After building MXNet's shared library, you can install other language bindings. | ||
|
|
||
| **NOTE:** The C++ API binding must be built when you build MXNet from source. See [Build MXNet with C++](#build-mxnet-with-c++). | ||
|
|
||
| The following table provides links to each language binding by operating system: | ||
| | | [Ubuntu](ubuntu_setup.html) | [macOS](osx_setup.html) | [Windows](windows_setup.html) | | ||
| | --- | ---- | --- | ------- | | ||
| | Python | [Ubuntu guide](ubuntu_setup.html#install-mxnet-for-python) | [OSX guide](osx_setup.html) | [Windows guide](windows_setup.html#install-mxnet-for-python) | | ||
| | C++ | [C++ guide](c_plus_plus.html) | [C++ guide](c_plus_plus.html) | [C++ guide](c_plus_plus.html) | | ||
| | Clojure | [Clojure guide](https://github.com/apache/incubator-mxnet/tree/master/contrib/clojure-package) | [Clojure guide](https://github.com/apache/incubator-mxnet/tree/master/contrib/clojure-package) | n/a | | ||
| | Julia | [Ubuntu guide](ubuntu_setup.html#install-the-mxnet-package-for-julia) | [OSX guide](osx_setup.html#install-the-mxnet-package-for-julia) | [Windows guide](windows_setup.html#install-the-mxnet-package-for-julia) | | ||
| | Perl | [Ubuntu guide](ubuntu_setup.html#install-the-mxnet-package-for-perl) | [OSX guide](osx_setup.html#install-the-mxnet-package-for-perl) | n/a | | ||
| | R | [Ubuntu guide](ubuntu_setup.html#install-the-mxnet-package-for-r) | [OSX guide](osx_setup.html#install-the-mxnet-package-for-r) | [Windows guide](windows_setup.html#install-the-mxnet-package-for-r) | | ||
| | Scala | [Scala guide](scala_setup.html) | [Scala guide](scala_setup.html) | n/a | | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The flow does not seem right. We are taking about Installing API bindings and we redirect the users to API page, we should redirect them to the language install page instead.