Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Commit

Permalink
Merge remote-tracking branch 'origin/master' into mkldnn-v1.0
Browse files Browse the repository at this point in the history
  • Loading branch information
TaoLv committed Sep 10, 2019
2 parents e0cbc92 + 9675a2d commit 99b4961
Show file tree
Hide file tree
Showing 42 changed files with 1,648 additions and 97 deletions.
2 changes: 1 addition & 1 deletion 3rdparty/ps-lite
12 changes: 5 additions & 7 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ mxnet_option(USE_OLDCMAKECUDA "Build with old cmake cuda" OFF)
mxnet_option(USE_NCCL "Use NVidia NCCL with CUDA" OFF)
mxnet_option(USE_OPENCV "Build with OpenCV support" ON)
mxnet_option(USE_OPENMP "Build with Openmp support" ON)
mxnet_option(USE_OPENMP_BUNDLED_LLVM "Build with bundled llvm openmp from 3rdparty" OFF)
mxnet_option(USE_CUDNN "Build with cudnn support" ON) # one could set CUDNN_ROOT for search path
mxnet_option(USE_SSE "Build with x86 SSE instruction support" ON IF NOT ARM)
mxnet_option(USE_F16C "Build with x86 F16C instruction support" ON) # autodetects support if ON
Expand Down Expand Up @@ -432,11 +431,11 @@ if(USE_OPENMP)
find_package(OpenMP REQUIRED)
# This should build on Windows, but there's some problem and I don't have a Windows box, so
# could a Windows user please fix?
if(USE_OPENMP_BUNDLED_LLVM AND EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/openmp/CMakeLists.txt
AND SYSTEM_ARCHITECTURE STREQUAL "x86_64"
AND NOT MSVC
AND NOT CMAKE_CROSSCOMPILING)
message("Using bundlded LLVM OpenMP")
if(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/openmp/CMakeLists.txt
AND SYSTEM_ARCHITECTURE STREQUAL "x86_64"
AND NOT MSVC
AND NOT CMAKE_CROSSCOMPILING)

# Intel/llvm OpenMP: https://github.com/llvm-mirror/openmp
set(OPENMP_STANDALONE_BUILD TRUE)
set(LIBOMP_ENABLE_SHARED TRUE)
Expand All @@ -450,7 +449,6 @@ if(USE_OPENMP)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenMP_CXX_FLAGS}")
add_definitions(-DMXNET_USE_OPENMP=1)
else()
message("Using platform provided OpenMP")
if(OPENMP_FOUND)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} ${OpenMP_C_FLAGS}")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenMP_CXX_FLAGS}")
Expand Down
26 changes: 13 additions & 13 deletions CONTRIBUTORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,37 +26,37 @@ Committers are people who have made substantial contribution to the project and
The committers are the granted write access to the project.

* [Bing Xu](https://github.com/antinucleon)
- Bing is the initiator and major contributor of operators and ndarray modules of mxnet.
- Bing is the initiator and major contributor of operators and ndarray modules of MXNet.
* [Tianjun Xiao](https://github.com/sneakerkg)
- Tianqjun is the master behind the fast data loading and preprocessing.
* [Yutian Li](https://github.com/hotpxl)
- Yutian is the ninja behind the dependency and storage engine of mxnet.
- Yutian is the ninja behind the dependency and storage engine of MXNet.
* [Mu Li](https://github.com/mli)
- Mu is the contributor of the distributed key-value store in mxnet.
- Mu is the contributor of the distributed key-value store in MXNet.
* [Tianqi Chen](https://github.com/tqchen)
- Tianqi is one of the initiator of the mxnet project.
- Tianqi is one of the initiator of the MXNet project.
* [Min Lin](https://github.com/mavenlin)
- Min is the guy behind the symbolic magics of mxnet.
- Min is the guy behind the symbolic magics of MXNet.
* [Naiyan Wang](https://github.com/winstywang)
- Naiyan is the creator of static symbolic graph module of mxnet.
- Naiyan is the creator of static symbolic graph module of MXNet.
* [Mingjie Wang](https://github.com/jermainewang)
- Mingjie is the initiator, and contributes the design of the dependency engine.
* [Chuntao Hong](https://github.com/hjk41)
- Chuntao is the initiator and provides the initial design of engine.
* [Chiyuan Zhang](https://github.com/pluskid)
- Chiyuan is the creator of MXNet Julia Package.
- Chiyuan is the creator of MXNet Julia package.
* [Junyuan Xie](https://github.com/piiswrong)
* [Haibin Lin](https://github.com/eric-haibin-lin)
* [Qiang Kou](https://github.com/thirdwing)
- KK is a R ninja, he makes mxnet available for R users.
* [Tong He](https://github.com/hetong007)
- Tong is the major maintainer of MXNetR, he designs the mxnet interface and wrote many of the tutorials on R.
- Tong is the major maintainer of MXNet R package, he designs the MXNet interface and wrote many of the tutorials on R.
* [Yizhi Liu](https://github.com/yzhliu)
- Yizhi is the main creator on mxnet scala project to make deep learning available for JVM stacks.
* [Zixuan Huang](https://github.com/yanqingmen)
- Zixuan is one of major maintainers of mxnet scala package.
- Zixuan is one of major maintainers of MXNet Scala package.
* [Yuan Tang](https://github.com/terrytangyuan)
- Yuan is one of major maintainers of mxnet scala package.
- Yuan is one of major maintainers of MXNet Scala package.
* [Chris Olivier](https://github.com/cjolivier01)
* [Sergey Kolychev](https://github.com/sergeykolychev)
- Sergey is original author and current maintainer of Perl5 interface.
Expand Down Expand Up @@ -89,9 +89,9 @@ List of Contributors
* [Full List of Contributors](https://github.com/apache/incubator-mxnet/graphs/contributors)
- To contributors: please add your name to the list when you submit a patch to the project:)
* [Feng Wang](https://github.com/happynear)
- Feng makes mxnet compatible with Windows Visual Studio.
- Feng makes MXNet compatible with Windows Visual Studio.
* [Jack Deng](https://github.com/jdeng)
- Jack created the amalgamation script and Go bind for mxnet.
- Jack created the amalgamation script and Go bind for MXNet.
* [Li Dong](https://github.com/donglixp)
* [Piji Li](https://github.com/lipiji)
* [Hu Shiwen](https://github.com/yajiedesign)
Expand All @@ -103,7 +103,7 @@ List of Contributors
* [Nan Xiao](https://github.com/road2stat)
* [Wei Wu](https://github.com/tornadomeet)
* [Michaël Benesty](https://github.com/pommedeterresautee)
-Michaël contributes the R visualization module of mxnet
-Michaël contributes the R visualization module of MXNet
* [Kublai Jing](https://github.com/Kublai-Jing)
* [chenjx1005](https://github.com/chenjx1005)
* [ry](https://github.com/ry)
Expand Down
8 changes: 4 additions & 4 deletions ci/docker/runtime_functions.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1089,7 +1089,7 @@ unittest_ubuntu_gpu_R() {
unittest_ubuntu_cpu_julia() {
set -ex
export PATH="$1/bin:$PATH"
export MXNET_HOME='/work/mxnet'
export MXNET_ROOT='/work/mxnet'
export JULIA_DEPOT_PATH='/work/julia-depot'
export INTEGRATION_TEST=1

Expand All @@ -1099,7 +1099,7 @@ unittest_ubuntu_cpu_julia() {
export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libjemalloc.so'
export LD_LIBRARY_PATH=/work/mxnet/lib:$LD_LIBRARY_PATH

# use the prebuilt binary from $MXNET_HOME/lib
# use the prebuilt binary from $MXNET_ROOT/lib
julia --project=./julia -e 'using Pkg; Pkg.build("MXNet")'

# run the script `julia/test/runtests.jl`
Expand Down Expand Up @@ -1254,7 +1254,7 @@ build_docs() {

# Setup environment for Julia docs
export PATH="/work/julia10/bin:$PATH"
export MXNET_HOME='/work/mxnet'
export MXNET_ROOT='/work/mxnet'
export JULIA_DEPOT_PATH='/work/julia-depot'

julia -e 'using InteractiveUtils; versioninfo()'
Expand Down Expand Up @@ -1466,7 +1466,7 @@ deploy_docs() {

# Setup for Julia docs
export PATH="/work/julia10/bin:$PATH"
export MXNET_HOME='/work/mxnet'
export MXNET_ROOT='/work/mxnet'
export JULIA_DEPOT_PATH='/work/julia-depot'

julia -e 'using InteractiveUtils; versioninfo()'
Expand Down
2 changes: 1 addition & 1 deletion ci/windows/test_jl07_cpu.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
# set default output encoding to utf8
$PSDefaultParameterValues['Out-File:Encoding'] = 'utf8'

$env:MXNET_HOME = [System.IO.Path]::GetFullPath('.\windows_package')
$env:MXNET_ROOT = [System.IO.Path]::GetFullPath('.\windows_package')
$env:JULIA_URL = "https://julialang-s3.julialang.org/bin/winnt/x64/0.7/julia-0.7.0-win64.exe"
$env:JULIA_DEPOT_PATH = [System.IO.Path]::GetFullPath('.\julia-depot')

Expand Down
2 changes: 1 addition & 1 deletion ci/windows/test_jl10_cpu.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
# set default output encoding to utf8
$PSDefaultParameterValues['Out-File:Encoding'] = 'utf8'

$env:MXNET_HOME = [System.IO.Path]::GetFullPath('.\windows_package')
$env:MXNET_ROOT = [System.IO.Path]::GetFullPath('.\windows_package')
$env:JULIA_URL = "https://julialang-s3.julialang.org/bin/winnt/x64/1.0/julia-1.0.3-win64.exe"
$env:JULIA_DEPOT_PATH = [System.IO.Path]::GetFullPath('.\julia-depot')

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -407,8 +407,8 @@
(-> (executor/forward exec-test) (executor/outputs) (first))))))

(deftest test-maximum
(let [data1 (sym/variable "data")
data2 (sym/variable "data")
(let [data1 (sym/variable "data1")
data2 (sym/variable "data2")
shape-vec [3 4]
data-tmp1 (random/uniform 0 100 shape-vec)
data-tmp2 (random/uniform 0 100 shape-vec)
Expand All @@ -424,8 +424,8 @@
out))))

(deftest test-minimun
(let [data1 (sym/variable "data")
data2 (sym/variable "data")
(let [data1 (sym/variable "data1")
data2 (sym/variable "data2")
shape-vec [3 4]
data-tmp1 (random/uniform 0 100 shape-vec)
data-tmp2 (random/uniform 0 100 shape-vec)
Expand Down
42 changes: 42 additions & 0 deletions include/mxnet/c_api.h
Original file line number Diff line number Diff line change
Expand Up @@ -2649,6 +2649,48 @@ MXNET_DLL int MXKVStorePullRowSparseEx(KVStoreHandle handle,
const NDArrayHandle* row_ids,
int priority);

/*!
* \brief push and pull a list of (key, value) pairs from the kvstore
* \param handle handle to the kvstore
* \param vnum the number of key-value pairs corresponding to vkeys
* \param vkeys the list of keys for the values to be pushed
* \param onum the number of key-value pairs corresponding to okeys
* \param okeys the list of keys for the values to be pulled
* \param vals the list of values
* \param outs the list of outputs
* \param priority the priority of the action
* \return 0 when success, -1 when failure happens
*/
MXNET_DLL int MXKVStorePushPull(KVStoreHandle handle,
mx_uint vnum,
const int* vkeys,
mx_uint onum,
const int* okeys,
NDArrayHandle* vals,
NDArrayHandle* outs,
int priority);
/*!
* \brief push and pull a list of (key, value) pairs from the kvstore,
* where each key is a string
* \param handle handle to the kvstore
* \param vnum the number of key-value pairs corresponding to vkeys
* \param vkeys the list of keys for the values to be pushed
* \param onum the number of key-value pairs corresponding to okeys
* \param okeys the list of keys for the values to be pulled
* \param vals the list of values
* \param outs the list of outputs
* \param priority the priority of the action
* \return 0 when success, -1 when failure happens
*/
MXNET_DLL int MXKVStorePushPullEx(KVStoreHandle handle,
mx_uint vnum,
const char** vkeys,
mx_uint onum,
const char** okeys,
NDArrayHandle* vals,
NDArrayHandle* outs,
int priority);

/*!
* \brief user-defined updater for the kvstore
* It's this updater's responsibility to delete \a recv and \a local
Expand Down
27 changes: 27 additions & 0 deletions include/mxnet/kvstore.h
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,33 @@ class KVStore {
const std::vector<NDArray*>& values,
int priority = 0, bool ignore_sparse = true) = 0;

/*!
* \brief push and pull a list of key-value pairs from the store
* \param vkeys the list of keys to be pushed
* \param okeys the list of keys to be pulled. Should be the same set of keys in vkeys.
* \param values the list of values to be pushed
* \param outs the list of buffers for the pulled data, they should be preallocated
* \param priority Priority of the action.
*/
virtual void PushPull(const std::vector<int>& vkeys,
const std::vector<int>& okeys,
const std::vector<NDArray>& values,
const std::vector<NDArray*>& outs,
int priority = 0) = 0;

/*!
* \brief push and pull a list of key-value pairs from the store
* \param vkeys the list of keys to be pushed in string format
* \param okeys the list of keys to be pulled in string format. Should be the same set of keys in vkeys.
* \param values the list of values to be pushed
* \param outs the list of buffers for the pulled data, they should be preallocated
* \param priority Priority of the action.
*/
virtual void PushPull(const std::vector<std::string>& str_vkeys,
const std::vector<std::string>& str_okeys,
const std::vector<NDArray>& values,
const std::vector<NDArray*>& outs,
int priority = 0) = 0;
/*!
* \brief pull a list of key-value pairs from the store.
* The NDArray pulled back will be in row_sparse storage with only the
Expand Down
7 changes: 7 additions & 0 deletions julia/NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,13 @@

# v1.6.0

* Rename environment variable `MXNET_HOME` to `MXNET_ROOT` (#15568).

* Environment variable `MXNET_LIBRARY_PATH` supports (#15568).

```shell
$ MXNET_LIBRARY_PATH=/path/to/libmxnet.so julia
```

# v1.5.0

Expand Down
33 changes: 27 additions & 6 deletions julia/deps/build.jl
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,35 @@ libmxnet_detected = false
libmxnet_curr_ver = get(ENV, "MXNET_COMMIT", "master")
curr_win = "20190608" # v1.5.0

# TODO: remove MXNET_HOME backward compatibility in v1.7
if haskey(ENV, "MXNET_HOME")
MXNET_HOME = ENV["MXNET_HOME"]
@info("MXNET_HOME environment detected: $MXNET_HOME")
@warn "The environment variable `MXNET_HOME` has been renamed, please use `MXNET_ROOT` instead."
end

# TODO: remove MXNET_HOME backward compatibility in v2.0
MXNET_ROOT = get(ENV, "MXNET_ROOT", get(ENV, "MXNET_HOME", ""))
search_locations = if !isempty(MXNET_ROOT)
!isabspath(MXNET_ROOT) && error("MXNET_ROOT should be a absolute path")
@info "env var: MXNET_ROOT -> $MXNET_ROOT"
[joinpath(MXNET_ROOT, "lib"), MXNET_ROOT]
else
[]
end

MXNET_LIBRARY_PATH = get(ENV, "MXNET_LIBRARY_PATH", "")
println(typeof(MXNET_LIBRARY_PATH))
# In case of macOS, if user build libmxnet from source and set the MXNET_ROOT,
# the output is still named as `libmxnet.so`.
search_names = ["libmxnet.$(Libdl.dlext)", "libmxnet.so"]
if !isempty(MXNET_LIBRARY_PATH)
!isabspath(MXNET_LIBRARY_PATH) && error("MXNET_LIBRARY_PATH should be a absolute path")
@info "env var: MXNET_LIBRARY_PATH -> $MXNET_LIBRARY_PATH"
pushfirst!(search_names, MXNET_LIBRARY_PATH)
end

if (!isempty(MXNET_ROOT)) || (!isempty(MXNET_LIBRARY_PATH))
@info("Trying to load existing libmxnet...")
# In case of macOS, if user build libmxnet from source and set the MXNET_HOME,
# the output is still named as `libmxnet.so`.
lib = Libdl.find_library(["libmxnet.$(Libdl.dlext)", "libmxnet.so"],
[joinpath(MXNET_HOME, "lib"), MXNET_HOME])
lib = Libdl.find_library(search_names, search_locations)
if !isempty(lib)
@info("Existing libmxnet detected at $lib, skip building...")
libmxnet_detected = true
Expand Down
20 changes: 12 additions & 8 deletions julia/docs/src/user-guide/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,13 @@ libmxnet.

There are several environment variables that change this behaviour.

- `MXNET_HOME`: If you already have a pre-installed version of mxnet
you can use `MXNET_HOME` to point the build-process in the right direction.
- `CUDA_HOME`: If the automatic cuda detection fails you can also set `CUDA_HOME`
- `MXNET_ROOT`: If you already have a pre-installed version of mxnet
you can use `MXNET_ROOT` to point the build-process in the right direction.
Note that you should set this variable as a absolute path.
- `MXNET_HOME`: This variable is replaced by `MXNET_ROOT`. It has been renamed
as of v1.6.
- `MXNET_LIBRARY_PATH`: The absolute path of the libmxnet share object.
- `CUDA_HOME`: If the automatic CUDA detection fails you can also set `CUDA_HOME`
to override the process.
- `MXNET_COMMIT`: To control which version of libmxnet will be compiled,
you can use the`MXNET_COMMIT` variable to point to either a version tag
Expand Down Expand Up @@ -80,26 +84,26 @@ to work with a separate, maybe customized libmxnet.

To build libmxnet, please refer to [the installation guide of
libmxnet](https://mxnet.incubator.apache.org/install/index.html). After
successfully installing libmxnet, set the `MXNET_HOME` *environment
successfully installing libmxnet, set the `MXNET_ROOT` *environment
variable* to the location of libmxnet. In other words, the compiled
`libmxnet.so` should be found in `$MXNET_HOME/lib`.
`libmxnet.so` should be found in `$MXNET_ROOT/lib`.

> **note**
>
> The constant `MXNET_HOME` is pre-compiled in MXNet.jl package cache.
> The constant `MXNET_ROOT` is pre-compiled in MXNet.jl package cache.
> If you updated the environment variable after installing MXNet.jl,
> make sure to update the pre-compilation cache by
> `Base.compilecache("MXNet")`.

When the `MXNET_HOME` environment variable is detected and the
When the `MXNET_ROOT` environment variable is detected and the
corresponding `libmxnet.so` could be loaded successfully, MXNet.jl will
skip automatic building during installation and use the specified
libmxnet instead.

Basically, MXNet.jl will search `libmxnet.so` or `libmxnet.dll` in the
following paths (and in that order):

- `$MXNET_HOME/lib`: customized libmxnet builds
- `$MXNET_ROOT/lib`: customized libmxnet builds
- `Pkg.dir("MXNet", "deps", "usr", "lib")`: automatic builds
- Any system wide library search path

Expand Down
Loading

0 comments on commit 99b4961

Please sign in to comment.