-
Notifications
You must be signed in to change notification settings - Fork 6.8k
import Julia binding #10149
import Julia binding #10149
Changes from all commits
d1eb318
c06b211
b81b26c
1ea44c9
4d0aa87
ab80048
2393d4a
179daa5
1b09fb4
030990a
22a3c72
2a66ea3
3ad5819
8d044b1
15c7052
b039771
6335eca
d9f9551
a6968a6
a79d002
a476d3e
11deaab
ba3c9fb
fecfcdd
fb2a0d0
cac5625
d0d7963
d59b214
0ee926e
79d80b6
75060bc
f33f1dc
17a5442
c58eb46
fa25ad6
f23b0a9
96191ea
ba97d93
09d8f27
cf7f32b
c9eb71b
1781290
9474d95
46440e3
9dd3291
fc2974f
5c02fcb
867b1c5
31b95fb
1be1b87
7a6120a
bfa966a
8947ead
44a6c36
558adb7
b4ece81
ced3f73
86e3fa0
3263b95
d0253f1
68b67e8
1697522
e12052c
def129f
c2044d5
349d2a7
4675ace
c4a2658
9aac7e1
7a530fa
06010b8
6cc9e99
4cd5f3f
5d16305
02c7661
8918f1b
60f5306
efff124
07777d6
28344a4
96bdfa7
aae9114
2ecf2c1
9c70922
b63fd9b
c7032f0
172af60
2b26bc2
841e9ee
c965235
18547b8
e7d1b62
10ad90b
0750c41
778529c
85a2d97
873715b
8bd307f
4e5ab13
d1bbcc4
0d54f4b
f34643b
b68ce5b
21ccc56
23866bd
2c5120d
4e439ed
7cc66d1
5d739da
f20534d
a51cba4
16c2c5a
d824631
e3ad29d
dcba690
3d4adfb
bbd0e66
b68ca2e
0b9bf78
4e89c91
63a34cc
68d6ff7
a2e270e
bb88d79
9e0844a
55ccbfa
3595403
df13ddd
6d1ba53
ff0556b
5b3c507
251a8b3
ebb03d0
077a8cb
0a53778
ac7bbd0
904eb20
aae016c
19082ef
9d90506
15def2c
81b134f
a79e33c
71f2d40
3824df0
41f8f55
9fcab40
4f182ee
a3317f1
8cc5c1c
a488d7a
57cc677
eea128a
f839be1
4919273
1fc03f2
00d61d2
3f93ffc
678b49e
e083e5f
09b9718
9304e6e
f8e1938
a19fc93
526857e
8071186
e4b6c0e
a51d9b5
73b856b
1dfb5b3
b56c44e
9efedbc
a4aab01
57b50d4
8564f19
ceb7fbf
1a7887c
010ea3c
1f50a14
b0556e6
91a410e
f126482
cb06a9a
45d6279
e0f625a
935eb35
2a5a284
cb042fd
914fab6
c43d0dd
09ee1f4
8a19651
fd7fb79
b9855c4
ad57be9
642b17b
ce0e237
f8d4f62
14f9748
daf787c
233fcfc
eb819b0
2ca5565
12198f0
2d7cdc6
db09528
27c66ec
bfb1cc4
6609616
881759f
5908d97
a941f3a
8135a63
1e20f50
d921225
4507598
39ecb32
cb443b7
883cdd3
bfeba81
3787895
813bfdc
d3cdbf6
9d60663
a4143e9
ae425bf
930090b
a1cef7f
86ffb5d
30852dd
0ef5966
49399fc
e79ce71
2ab0584
1c990bc
0fba766
39beaaf
ec2b699
157b088
9f4f533
2c09c26
6ac37f9
4038704
3c8e9d3
3df1214
0ccf8b4
b74a516
224ae96
06e1d36
9f8425b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
#!/bin/bash | ||
|
||
# Licensed to the Apache Software Foundation (ASF) under one | ||
# or more contributor license agreements. See the NOTICE file | ||
# distributed with this work for additional information | ||
# regarding copyright ownership. The ASF licenses this file | ||
# to you under the Apache License, Version 2.0 (the | ||
# "License"); you may not use this file except in compliance | ||
# with the License. You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
|
||
# build and install are separated so changes to build don't invalidate | ||
# the whole docker cache for the image | ||
|
||
set -ex | ||
|
||
export JLBINARY='julia.tar.gz' | ||
export JULIADIR='/work/julia' | ||
export JULIA="${JULIADIR}/bin/julia" | ||
|
||
mkdir -p $JULIADIR | ||
# The julia version in Ubuntu repo is too old | ||
# We download the tarball from the official link: | ||
# https://julialang.org/downloads/ | ||
wget -O $JLBINARY https://julialang-s3.julialang.org/bin/linux/x64/0.6/julia-0.6.2-linux-x86_64.tar.gz | ||
tar xzvf $JLBINARY -C $JULIADIR --strip 1 | ||
rm $JLBINARY | ||
|
||
$JULIA -e 'versioninfo()' |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -816,6 +816,35 @@ unittest_ubuntu_gpu_R() { | |
make rpkgtest R_LIBS=/tmp/r-site-library R_GPU_ENABLE=1 | ||
} | ||
|
||
unittest_ubuntu_cpu_julia06() { | ||
set -ex | ||
export PATH="/work/julia/bin:$PATH" | ||
export MXNET_HOME='/work/mxnet' | ||
export JULIA_PKGDIR='/work/julia-pkg' | ||
export DEPDIR=`julia -e 'print(Pkg.dir())'` | ||
|
||
julia -e 'versioninfo()' | ||
julia -e 'Pkg.init()' | ||
|
||
# install package | ||
ln -sf ${MXNET_HOME}/julia ${DEPDIR}/MXNet | ||
|
||
# install dependencies | ||
julia -e 'Pkg.resolve()' | ||
|
||
# FIXME | ||
export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libjemalloc.so' | ||
|
||
# use the prebuilt binary from $MXNET_HOME/lib | ||
julia -e 'Pkg.build("MXNet")' | ||
|
||
# run the script `julia/test/runtests.jl` | ||
julia -e 'Pkg.test("MXNet")' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could you add a comment that points to the file which defines the test behaviour? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sure, I added some in 40ae566 |
||
|
||
# See https://github.com/dmlc/MXNet.jl/pull/303#issuecomment-341171774 | ||
julia -e 'using MXNet; mx._sig_checker()' | ||
} | ||
|
||
unittest_centos7_cpu() { | ||
set -ex | ||
cd /work/mxnet | ||
|
@@ -1090,6 +1119,31 @@ deploy_docs() { | |
popd | ||
} | ||
|
||
deploy_jl_docs() { | ||
set -ex | ||
export PATH="/work/julia/bin:$PATH" | ||
export MXNET_HOME='/work/mxnet' | ||
export JULIA_PKGDIR='/work/julia-pkg' | ||
export DEPDIR=`julia -e 'print(Pkg.dir())'` | ||
|
||
julia -e 'versioninfo()' | ||
julia -e 'Pkg.init()' | ||
ln -sf ${MXNET_HOME}/julia ${DEPDIR}/MXNet | ||
julia -e 'Pkg.resolve()' | ||
|
||
# FIXME | ||
export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libjemalloc.so' | ||
|
||
# use the prebuilt binary from $MXNET_HOME/lib | ||
julia -e 'Pkg.build("MXNet")' | ||
# build docs | ||
julia -e 'Pkg.add("Documenter")' | ||
julia -e 'cd(Pkg.dir("MXNet")); include(joinpath("docs", "make.jl"))' | ||
|
||
# TODO: make Jenkins worker push to MXNet.jl ph-pages branch if master build | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You said you needed help here. How may I help you? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is a builtin deployment method in Documenter: Or... will it make deployment easier, if it outputs docs to a specific dir and let Jenkins do the pushing work? I'm not sure how to make Jenkins gain the permission of pushing to gh-pages. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see, we're basically using the same approach for our MXNet docs. I have to say that the name of this job is a bit off, because we're not deploying to production as part of this pipeline. We have a separate publish job for our website. It gets two env-vars passed into the job and uses them as follows:
I think we could follow the same approach here, right? About the private key: Usually you use a GitHub personal access token. The username is the GitHub-Accountname and the password is the generated access token. Is this suitable here? I'm happy to assist you with creating the jenkins job. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Oh, is it ok to put this publish job here?
Which GitHub bot account have you set up already? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The publish job here is just for staging - in order to verify document generation as part of the PR. The publish for the main website will be done with a different job. You might want to work together with @aaronmarkham on this one. We're not allowed to use bot accounts for Apache services, so right now, we're using a committers account credentials.. Shall I set one up for this purpose? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think using committer's account credentials isn't a good idea. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That sounds feasible :) |
||
# ... | ||
} | ||
|
||
# broken_link_checker | ||
|
||
broken_link_checker() { | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
NEWS.md merge=union |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
*.jl.cov | ||
*.jl.*.cov | ||
*.jl.mem | ||
*.pyc | ||
.ipynb_checkpoints | ||
data | ||
deps/src | ||
deps/usr | ||
deps/deps.jl | ||
docs/_build | ||
docs/build/ | ||
docs/site/ | ||
.vscode |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,59 @@ | ||
# Documentation: http://docs.travis-ci.com/user/languages/julia/ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it would be feasible to re-introduce Travis since you have been working with that package so far. I personally wouldn't really have time to maintain it though - would you be up to do it? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah. I can maintain it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Travis is too slow. We should bring CI in to jenkins There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Auto-deploying what document? I think it would be best if we migrate all jobs to Jenkins - feel free to let me know if you got any problems with required dependencies. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (follow #8727 (comment))
If you really want to merge outputs of two document system ( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. About the doc build: We can have to two separate doc jobs, that's not an issue. At the moment, this job is handled in the Deploy-stage - feel free to add another one for the Julia docs in case they use a different system. The problem with your blueprint is that it does not cover bugs being introduced with PRs. it would be desirable to have Julia tests running as part of our PR stage. Where exactly do you see a problem in migrating from Travis and AppVeyor to Jenkins? |
||
sudo: false | ||
|
||
language: julia | ||
|
||
os: | ||
- linux | ||
- osx | ||
osx_image: xcode8 | ||
|
||
julia: | ||
- 0.6 | ||
# - nightly 0.6 supports depends on #170 | ||
|
||
branches: | ||
only: | ||
- master | ||
- stable | ||
- /^v\d+\.\d+(\.\d+)?(-\S*)?$/ # for tagging | ||
|
||
cache: | ||
directories: | ||
- $TRAVIS_BUILD_DIR/deps/src | ||
|
||
# dependent apt packages | ||
addons: | ||
apt: | ||
sources: | ||
- ubuntu-toolchain-r-test | ||
packages: | ||
- doxygen | ||
- wget | ||
- git | ||
- libcurl4-openssl-dev | ||
- unzip | ||
- libatlas-base-dev | ||
- libatlas-dev | ||
- libopencv-dev | ||
- gcc-4.8 | ||
- g++-4.8 | ||
|
||
before_install: | ||
- export TRAVIS_DIR=test/travis | ||
- source ${TRAVIS_DIR}/setup_env.sh | ||
|
||
notifications: | ||
email: false | ||
|
||
script: | ||
- ${TRAVIS_DIR}/run_test.sh | ||
|
||
after_success: | ||
# See https://github.com/dmlc/MXNet.jl/pull/303#issuecomment-341171774 | ||
- julia -e 'using MXNet; mx._sig_checker()' | ||
|
||
- source ${TRAVIS_DIR}/run_coverage.sh | ||
- echo $TRAVIS_JULIA_VERSION | ||
- julia -e 'Pkg.add("Documenter")' | ||
- julia -e 'cd(Pkg.dir("MXNet")); include(joinpath("docs", "make.jl"))' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this actually doing any compilation? How would users use a julia package? At this stage, MXNet has already been compiled in the build stage and I (without having Julia experience) would not expect to do another build
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I add a env var block for build in Jenkins config,
This build script will use the prebuilt binary if available.
See https://github.com/dmlc/MXNet.jl/blob/master/deps/build.jl#L11
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These scripts run within a docker environment. The mxnet root directory is available under /work/mxnet. In general, please don't specify env vars in the Jenkinsfile. Instead, use this shellscript (export...).
What exactly are the dependencies here? Is this something we could bake into the Dockerfile?
Could you elaborate "if available"? What happens, if it is not available?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh...what's the different between
/workspace/ut-julia-cpu
and/work/mxnet
?All dependencies are Julia stuffs: https://github.com/dmlc/MXNet.jl/blob/master/REQUIRE.
Managing them via Julia's package manager is enough. There is nothing to do with Dockerfile, I think.
The build script will search the dir
MXNET_HOME/lib
first. If found alibmxnet.so
under that dir, it will try to load it viadlopen
. Ifdlopen
failed, that file will be ignored.In case of not available, it will build stuffs from source.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed in 1e0926e
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm afraid I don't understand your question. We specify the dependencies in our install scripts and then keep them consistent until we decide to change a version in the install script.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added an installation script for
julia
compiler (30aa3ca). (but not sure which version of Julia Ubuntu have.)The file
REQUIRE
is a Julia version ofrequirements.txt
.And all the Julia deps is listed in that file, end-user will use it / any PR can change it.
So, I want deps being installed during a normal build process.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can install these requirements as part of the docker build process. In order to make sure it is being re-run when the
REQUIRE
-file is changed, just add anADD
command to the Dockerfile before the script to install these dependencies is being called. This makes sure the underlying layers are automatically invalidated when the file is changed. This would look likeFeel free to pin a Julia-Version if you feel it's appropriate.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ADD ...
for cache management is pretty cool!Is it valid if I put
JULIA_PKGDIR
(the deps need to be cached) under/work
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please check 701aca9 out.