Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

import Julia binding #10149

Merged
merged 736 commits into from
Oct 5, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
736 commits
Select commit Hold shift + click to select a range
d1eb318
Bug fix updating the ACE loss function. Included a test case. Also pr…
JockLawrie Dec 8, 2016
c06b211
Merge pull request #156 from JockLawrie/pull-request/d1eb3183
vchuravy Dec 10, 2016
b81b26c
New Optimizers (#159)
Arkoniak Dec 29, 2016
1ea44c9
Fix hygiene bug (#164)
Arkoniak Jan 7, 2017
4d0aa87
Fixed visualize (#163)
Arkoniak Jan 7, 2017
ab80048
ENH: added verbosity training option to control printouts (#162)
sglyon Jan 7, 2017
2393d4a
Fix scalar operations in NDArray (#165)
Arkoniak Jan 9, 2017
179daa5
Fix of bugs in nnvm branch (#169)
Arkoniak Jan 13, 2017
1b09fb4
Merge branch 'nnvm'
pluskid Jan 13, 2017
030990a
BUG: fix two small mistakes with verbosity control (#168)
sglyon Jan 15, 2017
22a3c72
Drop support for v0.4 and Deprecations for v0.6 (#171)
vchuravy Jan 16, 2017
2a66ea3
Update build.jl (#174)
mdpradeep Jan 20, 2017
3ad5819
update NEWS.md
vchuravy Jan 26, 2017
8d044b1
refactor windows support
vchuravy Jan 27, 2017
15c7052
Merge pull request #186 from dmlc/vc/windows
vchuravy Jan 27, 2017
b039771
Using WebRequest function to download the JSON file
bmharsha Jan 27, 2017
6335eca
fix vectorized abs deprecation
vchuravy Jan 27, 2017
d9f9551
don't include code from within a function
vchuravy Jan 27, 2017
a6968a6
deprecation of filter and Array creation
vchuravy Jan 27, 2017
a79d002
readall deprecation
vchuravy Jan 27, 2017
a476d3e
MXNet.jl on v0.6 is currently subtlety broken due to broadcasting.
vchuravy Jan 27, 2017
11deaab
Merge pull request #188 from bmharsha/Fix-Windows
vchuravy Jan 27, 2017
ba3c9fb
Merge pull request #187 from oist/vc/travis
vchuravy Jan 28, 2017
fecfcdd
fix use of Symbol in symbolic-node.jl, fixes #189
vchuravy Jan 28, 2017
fb2a0d0
Merge pull request #190 from dmlc/vc/fixup
vchuravy Jan 28, 2017
cac5625
refactor sgd.jl to exploit multiple dispatch
ZacCranko Jan 24, 2017
d0d7963
remove BaseTestNext and bump Compat
vchuravy Jan 28, 2017
d59b214
Merge pull request #192 from dmlc/vc/review_comments
vchuravy Jan 28, 2017
0ee926e
remove usr/setupenv.cmd because it is too invasive
vchuravy Jan 29, 2017
79d80b6
Merge pull request #196 from dmlc/vc/fix_win
vchuravy Jan 29, 2017
75060bc
Fix for symbolic node (#199)
Arkoniak Feb 5, 2017
f33f1dc
get_name (#206)
MikeInnes Feb 24, 2017
17a5442
fix data URL error (#204)
pluskid Feb 28, 2017
c58eb46
fix for issue #207 (#208)
mcreel Mar 3, 2017
fa25ad6
metric: intro NMSE
iblislin Mar 9, 2017
f23b0a9
doc: update CDN link of MathJax
iblislin Mar 9, 2017
96191ea
metric: add test cases for NMSE
iblislin Mar 10, 2017
ba97d93
docs: add an explanation for normalized MSE
iblislin Mar 11, 2017
09d8f27
Merge pull request #211 from iblis17/nmse
vchuravy Mar 11, 2017
cf7f32b
Fixed data attribute
steventhornton Mar 9, 2017
c9eb71b
switch to Plots.jl from pyplot (#217)
mcreel Mar 19, 2017
1781290
Update ACE metric (#220)
vchuravy Mar 29, 2017
9474d95
provide an optional context in mx.load_checkpoint (#221)
vchuravy Mar 29, 2017
46440e3
Added initializers for Spatial Transformer Networks (STN) layers (#223)
facundoq Mar 30, 2017
9dd3291
docs: update renamed `every_n_batch`
iblislin Apr 2, 2017
fc2974f
Merge pull request #225 from iblis17/docs
vchuravy Apr 2, 2017
5c02fcb
Check if key stride exists in the node info when producing graphviz o…
facundoq Apr 2, 2017
867b1c5
Merge pull request #226 from facundoq/master
vchuravy Apr 3, 2017
31b95fb
Update mnist.md (#227)
ultradian Apr 10, 2017
1be1b87
fix conflict with Images
vchuravy Apr 12, 2017
7a6120a
Merge pull request #228 from dmlc/vc/images_subtype
vchuravy Apr 12, 2017
bfa966a
fixes bilinear initializer following approach in #34
vchuravy Apr 13, 2017
8947ead
adds test for bilinear initializer
vchuravy Apr 13, 2017
44a6c36
Merge pull request #229 from dmlc/vc/bilinear_initializer
vchuravy Apr 13, 2017
558adb7
add SeqMetric to apply different metrics for multi-output
vchuravy Mar 28, 2017
b4ece81
add NullMetric
vchuravy Mar 29, 2017
ced3f73
fix pointer(NDArray) for Float64
vchuravy Mar 30, 2017
86e3fa0
Merge pull request #231 from dmlc/vc/pointer
vchuravy Apr 14, 2017
3263b95
Merge pull request #230 from dmlc/vc/seqmetric
vchuravy Apr 14, 2017
d0253f1
XCode8 supports thread_local in C++
vchuravy Apr 14, 2017
68b67e8
Merge pull request #234 from dmlc/vc/mac_travis
vchuravy Apr 14, 2017
1697522
update MSE to be agnostic to dimensions.
vchuravy Mar 28, 2017
e12052c
convert single outputs to array so that inference has it easier
vchuravy Mar 30, 2017
def129f
convert eps to eltype of array
vchuravy Apr 14, 2017
c2044d5
adds THTT to enable metrics to support NDArray or Array
vchuravy Apr 14, 2017
349d2a7
convert MSE to use NDArray as an example for async
vchuravy Apr 14, 2017
4675ace
Merge pull request #233 from dmlc/vc/metrics
vchuravy Apr 15, 2017
c4a2658
ensure that new submodules are initialized
vchuravy Apr 15, 2017
9aac7e1
Update callback.jl (#235)
undefdev Apr 15, 2017
7a530fa
Fixed SpatialTransformerNetwork loc_bias initializer. Added lenet-stn…
facundoq Apr 26, 2017
06010b8
explicitly specify 7z (#239)
musm Apr 30, 2017
6cc9e99
add verbosity option to predict()
rdeits May 9, 2017
4cd5f3f
Merge pull request #244 from rdeits/predict-verbosity
vchuravy May 11, 2017
5d16305
explicitly specify 7z
musm May 12, 2017
02c7661
Merge pull request #246 from musm/patch-1
vchuravy May 12, 2017
8918f1b
update appveyor script (#195)
vchuravy May 12, 2017
60f5306
improve GPU detection.
vchuravy Jan 29, 2017
efff124
Add appveyor badge
pluskid May 12, 2017
07777d6
switch to vc14
vchuravy May 12, 2017
28344a4
actually copy the mxnet.dll to the right place on windows
vchuravy May 13, 2017
96bdfa7
Prebuilt binaries only work on windows 64bit
vchuravy May 13, 2017
aae9114
Merge pull request #194 from dmlc/vc/gpu_detection
vchuravy May 13, 2017
2ecf2c1
update NEWS.md
vchuravy May 14, 2017
9c70922
Update CI URLs to point to new caching infrastructure
staticfloat May 17, 2017
b63fd9b
Fix #242 (#247)
holl- May 23, 2017
c7032f0
Merge pull request #249 from staticfloat/ci_url
vchuravy May 26, 2017
172af60
Use `Sys.CPU_CORES` not `nprocs()` when doing a parallel `make` (#250)
staticfloat May 30, 2017
2b26bc2
update for Julia v0.6 (#248)
ExpandingMan Jun 1, 2017
841e9ee
update regression example for julia v0.6(#254)
mcreel Jun 2, 2017
c965235
Add extra check on CUDA library detection
joshbode Jun 9, 2017
18547b8
Merge pull request #255 from joshbode/cudalib_check
vchuravy Jun 10, 2017
e7d1b62
Added cblas_[ds]tr[sm]m to cblas include file (#259)
joshbode Jun 20, 2017
10ad90b
Added MXNET_COMMIT env variable for upstream commit/tag (#260)
joshbode Jun 23, 2017
0750c41
build: bump curr_win to 20170819, MXNet 0.11.0
iblislin Sep 8, 2017
778529c
Merge pull request #268 from iblis17/win-prebuild
vchuravy Sep 8, 2017
85a2d97
ndarray: add kInt8 and kInt64 into TypeFlag enum
iblislin Sep 9, 2017
873715b
cblas: import all func prototype into header
iblislin Sep 8, 2017
8bd307f
Merge pull request #270 from iblis17/nd-typeflag
vchuravy Sep 9, 2017
4e5ab13
travis : set make -j4 and travis_wait
iblislin Sep 6, 2017
d1bbcc4
base/build: using `Libdl.dlext` for searching lib
iblislin Sep 1, 2017
0d54f4b
build: force enable LAPACK when Julia's OpenBLAS is available
iblislin Sep 9, 2017
f34643b
Merge pull request #271 from iblis17/fix-travis
vchuravy Sep 10, 2017
b68ce5b
Merge pull request #265 from iblis17/ext
vchuravy Sep 10, 2017
21ccc56
Merge pull request #269 from iblis17/build-lapack
vchuravy Sep 10, 2017
23866bd
build: fix macOS LAPACK build
iblislin Sep 11, 2017
2c5120d
Merge pull request #277 from iblis17/mac-lapack
vchuravy Sep 11, 2017
4e439ed
build: remove FileRule on config.mk and fix CUDA config
iblislin Sep 10, 2017
7cc66d1
executor: fix docstring of `bind`
iblislin Sep 12, 2017
5d739da
ndarray: implement deepcopy via MXNDArrayGetDataNDArray
iblislin Sep 13, 2017
f20534d
Merge pull request #273 from iblis17/issue-273
vchuravy Sep 17, 2017
a51cba4
Merge pull request #281 from iblis17/bind-doc
vchuravy Sep 17, 2017
16c2c5a
base: update GRAD_REQ enumeration
iblislin Sep 18, 2017
d824631
Merge pull request #275 from iblis17/file-rule-cuda
vchuravy Sep 18, 2017
e3ad29d
travis: bring coverage report back
iblislin Sep 19, 2017
dcba690
build: CuDNN detection
iblislin Sep 11, 2017
3d4adfb
ndarray: `reshape` share the same API with Base
iblislin Sep 10, 2017
bbd0e66
Merge pull request #285 from iblis17/cov-report
vchuravy Sep 24, 2017
b68ca2e
Merge pull request #272 from iblis17/issue-272
vchuravy Sep 24, 2017
0b9bf78
sym-node: support `print` for debugging
iblislin Sep 11, 2017
4e89c91
sym-node: extract a until function _create_atomic_symbol
iblislin Sep 12, 2017
63a34cc
sym-node: implement reshape api as Base
iblislin Sep 12, 2017
68d6ff7
sym-node: update deprecation for reshape
iblislin Sep 25, 2017
a2e270e
sym-node: update NEWS for reshape API
iblislin Sep 25, 2017
bb88d79
Merge pull request #283 from iblis17/issue-283
vchuravy Sep 25, 2017
9e0844a
sym-node: add reverse keyword arg
iblislin Sep 25, 2017
55ccbfa
sym-node: more test cases for reshape
iblislin Sep 25, 2017
3595403
Merge pull request #280 from iblis17/build-cudnn
vchuravy Sep 25, 2017
df13ddd
Merge pull request #276 from iblis17/print-symbol
vchuravy Sep 25, 2017
6d1ba53
Merge pull request #279 from iblis17/sym-reshape
vchuravy Sep 25, 2017
ff0556b
random: rename srand! to srand
iblislin Sep 13, 2017
5b3c507
random: update NEWS and deprecated.jl
iblislin Sep 25, 2017
251a8b3
random: upstream api change
iblislin Oct 4, 2017
ebb03d0
Merge pull request #288 from iblis17/upstream-random
vchuravy Oct 4, 2017
077a8cb
build: improve cuda detection
iblislin Oct 4, 2017
0a53778
Merge pull request #282 from iblis17/srand
vchuravy Oct 5, 2017
ac7bbd0
base: support MXGetVersion
iblislin Oct 7, 2017
904eb20
rand: docstring
iblislin Oct 10, 2017
aae016c
ndarray: pretty printing
iblislin Oct 7, 2017
19082ef
Merge pull request #289 from iblis17/mx-get-ver
vchuravy Oct 10, 2017
9d90506
Merge pull request #295 from iblis17/doc-rand
vchuravy Oct 10, 2017
15def2c
Merge pull request #290 from iblis17/nd-pretty-print
vchuravy Oct 10, 2017
81b134f
ndarray: implement `fill` and `fill!` (#297)
iblislin Oct 24, 2017
a79e33c
Merge pull request #287 from iblis17/issue-286
vchuravy Oct 25, 2017
71f2d40
base: merge _julia_to_mx_param into dump_mx_param (#296)
iblislin Oct 25, 2017
3824df0
Fix mx.chain var reference in macro (#299)
iblislin Oct 26, 2017
41f8f55
visualize: fix node_attr on upstream master (#301)
iblislin Oct 31, 2017
9fcab40
symbol: add Base.show (#302)
iblislin Nov 6, 2017
4f182ee
ndarray: more Base-like APIs (#303)
iblislin Nov 6, 2017
a3317f1
ndarray: implement rdiv (#292)
iblislin Nov 6, 2017
8cc5c1c
util: docstring of _sig_checker (#305)
iblislin Nov 7, 2017
a488d7a
ndarray: intro TakingBroadcastSeriously and broadcast unfusing (#300)
iblislin Nov 7, 2017
57cc677
ndarray: elementwise power (#293)
iblislin Nov 8, 2017
eea128a
ndarray: fix `add` broadcasting and more tests (#308)
iblislin Nov 8, 2017
f839be1
Fix deprecations (#307)
iblislin Nov 8, 2017
4919273
ndarray: getindex/setindex! linear indexing (#294)
iblislin Nov 9, 2017
1fc03f2
ndarray: elementwise power for irrational (#310)
iblislin Nov 9, 2017
00d61d2
executor: return `outputs` on forward (#312)
iblislin Nov 9, 2017
3f93ffc
git: union merge policy for NEWS.md (#309)
iblislin Nov 9, 2017
678b49e
ndarray: regularize fill! API (#311)
iblislin Nov 9, 2017
e083e5f
doc: cleanup useless enc file (#315)
iblislin Nov 10, 2017
09b9718
build: prevent CUDA_HOME KeyError (#316)
iblislin Nov 13, 2017
9304e6e
sym: broadcast unfusion (#314)
iblislin Nov 13, 2017
f8e1938
Remove Compat and src/compat.jl (#318)
iblislin Nov 13, 2017
a19fc93
test: replace `reldiff` with `isapprox` (#321)
iblislin Nov 13, 2017
526857e
docs: address docstring override warning (#320)
iblislin Nov 13, 2017
8071186
travis: enable 'build branch updates' on master/stable only (#319)
iblislin Nov 13, 2017
e4b6c0e
docs: update examples in overview (#317)
iblislin Nov 13, 2017
a51d9b5
docs: fix more example rendering in overview (#322)
iblislin Nov 14, 2017
73b856b
Update README badge (#324)
iblislin Nov 15, 2017
1dfb5b3
Cut a release for mxnet v0.12.0 (#304)
iblislin Nov 16, 2017
b56c44e
Bump upstream to v0.12.1 (#326)
iblislin Nov 16, 2017
9efedbc
Update releasing note (#327)
iblislin Nov 16, 2017
a4aab01
docs: minor changes in overview (#330)
iblislin Nov 17, 2017
57b50d4
build: set upstream version back to master (#328)
iblislin Nov 17, 2017
8564f19
executor: add Base.show and Base.print (#337)
iblislin Nov 20, 2017
ceb7fbf
travis: enable caching for reducing build time (#329)
iblislin Nov 20, 2017
1a7887c
ndarray: add outer constrcutor for AbstractArray (#334)
iblislin Nov 20, 2017
010ea3c
sym: implement get_children (#333)
iblislin Nov 20, 2017
1f50a14
sym: fix printing of symbol generated via get_internals (#332)
iblislin Nov 20, 2017
b0556e6
Update regression-example.jl (#339)
mcreel Nov 22, 2017
91a410e
test: minor style changes for mlp-test (#340)
iblislin Nov 22, 2017
f126482
sym: handy macro `mx.var` for creating `mx.Variable` (#338)
iblislin Nov 22, 2017
cb06a9a
build: bump max number of make jobs to 32 (#343)
iblislin Nov 25, 2017
45d6279
ndarray: add `Base.show(io, MIME"text/plain")` (#347)
iblislin Nov 27, 2017
e0f625a
example: fix batchsize config in regression-example (#342)
iblislin Nov 27, 2017
935eb35
ndarray: make _minus type stable (#345)
iblislin Nov 27, 2017
2a5a284
model: refine and test cases for FeedForward constructor (#346)
iblislin Nov 27, 2017
cb042fd
parametric NDArray (#331)
iblislin Dec 1, 2017
914fab6
ndarray: minor style changes (#349)
iblislin Dec 2, 2017
c43d0dd
model: fix test cases macro (#355)
iblislin Dec 2, 2017
09ee1f4
ndarray: implement `size(x, dims...)` (#350)
iblislin Dec 2, 2017
8a19651
ndarray: implement endof (#351)
iblislin Dec 2, 2017
fd7fb79
ndarray: make _plus type stable (#352)
iblislin Dec 2, 2017
b9855c4
ndarray: type convert of _mul_scalar (#356)
iblislin Dec 4, 2017
ad57be9
ndarray: type convertion of _plus_scalar (#360)
iblislin Dec 4, 2017
642b17b
ndarray: type convertion of _rdiv_scalar (#358)
iblislin Dec 4, 2017
ce0e237
doc: sort out table for ndarray api (#362)
iblislin Dec 6, 2017
f8d4f62
ndarray: type convertion of _minus_scalar (#354)
iblislin Dec 7, 2017
14f9748
ndarray: do not auto-import _full, _ones and _zeros (#359)
iblislin Dec 8, 2017
daf787c
ndarray: make trigonometric functions dot-call only (#361)
iblislin Dec 9, 2017
233fcfc
ndarray: change internal api of plus to help autograd (#364)
iblislin Dec 9, 2017
eb819b0
ndarray: change internal api of mul/div to help autograd (#366)
iblislin Dec 9, 2017
2ca5565
ndarray: change internal api of minus to help autograd (#365)
iblislin Dec 9, 2017
12198f0
ndarray: copy(AbstractArray, context) (#367)
iblislin Dec 10, 2017
2d7cdc6
Reexporting NDArray and SymbolicNode (#306)
iblislin Dec 10, 2017
db09528
example: fix parametric NDArray in lstm example (#370)
iblislin Dec 10, 2017
27c66ec
sym: overload `bind` (#372)
iblislin Dec 12, 2017
bfb1cc4
ndarray: remap hyperbolic function (#374)
iblislin Dec 15, 2017
6609616
ndarray: support transpose on 1D array (#375)
iblislin Dec 15, 2017
881759f
example: fix MSE init (#376)
iblislin Dec 15, 2017
5908d97
base: cleanup stale code in _defstruct_impl (#377)
iblislin Dec 15, 2017
a941f3a
ndarray: add modulo operator (#373)
iblislin Dec 15, 2017
8135a63
ndarray: type convertion of _div_scalar (#357)
iblislin Dec 15, 2017
1e20f50
ndarray: protect from diving zero for non-inplace op (#378)
iblislin Dec 15, 2017
d921225
doc/ndarray: add ref table for hyperbolic function (#382)
iblislin Dec 16, 2017
4507598
ndarray: `cat`, `hcat`, `vcat` (#380)
iblislin Dec 16, 2017
39ecb32
base: improve MXError display (#379)
iblislin Dec 16, 2017
cb443b7
ndarray: porting Python's autograd (#274)
iblislin Dec 18, 2017
883cdd3
build: make appending lib to MXNET_HOME optional (#386)
nstiurca Dec 18, 2017
bfeba81
ndarray: support matrix/tensor multiplication (#384)
iblislin Dec 21, 2017
3787895
ndarray: remap expand_dims (#381)
iblislin Dec 21, 2017
813bfdc
random: Base-like APIs for rand, rand!, randn, randn! (#383)
iblislin Dec 21, 2017
d3cdbf6
ndarray: remap activation functions (#387)
iblislin Dec 28, 2017
9d60663
ndarray: refine signature of `clip` (#391)
iblislin Dec 30, 2017
a4143e9
Bump :copyright: (#397)
iblislin Jan 1, 2018
ae425bf
doc: append 'Defined in' into operator docstring (#393)
iblislin Jan 1, 2018
930090b
ndarray: fix 1D NDArray display (#399)
iblislin Jan 2, 2018
a1cef7f
ndarray: fix type convertion in setindex! (#398)
iblislin Jan 2, 2018
86ffb5d
ndarray: inplace modulo operators (#389)
iblislin Jan 4, 2018
30852dd
ndarray: support modulo operation in macro inplace (#390)
iblislin Jan 4, 2018
0ef5966
ndarray: add Base.ones(::NDArray) and Base.zeros (#363)
iblislin Jan 4, 2018
49399fc
ndarray: broadcasting along dimension on arith operators (#401)
iblislin Jan 7, 2018
e79ce71
ndarray: broadcast comparison operators (#402)
iblislin Jan 10, 2018
2ab0584
ndarray: copy!(NDArray, AbstractArray) (#405)
iblislin Jan 13, 2018
1c990bc
ndarray: new constructor NDArray(Type, AbstractArray) (#404)
iblislin Jan 17, 2018
0fba766
ndarray: broadcast_min/max (#403)
iblislin Jan 19, 2018
39beaaf
remap broadcast_to/axis/axes (#406)
iblislin Jan 22, 2018
ec2b699
ndarray: remap hypot (#407)
iblislin Jan 24, 2018
157b088
doc: update predict section of mnist tutorial (#411)
iblislin Jan 28, 2018
9f4f533
Optimizer module overhaul (#396)
iblislin Jan 31, 2018
2c09c26
Update README: `predict` section (#412)
iblislin Feb 4, 2018
6ac37f9
kvstore: refine and copy docstring from Python (#388)
iblislin Feb 5, 2018
4038704
build: Propagate more build flags from ENV (#414)
iblislin Feb 15, 2018
3c8e9d3
build: propagate CC/CXX into config.mk (#419)
iblislin Mar 4, 2018
3df1214
build: propagate USE_JEMALLOC (#422)
iblislin Mar 8, 2018
0ccf8b4
build: add `libmxnet.so` into lib search list. (#424)
iblislin Apr 3, 2018
b74a516
Update README.md
iblislin Aug 20, 2018
224ae96
Typo in README
iblislin Aug 20, 2018
06e1d36
Add 'julia/' from commit '224ae962db8ae252ce1ac317c6d4a2cfe67a7d77'
iblislin Oct 2, 2018
9f8425b
import Julia binding
iblislin Sep 28, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 35 additions & 9 deletions Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,30 @@ def python3_gpu_ut_nocudnn(docker_container_name) {
}
}

def deploy_docs() {
parallel 'Docs': {
node(NODE_LINUX_CPU) {
ws('workspace/docs') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
utils.docker_run('ubuntu_cpu', 'deploy_docs', false)
sh "ci/other/ci_deploy_doc.sh ${env.BRANCH_NAME} ${env.BUILD_NUMBER}"
}
}
}
},
'Julia docs': {
node(NODE_LINUX_CPU) {
ws('workspace/julia-docs') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('cpu', mx_lib)
utils.docker_run('ubuntu_cpu', 'deploy_jl_docs', false)
}
}
}
}
}

node('mxnetlinux-cpu') {
// Loading the utilities requires a node context unfortunately
checkout scm
Expand Down Expand Up @@ -746,6 +770,16 @@ core_logic: {
}
}
},
'Julia 0.6: CPU': {
node(NODE_LINUX_CPU) {
ws('workspace/ut-julia06-cpu') {
timeout(time: max_time, unit: 'MINUTES') {
utils.unpack_and_init('cpu', mx_lib)
utils.docker_run('ubuntu_cpu', 'unittest_ubuntu_cpu_julia06', false)
}
}
}
},

'Python 2: CPU Win':{
node(NODE_WINDOWS_CPU) {
Expand Down Expand Up @@ -911,15 +945,7 @@ core_logic: {
}

stage('Deploy') {
node(NODE_LINUX_CPU) {
ws('workspace/docs') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()
utils.docker_run('ubuntu_cpu', 'deploy_docs', false)
sh "ci/other/ci_deploy_doc.sh ${env.BRANCH_NAME} ${env.BUILD_NUMBER}"
}
}
}
deploy_docs()
}
}
,
Expand Down
3 changes: 3 additions & 0 deletions ci/docker/Dockerfile.build.ubuntu_cpu
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,9 @@ RUN /work/ubuntu_r.sh
COPY install/ubuntu_perl.sh /work/
RUN /work/ubuntu_perl.sh

COPY install/ubuntu_julia.sh /work/
RUN /work/ubuntu_julia.sh

COPY install/ubuntu_clang.sh /work/
RUN /work/ubuntu_clang.sh

Expand Down
37 changes: 37 additions & 0 deletions ci/docker/install/ubuntu_julia.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
#!/bin/bash

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.

# build and install are separated so changes to build don't invalidate
# the whole docker cache for the image

set -ex

export JLBINARY='julia.tar.gz'
export JULIADIR='/work/julia'
export JULIA="${JULIADIR}/bin/julia"

mkdir -p $JULIADIR
# The julia version in Ubuntu repo is too old
# We download the tarball from the official link:
# https://julialang.org/downloads/
wget -O $JLBINARY https://julialang-s3.julialang.org/bin/linux/x64/0.6/julia-0.6.2-linux-x86_64.tar.gz
tar xzvf $JLBINARY -C $JULIADIR --strip 1
rm $JLBINARY

$JULIA -e 'versioninfo()'
54 changes: 54 additions & 0 deletions ci/docker/runtime_functions.sh
Original file line number Diff line number Diff line change
Expand Up @@ -816,6 +816,35 @@ unittest_ubuntu_gpu_R() {
make rpkgtest R_LIBS=/tmp/r-site-library R_GPU_ENABLE=1
}

unittest_ubuntu_cpu_julia06() {
set -ex
export PATH="/work/julia/bin:$PATH"
export MXNET_HOME='/work/mxnet'
export JULIA_PKGDIR='/work/julia-pkg'
export DEPDIR=`julia -e 'print(Pkg.dir())'`

julia -e 'versioninfo()'
julia -e 'Pkg.init()'

# install package
ln -sf ${MXNET_HOME}/julia ${DEPDIR}/MXNet

# install dependencies
julia -e 'Pkg.resolve()'

# FIXME
export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libjemalloc.so'

# use the prebuilt binary from $MXNET_HOME/lib
julia -e 'Pkg.build("MXNet")'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this actually doing any compilation? How would users use a julia package? At this stage, MXNet has already been compiled in the build stage and I (without having Julia experience) would not expect to do another build

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I add a env var block for build in Jenkins config,

withEnv(['MXNET_HOME=/workspace/ut-julia-cpu', 'JULIA_PKGDIR=/tmp/julia-pkg']) {
   ...
}

This build script will use the prebuilt binary if available.
See https://github.com/dmlc/MXNet.jl/blob/master/deps/build.jl#L11

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These scripts run within a docker environment. The mxnet root directory is available under /work/mxnet. In general, please don't specify env vars in the Jenkinsfile. Instead, use this shellscript (export...).

What exactly are the dependencies here? Is this something we could bake into the Dockerfile?

Could you elaborate "if available"? What happens, if it is not available?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh...what's the different between /workspace/ut-julia-cpu and /work/mxnet?

What exactly are the dependencies here? Is this something we could bake into the Dockerfile?

All dependencies are Julia stuffs: https://github.com/dmlc/MXNet.jl/blob/master/REQUIRE.
Managing them via Julia's package manager is enough. There is nothing to do with Dockerfile, I think.

Could you elaborate "if available"? What happens, if it is not available?

The build script will search the dir MXNET_HOME/lib first. If found a libmxnet.so under that dir, it will try to load it via dlopen. If dlopen failed, that file will be ignored.
In case of not available, it will build stuffs from source.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed in 1e0926e

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid I don't understand your question. We specify the dependencies in our install scripts and then keep them consistent until we decide to change a version in the install script.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an installation script for julia compiler (30aa3ca). (but not sure which version of Julia Ubuntu have.)


This means that every time you call Pkg.update(), it will download and install the dependencies and throw everything away after the run has been finished.

The file REQUIRE is a Julia version of requirements.txt.
And all the Julia deps is listed in that file, end-user will use it / any PR can change it.
So, I want deps being installed during a normal build process.

Copy link
Contributor

@marcoabreu marcoabreu Mar 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can install these requirements as part of the docker build process. In order to make sure it is being re-run when the REQUIRE-file is changed, just add an ADD command to the Dockerfile before the script to install these dependencies is being called. This makes sure the underlying layers are automatically invalidated when the file is changed. This would look like

ADD ..../REQUIRE
COPY install/ubuntu_julia.sh /work/
RUN /work/ubuntu_julia.sh

Feel free to pin a Julia-Version if you feel it's appropriate.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ADD ... for cache management is pretty cool!
Is it valid if I put JULIA_PKGDIR (the deps need to be cached) under /work?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please check 701aca9 out.


# run the script `julia/test/runtests.jl`
julia -e 'Pkg.test("MXNet")'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a comment that points to the file which defines the test behaviour?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I added some in 40ae566


# See https://github.com/dmlc/MXNet.jl/pull/303#issuecomment-341171774
julia -e 'using MXNet; mx._sig_checker()'
}

unittest_centos7_cpu() {
set -ex
cd /work/mxnet
Expand Down Expand Up @@ -1090,6 +1119,31 @@ deploy_docs() {
popd
}

deploy_jl_docs() {
set -ex
export PATH="/work/julia/bin:$PATH"
export MXNET_HOME='/work/mxnet'
export JULIA_PKGDIR='/work/julia-pkg'
export DEPDIR=`julia -e 'print(Pkg.dir())'`

julia -e 'versioninfo()'
julia -e 'Pkg.init()'
ln -sf ${MXNET_HOME}/julia ${DEPDIR}/MXNet
julia -e 'Pkg.resolve()'

# FIXME
export LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libjemalloc.so'

# use the prebuilt binary from $MXNET_HOME/lib
julia -e 'Pkg.build("MXNet")'
# build docs
julia -e 'Pkg.add("Documenter")'
julia -e 'cd(Pkg.dir("MXNet")); include(joinpath("docs", "make.jl"))'

# TODO: make Jenkins worker push to MXNet.jl ph-pages branch if master build
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You said you needed help here. How may I help you?

Copy link
Member Author

@iblislin iblislin Mar 26, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a builtin deployment method in Documenter:
https://juliadocs.github.io/Documenter.jl/stable/man/hosting/#SSH-Deploy-Keys-1
It needs a key pair: add a public key on GitHub (only @pluskid has the permission) first, then
add an env var DOCUMENTER_KEY as private key in Jenkins.

Or... will it make deployment easier, if it outputs docs to a specific dir and let Jenkins do the pushing work? I'm not sure how to make Jenkins gain the permission of pushing to gh-pages.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, we're basically using the same approach for our MXNet docs. I have to say that the name of this job is a bit off, because we're not deploying to production as part of this pipeline.

We have a separate publish job for our website. It gets two env-vars passed into the job and uses them as follows:

git clone https://$APACHE_USERNAME:[email protected]/apache/incubator-mxnet-site

I think we could follow the same approach here, right?

About the private key: Usually you use a GitHub personal access token. The username is the GitHub-Accountname and the password is the generated access token. Is this suitable here? I'm happy to assist you with creating the jenkins job.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because we're not deploying to production as part of this pipeline

Oh, is it ok to put this publish job here?

I think we could follow the same approach here, right?

Which GitHub bot account have you set up already?
@pluskid: Could you add it as collaborator?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The publish job here is just for staging - in order to verify document generation as part of the PR. The publish for the main website will be done with a different job. You might want to work together with @aaronmarkham on this one.

We're not allowed to use bot accounts for Apache services, so right now, we're using a committers account credentials.. Shall I set one up for this purpose?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using committer's account credentials isn't a good idea.
Maybe deploy key is a better way: https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds feasible :)

# ...
}

# broken_link_checker

broken_link_checker() {
Expand Down
1 change: 1 addition & 0 deletions julia/.gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
NEWS.md merge=union
13 changes: 13 additions & 0 deletions julia/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
*.jl.cov
*.jl.*.cov
*.jl.mem
*.pyc
.ipynb_checkpoints
data
deps/src
deps/usr
deps/deps.jl
docs/_build
docs/build/
docs/site/
.vscode
59 changes: 59 additions & 0 deletions julia/.travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Documentation: http://docs.travis-ci.com/user/languages/julia/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be feasible to re-introduce Travis since you have been working with that package so far. I personally wouldn't really have time to maintain it though - would you be up to do it?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I can maintain it.
What I expect is that we still need travis build for the mirror, and it serves for auto-deploying the document.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Travis is too slow. We should bring CI in to jenkins

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-deploying what document? I think it would be best if we migrate all jobs to Jenkins - feel free to let me know if you got any problems with required dependencies.

Copy link
Member Author

@iblislin iblislin Mar 21, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(follow #8727 (comment))
The blueprint in my mind:

  1. We have Jenkins build in main repo. In this stage, Travis/Appveyor won't be triggered.
  2. Set up mirror link with MXNet.jl
  3. Just push to MXNet.jl and trigger Travis CI and Appveyor builds.
  4. (I want to keep current doc setup:) Travis CI will push document to MXNet.jl's GitHub page.

If you really want to merge outputs of two document system (Sphinx and Documenter.jl)....
I do not have any clear idea about detail of Sphinx.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About the doc build: We can have to two separate doc jobs, that's not an issue. At the moment, this job is handled in the Deploy-stage - feel free to add another one for the Julia docs in case they use a different system.

The problem with your blueprint is that it does not cover bugs being introduced with PRs. it would be desirable to have Julia tests running as part of our PR stage. Where exactly do you see a problem in migrating from Travis and AppVeyor to Jenkins?

sudo: false

language: julia

os:
- linux
- osx
osx_image: xcode8

julia:
- 0.6
# - nightly 0.6 supports depends on #170

branches:
only:
- master
- stable
- /^v\d+\.\d+(\.\d+)?(-\S*)?$/ # for tagging

cache:
directories:
- $TRAVIS_BUILD_DIR/deps/src

# dependent apt packages
addons:
apt:
sources:
- ubuntu-toolchain-r-test
packages:
- doxygen
- wget
- git
- libcurl4-openssl-dev
- unzip
- libatlas-base-dev
- libatlas-dev
- libopencv-dev
- gcc-4.8
- g++-4.8

before_install:
- export TRAVIS_DIR=test/travis
- source ${TRAVIS_DIR}/setup_env.sh

notifications:
email: false

script:
- ${TRAVIS_DIR}/run_test.sh

after_success:
# See https://github.com/dmlc/MXNet.jl/pull/303#issuecomment-341171774
- julia -e 'using MXNet; mx._sig_checker()'

- source ${TRAVIS_DIR}/run_coverage.sh
- echo $TRAVIS_JULIA_VERSION
- julia -e 'Pkg.add("Documenter")'
- julia -e 'cd(Pkg.dir("MXNet")); include(joinpath("docs", "make.jl"))'
Loading