Julia docs #15454

aaronmarkham · 2019-07-04T00:48:16Z

Description

Since the dmlc.ml domain is gone we lost our Julia docs (and other things).
This PR adds Julia docs to our regular website deployment pipeline. 💥 Links to the Julia docs will no longer go to a spam/malware website.

Fixes #15415

Update: It also adds a thorough Ubuntu setup guide for Julia. The existing one doesn't work. However, this needs more eyes and testing. I did get it to work, but after a ton of troubleshooting, so I might have missed some env var or some setting that is required.

Update 2: I removed the separate Julia pipeline for docs. I needed to have its installation and config steps and it was duplicating things anyways. So since I had to fix CI, I went ahead and removed the duplication. That being said, sometime in the future it would be nice to revamp the docs build and have stages for each of the doc sets and make it so you don't have to rerun every stage of CI if one particular part of docs generation fails.

Preview

The Julia docs site ends up in /site in the Julia API area.
http://34.201.8.176/versions/julia/api/julia/site/index.html

aaronmarkham · 2019-07-04T03:17:12Z

Hmm maven failed in CI - that should be unrelated to this PR.
@zachgk @lanking520 - any ideas? I'll restart it anyways....

iblislin · 2019-07-05T03:04:35Z

If we have this PR merged, do we still need the Julia doc build pipeline in Jenkins?
(this one: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwebsite/detail/PR-15454/2/pipeline/86)

roywei · 2019-07-08T15:11:52Z

@mxnet-label-bot add [Julia, Doc]

aaronmarkham · 2019-07-08T18:51:26Z

If we have this PR merged, do we still need the Julia doc build pipeline in Jenkins?
(this one: http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fwebsite/detail/PR-15454/2/pipeline/86)

I think having a separate test for the docs package like Julia is a good idea and wish we had that for each kind, like C++ docs, Scala, etc. With this PR, it is duplicating the test, but this was the easiest way and keeps all of the docs generated in the same pipeline as the website output.

Let's leave it in there for now. Make sure everything is working how we want it, (docs are being built!) then discuss further how things should work, and try to streamline.

lanking520 · 2019-07-08T18:59:23Z

It seemed to be something to do with the the libmxnet.so problem. I just restart the test to see if it is reproducible

aaronmarkham · 2019-07-08T19:01:52Z

It seemed to be something to do with the the libmxnet.so problem. I just restart the test to see if it is reproducible

It did it twice now already. This new test is the third attempt.

aaronmarkham · 2019-07-08T19:34:05Z

@iblis17 There's a line in the ubuntu setup guide that says "The MXNet package for Julia is hosted in a separate repository, MXNet.jl, which is available on GitHub." But this isn't true anymore is it?

iblislin · 2019-07-09T00:41:36Z

@aaronmarkham It isn't true anymore.

aaronmarkham · 2019-07-09T16:39:21Z

@aaronmarkham It isn't true anymore.

Thanks, I'll change that and rebase so maybe this CI/maven problem will be fixed.

How far should the Julia docs go back? I could trigger the build in 1.4.x or earlier in settings.ini, but I'm pretty sure that those versions must be compatible with Julia 0.7 and 1.0 since invoking Julia with MXNet and running docs seemed different for earlier versions. Also, I don't really think it is needed to engineer a solution for older versions. If so, I'd put that in a different PR.

For this PR though, I'd trigger whatever is compatible plus maybe note what's going on there if people browse the older versions.

aaronmarkham · 2019-07-09T17:53:44Z

@iblis17 I tried to add some more details on getting Ubuntu configured natively (since all I did here was get it working via Docker and to run the docs), but I ran into this:

julia> using Pkg; Pkg.build("MXNet")
ERROR: The following package names could not be resolved:
 * MXNet (not found in project or manifest)
Please specify by known `name=uuid`.
Stacktrace:
 [1] pkgerror(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:120
 [2] #ensure_resolved#43(::Bool, ::Function, ::Pkg.Types.EnvCache, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:900
 [3] ensure_resolved at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:870 [inlined]
 [4] #build#53(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:455
 [5] build at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:436 [inlined]
 [6] build(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:431
 [7] top-level scope at none:0

I saw a post that you might need to add this to the registry like this, but it didn't work:

   Cloning git-repo `https://github.com/apache/incubator-mxnet/tree/master/julia`
ERROR: failed to clone from https://github.com/apache/incubator-mxnet/tree/master/julia, error: GitError(Code:ERROR, Class:Net, unexpected HTTP status code: 404)

aaronmarkham · 2019-07-09T19:13:03Z

docs/install/ubuntu_setup.md


 ```bash
-    export MXNET_HOME=/<path to>/libmxnet
+wget -qO julia-10.tar.gz https://julialang-s3.julialang.org/bin/linux/x64/1.0/julia-1.0.3-linux-x86_64.tar.gz


I am using CI's Julia setup as a guide.
https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_julia.sh

aaronmarkham · 2019-07-09T19:15:06Z

docs/install/ubuntu_setup.md

+julia -e 'using InteractiveUtils; versioninfo()'
+```
+
+#### Setup Your MXNet-Julia Environment


These steps come from CI's runtime_functions.sh:
https://github.com/apache/incubator-mxnet/blob/master/ci/docker/runtime_functions.sh#L1059-L1073

szha

Could you paste the preview link?

aaronmarkham · 2019-07-09T23:54:23Z

Could you paste the preview link?

It's in the main description:
http://34.201.8.176/versions/julia/api/julia/site/index.html
But since the website CI is done you can see that one too:
http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-15454/8/index.html

docs/install/ubuntu_setup.md

iblislin · 2019-07-10T03:52:26Z

BTW, Julia v1.0.4 is out. Let's test it: #15502.

IvyBazan · 2019-07-10T22:54:53Z

LGTM! Verified Julia installation and doc building process on an Ubuntu EC2 instance.

iblislin · 2019-07-11T15:56:06Z

Nice work! @aaronmarkham

aaronmarkham requested a review from iblislin July 4, 2019 00:48

aaronmarkham requested a review from szha as a code owner July 4, 2019 00:48

marcoabreu added Doc Julia labels Jul 8, 2019

aaronmarkham added 6 commits July 9, 2019 10:41

add julia docs generation option

9be03e1

add julia docs to website build

690ad13

update links to point to local julia site

a950907

fix mxnet build setting bug

46a399c

fix link for site

ff687cc

update ubuntu guide for julia

3dca150

aaronmarkham force-pushed the julia-docs branch from 7091409 to 3dca150 Compare July 9, 2019 17:41

aaronmarkham added 2 commits July 9, 2019 11:45

turn building mxnet on by default

b203552

readd env var since CI uses it

6a6c5b3

aaronmarkham commented Jul 9, 2019

View reviewed changes

aaronmarkham added 3 commits July 9, 2019 13:37

fold julia docs into main website jenkins routine

a26b84b

fix ubuntu julia setup steps

9761646

cleanup mentions of the old julia docs pipeline

25112dd

aaronmarkham changed the title ~~[WIP] Julia docs~~ Julia docs Jul 9, 2019

szha reviewed Jul 9, 2019

View reviewed changes

iblislin reviewed Jul 10, 2019

View reviewed changes

docs/install/ubuntu_setup.md Show resolved Hide resolved

iblislin reviewed Jul 10, 2019

View reviewed changes

docs/install/ubuntu_setup.md Show resolved Hide resolved

IvyBazan approved these changes Jul 10, 2019

View reviewed changes

iblislin approved these changes Jul 11, 2019

View reviewed changes

lanking520 approved these changes Jul 11, 2019

View reviewed changes

aaronmarkham merged commit 1f3195f into apache:master Jul 11, 2019

This was referenced Jul 12, 2019

add julia env settings #15523

Merged

Julia path patch #15561

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Julia docs #15454

Julia docs #15454

aaronmarkham commented Jul 4, 2019 •

edited

Loading

aaronmarkham commented Jul 4, 2019

iblislin commented Jul 5, 2019

roywei commented Jul 8, 2019

aaronmarkham commented Jul 8, 2019

lanking520 commented Jul 8, 2019

aaronmarkham commented Jul 8, 2019

aaronmarkham commented Jul 8, 2019

iblislin commented Jul 9, 2019

aaronmarkham commented Jul 9, 2019

aaronmarkham commented Jul 9, 2019

aaronmarkham Jul 9, 2019

aaronmarkham Jul 9, 2019

szha left a comment

aaronmarkham commented Jul 9, 2019

iblislin commented Jul 10, 2019

IvyBazan commented Jul 10, 2019

iblislin commented Jul 11, 2019

Julia docs #15454

Julia docs #15454

Conversation

aaronmarkham commented Jul 4, 2019 • edited Loading

Description

Preview

aaronmarkham commented Jul 4, 2019

iblislin commented Jul 5, 2019

roywei commented Jul 8, 2019

aaronmarkham commented Jul 8, 2019

lanking520 commented Jul 8, 2019

aaronmarkham commented Jul 8, 2019

aaronmarkham commented Jul 8, 2019

iblislin commented Jul 9, 2019

aaronmarkham commented Jul 9, 2019

aaronmarkham commented Jul 9, 2019

aaronmarkham Jul 9, 2019

Choose a reason for hiding this comment

aaronmarkham Jul 9, 2019

Choose a reason for hiding this comment

szha left a comment

Choose a reason for hiding this comment

aaronmarkham commented Jul 9, 2019

iblislin commented Jul 10, 2019

IvyBazan commented Jul 10, 2019

iblislin commented Jul 11, 2019

aaronmarkham commented Jul 4, 2019 •

edited

Loading