jenkins: julia build script

[iblislin]

• meging this patch (WIP: Julia CI build #8175 (comment)) into the ci-test branch

cc @mbaijal

[iblislin]

hmm, @piiswrong could you tell more detail about why closing this?

[marcoabreu]

@iblis17 Oh sorry for that, lets review this from scratch, what do you think?

I see a few issues with your proposed approach. During build, you're checking out another repository that is not in sync with this repo. Is it possible to auto-generate the structure the structure Julia expects? If that's possible, I'd propose that as part of testing, we generate the appropriate Julia environment on CI and then run tests on top of that - no checkout of another repository happens that way.

[marcoabreu]

Additionally, we could create a job on CI that automatically keeps the Julia repo in sync with the main repo - e.g. copying all commits to the Julia Repo.

[iblislin]

Let me tell more about my approach.
The basic idea is adding the dmlc/MXNet.jl repo as a git submodule (See: https://github.com/apache/incubator-mxnet/pull/8175/files#diff-8903239df476d7401cf9e76af0252622R19).
Then, since it's a submodule, git clone and work correctly for Julia, thus CI can run the testing.

Julia's package manager is Pkg. To install a package, Pkg.clone("a_repo"), this instruction is same as git clone. And it clones stuffs into the dir ~/.julia/v0.x/MXNet by default.

During build, you're checking out another repository that is not in sync with this repo.

So, yeah, git submodule does have sync problem.

Is it possible to auto-generate the structure the structure Julia expects?

(Yes, but) It's only possible for testing/dev purpose. For releasing a package, you still need a standalone repo to let user Pkg.add/Pkg.clone it directly.
An easy solution is making a soft link ~/.julia/v0.x/MXNet which point to the real path.

e.g.

% cd ~/.julia/v0.6 && ln -s ~/dev/mxnet/julia MXNet

Additionally, we could create a job on CI that automatically keeps the Julia repo in sync with the main repo - e.g. copying all commits to the Julia Repo.

So, you propose that moving all Julia related development to this repo, and treat dmlc/MXNet.jlas kind of mirror, right?

[marcoabreu]

It's only possible for testing/dev purpose
That's the intention: For CI.

So, you propose that moving all Julia related development to this repo, and treat dmlc/MXNet.jlas kind of mirror, right?
Exactly.

Since the Julia requires its own dedicated GitHub repository, I propose that all Julia-related development gets moved into this repository and that we run a CI job that compiles MXNet and Julia packages and then publishes it to a new repository, e.g. incubator-mxnet-julia. Users would close off that repository while actual development happens in incubator-mxnet.

I'm totally unfamiliar with Julia, so please bare with me if there's a mistake in my thought-process.

[iblislin]

I agree your thought-process.
cc (other MXNet.jl maintainer) @pluskid, @vchuravy .

then publishes it to a new repository, e.g. incubator-mxnet-julia

I want that we keep using the original dmlc/MXNet.jl repo without renaming.
Since Julia community has a releasing bot set up already; we are using it, also. (I think) it requires the consistency of repo name. (bot: https://github.com/attobot)

Users would close off that repository while actual development happens in incubator-mxnet.

I prefer to have a standalone issue tracker for Julia stuffs, as the status quo. I can keep eye on it, if user open issue on it. (Watching this repo is too noisy for me.)

Also, If we move dev to this repo, there are some issues I want to check:

• How to git tag a new release? Julia binding can have different version and release cycle, right?
• Document hosting, Julia's doc system is integrated Github page, and able to auto-depoly with Travis build. And Julia's docstring is written in MD, it's unable to render with Sphinx.
  Current one: http://dmlc.ml/MXNet.jl/stable/.
  For this issue, we can just keep current repo and CI build to deploy docs.
• Committer bit: There are 3 developers own the committer bit of MXNet.jl. @pluskid, @vchuravy, and me. Will this be able to migrate as well?

[vchuravy]

Having MXNet.jl being part of the main incubator-mxnet would actually be helpful, since their have been issues in the past with keeping the code synchronised. I would propose that we do a git subtree import of MXNet.jl into incubator-mxnet/julia
We can then use the git subtree mechanism to push changes to MXNet.jl and tag releases that way.

[vchuravy]

It will also be possible with the next major release of Julia and the new package manager to install packages that are not primary repositories. Which means that in the long-term we can deprecate the MXNet.jl repository.

…

On Tue, 27 Feb 2018 at 16:03 Valentin Churavy ***@***.***> wrote: Having MXNet.jl being part of the main incubator-mxnet would actually be helpful, since their have been issues in the past with keeping the code synchronised. I would propose that we do a git subtree import of MXNet.jl into incubator-mxnet/julia We can then use the git subtree mechanism to push changes to MXNet.jl and tag releases that way. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8727 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAI3akkPGnRoHLiVEMQtr6WG5iAdSuZsks5tZG1-gaJpZM4Qkij_> .

[marcoabreu]

I want that we keep using the original dmlc/MXNet.jl repo without renaming.

Would you be fine with that repo being a mirror of the Apache repository?

I prefer to have a standalone issue tracker for Julia stuffs, as the status quo. I can keep eye on it, if user open issue on it. (Watching this repo is too noisy for me.)

We are make use of the tagging feature and label each issue according to the language binding. In future (see the current vote thread on dev@), we will make use of Jira. This should give you all the tools and overview you need.

How to git tag a new release? Julia binding can have different version and release cycle, right?

At the moment, every language binding shares the same release cycle as MXNet itself. I agree that this is not optimal and I'd be happy to see some improvements in that direction.

Document hosting

From a first clance it seems like it is not too complicated to migrate to a Sphinx compatible layout, but I could underestimate the required effort. Do you think this would be an issue?

Committer bit: There are 3 developers own the committer bit of MXNet.jl

I have to check back, but since you three are putting a lot of time and effort into supporting MXNet, I don't see many issues there. I'm unfamiliar with the Apache process if two projects get merged, but I'm sure we will find a solution.

Generally, this sounds like a discussion we should involve the community into. Would you mind creating a thread on dev@? In the meantime, I will check back about the committership.

[iblislin]

Would you be fine with that repo being a mirror of the Apache repository?

yeah, mirroring to that repo sounds good to me.

We are make use of the tagging feature and label each issue according to the language binding. In future (see the current vote thread on dev@), we will make use of Jira. This should give you all the tools and overview you need.

👍

From a first clance it seems like it is not too complicated to migrate to a Sphinx compatible layout, but I could underestimate the required effort. Do you think this would be an issue?

Well, I have no idea about any detail of Sphinx at all.
Julia's doc system is Documenter.jl, which launches Julia compiler, collects docstring from package and
render the static html as output. If Sphinx can accept some extra html files from external source, I guess most of work is done.

Generally, this sounds like a discussion we should involve the community into. Would you mind creating a thread on dev@? In the meantime, I will check back about the committership.

Ok, will do.

[pluskid]

I think it is OK to keep the development in the main repo side: either way seems to have trade offs. I guess the important thing is to keep both sides in sync. Currently hard constraints is that the Julia package should be in a separate repo (being a mirror or not).

As for documents, sphinx might work but there might be some extra work to make the dynamically generated doc work. Also I see a much less motivation to merge and migrate the docs. The original setup should still work, right?

[iblislin]

As for documents, sphinx might work but there might be some extra work to make the dynamically generated doc work. Also I see a much less motivation to merge and migrate the docs. The original setup should still work, right?

yeah, the easiest solution is just keeping the original setup; keeping travis build and auto-deployment.