Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[MXNET-1191] Maven Jenkins Pipeline with Static Binary and GPU #13767

Merged
merged 1 commit into from
Jan 16, 2019

Conversation

zachgk
Copy link
Contributor

@zachgk zachgk commented Jan 3, 2019

Description

This PR is the third and final part of the maven Jenkins publishing that adds integration with the static building scripts placed in tools as well as enabling GPU.

Note that this PR depends on (and is based on top of until they are merged) #13450 and #13707. The changes as part of this PR are only those located in a3c7d46.

Please review so the pipeline can be put into production:
@nswamy @lanking520 @andrewfayres @piyushghai @marcoabreu @yzhliu @szha

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

@Roshrini
Copy link
Member

Roshrini commented Jan 3, 2019

@mxnet-label-bot Add [pr-awaiting-review]

@marcoabreu marcoabreu added the pr-awaiting-review PR is waiting for code review label Jan 3, 2019
@zachgk
Copy link
Contributor Author

zachgk commented Jan 3, 2019

@mxnet-label-bot add [CI, Java, Maven, Scala]

@marcoabreu marcoabreu added CI Java Label to identify Java API component Maven Scala labels Jan 3, 2019
@lanking520
Copy link
Member

Please do a rebase with master

# the whole docker cache for the image

set -ex
apt-get update || true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why add the boolean OR true condition here ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "|| true" means that the line will always evaluate with a zero exit code. For scripts with "set -e", any line evaluating with a nonzero exit code will exit the script. I am not sure why we ignore problems on the update, but this is done throughout the CI install scripts.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super! Thanks.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes, some apt repositories are not available and apt-get update fails. If that's the case, we still want to go ahead since all other repositories still updated (think of 1 out of 500 repositories not being available). If the repository we actually need for the installation is actually not available, the install will fail properly and that's fine. But in most cases, it's just an unused and unrelated repository that's not available and thus we just ignore it.

Copy link
Contributor

@marcoabreu marcoabreu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a3c7d46 LGTM

Copy link
Member

@lanking520 lanking520 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran it for a couple of times on my local computer and it is working just fine. Also see @zachgk tested a couple of times on Jenkins, overall looks good.

@lanking520 lanking520 merged commit 13f7c58 into apache:master Jan 16, 2019
stephenrawls pushed a commit to stephenrawls/incubator-mxnet that referenced this pull request Feb 16, 2019
@zachgk zachgk deleted the allMaven branch April 12, 2019 22:25
lanking520 pushed a commit to lanking520/incubator-mxnet that referenced this pull request Apr 26, 2019
lanking520 pushed a commit to lanking520/incubator-mxnet that referenced this pull request Apr 30, 2019
zachgk added a commit to zachgk/incubator-mxnet that referenced this pull request May 16, 2019
haohuanw pushed a commit to haohuanw/incubator-mxnet that referenced this pull request Jun 23, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
CI Java Label to identify Java API component Maven pr-awaiting-review PR is waiting for code review Scala
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants