[Discussion] 1.5.1 Patch Release #15613

samskalicky · 2019-07-19T18:45:15Z

Let's start a discussion here about the known issues with 1.5.0 to put into a patch release.

Create (or locate existing) issue/pull request for the item, note the issue/pull request number.
Comment in this issue: 1) the above issue number, 2) one sentence of what the item is about and why it's important.
Indicate whether you'd be willing to help out on the item.
Share the ETA if you're driving the item and have an guesstimate on when it will be done.

cc @apache/mxnet-committers

samskalicky · 2019-07-19T18:46:29Z

Theres a problem with the WarpCTC plugin in 1.5.0 #15612. A PR has been created with a fix #15601

ThomasDelteil · 2019-07-19T18:48:30Z

Reports of inference errors on 1.5.0:
https://discuss.mxnet.io/t/mxnet-1-5-0-is-not-downward-compatible-to-mxnet-1-4-1/4474
#15281

roywei · 2019-07-19T18:52:20Z

Possible issue with asnumpy(), but not able to reproduce yet #15431

roywei · 2019-07-19T18:52:51Z

license issues need to be fixed before next release #15542
MKL-DNN can be resolved in 1.6.0, Cub issue depends on Nvidia to release new version of Cub, other issue should be easy to resolve in 1.5.1.

roywei · 2019-07-19T19:43:42Z

FYI #15281 (comment) is fixed in 1.5.0

roywei · 2019-07-19T21:32:42Z

FYI #15337 is also resolved

keerthanvasist · 2019-07-19T22:05:06Z

@mxnet-label-bot add [discussion]

iblislin · 2019-07-20T06:10:06Z

I will track Julia stuffs that need to be backported here:

iblislin · 2019-07-20T14:22:23Z

Hi, I just created a label "Backport 1.5" (https://github.com/apache/incubator-mxnet/labels/Backport%201.5).
I think we can create a single PR to collecting the stuffs that need to be backported, use this label as a checklist, and keep that PR open until we want to make a release candidate.

marcoabreu · 2019-07-22T12:46:47Z

I'd prefer if we use multiple PRs which only contain cherry-picks from master. That way, we don't squash and keep the references.

iblislin · 2019-07-23T06:28:23Z

ah, okay

szha · 2019-07-26T20:22:58Z

@samskalicky thanks for starting the discussion. @PatricZhao @TaoLv could you drive this release?

pengzhao-intel · 2019-07-28T04:37:32Z

It's great to lead the release. @TaoLv @juliusshufan will be working on this.

roywei · 2019-07-31T17:08:57Z

Update: moved this to 1.6.0 scope

nightly test failure need to be fixed:
#15374

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/NightlyTestsForBinaries/detail/master/395/pipeline/

sandeep-krishnamurthy · 2019-07-31T23:56:09Z

#15672

samskalicky · 2019-08-05T22:49:56Z

@TaoLv Lets include #15692 in the 1.5.1 release to fix #15737

TaoLv · 2019-08-06T05:43:34Z

@TaoLv Lets include #15692 in the 1.5.1 release to fix #15737

Thanks for reporting. Added it to https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status

wkcn · 2019-08-06T23:47:28Z

Hi, I’m fixing the bug of two C APIs MXEnginePushAsyncND and MXEnginePushSyncND
#15751

It will be finished this week.

TaoLv · 2019-08-07T02:14:53Z

Thanks for proposing, @wkcn .
Is this bug firstly introduced in the 1.5.0 release? Why do you think it should be included into the 1.5.1 patch release?

wkcn · 2019-08-07T02:41:59Z

@TaoLv Yes, it is a bug in 1.5.0 release https://github.com/apache/incubator-mxnet/blob/v1.5.x/src/c_api/c_api.cc#L1537

It need to be fixed, otherwise users couldn't use these two C APIs in the release version of MXNet.

TaoLv · 2019-08-07T02:45:34Z

I mean does it also exist in v1.4.x releases?

wkcn · 2019-08-07T02:48:15Z

@TaoLv It doesn't exist in v1.4.x release. The bug was introdueced firstly in v1.5.x release.

TaoLv · 2019-08-07T05:44:43Z

Thanks for the clarification. Do you have a github issue for that? I add it to the track list.

wkcn · 2019-08-07T06:36:30Z

@TaoLv I have added the issue #15774

TaoLv · 2019-08-07T06:48:39Z

@wkcn, thanks for your report and contribution! The issue and your fix are added to https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status.

KellenSunderland · 2019-08-08T05:22:55Z

What's the issue with MShadow? Did they have to do a force push / history re-write again? We may indeed have to backport that to other release branches.

TaoLv · 2019-08-08T14:03:21Z

@KellenSunderland mshadow source code is donated and merged to MXNet. See #15600 and dmlc/mshadow#373. So it's not a git submodule any more.

wkcn · 2019-08-08T14:43:19Z

For the source downloaded before mshadow merged into MXNet, users need to remove
the directory mshadow firstly, then git pull to update the code.

samskalicky · 2019-08-08T15:35:49Z

@wkcn that doesnt work for me:

ubuntu@ip-172-31-76-214:~$ git clone --recursive https://github.com/apache/incubator-mxnet.git mxnet2
Cloning into 'mxnet2'...

ubuntu@ip-172-31-76-214:~/mxnet2$ git checkout v1.5.x
M	3rdparty/dlpack
M	3rdparty/dmlc-core
M	3rdparty/mkldnn
M	3rdparty/tvm
Branch v1.5.x set up to track remote branch v1.5.x from origin.
Switched to a new branch 'v1.5.x'
ubuntu@ip-172-31-76-214:~/mxnet2$ cd 3rdparty/
ubuntu@ip-172-31-76-214:~/mxnet2/3rdparty$ ls
ctc_include  dlpack  dmlc-core  googletest  mkldnn  mshadow  nvidia_cub  onnx-tensorrt  openmp  ps-lite  tvm
ubuntu@ip-172-31-76-214:~/mxnet2/3rdparty$ ls mshadow/
ubuntu@ip-172-31-76-214:~/mxnet2/3rdparty$ rm -rf mshadow/
ubuntu@ip-172-31-76-214:~/mxnet2/3rdparty$ git pull
Already up-to-date.
ubuntu@ip-172-31-76-214:~/mxnet2/3rdparty$ ls
ctc_include  dlpack  dmlc-core  googletest  mkldnn  nvidia_cub  onnx-tensorrt  openmp  ps-lite  tvm
ubuntu@ip-172-31-76-214:~/mxnet2/3rdparty$ git submodule update --recursive
Submodule path 'dlpack': checked out '10892ac964f1af7c81aae145cd3fab78bbccd297'
Submodule path 'dmlc-core': checked out '3943914eed66470bd010df581e29e4dca4f7df6f'
Submodule path 'mkldnn': checked out '41bee20d7eb4a67feeeeb8d597b3598994eb1959'
Submodule path 'tvm': checked out '21935dcbf56ad3bd66ebff9891a6bc3865b8106d'
Submodule path 'tvm/3rdparty/dlpack': checked out '5c792cef3aee54ad8b7000111c9dc1797f327b59'

Submodule path 'tvm/3rdparty/dmlc-core': checked out '82bf4c2e2af312b3d52513aa727483803a2f8734'
ubuntu@ip-172-31-76-214:~/mxnet2/3rdparty$ 
ubuntu@ip-172-31-76-214:~/mxnet2/3rdparty$ ls
ctc_include  dlpack  dmlc-core  googletest  mkldnn  nvidia_cub  onnx-tensorrt  openmp  ps-lite  tvm
ubuntu@ip-172-31-76-214:~/mxnet2/3rdparty$

wkcn · 2019-08-08T22:58:51Z

Hi @samskalicky , I can download mshadow by git submodule update --init --recursive

TaoLv · 2019-08-09T15:53:03Z

@KellenSunderland Could you please include the TensorRT patches you just mentioned into the cwiki? https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status

Thanks for your support!

TaoLv · 2019-08-12T02:10:31Z

@iblis17 Could you help to pick #15608 #15609 to the v1.5.x branch?

KellenSunderland · 2019-08-13T04:57:08Z

For some reason not able to edit the cwiki?

Proposals for fixes are:
#15645
#15666
#14898
#15344
#14860

TaoLv · 2019-08-13T05:31:38Z

Thank you @KellenSunderland . I just included these patches to the table on cwiki. Please help take a look: https://cwiki.apache.org/confluence/display/MXNET/1.5.1+Release+Plan+and+Status
BTW, do you know someone with TensorRT background can help to pick them to the v1.5.x branch? Because I'm wondering if they need be ported in a certain order.

KellenSunderland · 2019-08-13T16:15:25Z

No problem Tao, I can probably cherry-pick them a bit later today.

KellenSunderland · 2019-08-13T16:35:03Z

First three cherry-picked PRs:
#15875
#15874
#15877

ChaiBapchya · 2019-08-19T14:41:41Z

Benchmark doc fix PR #15769

samskalicky · 2019-08-21T00:00:07Z

@TaoLv I'd like to pull in the following PRs, they are necessary fixes for some of my use-cases:
#15245
#15917

apeforest · 2019-08-21T00:06:43Z

This PR fixes a memory misalignment bug in topk operator introduced recently. Please add it to 1.5.1 patch release:
#15948

Thanks

TaoLv · 2019-08-28T14:59:01Z

First three cherry-picked PRs:
#15875
#15874
#15877

@KellenSunderland, the 3 PRs were already merged to v1.5.x. Please update for the other two if we still need them.

TaoLv · 2019-08-28T14:59:20Z

Benchmark doc fix PR #15769

@ChaiBapchya, thank you for the report. Could you please cherry pick it to the v1.5.x branch and tag me in the PR?

TaoLv · 2019-08-28T15:00:07Z

@TaoLv I'd like to pull in the following PRs, they are necessary fixes for some of my use-cases:
#15245
#15917

@samskalicky , please pick them to v1.5.x and tag me in the PR. I will update the cwiki page accordingly.

TaoLv · 2019-08-28T15:01:35Z

This PR fixes a memory misalignment bug in topk operator introduced recently. Please add it to 1.5.1 patch release:
#15948

Thanks

@apeforest Any update? Is it still true that we can pick this to v1.5.x?

KellenSunderland · 2019-08-30T04:13:05Z

Hey @TaoLv . Updated the last two:

#16043
#16044

iblislin · 2019-09-11T04:24:51Z

@iblis17 Could you help to pick #15608 #15609 to the v1.5.x branch?

@TaoLv Just can I just push to the v1.5.x branch direcetly?

marcoabreu · 2019-09-11T06:16:07Z

No, please create pull requests instead

apeforest · 2019-09-11T06:26:45Z

Haven't we already passed code freeze?

…

On Tue, Sep 10, 2019 at 11:16 PM Marco de Abreu ***@***.***> wrote: No, please create pull requests instead — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15613>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABT54SKA7OO3WMTGBRTZXTTQJCEMVANCNFSM4IFKN5YA> .

TaoLv · 2019-09-11T08:17:49Z

Yes, we have passed code freeze and the rc0 was already tagged. Front-end packaging is in progress. One can still commit code change to the v1.5.x branch but that will be not included in the vote for rc0.

lostella · 2019-09-11T12:54:41Z

1.5.0 apparently broke sampling operators on Linux, see #16135. A PR is open for this #16139. I hope this has a chance to make it to 1.5.1.

eric-haibin-lin · 2019-09-20T00:47:44Z

If we have a chance for another RC, we should port #15240 and #15342

iblislin · 2019-09-22T07:21:25Z

~~No idea why my backporting PR got closed (#16142).~~ reopened

strawberrypie · 2019-10-25T14:39:05Z

Any news on 1.5.1?

eric-haibin-lin · 2019-10-25T17:31:32Z

released: https://github.com/apache/incubator-mxnet/releases/tag/1.5.1

marcoabreu added the Discussion label Jul 19, 2019

pengzhao-intel assigned pengzhao-intel and unassigned pengzhao-intel Jul 28, 2019

pengzhao-intel assigned pengzhao-intel and TaoLv Aug 3, 2019

roywei mentioned this issue Aug 29, 2019

[Discussion] 1.6.0 Roadmap #15589

Closed

eric-haibin-lin closed this as completed Oct 25, 2019

[Discussion] 1.5.1 Patch Release #15613

[Discussion] 1.5.1 Patch Release #15613

Comments

samskalicky commented Jul 19, 2019

samskalicky commented Jul 19, 2019

ThomasDelteil commented Jul 19, 2019

roywei commented Jul 19, 2019

roywei commented Jul 19, 2019 • edited Loading

roywei commented Jul 19, 2019

roywei commented Jul 19, 2019

keerthanvasist commented Jul 19, 2019

iblislin commented Jul 20, 2019 • edited Loading

iblislin commented Jul 20, 2019

marcoabreu commented Jul 22, 2019

iblislin commented Jul 23, 2019

szha commented Jul 26, 2019

pengzhao-intel commented Jul 28, 2019

roywei commented Jul 31, 2019 • edited Loading

Update: moved this to 1.6.0 scope

sandeep-krishnamurthy commented Jul 31, 2019

samskalicky commented Aug 5, 2019 • edited Loading

TaoLv commented Aug 6, 2019

wkcn commented Aug 6, 2019

TaoLv commented Aug 7, 2019

wkcn commented Aug 7, 2019

TaoLv commented Aug 7, 2019

wkcn commented Aug 7, 2019

TaoLv commented Aug 7, 2019

wkcn commented Aug 7, 2019

TaoLv commented Aug 7, 2019

KellenSunderland commented Aug 8, 2019

TaoLv commented Aug 8, 2019

wkcn commented Aug 8, 2019

samskalicky commented Aug 8, 2019

wkcn commented Aug 8, 2019 • edited Loading

TaoLv commented Aug 9, 2019

TaoLv commented Aug 12, 2019

KellenSunderland commented Aug 13, 2019 • edited Loading

TaoLv commented Aug 13, 2019

KellenSunderland commented Aug 13, 2019

KellenSunderland commented Aug 13, 2019 • edited Loading

ChaiBapchya commented Aug 19, 2019

samskalicky commented Aug 21, 2019

apeforest commented Aug 21, 2019

TaoLv commented Aug 28, 2019

TaoLv commented Aug 28, 2019

TaoLv commented Aug 28, 2019

TaoLv commented Aug 28, 2019

KellenSunderland commented Aug 30, 2019 • edited Loading

iblislin commented Sep 11, 2019

marcoabreu commented Sep 11, 2019

apeforest commented Sep 11, 2019 via email

TaoLv commented Sep 11, 2019

lostella commented Sep 11, 2019

eric-haibin-lin commented Sep 20, 2019

iblislin commented Sep 22, 2019 • edited Loading

strawberrypie commented Oct 25, 2019

eric-haibin-lin commented Oct 25, 2019

roywei commented Jul 19, 2019 •

edited

Loading

iblislin commented Jul 20, 2019 •

edited

Loading

roywei commented Jul 31, 2019 •

edited

Loading

samskalicky commented Aug 5, 2019 •

edited

Loading

wkcn commented Aug 8, 2019 •

edited

Loading

KellenSunderland commented Aug 13, 2019 •

edited

Loading

KellenSunderland commented Aug 13, 2019 •

edited

Loading

KellenSunderland commented Aug 30, 2019 •

edited

Loading

iblislin commented Sep 22, 2019 •

edited

Loading