Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

updated NCCL version breaks nightly pip wheel builds #17252

Closed
samskalicky opened this issue Jan 8, 2020 · 3 comments
Closed

updated NCCL version breaks nightly pip wheel builds #17252

samskalicky opened this issue Jan 8, 2020 · 3 comments
Assignees
Labels

Comments

@samskalicky
Copy link
Contributor

Description

nightly pip wheel builds were building successfully until December 30th. On December 31st the builds started failing for these build flavors:
cu90
cu100
cu101
cu90mkl
cu100mkl
cu101mkl

Error Message

All builds failed with the same error message:

/usr/bin/ld: /root/pip_build/mxnet-build/staticdeps/usr/lib/x86_64-linux-gnu/libnccl.a(init.o): unrecognized relocation (0x2a) in section `.text' 
/usr/bin/ld: final link failed: Bad value 
collect2: error: ld returned 1 exit status 
make: *** [lib/libmxnet.so] Error 1 
make: *** Waiting for unfinished jobs.... 
/usr/bin/ld: /root/pip_build/mxnet-build/staticdeps/usr/lib/x86_64-linux-gnu/libnccl.a(transport.o): unrecognized relocation (0x2a) in section `.text.startup' 
/usr/bin/ld: final link failed: Bad value 
collect2: error: ld returned 1 exit status 
make: *** [bin/im2rec] Error 1 

To Reproduce

Build using the docker container used in the CI:
mxnetci/publish.ubuntu1404_cpu

./tools/staticbuild/build.sh $mxnet_variant pip
@samskalicky samskalicky added the Bug label Jan 8, 2020
@samskalicky
Copy link
Contributor Author

@mxnet-label-bot update [Build]

@lanking520 lanking520 added Build and removed Bug labels Jan 8, 2020
@szha
Copy link
Member

szha commented Jan 8, 2020

working on it in https://github.com/szha/mxnet/tree/fix_cd_s3 (szha@f2035bf)

@szha szha self-assigned this Jan 8, 2020
@szha
Copy link
Member

szha commented Jan 19, 2020

fixed in #17259

@szha szha closed this as completed Jan 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants