Ship Modin with Ray. by devin-petersohn · Pull Request #3109 · ray-project/ray

devin-petersohn · 2018-10-22T20:47:03Z

What do these changes do?

Adds modin to the build process for Ray

Related issue number

Resolves #3108

devin-petersohn · 2018-10-22T21:34:11Z

cc @robertnishihara @pcmoritz I am not sure if this will also make it in the wheels.

thirdparty/scripts/build_modin.sh

robertnishihara · 2018-10-22T22:07:22Z

thirdparty/scripts/setup.sh

I'm not sure if this is the right way to do this, since it seems like we're deprecating this file, though I'm not sure. @chuxi can you comment?

I am certainly willing to move this, but I am not sure where it will go.

AmplabJenkins · 2018-10-22T23:28:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8812/
Test FAILed.

AmplabJenkins · 2018-10-23T23:19:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8832/
Test PASSed.

thirdparty/scripts/build_modin.sh

python/ray/__init__.py

AmplabJenkins · 2018-10-24T16:43:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8851/
Test FAILed.

AmplabJenkins · 2018-10-25T05:49:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8880/
Test FAILed.

AmplabJenkins · 2018-10-25T23:12:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8899/
Test FAILed.

AmplabJenkins · 2018-10-31T03:51:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8978/
Test FAILed.

AmplabJenkins · 2018-10-31T04:29:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8979/
Test FAILed.

devin-petersohn · 2018-10-31T05:17:20Z

This will not work because of the use_raylet parameter being dropped from the ray.init API. Is there any chance to add it back for one release, even though it is not doing anything? cc @robertnishihara

atumanov · 2018-10-31T16:10:05Z

Hey @devin-petersohn , could you elaborate on this, why do you need use_raylet to be there even though it will be a no-op? Are you concerned about existing binary releases of Modin that currently use ray.init and would automatically start using xray instead? Nothing changes from the application developer's perspective, and ray.init invocation is controlled by the Modin code from what I understand. So, just wondering about context.

devin-petersohn · 2018-10-31T16:28:06Z

I am not worried about defaulting to xray, we are already using xray. The problem comes with how our binary calls ray.init, because we use ray.init(use_raylet=True). Since that was completely dropped from the API, we get issues like this: modin-project/modin#236

I prefer that we don't break backwards compatibility with Modin, because we have been using xray for a month or so now (and two of our releases use xray).

atumanov · 2018-10-31T16:49:10Z

I see, the issue is that existing binary releases of Modin are explicitly passing use_raylet. So dropping use_raylet (which I agree is not back-compat) would only be an issue for the Modin releases that are already out, right? Going forward, as Modin ships with Ray (i.e., for this PR), we have full control over synchronizing their versions. Specifying a required ray package version for the modin pypi package is also an option to guarantee that you get the expected ABI.

devin-petersohn · 2018-10-31T17:03:31Z

Yes, we already specify that requirement. The problem is that we explicitly pull a the most recent binary of Modin into the Ray repo. It's a chicken and egg problem.

Modin depends on the most recent binary of Ray, which currently requires that we have use_raylet if we want to use xray. We can remove that in a subsequent release, but then the people that install modin with pip install modin will end up using legacy Ray, at least until we release again bumping the Ray version we depend on. If we do that, the version shipped with Ray will always lag the pypi binary by 1 release. This is because we have to release a version removing use_raylet so that Ray can release with a working version of Modin, then Modin will have to release again bumping the version of Ray. The easiest way to solve the chicken and egg problem is to release Ray for a single release with use_raylet still as optional in ray.init, but as a no-op (we can even throw an error if someone tries to set use_raylet=False). Then we can safely remove it in a subsequent release without having a release that uses legacy Ray.

richardliaw · 2018-10-31T18:46:06Z

Maybe keep use_raylet since it's an end-user API change, and raise DeprecationWarning() noting that this will do nothing and will be removed after 0.6 (or some specific point release)?

AmplabJenkins · 2018-11-02T01:06:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/8997/
Test PASSed.

thirdparty/scripts/build_modin.sh

AmplabJenkins · 2018-11-04T05:24:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9044/
Test FAILed.

AmplabJenkins · 2018-11-05T02:41:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9064/
Test FAILed.

AmplabJenkins · 2018-11-07T06:50:55Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9156/
Test PASSed.

AmplabJenkins · 2018-11-07T07:06:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9157/
Test PASSed.

AmplabJenkins · 2018-11-07T07:18:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9158/
Test PASSed.

AmplabJenkins · 2018-11-07T16:56:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9172/
Test PASSed.

AmplabJenkins · 2018-11-22T03:48:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9523/
Test FAILed.

AmplabJenkins · 2018-11-24T20:46:21Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9573/
Test FAILed.

robertnishihara · 2018-11-26T23:53:27Z

Maybe use curl instead of wget. If that fails, maybe try upgrading openssl in https://github.com/ray-project/ray/blob/master/python/build-wheel-manylinux1.sh.

AmplabJenkins · 2018-11-27T22:10:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9631/
Test FAILed.

AmplabJenkins · 2018-11-28T19:31:12Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9643/
Test FAILed.

AmplabJenkins · 2018-11-28T20:33:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9645/
Test FAILed.

AmplabJenkins · 2018-11-28T21:13:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9648/
Test FAILed.

AmplabJenkins · 2018-11-28T21:53:46Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9646/
Test FAILed.

AmplabJenkins · 2018-11-28T21:53:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9644/
Test FAILed.

AmplabJenkins · 2018-11-28T23:21:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9649/
Test FAILed.

AmplabJenkins · 2018-11-28T23:26:34Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9650/
Test FAILed.

AmplabJenkins · 2018-11-28T23:58:02Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/9651/
Test FAILed.

pcmoritz reviewed Oct 22, 2018

View reviewed changes

thirdparty/scripts/build_modin.sh Outdated Show resolved Hide resolved

robertnishihara reviewed Oct 22, 2018

View reviewed changes

thirdparty/scripts/build_modin.sh Outdated Show resolved Hide resolved

robertnishihara reviewed Oct 22, 2018

View reviewed changes

robertnishihara reviewed Oct 23, 2018

View reviewed changes

thirdparty/scripts/build_modin.sh Outdated Show resolved Hide resolved

robertnishihara reviewed Oct 23, 2018

View reviewed changes

python/ray/__init__.py Outdated Show resolved Hide resolved

robertnishihara changed the title ~~Ship modin~~ Ship Modin with Ray. Oct 25, 2018

devin-petersohn force-pushed the ship_modin branch from 1042c64 to f10a1bb Compare October 31, 2018 03:14

devin-petersohn force-pushed the ship_modin branch from f10a1bb to a76057e Compare November 1, 2018 23:27

robertnishihara reviewed Nov 2, 2018

View reviewed changes

thirdparty/scripts/build_modin.sh Outdated Show resolved Hide resolved

devin-petersohn added 2 commits November 21, 2018 16:35

Addressing comments

6131b5e

Lint

7f26369

devin-petersohn force-pushed the ship_modin branch from e597fc0 to 7f26369 Compare November 22, 2018 00:43

devin-petersohn added 2 commits November 24, 2018 12:21

Update lint

5f78131

Debug failure

a13e3cb

devin-petersohn added 2 commits November 27, 2018 11:04

Try with pip download

a6a6d5c

Ensure correct usage of pip

c52eb92

devin-petersohn added 2 commits November 28, 2018 10:53

Trying with wget

070579e

Trying with curl

5a35bf6

devin-petersohn added 5 commits November 28, 2018 11:36

fix curl

3a824ad

fix curl

83bf78f

fix

755911c

fix

f7d857f

test

d9637bf

fix

08f545a

pcmoritz approved these changes Nov 29, 2018

View reviewed changes

pcmoritz merged commit 4d2010a into ray-project:master Nov 29, 2018

williamma12 mentioned this pull request Feb 8, 2019

modin does not ship with ray #3998

Closed

Conversation

devin-petersohn commented Oct 22, 2018

What do these changes do?

Related issue number

Uh oh!

devin-petersohn commented Oct 22, 2018

Uh oh!

Uh oh!

Uh oh!

robertnishihara Oct 22, 2018

Choose a reason for hiding this comment

Uh oh!

devin-petersohn Oct 24, 2018

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Oct 22, 2018

Uh oh!

AmplabJenkins commented Oct 23, 2018

Uh oh!

Uh oh!

Uh oh!

AmplabJenkins commented Oct 24, 2018

Uh oh!

AmplabJenkins commented Oct 25, 2018

Uh oh!

AmplabJenkins commented Oct 25, 2018

Uh oh!

AmplabJenkins commented Oct 31, 2018

Uh oh!

AmplabJenkins commented Oct 31, 2018

Uh oh!

devin-petersohn commented Oct 31, 2018

Uh oh!

atumanov commented Oct 31, 2018

Uh oh!

devin-petersohn commented Oct 31, 2018

Uh oh!

atumanov commented Oct 31, 2018

Uh oh!

devin-petersohn commented Oct 31, 2018

Uh oh!

richardliaw commented Oct 31, 2018

Uh oh!

AmplabJenkins commented Nov 2, 2018

Uh oh!

Uh oh!

AmplabJenkins commented Nov 4, 2018

Uh oh!

AmplabJenkins commented Nov 5, 2018

Uh oh!

AmplabJenkins commented Nov 7, 2018

Uh oh!

AmplabJenkins commented Nov 7, 2018

Uh oh!

AmplabJenkins commented Nov 7, 2018

Uh oh!

AmplabJenkins commented Nov 7, 2018

Uh oh!

AmplabJenkins commented Nov 22, 2018

Uh oh!

AmplabJenkins commented Nov 24, 2018

Uh oh!

robertnishihara commented Nov 26, 2018

Uh oh!

AmplabJenkins commented Nov 27, 2018

Uh oh!

AmplabJenkins commented Nov 28, 2018

Uh oh!

AmplabJenkins commented Nov 28, 2018

Uh oh!

AmplabJenkins commented Nov 28, 2018

Uh oh!

AmplabJenkins commented Nov 28, 2018

Uh oh!

AmplabJenkins commented Nov 28, 2018

Uh oh!

AmplabJenkins commented Nov 28, 2018

Uh oh!

AmplabJenkins commented Nov 28, 2018

Uh oh!