Skip to content

Update Dask/Distributed version - #3978

Merged
rapids-bot[bot] merged 3 commits into
NVIDIA:branch-21.08from
pentschev:update-dask-versions
Jun 14, 2021
Merged

Update Dask/Distributed version#3978
rapids-bot[bot] merged 3 commits into
NVIDIA:branch-21.08from
pentschev:update-dask-versions

Conversation

@pentschev

@pentschev pentschev commented Jun 10, 2021

Copy link
Copy Markdown
Contributor

No description provided.

@pentschev
pentschev requested a review from a team as a code owner June 10, 2021 20:02
@github-actions github-actions Bot added conda conda issue gpuCI gpuCI issue labels Jun 10, 2021
@ajschmidt8 ajschmidt8 added improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jun 10, 2021
@dantegd

dantegd commented Jun 10, 2021

Copy link
Copy Markdown
Contributor

@gpucibot merge

@dantegd

dantegd commented Jun 10, 2021

Copy link
Copy Markdown
Contributor

@ajschmidt8 @pentschev CI seems to be installing an older nightly of dask-cudf:

dask-cudf                 21.08.00a210609 py38_g83bc0320b7_125    rapidsai-nightly

which probably is the reason we're getting these fails?

@jakirkham

jakirkham commented Jun 10, 2021

Copy link
Copy Markdown
Contributor

Guessing this is relevant ( rapidsai/cugraph#1637 (comment) )

@dantegd

dantegd commented Jun 11, 2021

Copy link
Copy Markdown
Contributor

rerun tests

1 similar comment
@jakirkham

jakirkham commented Jun 11, 2021

Copy link
Copy Markdown
Contributor

rerun tests

dantegd
dantegd requested changes Jun 11, 2021
View reviewed changes

@dantegd dantegd left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the change

Comment thread conda/environments/cuml_dev_cuda11.0.yml Outdated
Comment on lines +17 to +18
- dask>=2021.6.0
- distributed>=2021.6.0

@dantegd dantegd Jun 11, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change from installing from main? Since CI is installing from main, we should keep this so dev environments behave like CI (and mirror cudf https://github.com/rapidsai/cudf/blob/0a4e8a1c860518c9e30f5289dea07bd0879fb642/conda/environments/cudf_dev_cuda11.2.yml#L64)

@jakirkham jakirkham Jun 11, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's still done above. Peter is just setting a minimum version

@pentschev pentschev Jun 11, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think what Dante was asking for is that the conda dev environment files install Dask/Distributed main to match the same as CI is running on. I did that in 80c340d now.

@pentschev

pentschev commented Jun 11, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Failures seem legit to the changes that were introduced in Distributed 2021.6.0, for example:

ValueError: Metadata inference failed in `len`.

You have supplied a custom function and Dask is unable to 
determine the type of output that that function returns. 

To resolve this please provide a meta= keyword.
The docstring of the Dask function you ran should have more information.

I'm guessing cuML will require an update to adapt to the changes above, similar to what has been done in NVIDIA/cudf#8426 . Also ccing @galipremsagar in case he has thoughts or could work on addressing those.

Also FYI, I'll be on PTO for the next couple weeks, so maybe someone will need to take this over or ensure those issues are addressed on a different PR before merging this.

@pentschev pentschev mentioned this pull request Jun 11, 2021
Merged
@galipremsagar

galipremsagar commented Jun 11, 2021

Copy link
Copy Markdown
Contributor

Failures seem legit to the changes that were introduced in Distributed 2021.6.0, for example:

ValueError: Metadata inference failed in `len`.

You have supplied a custom function and Dask is unable to 
determine the type of output that that function returns. 

To resolve this please provide a meta= keyword.
The docstring of the Dask function you ran should have more information.

I'm guessing cuML will require an update to adapt to the changes above, similar to what has been done in rapidsai/cudf#8426 . Also ccing @galipremsagar in case he has thoughts or could work on addressing those.

Just looked at the errors, this bug is in dask-cudf which is introduced by : NVIDIA/cudf#8309, I can throw in a fix for this real quick.

@pentschev

pentschev commented Jun 11, 2021

Copy link
Copy Markdown
Contributor Author

Just looked at the errors, this bug is in dask-cudf which is introduced by : rapidsai/cudf#8309, I can throw in a fix for this real quick.

That would be great, thanks @galipremsagar !

@galipremsagar

galipremsagar commented Jun 12, 2021 •
edited
Loading

Copy link
Copy Markdown
Contributor

I've checked out cugraph(cugraph CI is also having a similar issue: rapidsai/cugraph#1637 (comment)) locally and built from source with latest dask and don't find the errors that I see in CI. This looks like somehow the older version of dask is not being uninstalled correctly in those images. I see that tests in the jobs that are currently running are passing, so will keep an eye out on this one.

@dantegd

dantegd commented Jun 12, 2021

Copy link
Copy Markdown
Contributor

rerun tests

@dantegd

dantegd commented Jun 12, 2021

Copy link
Copy Markdown
Contributor

@gpucibot merge

@galipremsagar

galipremsagar commented Jun 12, 2021

Copy link
Copy Markdown
Contributor

rerun tests

4 similar comments
@jakirkham

jakirkham commented Jun 12, 2021

Copy link
Copy Markdown
Contributor

rerun tests

@jakirkham

jakirkham commented Jun 12, 2021

Copy link
Copy Markdown
Contributor

rerun tests

@jakirkham

jakirkham commented Jun 12, 2021

Copy link
Copy Markdown
Contributor

rerun tests

@jakirkham

jakirkham commented Jun 12, 2021

Copy link
Copy Markdown
Contributor

rerun tests

@codecov-commenter

codecov-commenter commented Jun 12, 2021

Copy link
Copy Markdown

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.08@8fe1b05). Click here to learn what that means.
The diff coverage is n/a.

Impacted file tree graph

@@               Coverage Diff               @@
##             branch-21.08    #3978   +/-   ##
===============================================
  Coverage                ?   85.32%           
===============================================
  Files                   ?      230           
  Lines                   ?    18093           
  Branches                ?        0           
===============================================
  Hits                    ?    15437           
  Misses                  ?     2656           
  Partials                ?        0
Flag Coverage Δ
dask 47.90% <0.00%> (?)
non-dask 77.66% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8fe1b05...80c340d. Read the comment docs.

@galipremsagar

galipremsagar commented Jun 14, 2021

Copy link
Copy Markdown
Contributor

rerun tests

@rapids-bot
rapids-bot Bot merged commit 4f13873 into NVIDIA:branch-21.08 Jun 14, 2021
@pentschev
pentschev deleted the update-dask-versions branch June 28, 2021 10:59
vimarsh6739 pushed a commit to vimarsh6739/cuml that referenced this pull request Oct 9, 2023
@pentschev
Update Dask/Distributed version (NVIDIA#3978)
99cfdd2 
Authors:
  - Peter Andreas Entschev (https://github.com/pentschev)

Approvers:
  - AJ Schmidt (https://github.com/ajschmidt8)
  - https://github.com/jakirkham
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: NVIDIA#3978
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jakirkham jakirkham jakirkham approved these changes
@dantegd dantegd dantegd approved these changes
@ajschmidt8 ajschmidt8 ajschmidt8 approved these changes

Assignees

No one assigned

Labels

conda conda issue gpuCI gpuCI issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@pentschev @dantegd @jakirkham @galipremsagar @codecov-commenter @ajschmidt8