[REVIEW] Upgrade `arrow` & `pyarrow` to `5.0.0` by galipremsagar · Pull Request #8908 · NVIDIA/cudf

galipremsagar · 2021-07-30T02:24:15Z

This PR upgrades arrow to 5.0.0.

Upgrade & test arrow 5.0.0.
Fix pytest failures related to decimal arrays.

codecov · 2021-07-30T09:02:21Z

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.10@59b84f3). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff               @@
##             branch-21.10    #8908   +/-   ##
===============================================
  Coverage                ?   10.58%           
===============================================
  Files                   ?      116           
  Lines                   ?    19060           
  Branches                ?        0           
===============================================
  Hits                    ?     2018           
  Misses                  ?    17042           
  Partials                ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59b84f3...e493f4f. Read the comment docs.

robertmaynard

CMake changes LGTM

brandon-b-miller

The changes in the column constructor lgtm but I am not quite following why all these tests were wrong. Can you help me with a simple example?

rjzamora · 2021-07-30T15:34:01Z

I'm a bit surprised that CI passes for dask-cudf (I would expect that we need #8871 to avoid the ParquetDataset deprecations in pyarrow-5, but we could be narrowly avoiding the problem for now). Also, note that Dask is temporarily pinned to pyarrow<5

galipremsagar · 2021-07-30T15:37:18Z

The changes in the column constructor lgtm but I am not quite following why all these tests were wrong. Can you help me with a simple example?

Sure, previously arrow had a bug of incorrectly allowing value with a lower precision, in 5.0, this is fixed:

>>> pa.array([10, 20], type=pa.decimal128(precision=2, scale=2))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/array.pxi", line 306, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Decimal type with precision 2 does not fit into precision inferred from first array element: 2

If we see here, we cannot fit 10 & 20 into a decimal of precision 2 & scale 2.

galipremsagar · 2021-07-30T15:48:55Z

I'm a bit surprised that CI passes for dask-cudf (I would expect that we need #8871 to avoid the ParquetDataset deprecations in pyarrow-5, but we could be narrowly avoiding the problem for now). Also, note that Dask is temporarily pinned to pyarrow<5

I can confirm I didn't see dask-cudf failures locally aswell. Probably because they are just DeprecationWarnings now and not errors?

brandon-b-miller · 2021-07-30T16:02:29Z

The changes in the column constructor lgtm but I am not quite following why all these tests were wrong. Can you help me with a simple example?

Sure, previously arrow had a bug of incorrectly allowing value with a lower precision, in 5.0, this is fixed:
>>> pa.array([10, 20], type=pa.decimal128(precision=2, scale=2))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow/array.pxi", line 306, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Decimal type with precision 2 does not fit into precision inferred from first array element: 2
If we see here, we cannot fit 10 & 20 into a decimal of precision 2 & scale 2.

Oh I see. And thus the result precision must be updated as well.

Thank you for clarifying!

galipremsagar · 2021-07-30T22:49:20Z

rerun tests

galipremsagar · 2021-07-30T23:17:52Z

rerun tests

galipremsagar · 2021-08-03T21:28:12Z

rerun tests

galipremsagar · 2021-08-03T22:14:51Z

rerun tests

galipremsagar · 2021-08-03T23:54:16Z

rerun tests

msadang

Looks good to me.

galipremsagar · 2021-08-13T08:08:31Z

@gpucibot merge

upgrade arrow

f127d8e

galipremsagar added 2 - In Progress Currently a work in progress Python Affects Python cuDF API. improvement Improvement / enhancement to an existing function non-breaking Non-breaking change labels Jul 30, 2021

galipremsagar self-assigned this Jul 30, 2021

galipremsagar requested review from a team as code owners July 30, 2021 02:24

github-actions Bot added CMake CMake build issue conda libcudf Affects libcudf (C++/CUDA) code. and removed Python Affects Python cuDF API. labels Jul 30, 2021

galipremsagar added 2 commits July 29, 2021 23:51

fix decimal creation

4f40b2c

add error tests

9356ae1

github-actions Bot added the Python Affects Python cuDF API. label Jul 30, 2021

galipremsagar changed the title ~~[WIP] Upgrade arrow & pyarrow to 5.0.0~~ [REVIEW] Upgrade arrow & pyarrow to 5.0.0 Jul 30, 2021

galipremsagar added 4 - Needs cuDF (Python) Reviewer 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Jul 30, 2021

galipremsagar mentioned this pull request Jul 30, 2021

Upgrade arrow to 5.0.0 rapidsai/integration#330

Merged

robertmaynard approved these changes Jul 30, 2021

View reviewed changes

galipremsagar requested a review from trxcllnt July 30, 2021 14:48

brandon-b-miller reviewed Jul 30, 2021

View reviewed changes

shwina reviewed Jul 30, 2021

View reviewed changes

Comment thread python/cudf/cudf/core/column/column.py Outdated

remove try/except

d77683a

galipremsagar requested a review from a team as a code owner July 30, 2021 15:40

galipremsagar requested a review from isVoid July 30, 2021 15:40

galipremsagar added the DO NOT MERGE Hold off on merging; see PR for details label Jul 30, 2021

galipremsagar removed the DO NOT MERGE Hold off on merging; see PR for details label Jul 30, 2021

galipremsagar added the DO NOT MERGE Hold off on merging; see PR for details label Jul 30, 2021

Merge branch 'rapidsai:branch-21.10' into arrow_5.0

5b9124a

galipremsagar removed the DO NOT MERGE Hold off on merging; see PR for details label Aug 3, 2021

shwina approved these changes Aug 3, 2021

View reviewed changes

galipremsagar removed the 4 - Needs cuDF (Python) Reviewer label Aug 3, 2021

galipremsagar added 2 commits August 3, 2021 13:31

Merge remote-tracking branch 'upstream/branch-21.10' into arrow_5.0

0371448

remove temporary build commands

69ea903

github-actions Bot removed the gpuCI label Aug 3, 2021

Merge branch 'rapidsai:branch-21.10' into arrow_5.0

a20af37

galipremsagar mentioned this pull request Aug 10, 2021

Upgrade arrow to 5.0.0 rapidsai/integration#337

Merged

msadang approved these changes Aug 13, 2021

View reviewed changes

galipremsagar added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 3 - Ready for Review Ready for review by team labels Aug 13, 2021

Merge branch 'rapidsai:branch-21.10' into arrow_5.0

e493f4f

dillon-cullinan approved these changes Aug 13, 2021

View reviewed changes

rapids-bot Bot merged commit 2c5a2ad into NVIDIA:branch-21.10 Aug 13, 2021

Uh oh!

Conversation

galipremsagar commented Jul 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

robertmaynard left a comment

Choose a reason for hiding this comment

Uh oh!

brandon-b-miller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rjzamora commented Jul 30, 2021

Uh oh!

galipremsagar commented Jul 30, 2021

Uh oh!

galipremsagar commented Jul 30, 2021

Uh oh!

brandon-b-miller commented Jul 30, 2021

Uh oh!

galipremsagar commented Jul 30, 2021

Uh oh!

galipremsagar commented Jul 30, 2021

Uh oh!

galipremsagar commented Aug 3, 2021

Uh oh!

galipremsagar commented Aug 3, 2021

Uh oh!

galipremsagar commented Aug 3, 2021

Uh oh!

msadang left a comment

Choose a reason for hiding this comment

Uh oh!

galipremsagar commented Aug 13, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

galipremsagar commented Jul 30, 2021 •

edited

Loading

codecov Bot commented Jul 30, 2021 •

edited

Loading