Update of numpy to version 1.21.0 causing conflict with pyarrow version

[AlexanderMeldonWhite]

Hi, I believe the recent upgrade of numpy to version 1.21.0 has caused the below error to occur when converting a dataframe to parquet format (df.to_parquet(f)).

pyarrow.lib.ArrowTypeError: ('Did not pass numpy.dtype object', 'Conversion failed for column xxxxxxxx with type int64')

This was fixed locally by upgrading pyarrow to version 3.0.

Would it be possible to upgrade the pyarrow version in the image?

Many thanks.

[yiyi-zz]

Have the same issue... It would be great if the image gets fixed

[mabunday]

I'm investigating this but don't have an ETA yet for the fix. The main difficulty is the dependency on MLIO - bumping the PyArrow version causes some integration failures.

To confirm, this only started occurring when the numpy version was bumped, right? If so, I may investigate bumping the numpy version instead as an easier path forward.

[AlexanderMeldonWhite]

Hi, yes, that's correct, the issue only occurred after the upgrade of numpy.