Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Emitting a Dataframe with dtype float16 from a Python-UDF leads to silent data corruption #796

Closed
tkilias opened this issue May 8, 2023 · 0 comments · Fixed by #795
Closed

Emitting a Dataframe with dtype float16 from a Python-UDF leads to silent data corruption #796

tkilias opened this issue May 8, 2023 · 0 comments · Fixed by #795
Assignees
@tkilias
Labels
bug Unwanted / harmful behavior

Comments

@tkilias
Copy link
Collaborator

tkilias commented May 8, 2023 •
edited
Loading

Background

  • float16 columns are currently incorrectly converted from float16 to double.
  • Our protocol only speaks double for that reason, we need to convert it.
  • The current conversion casts the numpy float16 representation into an uint8 and from there into a double.
  • However, the uint8 doesn't get interpreted as double on bit level, instead the uint8 value is used as the double value

https://github.com/exasol/script-languages/blob/4cb6de499dd9c724b5a21db2450a31d36e19c732/exaudfclient/base/python/python3/python_ext_dataframe.cc#L737

Solution

  • We can solve this using astype from numpy to do the conversion

Acceptance Criteria

  • Test for all data types
  • Fix
  • Documentation of the data corruption
@tkilias tkilias added the bug Unwanted / harmful behavior label May 8, 2023
@tkilias tkilias mentioned this issue May 26, 2023
Merged
8 tasks
@tkilias tkilias linked a pull request May 26, 2023 that will close this issue
#793: Added support Pandas 2 pyarrow dtype columns for emitting data from Python UDFs #795
Merged
8 tasks
@tkilias tkilias self-assigned this May 26, 2023
@tkilias tkilias mentioned this issue Jun 1, 2023
Merged
tkilias added a commit that referenced this issue Jun 2, 2023
@tkilias @tomuben
#793: Added support for Pandas 2 pyarrow dtype columns for emitting d…
7b16444 
…ata from Python UDFs (#795)

* #796: Fixed silent data corruption when emitting dataframes with float16 dtype columns from Python UDFs
* Replace asscalar with item in test dataframe.py, because asscalar was removed
* Added handleEmitPyFloat to also support float pyarrow dtype columns with NAN and object-dtype columns with float
* Refactored and split Pandas Tests
* Added tests for more dtypes to Pandas Tests
* Documented supported dtypes when emitting dataframes

Co-authored-by: Thomas Ubensee <[email protected]>
tkilias added a commit that referenced this issue Jun 2, 2023
@tkilias @tomuben
Merge develop to master (#817)
e2fe7c4 
* #792: Updated script-languages-container-tool to 0.17.0 and script-languages-container-ci to 1.1.0 (#799)
* #800: Updated Ubuntu packages (#801)
* #803: Updated Ubuntu packages (#804)
* #814: Updated packages (#815)
* #793: Added support for Pandas 2 pyarrow dtype columns for emitting data from Python UDFs (#795)
* #796: Fixed silent data corruption when emitting dataframes with float16 dtype columns from Python UDFs

---------

Co-authored-by: Thomas Ubensee <[email protected]>
tkilias added a commit that referenced this issue Jun 2, 2023
@tkilias
Merge pull request #818 from exasol/develop
e76a80c 
 - #789: Updated Ubuntu packages
 - #777: Updated redis-py to fix CVE-2023-28859
 - #800: Updated Ubuntu packages
 - #803: Updated Ubuntu packages
 - #815: Updated Ubuntu packages and poetry.lock
 - #793: Added support for Pandas 2 pyarrow dtype columns for emitting data from Python UDFs
 - #792: Fixed Github workflow publish-test-container by updating script-languages-container-tool to 0.17.0 and script-languages-container-ci to 1.1.0
 - #796: Fixed silent data corruption when emitting dataframes with float16 dtype columns from Python UDFs
@tkilias tkilias closed this as completed Jun 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tkilias tkilias

Labels
bug Unwanted / harmful behavior
Projects
None yet
Milestone
No milestone
1 participant
@tkilias