Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TsDB load from/export to pickle and dataframe #124

Merged
merged 11 commits into from
Aug 6, 2024

Conversation

mariustk
Copy link

@mariustk mariustk commented May 13, 2024

Summary

  • Added support for loading from/exporting to pickle file (using a pandas dataframe format)
    • Implemented to TsDB.load() (strictly, to TsDB._read()) and TsDB.export()
    • Implemented in GUI
  • New method TsDB.to_dataframe() generates a pandas dataframe with the common time array as the dataframe index.
  • TsDB.export(): new parameter force_common_time enables enforcing a common time array automatically (when possible). Default behavior is not changed (raising exception if not common time array across specified series).
  • Minor adjustment to TsDB.create_common_time(): if time array is already common, the time array from either of the time series objects is returned (avoid constructing new array, which may introduce minor inaccuracies/changes to the time array).

Closes #123

@eneelo eneelo requested review from tovop and eneelo May 15, 2024 17:02
@eneelo eneelo added enhancement New feature or request gui Issue is related to GUI labels May 15, 2024
@eneelo
Copy link
Collaborator

eneelo commented May 15, 2024

Tasks/checks before approval (@eneelo, @tovop)

  • Test new export/import in GUI
  • Consider if the test file (data/df_dump.pkl) should be generated during testing initiation (previous experience suggests that pandas dataframes on pickle files may fail to be loaded in a different pandas version than that used to create it).
  • Review code

@eneelo
Copy link
Collaborator

eneelo commented May 31, 2024

@mariustk Apologies for delayed response. This looks very promising, but still need to find time to look through the details before approving the pull request. Just wanted to let you know that it isn't forgotten.

Copy link

sonarcloud bot commented Aug 6, 2024

@eneelo
Copy link
Collaborator

eneelo commented Aug 6, 2024

I have reviewed the code, good job @mariustk 👍
I have also pushed some additional code, including support for export of pickle and to generate a pandas dataframe from a TsDB instance. Will now update the PR header/documentation, then make a new release (hopefully shortly).

@eneelo eneelo changed the title New pickle format TsDB export to pickle and dataframe Aug 6, 2024
@eneelo eneelo changed the title TsDB export to pickle and dataframe TsDB load from/export to pickle and dataframe Aug 6, 2024
@eneelo eneelo merged commit 610b0b7 into dnvgl:master Aug 6, 2024
6 checks passed
@eneelo eneelo mentioned this pull request Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request gui Issue is related to GUI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pandas pickle dump format
2 participants