Skip to content

DOC: warning section on memory overflow when joining/merging dataframes on index with duplicate keys #14788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Dec 11, 2016
Merged

DOC: warning section on memory overflow when joining/merging dataframes on index with duplicate keys #14788

Changes from 2 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
93a63a5
add doc regarding #14736
xgdgsc Dec 2, 2016
795a369
remove blank line
xgdgsc Dec 2, 2016
06d4430
change index to keys , fix spaces and format
xgdgsc Dec 3, 2016
fb021fb
fix format
xgdgsc Dec 3, 2016
403b099
add example , change warning position
xgdgsc Dec 11, 2016
ee9fc60
fix format
xgdgsc Dec 11, 2016
7581ce0
change fig name, change sentence
xgdgsc Dec 11, 2016
48e5322
remove blank line
xgdgsc Dec 11, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions doc/source/merging.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -817,6 +817,10 @@ equivalent:
left.join(right, on=key_or_keys)
pd.merge(left, right, left_on=key_or_keys, right_index=True,
how='left', sort=False)

.. warning::

* Joining on index with duplicate keys when joining large dataframes would cause severe memory overflow, sometimes freezes the computer and user has to hard reboot, which can be dangerous for unsaved work. Please make sure no duplicate keys in index before joining.
Copy link
Member

@jorisvandenbossche jorisvandenbossche Dec 2, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not needed to start with * (as this will give a one-bullet list). You only need an indentation so the text belongs to the warning.
Further, you have some wonky spaces in the text.

On the content: I don't think this is specific to joining on the index? But just on joining on a key with duplicate values?


Obviously you can choose whichever form you find more convenient. For
many-to-one joins (where one of the DataFrame's is already indexed by the join
Expand Down