Use from_map for offset / limit computation
#514
Conversation
| def select_from_to(df, partition_index, partition_borders): | ||
| partition_borders = partition_borders.cumsum().to_dict() | ||
| def select_from_to(index, partition, partition_borders): | ||
| partition_borders = partition_borders.compute().cumsum().to_dict() |
There was a problem hiding this comment.
One difference between from_map and from_delayed is that Dask objects would already be computed by the time the function is called - using from_map, we would need to manually compute Dask objects either before calling from_map or within the function being called by from_map (which I opted for here).
Is this an intended difference for from_map?
|
Given that the function is performed on all partitions of an existing dataframe instead of creating a new one, this might be a place where |
|
@charlesbluca and I discussed this PR briefly offline, but just to make a note here: The ideal appraoch is probably to use |
|
Closing in favor of #517 |
3 participants
Following up on dask/dask#8911, this PR switches over to
from_mapfor the computation of limit/offsets instead offrom_delayed.Additionally, I opted to consolidate the logic of
map_on_partition_indexdirectly into limit.py, as it doesn't seem to be used anywhere else within the codebase - if this stops being the case later on, we can explore breaking the function off into a separate utility again.cc @rjzamora