Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: allow cell_widths and nproc > 1 for load_uniform_grid #4732

Merged
merged 2 commits into from
Dec 5, 2023

Conversation

chrishavlin
Copy link
Contributor

@chrishavlin chrishavlin commented Nov 3, 2023

PR Summary

Working on getting load_uniform_grid to work with nproc > 1 for stretched grids (when cell_widths is allowed). Most the changes here are to the grid decomposition.

Closes #4330

PR Checklist

  • New features are documented, with docstrings and narrative docs
  • Adds tests for new features.

Sample from the version just pushed:

import numpy as np
import yt

N = 64
data = {"density": np.random.random((N, N, N))}

cell_widths = []
for _ in range(3):
    cw = 0.1 + np.random.random(N)
    cw /= cw.sum()
    cell_widths.append(cw)

ds = yt.load_uniform_grid(
        data,
        data["density"].shape,
        bbox=np.array([[0.0, 1.0], [0.0, 1.0], [0.0, 1.0]]),
        cell_widths=cell_widths,
        nprocs=32,
    )

slc = yt.SlicePlot(ds, 'x', 'density')
slc.set_log('density', False)
slc.annotate_grids(edgecolors='w', linewidth=4)

image

@chrishavlin chrishavlin added enhancement Making something better index: grid labels Nov 3, 2023
@chrishavlin chrishavlin marked this pull request as draft November 3, 2023 22:16
@chrishavlin
Copy link
Contributor Author

needs tests! but seems to work?

@chrishavlin
Copy link
Contributor Author

chrishavlin commented Nov 3, 2023

but seems to work?

also seems to have broken many unexpected things :) definitely WIP

@chrishavlin chrishavlin force-pushed the tiled_stretched_grids branch from 5bd5583 to 554fc83 Compare November 6, 2023 19:56
@chrishavlin chrishavlin force-pushed the tiled_stretched_grids branch from 554fc83 to df8efbf Compare November 6, 2023 21:39
@chrishavlin
Copy link
Contributor Author

@yt-fido test this please

@chrishavlin chrishavlin marked this pull request as ready for review November 7, 2023 17:08
@chrishavlin
Copy link
Contributor Author

@yt-fido test this please

Copy link
Member

@neutrinoceros neutrinoceros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As long as the test pass I'm okay with this, but I believe that the signature changes in what looks like public API functions should be discussed, or changed for backward compatibility.

left_edges.append(lle)
right_edges.append(lre)
shapes.append(rei - lei)
slices.append(np.s_[lei[0] : rei[0], lei[1] : rei[1], lei[2] : rei[2]])

return left_edges, right_edges, shapes, slices
return left_edges, right_edges, shapes, slices, cell_widths_by_grid
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like a backward incompatible change. Returning cell_widths_by_grid should be optional, and the default behaviour should remain stable, unless there's a consensus that yt.utilities.decompose isn't considered public API ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not public API, so this should be ok by me.

Copy link
Contributor Author

@chrishavlin chrishavlin Nov 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh! ya, I had assumed without too much thought that this was not public API. I think there is a small chance that there's an external frontend out there that utilizes this function. Happy to change this up to cover that scenario: the simplest change might be to have the return from this function vary and include a deprecation, e.g.,

if cell_widths is not None:
    return left_edges, right_edges, shapes, slices, cell_widths_by_grid
else:
    deprecation warning ("this function will have an additional return value in the future...")
    return left_edges, right_edges, shapes, slices

Could leave out the deprecation but that might make future typing efforts annoying.

Or I could add a split_array_with_cell_widths function and just call that directly from the load_uniform_grid when needed. That'd probably result in a bit of code duplication, but it might be clearer to have a separate function.

Also happy to leave this change as-is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following John's remark I think the best course of action is to keep it simple and treat it as private. We can always inject additional backward-compatibility fixes later if anyone asks for it.

@neutrinoceros neutrinoceros dismissed their stale review November 13, 2023 17:24

no public API break

@chrishavlin chrishavlin merged commit 414b617 into yt-project:main Dec 5, 2023
1 check passed
@chrishavlin chrishavlin deleted the tiled_stretched_grids branch December 5, 2023 18:40
@neutrinoceros neutrinoceros added this to the 4.4.0 milestone Dec 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Making something better index: grid
Projects
None yet
Development

Successfully merging this pull request may close these issues.

load_uniform_grid with nprocs>1 and cell_widths
4 participants