Expose apply_ufunc as public API and add documentation #1619
Conversation
TODO: turn the description from GH1517 into an auto-gallery example, and link to it from `computation.rst` and `dask.rst`.
doc/whats-new.rst
Outdated
| written to work on NumPy arrays to support labels on xarray objects. | ||
| ``apply_ufunc`` also support automatic parallelization for many functions | ||
| with dask. See :ref:`computation.wrapping-custom` for details. | ||
|
|
There was a problem hiding this comment.
Don't forget:
(:issue:`XXXX`). By `Stephan Hoyer <https://github.com/shoyer>`_.
doc/computation.rst
Outdated
| many functions when using dask by setting ``dask='parallelized'``. This is | ||
| illustrated in a separate example. | ||
|
|
||
| .. TODO: add link! |
There was a problem hiding this comment.
I'm thinking the recipes would be a good place for this: http://xarray.pydata.org/en/stable/auto_gallery/index.html
There was a problem hiding this comment.
Yes, if I can come up with a figure!
doc/computation.rst
Outdated
| Unfortunately, a limitation of the current version of numpy means that we | ||
| cannot override ufuncs for datasets, because datasets cannot be written as | ||
| a single array [1]_. :py:meth:`~xarray.Dataset.apply` works around this | ||
| Unfortunately, we current do not support NUmPy ufuncs for datasets [1]_. |
doc/computation.rst
Outdated
|
|
||
| squared_error = lambda x, y: (x - y) ** 2 | ||
| arr1 = xr.DataArray([0, 1, 2, 3], dims='x') | ||
| xr.apply_func(squared_error, arr1, 1) |
doc/dask.rst
Outdated
| Another option is to use xarray's :py:func:`~xarray.apply_ufunc`, which can | ||
| automatically parallelize functions written for processing NumPy arrays when | ||
| applied to dask arrays. It works similarly to :py:func:`dask.array.map_blocks` | ||
| and :py:func:`dask.array.atop`, but without requiring an immediate layer of |
doc/computation.rst
Outdated
|
|
||
| It doesn't always make sense to do computation directly with xarray objects: | ||
|
|
||
| - When working with small arrays (less than ~1e7 elements), applying an |
There was a problem hiding this comment.
Is the point on speed is a distraction? If an array is that small, the absolute difference in speed is still very small, so it really only makes a difference if you're doing those operations in a loop.
There was a problem hiding this comment.
Agreed. I had "in the inner loop" in mind when I wrote this, but I see now that that never made it into the text. Let me know if this latest update seems more reasonable to you, or if I'm still pushing too hard on performance considerations.
| @@ -724,7 +727,7 @@ def apply_ufunc(func, *args, **kwargs): | |||
| ``apply_ufunc`` to write functions to (very nearly) replicate existing | |||
| xarray functionality: | |||
|
|
|||
| Calculate the vector magnitude of two arguments: | |||
| Calculate the vector magnitude of two arguments:: | |||
There was a problem hiding this comment.
When you write ::, Sphinx formats the block below as code in the generated API docs.
|
Any additional comments? If not, I'll merge this in about 24 hours. |
apply_ufunc()does not meet all our needs for wrapping unlabeled array routines with xarray (see #1618 for a proposal forapply_raw()), but it should be useful for many advanced users and isn't doing much in its current state as non-public API. So I'd like to make it public for the xarray 0.10 release.git diff upstream/master | flake8 --diffwhats-new.rstfor all changes andapi.rstfor new APIcomputation.rstanddask.rst.@MaximilianR @rabernat @jhamman Review from any of you would be appreciated here!