xarray >= 2025.8.0: Respect ulimit -n with dask.distributed

[SarahG-579462]

What is your issue?

It's hard to tell whether this is a bug in xarray or dask, but since the async update for 2025.8.0, I have been getting many OSError: [Errno 24] Too many open files errors when opening large zarr datasets. I don't have a minimal reproducible example readily available since I believe this only occurs for large files. My ulimit -n is set to 1024. Example traceback below:

Traceback

Traceback (most recent call last):                                                                                                                        
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/scheduler.py", line 4874, in update_graph                             
    expr = deserialize(expr_ser.header, expr_ser.frames)                                                                                                   
    ^^^^^^^                                                                                                                                               
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/protocol/serialize.py", line 452, in deserialize                      
    return loads(header, frames)                                                                                                                           
      ^^^^^^^^^^^^^^^^^                                                                                                                                   
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/protocol/serialize.py", line 111, in pickle_loads                     
    return pickle.loads(pik, buffers=buffers)                                                                                                              
      ^^^^^^^^^^^^^^^^^                                                                                                                                   
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/protocol/pickle.py", line 93, in loads                                
    return pickle.loads(x, buffers=buffers)                                                                                                                
    ^^^^^^^^^^^                                                                                                                                           
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/zarr/storage.py", line 1835, in __setstate__                                      
    self.__init__(path=path, compression=compression, allowZip64=allowZip64, mode=mode)                                                                   
    ^^^^^^^^^^^^^^^                                                                                                                                       
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/zarr/storage.py", line 1823, in __init__                                           
    self.zf = zipfile.ZipFile(path, mode=mode, compression=compression, allowZip64=allowZip64)                                                            
    ^^^^^^^^^^^^^^^                                                                                                                                          File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/zipfile/__init__.py", line 1367, in __init__                                                    
    self.fp = io.open(file, filemode)                                                                                                                     
    ^^^                                                                                                                                                   
OSError: [Errno 24] Too many open files: '/baril/scenario/derived/Portraits-Climatiques-v2/simulation/ESPO-G6-AHCCD_v1-patch/CMIP6/ScenarioMIP/Ouranos/MIRO
C6/ssp585/r1i1p1f1/IDM/indicators/QS-DEC/liquidprcptot/liquidprcptot_QS-DEC_ESPO-G6-AHCCD_v1-patch_CMIP6_ScenarioMIP_Ouranos_MIROC6_ssp585_r1i1p1f1_IDM_ind
icators_1950-2100.zarr.zip'                                                                                                                               
                                                                                                                                                          
The above exception was the direct cause of the following exception:                                                                                       
                                                                                                                                                          
Traceback (most recent call last):                                                                                                                         
  File "/home/sgammon/projects/portraits/preparation/2_statistics.py", line 151, in <module>                                                              
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/xarray/core/dataset.py", line 2109, in to_netcdf                                  
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/xarray/backends/api.py", line 2096, in to_netcdf                                  
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/xarray/backends/common.py", line 413, in sync                                     
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/xarray/namedarray/daskmanager.py", line 247, in store                             
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/dask/array/core.py", line 1227, in store                                           
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/dask/base.py", line 681, in compute                                        
  File "/exec/sgammon/.conda/envs/portclim/lib/python3.13/site-packages/distributed/client.py", line 2417, in _gather
    raise exception.with_traceback(traceback)
RuntimeError: Error during deserialization of the task graph. This frequently
occurs if the Scheduler and Client have different environments.
For more information, see
https://docs.dask.org/en/stable/deployment-considerations.html#consistent-software-environments

[welcome]

Thanks for opening your first issue here at xarray! Be sure to follow the issue template!
If you have an idea for a solution, we would really welcome a Pull Request with proposed changes.
See the Contributing Guide for more.
It may take us a while to respond here, but we really value your contribution. Contributors like you help make xarray better.
Thank you!

[TomNicholas]

Thanks for reporting!

but since the async update for 2025.8.0

This should not matter... Are you able to git bisect the issue easily?

[SarahG-579462]

You’re right, I just tried it with 2025.7.0 and encountered the same issue. I’ll try updating ulimit higher, thanks!

…

On Fri, Sep 12, 2025 at 13:54 Tom Nicholas ***@***.***> wrote: *TomNicholas* left a comment (pydata/xarray#10736) <#10736 (comment)> Thanks for reporting! but since the async update for 2025.8.0 This should not matter... Are you able to git bisect the issue easily? — Reply to this email directly, view it on GitHub <#10736 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV4AH6MU3LDXCTC55JYYEOL3SMCFDAVCNFSM6AAAAACGLP5UKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEOBWGMZDSOBYGA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[SarahG-579462]

Increasing my ulimit -n to 20000 solved this problem for me. It would be nice if dask/xarray was aware of ulimits though, not sure how feasible that is.

[TomNicholas]

There's a lot to unpack here.

• How big of a zarr store are you trying to open?
• Are you doing this on a local filesystem?
• Is there actually a regression - i.e. did the behavior actually change between versions?
• If so, was it a change in xarray, dask, or zarr?
• Separately, should xarray/zarr read or set ulimit (I think they perhaps you warn you in advance, but should not alter your system environmental variables)

[dcherian]

The knob you have is zarr.config.set({"async.concurrency": 1}) for example. The default is 10, but that limit is not applied library-wide at the moment. so you will have to tune it depending on number of processes that dask is starting up for example.

See https://icechunk.io/en/latest/performance/#concurrency