Time slider widget broken for pandas > 1.0.5

[marfel]

ALL software version info

hvplot                    0.6.0              pyh9f0ad1d_0    conda-forge
holoviews                 1.13.3             pyh9f0ad1d_0    conda-forge
pandas                    1.1.0            py37h3340039_0    conda-forge
xarray                    0.16.0                     py_0    conda-forge

Description of expected behavior and the observed behavior

When creating a 2d plot plus slider widget from a 3d dataset, the widget apparently indexes the underlying xarray time dimension with a long integer (ie. microseconds), which worked fine with pandas up to 1.0.5, but now raises a KeyError.

Complete, minimal, self-contained example code that reproduces the issue

import xarray as xr
import numpy as np
import hvplot.xarray
a = xr.DataArray(np.arange(27).reshape(3,3,3), 
                 dims=["x", "y", "time"], 
                 coords=dict(x=[1,2,3], y=[1,2,3], time=pd.date_range("2020-01-01", freq="1d", periods=3)))
a.hvplot.image(x="x", y="y", groupby="time")  # same with widget_type='scrubber'

Stack traceback and/or browser JavaScript console output

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/IPython/core/formatters.py in __call__(self, obj, include, exclude)
    968 
    969             if method is not None:
--> 970                 return method(include=include, exclude=exclude)
    971             return None
    972         else:

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/dimension.py in _repr_mimebundle_(self, include, exclude)
   1310         combined and returned.
   1311         """
-> 1312         return Store.render(self)
   1313 
   1314 

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/options.py in render(cls, obj)
   1393         data, metadata = {}, {}
   1394         for hook in hooks:
-> 1395             ret = hook(obj)
   1396             if ret is None:
   1397                 continue

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in pprint_display(obj)
    280     if not ip.display_formatter.formatters['text/plain'].pprint:
    281         return None
--> 282     return display(obj, raw_output=True)
    283 
    284 

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in display(obj, raw_output, **kwargs)
    256     elif isinstance(obj, (HoloMap, DynamicMap)):
    257         with option_state(obj):
--> 258             output = map_display(obj)
    259     elif isinstance(obj, Plot):
    260         output = render(obj)

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in wrapped(element)
    144         try:
    145             max_frames = OutputSettings.options['max_frames']
--> 146             mimebundle = fn(element, max_frames=max_frames)
    147             if mimebundle is None:
    148                 return {}, {}

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in map_display(vmap, max_frames)
    204         return None
    205 
--> 206     return render(vmap)
    207 
    208 

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/ipython/display_hooks.py in render(obj, **kwargs)
     66         renderer = renderer.instance(fig='png')
     67 
---> 68     return renderer.components(obj, **kwargs)
     69 
     70 

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/plotting/renderer.py in components(self, obj, fmt, comm, **kwargs)
    386                 doc = Document()
    387                 with config.set(embed=embed):
--> 388                     model = plot.layout._render_model(doc, comm)
    389                 if embed:
    390                     return render_model(model, comm)

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/panel/viewable.py in _render_model(self, doc, comm)
    416         if comm is None:
    417             comm = state._comm_manager.get_server_comm()
--> 418         model = self.get_root(doc, comm)
    419 
    420         if config.embed:

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/panel/viewable.py in get_root(self, doc, comm)
    645         """
    646         doc = doc or _curdoc()
--> 647         root = self._get_model(doc, comm=comm)
    648         self._preprocess(root)
    649         ref = root.ref['id']

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/panel/layout.py in _get_model(self, doc, root, parent, comm)
    118         if root is None:
    119             root = model
--> 120         objects = self._get_objects(model, [], doc, root, comm)
    121         props = dict(self._init_properties(), objects=objects)
    122         model.update(**self._process_param_change(props))

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/panel/layout.py in _get_objects(self, model, old_objects, doc, root, comm)
    108             else:
    109                 try:
--> 110                     child = pane._get_model(doc, root, model, comm)
    111                 except RerenderError:
    112                     return self._get_objects(model, current_objects[:i], doc, root, comm)

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/panel/pane/holoviews.py in _get_model(self, doc, root, parent, comm)
    225             plot = self.object
    226         else:
--> 227             plot = self._render(doc, comm, root)
    228 
    229         plot.pane = self

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/panel/pane/holoviews.py in _render(self, doc, comm, root)
    284             kwargs = {}
    285 
--> 286         return renderer.get_plot(self.object, **kwargs)
    287 
    288     def _cleanup(self, root):

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/plotting/bokeh/renderer.py in get_plot(self_or_cls, obj, doc, renderer, **kwargs)
     71         combining the bokeh model with another plot.
     72         """
---> 73         plot = super(BokehRenderer, self_or_cls).get_plot(obj, doc, renderer, **kwargs)
     74         if plot.document is None:
     75             plot.document = Document() if self_or_cls.notebook_context else curdoc()

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/plotting/renderer.py in get_plot(self_or_cls, obj, doc, renderer, comm, **kwargs)
    214 
    215         # Initialize DynamicMaps with first data item
--> 216         initialize_dynamic(obj)
    217 
    218         if not renderer:

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/plotting/util.py in initialize_dynamic(obj)
    249             continue
    250         if not len(dmap):
--> 251             dmap[dmap._initial_key()]
    252 
    253 

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/spaces.py in __getitem__(self, key)
   1285         # Not a cross product and nothing cached so compute element.
   1286         if cache is not None: return cache
-> 1287         val = self._execute_callback(*tuple_key)
   1288         if data_slice:
   1289             val = self._dataslice(val, data_slice)

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/spaces.py in _execute_callback(self, *args)
   1059 
   1060         with dynamicmap_memoization(self.callback, self.streams):
-> 1061             retval = self.callback(*args, **kwargs)
   1062         return self._style(retval)
   1063 

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/spaces.py in __call__(self, *args, **kwargs)
    693 
    694         try:
--> 695             ret = self.callable(*args, **kwargs)
    696         except KeyError:
    697             # KeyError is caught separately because it is used to signal

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/util/__init__.py in dynamic_operation(*key, **kwargs)
   1007 
   1008         def dynamic_operation(*key, **kwargs):
-> 1009             key, obj = resolve(key, kwargs)
   1010             return apply(obj, *key, **kwargs)
   1011 

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/util/__init__.py in resolve(key, kwargs)
    996             elif isinstance(map_obj, DynamicMap) and map_obj._posarg_keys and not key:
    997                 key = tuple(kwargs[k] for k in map_obj._posarg_keys)
--> 998             return key, map_obj[key]
    999 
   1000         def apply(element, *key, **kwargs):

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/spaces.py in __getitem__(self, key)
   1285         # Not a cross product and nothing cached so compute element.
   1286         if cache is not None: return cache
-> 1287         val = self._execute_callback(*tuple_key)
   1288         if data_slice:
   1289             val = self._dataslice(val, data_slice)

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/spaces.py in _execute_callback(self, *args)
   1059 
   1060         with dynamicmap_memoization(self.callback, self.streams):
-> 1061             retval = self.callback(*args, **kwargs)
   1062         return self._style(retval)
   1063 

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/spaces.py in __call__(self, *args, **kwargs)
    693 
    694         try:
--> 695             ret = self.callable(*args, **kwargs)
    696         except KeyError:
    697             # KeyError is caught separately because it is used to signal

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/data/__init__.py in load_subset(*args)
    956             def load_subset(*args):
    957                 constraint = dict(zip(dim_names, args))
--> 958                 group = self.select(**constraint)
    959                 if np.isscalar(group):
    960                     return group_type(([group],), group=self.group,

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/data/__init__.py in pipelined_fn(*args, **kwargs)
    214 
    215             try:
--> 216                 result = method_fn(*args, **kwargs)
    217 
    218                 op = method_op.instance(

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/data/__init__.py in select(self, selection_expr, selection_specs, **selection)
    601         # Handle selection kwargs
    602         if selection:
--> 603             data = dataset.interface.select(dataset, **selection)
    604         else:
    605             data = dataset.data

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/holoviews/core/data/xarray.py in select(cls, dataset, selection_mask, **selection)
    544             else:
    545                 validated[dim] = v
--> 546         data = dataset.data.sel(**validated)
    547 
    548         # Restore constant dimensions

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/xarray/core/dataset.py in sel(self, indexers, method, tolerance, drop, **indexers_kwargs)
   2100         indexers = either_dict_or_kwargs(indexers, indexers_kwargs, "sel")
   2101         pos_indexers, new_indexes = remap_label_indexers(
-> 2102             self, indexers=indexers, method=method, tolerance=tolerance
   2103         )
   2104         result = self.isel(indexers=pos_indexers, drop=drop)

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/xarray/core/coordinates.py in remap_label_indexers(obj, indexers, method, tolerance, **indexers_kwargs)
    395 
    396     pos_indexers, new_indexes = indexing.remap_label_indexers(
--> 397         obj, v_indexers, method=method, tolerance=tolerance
    398     )
    399     # attach indexer's coordinate to pos_indexers

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/xarray/core/indexing.py in remap_label_indexers(data_obj, indexers, method, tolerance)
    268             coords_dtype = data_obj.coords[dim].dtype
    269             label = maybe_cast_to_coords_dtype(label, coords_dtype)
--> 270             idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance)
    271             pos_indexers[dim] = idxr
    272             if new_idx is not None:

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/xarray/core/indexing.py in convert_label_indexer(index, label, index_name, method, tolerance)
    188             else:
    189                 indexer = index.get_loc(
--> 190                     label.item(), method=method, tolerance=tolerance
    191                 )
    192         elif label.dtype.kind == "b":

/soft/conda-env/envs/tutorial/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py in get_loc(self, key, method, tolerance)
    620         else:
    621             # unrecognized type
--> 622             raise KeyError(key)
    623 
    624         try:

KeyError: 1577836800000000000

[robertjwilson]

I have run into a similar problem with my package nctoolkit, which relies on hvplot. However, this appears to be some subtle issue with the conda version of Python.

The code sample of @marfel fails for me (as do any of my plotting methods that have a time slider) on conda. This is regardless of whether hvplot is installed using conda or pip. However, it is totally fine if I use the version of Python installed on my machine and jupyter notebook installed through pip. So this is potentially a conda bug, not a hvplot bug.

[marfel]

Interesting... for completeness, my Python version is

python                    3.7.7           hcf32534_0_cpython    defaults

[philippjfr]

Appears to be an issue with xarray, I would expect this to work, but it raises the same error:

import numpy as np
import pandas
import xarray as xr

a = xr.DataArray(np.arange(27).reshape(3,3,3), 
                 dims=["x", "y", "time"], 
                 coords=dict(x=[1,2,3], y=[1,2,3], time=pd.date_range("2020-01-01", freq="1d", periods=3)))

a.sel(time=a.time.values[0])

Filed an issue with xarray here: pydata/xarray#4377

[philippjfr]

Appears there's already a PR with a fix: pydata/xarray#4292

Hopefully that'll be merged soon.

[robertjwilson]

Thanks @philippjfr. Very useful. Switching to pandas 1.0.5 seems to solve the problems my package. I'll just have to update the update its website to show how to solve the problem.

[marfel]

The release notes for pandas 1.1.0 contain a lot of datetime-related stuff. I suspect somewhere along the way the indexing behaviour changed subtly.

[robertjwilson]

I'm surprised that's not causing xarray's tests to fail on continuous integration. Though that maybe implies it's not affecting that much. Probably very subtle.

[philippjfr]

May well be causing test issues, I think xarray 0.16 was released before pandas 1.1 so it wouldn't have been caught at the time.

[robertjwilson]

True. I think I should worry more about the fact that I currently don't have a way to test plotting in my own package, which would have told me about this a week ago....