[BUG] UHI slices need to treat flow specially for unordered axes

[agoose77]

Describe the bug
When taking slices (as described by the UHI spec) to perform a projection, the start value is ignored (although must be in-range) unless a stop value is also provided.

Steps to reproduce

• An example notebook
• 

[HDembinski]

This needs to be moved to boost-histogram.

There is no need for an example notebook. Example code:

import boost_histogram as bh
h = bh.Histogram(bh.axis.IntCategory([0,1, 2]))
h.view(True)[:] = 1
h[1::sum] # returns 4
h[2::sum] # returns 4
h[3::sum] # returns 4

[agoose77]

Upon further investigation, the issue here is when the stop index is calculated in Axis._process_loc. It seems that a value of None resolves to the value len(axis)+1. I am under the impression that the indices should be in the inclusive range[0, 2] , with -1 and 3 being the overflow bins.

In any case, this is where the len(axis)+1 is set:

boost-histogram/src/boost_histogram/_internal/axis.py

Line 175 in c3da299

stop, len(self) + (1 if self._ax.traits_overflow else 0)

[henryiii]

Leaving off the end gives you the overflow bin. There's no "easy" way to get the overflow bin otherwise, and when you do, say, 5::sum, you want the sum from 5 going to infinity. That's why we have flow bins, for projections. If you want to avoid the overflow bin, you use 5:len:sum.

The issue with with IntCategory, not with the computation of the indices. Since the boost-histogram code does not care about the axes type, I think this might be a bug in Boost.Histogram. Using integer ranges on Categorical axes is slightly suspect, as the order doesn't have a strong meaning. [2,0,1] is also a valid IntCategory. (This should still work, but that might be related to why it's not working).

This is the commend being passed to Boost.Histogram for either Integer or IntCategory axes; only the former one works properly:

h[1::sum]
DEBUG:boost_histogram._internal.hist:Reduce with [reduce_command(slice_and_rebin(iaxis=0, begin=1, end=4, merge=1, mode=True))]

[agoose77]

After reading your comment I can see that I forgot about the n+1 slice behaviour to include the overflow, an oversight on my part.

I can raise a new issue over at https://github.com/boostorg/histogram. I'll probably reference this issue there, as I might not find time to create a reproducer in C++ until later this week.

[henryiii]

Found the problem. Since there is no "under/over" distinction for categorical axes, the extra items "below" get added into the (over)flow bin, then get included. Boost-histogram (Python) needs special handling for categories; it probably disable all upper/lower handling when slicing with any endpoints for categorical axes.

[agoose77]

Right, so the 0th bin is the over/underflow bin? Thanks for clarifying that, and for creating the PR.

[henryiii]

For unordered axes, the flow bin should not be viewed as just one bin; both h[:3:sum] and h[3::sum] should avoid the "flow" bin; before the patch the latter one was including the flow bin because it happens to be called the "overflow" bin. You can still access it directly via locators in the overflow spot, like h[len], but if you sum over it, you should only get it if you ask for all bins, h[::sum].

[henryiii]

Probably worth mentioning in the UHI spec, too.