Add a blocked K-Fold cross-validator

[leouieda]

The BlockKFold class splits the data into spatial blocks and does
K-fold cross-validation splits on the blocks. Unlike BlockShuffleSplit
it's difficult to balance the folds to have the same amount of data
(since blocks can have different numbers of points) so each fold will
have the same number of blocks but not necessarily data. Refactor code
from BlockShuffleSplit into a base class that can be shared with the
new class.

Reminders:

• Run make format and make check to make sure the code follows the style guide.
• Add tests for new features or tests that would have caught the bug that you're fixing.
• Add new public functions/methods/classes to doc/api/index.rst and the base __init__.py file for the package.
• Write detailed docstrings for all functions/classes/methods. It often helps to design better code if you write the docstrings first.
• If adding new functionality, add an example to the docstring, gallery, and/or tutorials.
• Add your full name, affiliation, and ORCID (optional) to the AUTHORS.md file (if you haven't already) in case you'd like to be listed as an author on the Zenodo archive of the next release.

[leouieda]

Here is a follow-up to #251 adding a blocked K-fold cross-validator. I couldn't think of a good way to do the balancing on this one. Any ideas?

[jessepisel]

That's tough to get the number of points balanced while keeping the block sizes consistent from fold to fold. Is there any way that it could use some kind of nearest neighbor search and adjust the size of blocks for each fold ensuring the number is the same. Or some kind of weighting scheme for blocks? That could be a rabbit hole itself...

[leouieda]

That could be a rabbit hole itself...

It certainly is.

I've been trying to think of a way to make sure the folds have close to the same number of data points without interfering with the block size. That would likely mean that the block order wouldn't be random, but would be such that folds are balanced. That could be a good extra option, so you could have a shuffle mode and a balanced mode.

[santisoler]

Nice feature for Verde!
I only have one question regarding the X array-like argument. Why not ask for coordinates and rearrange them inside the split method? (I've just seen this is also present in BlockShuffleSplit added in #251, although I haven't spot it).

[leouieda]

@santisoler that was how I had implemented them originally. The problem is they are not compatible with their sklearn counterparts if that is the case. And that meant that cross_val_score needed to know which type of class it was getting: a sklearn one or a Verde one. It would also mean that people doing machine learning on spatial data would be able to use our classes.

The memory copy when generating X is not ideal but I thought it was a good trade-off to making these classes generally applicable.

[leouieda]

I think I might have found a way to always balance the folds, even without shuffling.

1. Make a list of the block sizes
2. Split this list into N partitions of equal sum. The split preserves the order of the blocks.
3. Do the k-fold on these N partitions instead of the actual blocks. Basically do KFold(n_splits=N).split(range(N)).

The number of blocks per partition will vary depending on how much data is in each block. The folds will not be shuffled since the partition split preserves the order of blocks. To make this work shuffled, we can shuffle the blocks before doing the partition.

Splitting the list is the challenging part but I found this solution: https://stackoverflow.com/a/54024280

It could be implemented as a balanced argument that does the partitioning if True or doesn't if False. Probably best to default to True.

[jessepisel]

So the larger blocks are all the same size, but the sub chunks are varying sizes that contain equal numbers? Or do I have it backwards, vary the large blocks but keep the sub chunks the same size? I think the first option would work because you could hold the number of samples the same for each fold.

[leouieda]

This got a bit confusing in terms of terminology:

• Block: spatial block defined by block_split (a regular mesh). All blocks have the same size but different amounts of data. All blocks have at least 1 data point.
• Fold: partition of "data" in KFold CV. One fold is used for testing, the rest are used for training. Each data point can only be assigned to one fold (no repetition).

The current implementation does this:

1. Split the data into blocks.
2. Optionally shuffle the blocks.
3. Partition the blocks evenly into folds (each fold has the same number of blocks).
4. Assign data to folds based on which blocks are in each fold (if fold 1 contains blocks 3 and 4, the data in those blocks is assigned to fold 1).

What I'm proposing is:

1. Split the data into blocks.
2. Optionally shuffle the blocks.
3. Make a list with the number of data points in each block (block_n_data).
4. Split the blocks into folds in a way that each fold has the same number of data points. To do this, we can find the partition points by splitting the list block_n_data into parts of near equal sum. This will tell use which blocks belong in each fold.
5. Assign data to folds based on which blocks are in each fold.

[jessepisel]

Thanks for the clarification. What you are proposing makes sense, the equal number of points for each fold seems reasonable, and like you said it will help with understanding which blocks are in which fold.

[leouieda]

@jessepisel just coded that method it does work pretty well! See the gallery example comparing balanced and unbalaced results:

There were a few tricks involved since the method fails if the number of splits is too close to the number of blocks.

@santisoler and @jessepisel would either of you mind doing another round of reviews? Thank you for all the input!

[jessepisel]

@leouieda I like this visual. It turned out really nice, I will look through the changes this week and see what I can find

[leouieda]

Merging this in since all checks pass.

@jessepisel sorry to fast track this but I wanted to get it in for Transform next week. I think the API is settled so any changes that may be needed can be handled in a future release easily. Thanks for all your input!

[jessepisel]

No worries, I apologize for not being more prompt.

[leouieda]

I'm the last person you should apologize to for that 😉