[ML-DataFrame] Rewrite continuous logic to prevent terms count limit

[hendrikmuhs]

Rewrites how continuous data frame transforms calculates and handles buckets that require an update. Instead of storing the whole set in memory it pages through the updates using a 2nd cursor. This lowers not only memory consumption but prevents problems with limits at query time (max_terms_count). Apart from that the list of updates can be re-retrieved in a failure case (#43662)

Todo:

• disable/enable BWC, adapt versions
• cleanup, re-visit naming
• test correctness
• check behavior for histogram only pivots

[elasticmachine]

Pinging @elastic/ml-core

[droberts195]

Looks like good progress.

These are just a few minor comments from an initial read through.

[hendrikmuhs]

run elasticsearch-ci/1

[droberts195]

The constructor doesn't set runState, so it starts off null. This will cause an NPE if any switch (runState) is called while it's null.

Would it be appropriate to call runState = determineRunStateAtStart(); in the constructor? Or if that would cause problems then consider changing:

    switch (runState) {
        case x:
            return x();
        case y:
            return y();
        case z:
            return z();
        default:
            logger.warn("wrong");
            throw error;
    }

to:

    if (runState != null) {
        switch (runState) {
            case x:
                return x();
            case y:
                return y();
            case z:
                return z();
        }
    }
    logger.warn("wrong");
    throw error;

It would give us more information if runState still was null at the time of a switch.

[hendrikmuhs]

onStart gets called before any of the reads of runState happen, so an NPE should not be possible.

Because determineRunStateAtStart() depends on steps before, it would not return the same value if moved to the constructor.

However, I will assign a default to runState at the constructor, to be on the safe side.

Good spot!

[droberts195]

run elasticsearch-ci/packaging-sample

[droberts195]

LGTM