Move some security APIs to using promises in place of callbacks

[original-brownbear]

We ahve some incredibly deep callstacks in security that seem to visibly raise context switch costs, make profiling more complicated and generally make the code rather hard to follow.
Also, deeply nested listener chains tend to make logic consume more heap because we almost inevitably continue holding on to things we no longer need as we go down the stack.
Since the methods adjusted here mostly return a result synchronously on the hot path, we can both save overhead (we can do a couple follow-ups to that effect) and make things a little easier to follow by using promises as returns in place of consuming callbacks.

A good example of the effects of this change would be this piece of search request REST handling that flattens out enormously.

Before:

After:

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[DaveCTurner]

Should we do something similar to flatten out RequestFilterChain#proceed rather than using recursion there too?

[DaveCTurner]

I'd rather add something like this to SubscribableListener over exposing rawResult():

    public boolean completeIfFailed(ActionListener<?> listener) {
        final Object currentState = state;
        if (currentState instanceof FailureResult result) {
            listener.onFailure(result.exception());
            return true;
        } else {
            return false;
        }
    }

[original-brownbear]

I went for just an isSuccess now like other solutions have. There isn't much of a reason to be tricky for the failure path and this way we have a fast solution for the hot success path. Let me know what you think :) Combining this approach with the get-results on listenable futures would allow neat simplifications + speedups in some places I believe (by avoiding allocating new listeners).

[original-brownbear]

Should we do something similar to flatten out RequestFilterChain#proceed rather than using recursion there too?

I think we should yea, gotta be a little careful maybe because of thread-context though. That kind of thing is made it impossible to go further with this kind of change in security for the time being when I tried because any thread-local context that only exists around the onResponse is lost if the listener hasn't been added before.

[DaveCTurner]

This bit LGTM, I'll leave the rest to the security team