KAFKA-12467: Add controller-side snapshot generation

[cmccabe]

Implement writing snapshots in the controller, and reading them back.

[cmccabe]

I split off parts of this into:
#10455
#10456

[mumrah]

Thanks for the patch @cmccabe, this looks nice 👍

Returning an iterator of batches works well when the control managers only have a single iterable thing that needs to be translated to records. Do you think we might have cases (eventually) where the control classes have more complex state? It might be more flexible to take a consumer rather than return an iterable:

// In the control manager
void toRecords(Consumer<List<ApiMessageAndVersion>> batchConsumer) { 
  ...
}

A few other comments/questions inline.

[mumrah]

Since we're getting the entry set of a particular epoch, does this mean we'll be getting consistent snapshots?

[cmccabe]

Yes, snapshots are consistent.

[mumrah]

Should we use result here instead of numWriteTries? If not, maybe we can change generateBatch() to return a boolean instead

[cmccabe]

It's fine to use result here, I guess.

[cmccabe]

Returning an iterator of batches works well when the control managers only have a single iterable thing that needs to be translated to records. Do you think we might have cases (eventually) where the control classes have more complex state? It might be more flexible to take a consumer rather than return an iterable:

Hmm... I don't think that would be a problem. You could have a single iterator that iterates over multiple data structures X and Y. You would just need to store the state of whether you were done with X before starting Y.

In general one of the goals of controller snapshots is to avoid "stopping the world" while the snapshot is generated. So a synchronous API would not be useful here. The iterator API allows us to take only a few batches of records at a time.

[cmccabe]

Fix spotbugs

[mumrah]

Good point about the iterator vs consumer. All this LGTM

The only thing I might consider is to move SnapshotGeneratorManager into its own class, but that's not critical to for this PR