Skip to content

Commit

Permalink
kgo.Client: add explicit ShardedRequest; .Broker: add RetriableRequest
Browse files Browse the repository at this point in the history
The prior Request always merged responses, which was not perfect all of
the time. If one request to a broker failed of many requests, then we
would only have a partial response, and it would not be clear that the
response was partial.

The new ShardedRequest returns _all_ issued request/response/error pairs
and the broker that the requests went to, or an unknown broker if the
request could not be issued.

This is a rather large change that has a couple of other side effects:

- the admin controller is un-cached whenever a NOT_CONTROLLER error is
returned from an admin request. Previously, we would continue using the
cached controller until a metadata reload.

- this adds sharding for DescribeConfigs, AlterConfigs, and
IncrementalAlterConfigs

- for AlterReplicaLogDirs and DescribeLogDirs, requests are issued to
all replicas, rather than just the leader

- some requests in the Request flow are now retried, rather than failing
(such as if we cannot load metadata on a certain broker when looking up
where to send the request)

Since some aspect of retrying has been abstracted into a function, this
also now easily adds retrying an individual, broker dedicated request to
the Broker type.
  • Loading branch information
twmb committed Nov 9, 2020
1 parent 0eddb35 commit 9467a95
Show file tree
Hide file tree
Showing 3 changed files with 1,437 additions and 885 deletions.
4 changes: 4 additions & 0 deletions pkg/kgo/broker.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ type waitingResp struct {
err error
}

var unknownMetadata = BrokerMetadata{
NodeID: -1,
}

// BrokerMetadata is metadata for a broker.
//
// This struct mirrors kmsg.MetadataResponseBroker.
Expand Down
Loading

0 comments on commit 9467a95

Please sign in to comment.