Added resource pool wait time histogram metrics

[lcabancla]

Currently, the vttablet currently only exposes resource pool wait count and aggregated wait times. Adding timing histogram metrics to the resource pool would provide information on the distribution of the wait time latencies. This is specially useful to in determining tail latencies.

Example output from /debug/vars (look for TransactionPoolResourceWaitTime):

"Waits": {"TotalCount":186,"TotalTime":1559667140,"Histograms":{"Consolidations":{"500000":51,"1000000":96,"5000000":159,"10000000":163,"50000000":163,"100000000":163,"500000000":163,"1000000000":163,"5000000000":163,"10000000000":163,"inf":163,"Count":163,"Time":200226919},"TransactionPoolResourceWaitTime":{"500000":0,"1000000":0,"5000000":0,"10000000":1,"50000000":11,"100000000":20,"500000000":23,"1000000000":23,"5000000000":23,"10000000000":23,"inf":23,"Count":23,"Time":1359440221}}}

[mpawliszyn]

Doesn't the name get added automatically farther up the stack?

[lcabancla]

Yes. As far as I know it is only used to publish stats.

[sougou]

I would recommend passing in a logWait func(elapsed time.Dration) instead. This will allow the name handling to live with the caller, which will be more readable.

Also, is this a production use case? I'm asking because there are other changes that are saying that we are over-reporting, and asking for more consolidations.

[lcabancla]

@sougou
That sounds good. Will look into it.
Yes, this is a production use case. Would you mind citing the other changes so I could get some context?

[sougou]

Yes, this is a production use case. Would you mind citing the other changes so I could get some context?

#5649 is one. I also know that vtgate over-reports on calls to vttablets. It's an issue if you have too many shards. I'm about to publish a proposal to help solve this problem. RFC will be out soon.

[sougou]

@lcabancla see #5809. I think it alleviates the problem enough that this can be pushed through. Can you address the review comment?

[lcabancla]

@sougou Yes, will work on the comment and update this. Thanks.

[lcabancla]

@sougou Updated the PR per your comment. Please take a look. Thanks!

[sougou]

Looks like this feel through the cracks. Merging.