Enhance TiSpark start up task speed when there are a large amount of regions #2520
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: qidi1 [email protected]
What problem does this PR solve?
What is changed and how it works?
The origin logical of get regions info in the range is like this:
As you can see, a lot of rpc requests need to be made when a lot of region information is not in the cache.
In order to reduce rpc requests, when we request region information from pd, we not only request the region information containing the startKey, but also request other region information in this range.
We do not request all the regions of this range at once, because most of the regions we need may already be in the cache, and if we request all the regions it may put a large burden on the pd. If we request all the regions we need, we will need to request all the regions again, which will cause a huge consumption.
we limit the number of returned region information per request. The default number is 128.
We tested the new method against the old method.
When the region number than we need to request is 1w.
The old method took about seven seconds,The new method took about four seconds.
new
old
When the region number than we need to request is 10w.
The old method took about thirteen seconds,The new method took about thirty seconds.
new
old
Check List
Tests