Slowness in schema reload makes all requests on the same tidb slow

[AndreMouche]

Enhancement

When TiKV restarts, the latency increases due to the shared lock between expiredTimeStamp4PC and load-schema.
The related issue #45385 that if Load infoschema operation in domain takes a long time, then it may block the compile process, then makes the request on this tidb slow.

more details:
the reload process locks a mutex https://github.com/pingcap/tidb/blob/v6.5.0/domain/domain.go#L455.

however, the compile process calls Optimize https://github.com/pingcap/tidb/blob/v6.5.0/executor/compiler.go#L116, then OptimizeExecStmt https://github.com/pingcap/tidb/blob/v6.5.0/planner/optimize.go#L129, then calls GetPlanFromSessionPlanCache https://github.com/pingcap/tidb/blob/v6.5.0/planner/optimize.go#L439, then planCachePreprocess https://github.com/pingcap/tidb/blob/v6.5.0/planner/core/plan_cache.go#L118, then ExpiredTimeStamp4PC https://github.com/pingcap/tidb/blob/v6.5.0/planner/core/plan_cache.go#L102, which eventually reaches https://github.com/pingcap/tidb/blob/v6.5.0/domain/domain.go#L395

thus a slow batch-get request about the shcema version would block all requests on the same tidb, which doesn't make sense, particularly, the lock https://github.com/pingcap/tidb/blob/v6.5.0/domain/domain.go#L394-L407 doesn't need to be the same as the one used in Reload

using v6.5.0 as an example, the the issue applies to master as well.

[AndreMouche]

@zimulala @qw4990 Please help take a look, thanks

[aidendou]

Hi @AndreMouche, I'll take this one.

[jebter]

/sig sql-infra