fix: Make GetShardLeaders only retries on retriable error

[weiliu1031]

issue: #37532
pr: #37684

[mergify]

@weiliu1031 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[weiliu1031]

/run-cpu-e2e

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.60%. Comparing base (d23da2d) to head (852794a).
Report is 3 commits behind head on 2.4.

Additional details and impacted files

@@            Coverage Diff             @@
##              2.4   #37687      +/-   ##
==========================================
+ Coverage   72.31%   80.60%   +8.28%     
==========================================
  Files        1069     1069              
  Lines      166879   166879              
==========================================
+ Hits       120681   134506   +13825     
+ Misses      41801    27954   -13847     
- Partials     4397     4419      +22     

Files with missing lines	Coverage Δ
internal/proxy/lb_policy.go	97.90% <100.00%> (ø)

... and 230 files with indirect coverage changes

[czs007]

/approve
/lgtm

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, weiliu1031

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• internal/proxy/OWNERS [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment