Bring back "client-go: add retry logic for Watch and Stream" and fix test flake

[tkashem]

What type of PR is this?

/kind failing-test
/kind flake

What this PR does / why we need it:

We had to revert #102107 due to flakes - the flaking tests can be found here https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=unit&test=k8s.io%2Fclient-go%2Frest

• This PR reverts the revert Revert "client-go: add retry logic for Watch and Stream" #102581 to bring back client-go: add retry logic for Watch and Stream #102107 - client-go: add retry logic for Watch and Stream
• It fixes the flake in test TestRequestWatchWithRetry

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

• The revert commit is clean and pristine
• The fix for test flake is in a separate commit so it is easier to review

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[tkashem]

/assign @wojtek-t @deads2k

/cc @liggitt

[tkashem]

I reproduced the issue using stress binary:

$ go test -v -race -count=1 k8s.io/client-go/rest -c

Before the fix:

$ stress ./rest.test -test.run "TestRequestWatchWithRetry"

/tmp/go-stress-20210604T134627-494103019
--- FAIL: TestRequestWatchWithRetry (0.00s)
    --- FAIL: TestRequestWatchWithRetry/server_returns_success_on_the_final_retry (0.00s)
        request_test.go:2932: Expected response body Close to be invoked 1 times, but got: 2
FAIL


ERROR: exit status 1

After the fix:

$ stress ./rest.test -test.run "TestRequestWatchWithRetry"

.
.
.
53m5s: 299715 runs so far, 0 failures
53m10s: 300277 runs so far, 0 failures
53m15s: 300871 runs so far, 0 failures

[wojtek-t]

/lgtm
/approve

/hold
To give @liggitt a chance to look if he wants.

[wojtek-t]

/triage accepted

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tkashem, wojtek-t

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• staging/src/k8s.io/client-go/OWNERS [wojtek-t]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wojtek-t]

It seems @liggitt is OOO for the next week. So let me unhold.

@tkashem - can you please monitor this after merge that it won't start flaking again?

/hold cancel

[tkashem]

cc @MikeSpreitzer

[tkashem]

/test pull-kubernetes-e2e-gce-ubuntu-containerd