when etcdserver down, Reject client CPU leakage.

[barnettZQG]

when etcd-server down and not restart quickly. client will CPU leakage. It's mainly the following code.
https://github.com/coreos/etcd/blob/ad4e8e94905a3feac3b18511774e242f9d0f860b/clientv3/watch.go#L836:27

func (w *watchGrpcStream) openWatchClient() (ws pb.Watch_WatchClient, err error) {
	for {
		select {
		case <-w.ctx.Done():
			if err == nil {
				return nil, w.ctx.Err()
			}
			return nil, err
		default:
		}
		if ws, err = w.remote.Watch(w.ctx, w.callOpts...); ws != nil && err == nil {
			break
		}
		if isHaltErr(w.ctx, err) {
			return nil, v3rpc.Error(err)
		}
	}
	return ws, nil
}

if error not nil, there are always errors on the server side. so, Why not delay retry?

func (w *watchGrpcStream) openWatchClient() (ws pb.Watch_WatchClient, err error) {
	for {
		select {
		case <-w.ctx.Done():
			if err == nil {
				return nil, w.ctx.Err()
			}
			return nil, err
		default:
		}
		if ws, err = w.remote.Watch(w.ctx, w.callOpts...); ws != nil && err == nil {
			break
		}
		if isHaltErr(w.ctx, err) {
			return nil, v3rpc.Error(err)
		}
		time.Sleep(time.Millisecond * 200)
	}
	return ws, nil
}

[gyuho]

Related #9578.

We are actively working on clientv3 balancer rewrite. Once ready, we will address this as well.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.