server: v1.45 of grpc returns context cancelled on server shutdown

[DarrylWong]

When trying to upgrade GCS packages, upgrading from v1.43 to. v1.45 of grpc causes the test TestDirectoryConnect/drain_connection to fail.

This test fails now as draining the server no longer does anything. Looking into it, drain() does not do anything as there is no event listener ever added to the server. No event listener is added as the watchPods() in line 292 of directory.go shuts down before the second server, tds2, can start up.

The reason why the handler shuts down earlier than it should be is due to the following lines in watchPods():

if grpcutil.IsContextCanceled(err) {
break
}

where IsContextCanceled() is:

func IsContextCanceled(err error) bool {
if s, ok := status.FromError(errors.UnwrapAll(err)); ok {
return s.Code() == codes.Canceled && s.Message() == context.Canceled.Error()
}
return false
}

In v1.44 of grpc and before, when the context was canceled from grpc's side, it would return an unknown error. As of v1.45, it now returns a context.Canceled error. As a result, IsContextCanceled() now returns true even if the context was canceled by the servers side, which makes the two currently indistinguishable. The way the code currently works is that it only wants to break out if the context was canceled from our end, otherwise, try starting up the watchPods() handler again.

This functionality is now broken as stopping the server in line 707 of TestDirectoryConnect/drain_connection incorrectly stops the watchPods() handler as stopping the server returns a context.Canceled error.

[DarrylWong]

oops made this in wrong place