xds: Poor RPC error messages when unable to get listener resource

[ejona86]

This is related to #4939 (comment).

I created a trash bootstrap (td-grpc-bootstrap --gcp-project-number 5) and ran the xds client on my local machine without any credentials.

examples/features/xds/client$ time ./client 
2021/11/30 16:19:13 could not greet: rpc error: code = DeadlineExceeded desc = context deadline exceeded

real	0m10.157s
user	0m0.087s
sys	0m0.193s

I'd have hoped the RPC wouldn't need to timeout, because xds client is unable to do a successful RPC. I thought that was supposed to produce an error quickly from xds client. It looks like Java does produce an error quickly ("Permission denied on resource project #5"). So that seems like a bug, but let's work around it.

Changing the RPC's timeout in main.go to 30s, let's try again:

examples/features/xds/client$ time ./client 
2021/11/30 16:20:31 could not greet: rpc error: code = Unavailable desc = name resolver error: xds: ListenerResource target localhost:50051 not found, watcher timeout

real	0m15.147s
user	0m0.095s
sys	0m0.161s

That's... not that helpful. You have to be a grpc developer to understand that error message properly (others might think they understand it, but they'd be wrong). I could accept that error if the server was slow or something. But that's not the case.

examples/features/xds/client$ GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info ./client
...
2021/11/30 16:21:20 WARNING: xds: ADS stream creation failed: rpc error: code = Unauthenticated desc = transport: context canceled
2021/11/30 16:21:22 WARNING: xds: ADS stream creation failed: rpc error: code = Unauthenticated desc = transport: context canceled
2021/11/30 16:21:24 WARNING: xds: ADS stream creation failed: rpc error: code = Unauthenticated desc = transport: context canceled
2021/11/30 16:21:29 WARNING: xds: ADS stream creation failed: rpc error: code = Unauthenticated desc = transport: context canceled
2021/11/30 16:21:35 WARNING: xds: ADS stream creation failed: rpc error: code = Unauthenticated desc = transport: context canceled
2021/11/30 16:21:35 INFO: [xds] [xds-resolver 0xc00040d560] received LDS update: {
  "RouteConfigName": "",
  "InlineRouteConfig": null,
  "MaxStreamDuration": 0,
  "HTTPFilters": null,
  "InboundListenerCfg": null,
  "Raw": null
}, err: xds: ListenerResource target localhost:50051 not found, watcher timeout
2021/11/30 16:21:35 WARNING: [xds] [xds-resolver 0xc00040d560] Watch error on resource localhost:50051 from xds-client 0x173e760, xds: ListenerResource target localhost:50051 not found, watcher timeout
2021/11/30 16:21:35 WARNING: [core] ccResolverWrapper: reporting error to cc: xds: ListenerResource target localhost:50051 not found, watcher timeout
2021/11/30 16:21:35 INFO: [core] ClientConn switching balancer to "pick_first"
2021/11/30 16:21:35 INFO: [core] Channel switches to new LB policy "pick_first"
2021/11/30 16:21:35 INFO: [core] Channel Connectivity change to TRANSIENT_FAILURE
2021/11/30 16:21:35 INFO: [core] pickfirstBalancer: ResolverError called with error xds: ListenerResource target localhost:50051 not found, watcher timeout
2021/11/30 16:21:35 could not greet: rpc error: code = Unavailable desc = name resolver error: xds: ListenerResource target localhost:50051 not found, watcher timeout

Given that this situation is super-likely for new users, it is very important that we promptly provide helpful error messages in situations like this one.

[dfawley]

@menghanl was #5054 enough to resolve this issue or is there more to be done?

[ejona86]

It didn't change the state of this issue. I had re-tested after that PR and it was realized to be an accident to close it.

[menghanl]

This is now failing with

2022/05/20 10:58:08 could not greet: rpc error: code = Unavailable desc = name resolver error: xds: error received from xDS stream: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: context canceled, xDS client nodeID: id:"projects/5/networks/default/nodes/5de01256-7137-4488-8f80-5cdd836ce049" cluster:"cluster" metadata:{fields:{key:"INSTANCE_IP" value:{string_value:"127.0.1.1"}} fields:{key:"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER" value:{string_value:"5"}} fields:{key:"TRAFFICDIRECTOR_NETWORK_NAME" value:{string_value:"default"}}} locality:{} user_agent_name:"gRPC Go" user_agent_version:"1.47.0-dev" client_features:"envoy.lb.does_not_support_overprovisioning" client_features:"xds.config.resource-in-sotw"

In xds: error received from xDS stream: rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: context canceled, it's not clear where context canceled comes from (probably from OAuth2)

[menghanl]

I think it's OK to close this issue now.

[ejona86]

@menghanl, "per-RPC creds failed due to error: context canceled" makes it seem like a timeout. How is it clear there is a permission issue? We shouldn't be losing the ADS failure status. It seems even low-level debugging doesn't include it.

It looks like in Java we deliver ADS stream closures as errors to all the watchers. We might want to discuss if that is appropriate cross-language.

[ejona86]

@menghanl, in my test I had local credentials available (for the wrong project). In your test it is probably failing to talk to the metadata server or something.