Add gRPC error interceptors to API client

[Joerger]

Moves the gRPC error interceptors to api/utils/grpc/interceptors and uses them in the API client.

In a follow up PR, after https://github.com/gravitational/teleport.e/pull/2007 is merged, the alias functions in lib/utils/grpc.go will be removed.

[codingllama]

Many thanks!

Let's make sure we backport this, so other backported code won't work under the wrong assumptions.

[rosstimothy]

I added similar behavior in #25694 and then reverted it in #26169. It inadvertently broke tsh request ls and caused error traces to be rooted at the interceptor instead of the client. So instead of something which showed exactly where the error came from like the following

ERROR REPORT:
Original Error: *trace.TraceErr EOF
Stack Trace:
	github.com/gravitational/teleport/api/client/client.go:875 github.com/gravitational/teleport/api/client.(*Client).GetAccessRequests
	github.com/gravitational/teleport/tool/tsh/access_request.go:55 main.onRequestList.func1
	github.com/gravitational/teleport/lib/client/api.go:1395 github.com/gravitational/teleport/lib/client.(*TeleportClient).WithRootClusterClient
	github.com/gravitational/teleport/tool/tsh/access_request.go:54 main.onRequestList
	github.com/gravitational/teleport/tool/tsh/tsh.go:1215 main.Run
	github.com/gravitational/teleport/tool/tsh/tsh.go:506 main.main
	runtime/proc.go:250 runtime.main
	runtime/asm_arm64.s:1172 runtime.goexit
User Message: EOF

we would get something like this instead

ERROR REPORT:
Original Error: *trace.TraceErr EOF
Stack Trace:
	github.com/gravitational/teleport/api/utils/grpc/interceptors:55 github.com/gravitational/teleport/api/utils/grpc/interceptors.(* grpcClientStreamWrapper).RecvMsg
	github.com/gravitational/teleport/tool/tsh/access_request.go:55 main.onRequestList.func1
	github.com/gravitational/teleport/lib/client/api.go:1395 github.com/gravitational/teleport/lib/client.(*TeleportClient).WithRootClusterClient
	github.com/gravitational/teleport/tool/tsh/access_request.go:54 main.onRequestList
	github.com/gravitational/teleport/tool/tsh/tsh.go:1215 main.Run
	github.com/gravitational/teleport/tool/tsh/tsh.go:506 main.main
	runtime/proc.go:250 runtime.main
	runtime/asm_arm64.s:1172 runtime.goexit
User Message: EOF

[codingllama]

@rosstimothy is not trace-wrapping in the interceptors enough to avoid it? Any tips to avoid making the same mistakes?

[rosstimothy]

@rosstimothy is not trace-wrapping in the interceptors enough to avoid it? Any tips to avoid making the same mistakes?

If we don't do a trail.FromGRPC in the interceptors then what is the point of using them?

[Joerger]

@rosstimothy How did your PR break tsh request ls? It is working fine with this PR.

Also, Alan is right, we just need to trace.Unwrap(trail.FromGRPC()). This way, we get a proper error from the error code without adding the middleware trace:

ERROR REPORT:
Original Error: *status.Error rpc error: code = Unknown desc = EOF
Stack Trace:
	github.com/gravitational/teleport/api/client/client.go:998 github.com/gravitational/teleport/api/client.(*Client).GetAccessRequests
	github.com/gravitational/teleport/tool/tsh/common/access_request.go:58 github.com/gravitational/teleport/tool/tsh/common.onRequestList.func1
	github.com/gravitational/teleport/lib/client/api.go:1468 github.com/gravitational/teleport/lib/client.(*TeleportClient).WithRootClusterClient
	github.com/gravitational/teleport/tool/tsh/common/access_request.go:57 github.com/gravitational/teleport/tool/tsh/common.onRequestList
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:1403 github.com/gravitational/teleport/tool/tsh/common.Run
	github.com/gravitational/teleport/tool/tsh/common/tsh.go:545 github.com/gravitational/teleport/tool/tsh/common.Main
	github.com/gravitational/teleport/tool/tsh/main.go:24 main.main
	runtime/proc.go:267 runtime.main
	runtime/asm_amd64.s:1650 runtime.goexit
User Message: rpc error: code = Unknown desc = EOF

[codingllama]

Adding this comment here to record the history.

Since the interceptors landed various server-side errors that used to be treated as status.Error are now trace.BadParameter errors, which triggers relogin in various situations where it shouldn't. See #36749 and #36866 (comment).