Skip to content

Filter api-gateway cache logging to reduce log output on server disconnect#2880

Merged
curtbushko merged 2 commits intomainfrom
reduce-gateway-noise
Sep 5, 2023
Merged

Filter api-gateway cache logging to reduce log output on server disconnect#2880
curtbushko merged 2 commits intomainfrom
reduce-gateway-noise

Conversation

@curtbushko
Copy link
Contributor

Changes proposed in this PR:

  • I noticed this when I was working on a customer issue and it made debugging more difficult
  • I was working on an issue that was reproduced by deleting the consul-server-0 pod and the api-gateway logging spit out about 20k log lines in about 5 seconds
  • I don't think we need to log these errors out at the api-gateway level as other parts of the controller will log a similar error out but less frequently.
  • The customer will still see that the consul server is unreachable in the logs (see below)

Instead of 20k of these:

2023-08-31T14:33:54.861Z ERROR error fetching config entries for kind: http-route {"error": "Get "http://10.244.0.8:8500/v1/config/http-route?dc=dc1&index=10\": dial tcp 10.244.0.8:8500: connect: connection refused"}

They still see:

2023-08-31T14:41:43.031Z ERROR Reconciler error {"controller": "endpoints", "controllerGroup": "", "controllerKind": "Endpoints", "Endpoints": {"name":"consul-ui","namespace":"consul"}, "namespace": "consul", "name": "consul-ui", "reconcileID": "ce66b001-0173-40d1-95ab-a7a5363a6612", "error": "1 error occurred:\n\t* Get "http://:8500/v1/catalog/nodes?dc=dc1&filter=Meta%5B%22synthetic-node%22%5D+%3D%3D+%22true%22+&ns=%2A": dial tcp: lookup : no such host\n\n"}

How I've tested this PR:

  • build locally, delete consul-server-0 and look at the connect injector logs.

How I expect reviewers to test this PR:

👀

Checklist:

@curtbushko curtbushko self-assigned this Aug 31, 2023
@curtbushko curtbushko marked this pull request as ready for review August 31, 2023 14:54
Copy link
Contributor

@t-eckert t-eckert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Thank you

@t-eckert t-eckert added the theme/api-gateway Related to Consul API Gateway label Aug 31, 2023
@t-eckert
Copy link
Contributor

I think you will want to backport this into 1.2.x

@curtbushko curtbushko added the backport/1.2.x This release branch is no longer active. label Aug 31, 2023
@curtbushko curtbushko merged commit decdca5 into main Sep 5, 2023
@curtbushko curtbushko deleted the reduce-gateway-noise branch September 5, 2023 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport/1.2.x This release branch is no longer active. theme/api-gateway Related to Consul API Gateway

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants