Unexpected error validating SSL certificate

[sj-williams]

Background

We are seeing a large number of events logged in ingress logs for the error

Unexpected error validating SSL certificate CERT for server INGRESS: x509: certificate is valid for X, not Y

This is firing continuously, many millions of times over 24 hour period.

OS query example:
https://app-logs.cloud-platform.service.justice.gov.uk/_dashboards/app/data-explorer/discover#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:ef705d70-0d2e-11ef-afac-8f79b1004d33,view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2h,to:now))&_q=(filters:!(),query:(language:kuery,query:'%22Unexpected%20error%20validating%20SSL%20certificate%22'))

Investigate this error, record findings in this ticket.

Questions / Assumptions

Observing some of these specific events, we have seen cases where ingress resources are referencing certificate CRDs with non matching dnsName values, and the ingress complains of invalid certifcate.

Definition of done

• cause of error is understood / resolved
• runbook has been updated
• user docs have been updated
• another team member has reviewed
• smoke tests are green
• prepare demo for the team

Reference

How to write good user stories

[sj-williams]

Picking a dev namespace candidate:

https://app-logs.cloud-platform.service.justice.gov.uk/_dashboards/app/discover#/doc/ef705d70-0d2e-11ef-afac-8f79b1004d33/live_kubernetes_ingress-2024.11.13?id=893B8095-0A01-4A6D-FE89-78442AED9492

prison-visits team going to fix cert ref, and report back.

This is probably going to be another case of chasing up people with similar misconfigs

[sj-williams]

Capturing offending certs & ingresses here:

https://docs.google.com/document/d/1q6K2hLVZKtL-FyBa7IwxY7P_9FHP_ONL-W4IupEfF3s/edit?tab=t.0

[sj-williams]

All namespaces with problematic certificates now identified:

namespace	team contacted	fixed?
prison-visits-booking-dev	🟢	🟢
prisoner-content-hub-production	🟢
prison-visits-booking-production	🟢
makeaplea-prod	🟢
makeaplea-preprod	🟢
makeaplea-dev	🟢
sg-test	🟢
hmpps-manage-intelligence-preprod	🟢	🟢
hmpps-manage-intelligence-stage	🟢
hmpps-find-and-refer-an-intervention-dev	🟢	🟢
iac-fees-dev	🟢
iac-fees-staging	🟢
hmpps-identify-remand-periods-prod	🟢	🟢
hmpps-book-secure-move-frontend-production	🟢
hmpps-book-secure-move-api-production	🟢
hmpps-sentence-plan-prod	🟢	🟢
hmpps-sentence-plan-preprod	🟢	🟢
hmpps-breach-notice-dev	🟢	🟢

verified with OS query:

"Unexpected error validating SSL certificate" AND NOT "manage-intelligence" AND NOT "prison-visits-booking-dev" AND NOT "hmpps-find-and-refer-an-intervention-dev" AND NOT "hmpps-breach-notice-dev" AND NOT "sg-test" AND NOT "makeaplea" AND NOT "hmpps-sentence-plan-preprod" AND NOT "prisoner-content-hub-production" AND NOT "hmpps-book-secure-move-api-production" AND NOT "hmpps-book-secure-move-frontend-production" AND NOT "hmpps-identify-remand-periods-prod" AND NOT "prison-visits-booking-production" AND NOT "iac-fees-staging" AND NOT "iac-fees-dev" AND NOT "hmpps-sentence-plan-prod"

[sj-williams]

🔴 BLOCKED - pending user action

[sj-williams]

book-secure-move-api errors are a bit strange, need to take a look at these and understand why the wildcard cert is complaining:
https://mojdt.slack.com/archives/C57UPMZLY/p1734536492060909

[sj-williams]

Closing, replacing with tickets:

#6827

#6826