Fix up case where subscription is terminated due to ACLs changing or a snapshot restore occurring

[andrewstucki]

Description

Due to swallowing and only logging xDS errors here:

consul/agent/proxycfg/state.go

Lines 375 to 386 in 88951bf

	if IsTerminalError(u.Err) {
	s.logger.Error("Data source in an irrecoverable state; exiting", "error", u.Err, "correlationID", u.CorrelationID)
	s.Close(true)
	return
	}
	if err := s.handler.handleUpdate(ctx, u, snap); err != nil {
	s.logger.Error("Failed to handle update from watch",
	"id", u.CorrelationID, "error", err,
	)
	continue
	}

We were getting into a state where if ACLs were modified while connected to xDS, the stream wasn't properly being re-established and instead the Consul logs were filling up with error logs such as:

Failed to handle update from watch: kind=api-gateway proxy=default/default/gateway-d6b4d69c-krz5b service_id=default/default/gateway-d6b4d69c-krz5b id=gateway-config error="error filling agent cache: subscription closed by server, client must reset state and resubscribe"

Basically we were missing some casing for marking certain errors as TerminalErrors in proxycfg-glue, this fixes that.

Testing & Reproduction steps

Get a proxy and/or gateway to connect up to xDS and then modify the token ACLs that are associated with it, you should see a bunch of these in the logs.

PR Checklist

• updated test coverage
• external facing docs updated
• appropriate backport labels added
• not a security concern

[boxofrad]

Thanks so much for fixing this! We should probably backport to 1.15 if it does affect snapshot restoration.