Logging audit trail exclude/include settings are not validated if disabled

[DaveCTurner]

It is possible to end up with an invalid persistent cluster state by adjusting the include/exclude filters for the LoggingAuditTrail while xpack.security.audit.enabled: false as follows:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.security.audit.logfile.events.exclude": "invalid"
  }
}

Any nodes that start up with xpack.security.audit.enabled: true will, however, validate this setting and this will cause them to fail to apply this cluster state. In severe cases, this means that the cluster cannot form:

[2020-02-14T10:55:45,618][WARN ][o.e.c.s.ClusterApplierService] [node-0] failed to apply updated cluster state in [0s]:
version [43], uuid [RhJIgoX7Qmunrf8Yn3dhlw], source [becoming candidate: clusterApplier#onNewClusterState]
java.lang.IllegalArgumentException: invalid event name specified [invalid]
        at org.elasticsearch.xpack.security.audit.AuditLevel.parse(AuditLevel.java:74) ~[?:?]
        at org.elasticsearch.xpack.security.audit.AuditLevel.parse(AuditLevel.java:82) ~[?:?]
        at org.elasticsearch.xpack.security.audit.logfile.LoggingAuditTrail.lambda$new$4(LoggingAuditTrail.java:195) ~[?:?]
        at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:661) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.settings.Setting$2.apply(Setting.java:634) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings$SettingUpdater.lambda$updater$0(AbstractScopedSettings.java:609) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.settings.AbstractScopedSettings.applySettings(AbstractScopedSettings.java:191) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:480) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService.access$100(ClusterApplierService.java:73) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:176) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) [elasticsearch-7.5.1.jar:7.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:835) [?:?]

You can work around this by setting xpack.security.audit.enabled: false again since this will bypass this validation and allow the cluster to form again, which will allow the removal of this invalid setting. However I think we should ensure that these settings are valid even if auditing is disabled to prevent clusters from getting into this state.

Relates #47038.

[elasticmachine]

Pinging @elastic/es-security (:Security/Audit)