Third party audit task is fails on rhel 8

[danielmitterdorfer]

Over the last few days we see a noticeable increase in builds that fail with

Third party audit task is not configured correctly

Here are some examples:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=redhat-8/232/console:

21:41:22 > Task :test:framework:thirdPartyAudit
21:41:22 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
21:41:23 Unnecessary exclusions, following classes are not missing:
21:41:23    * javax.servlet.ServletContextEvent
21:41:23   * javax.servlet.ServletContextListener
21:41:23 
21:41:23 > Task :test:framework:thirdPartyAudit FAILED
21:41:23 
21:41:23 > Task :qa:vagrant:thirdPartyAudit
21:41:23 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
21:41:29 
21:41:29 > Task :plugins:repository-hdfs:thirdPartyAudit
21:41:29 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
21:41:30 
21:41:30 FAILURE: Build failed with an exception.
21:41:30 
21:41:30 * What went wrong:
21:41:30 Execution failed for task ':test:framework:thirdPartyAudit'.
21:41:30 > Third party audit task is not configured correctly

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.2+multijob-unix-compatibility/os=redhat-8/11/console

17:19:27 > Task :plugins:discovery-azure-classic:thirdPartyAudit FAILED
17:19:27 Unnecessary exclusions, following classes are not missing:
17:19:27    * com.sun.xml.fastinfoset.stax.StAXDocumentParser
17:19:27   * com.sun.xml.fastinfoset.stax.StAXDocumentSerializer
17:19:27   * javax.servlet.ServletContextEvent
17:19:27   * javax.servlet.ServletContextListener
17:19:27   * org.apache.avalon.framework.logger.Logger
17:19:27   * org.apache.log.Hierarchy
17:19:27   * org.apache.log.Logger
17:19:27   * org.eclipse.persistence.descriptors.ClassDescriptor
17:19:27   * org.eclipse.persistence.internal.oxm.MappingNodeValue
17:19:27   * org.eclipse.persistence.internal.oxm.TreeObjectBuilder
17:19:27   * org.eclipse.persistence.internal.oxm.XPathFragment
17:19:27   * org.eclipse.persistence.internal.oxm.XPathNode
17:19:27   * org.eclipse.persistence.internal.queries.ContainerPolicy
17:19:27   * org.eclipse.persistence.jaxb.JAXBContext
17:19:27   * org.eclipse.persistence.jaxb.JAXBHelper
17:19:27   * org.eclipse.persistence.mappings.DatabaseMapping
17:19:27   * org.eclipse.persistence.mappings.converters.TypeConversionConverter
17:19:27   * org.eclipse.persistence.mappings.foundation.AbstractCompositeDirectCollectionMapping
17:19:27   * org.eclipse.persistence.oxm.XMLContext
17:19:27   * org.eclipse.persistence.oxm.XMLDescriptor
17:19:27   * org.eclipse.persistence.oxm.XMLField
17:19:27   * org.eclipse.persistence.oxm.mappings.XMLCompositeCollectionMapping
17:19:27   * org.eclipse.persistence.sessions.DatabaseSession
17:19:27   * org.jvnet.fastinfoset.VocabularyApplicationData
17:19:27   * org.jvnet.staxex.Base64Data
17:19:27   * org.jvnet.staxex.XMLStreamReaderEx
17:19:27   * org.jvnet.staxex.XMLStreamWriterEx
17:19:27   * org.osgi.framework.BundleEvent
17:19:27   * org.osgi.framework.SynchronousBundleListener
17:19:28 
17:19:28 > Task :plugins:ingest-attachment:processTestResources
17:19:30 
17:19:30 > Task :plugins:discovery-ec2:thirdPartyAudit
17:19:30 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
17:19:36 
17:19:36 > Task :server:compileJava9Java
17:19:36 
17:19:36 FAILURE: Build failed with an exception.
17:19:36 
17:19:36 * What went wrong:
17:19:36 Execution failed for task ':plugins:discovery-azure-classic:thirdPartyAudit'.
17:19:36 > Third party audit task is not configured correctly

I could not reproduce these failures locally.

An analysis in build-stats shows the increase in build failures. We have between 16 - 18 of those per day across all CI builds and it started on May 25th 2019, 05:41:19.570 (see
failed-builds.txt for a full list of the builds that failed so far):

I don't want to jump to conclusions but one possibly related commit that happened before the build failures have started, could be #41392 which addresses the configuration phase in our build (there are a few other commits after that one which also address configuration caching so it's hard to tell). @mark-vieira could you please have a look at this as you're probably most familiar here?

[elasticmachine]

Pinging @elastic/es-core-infra

[danielmitterdorfer]

More instances in:

• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=redhat-8/235/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=redhat-8/234/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.2+multijob-unix-compatibility/os=redhat-8/14/console
• https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.1+multijob-unix-compatibility/os=redhat-8/13/console

[mark-vieira]

I don't believe this is related to #41392 as those changes exist only in master. These failures seem to be isolated to our Redhat 8 tests so there's something with that environment. My first thought is that probably that system is producing slightly different output from the forbidden APIs CLI which is no longer matching this regex.

@atorok any other thoughts on why this is specific to Redhat 8? If not I think I'll add some logging here so we dump the output from forbidden APIs on failure.

[alpar-t]

I got some idle time on the rhel worker but couldn't reproduce the issue.
Dumping the output on failure has it's own issues.
The problem is that a lot of times forbiddenapis fails because of missing classes, exceptions etc. For any project that has any configuration it will fail and we don't want to see the actual failure as we are interpreting it.
The problem with the lather is that it's hard to interpret missing classes: was it because the class was not listed as missing or because some other failure prevented that.

I think the solution is to write the output to a log file and make sure it's captured by the log upload in CI. This way it's available to diagnose when we need it without clobbering the output of a failure in ways most users won't easily understand.

[tvernum]

There's something weird about the RHEL8 environment. Some X-Pack tribe tests were consistently failing on the 6.7 RHEL8 workers but not on any other OS: #42004

This issue is preventing those tests from running on the new 6.8 matrix builds, but I suspect they're still broken.

[jbaiera]

Some additional examples from today. All RHEL8

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.1+multijob-unix-compatibility/os=redhat-8/18/console
https://scans.gradle.com/s/mnnbvdiestnqq/console-log?task=:plugins:discovery-gce:thirdPartyAudit

:plugins:discovery-gce:thirdPartyAudit FAILED
--
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Unnecessary exclusions, following classes are not missing:
* javax.servlet.ServletContextEvent
* javax.servlet.ServletContextListener
* org.apache.log.Hierarchy
* org.apache.log.Logger

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.1+multijob-unix-compatibility/os=redhat-8/17/console
https://scans.gradle.com/s/7a5gbu3sz55f6/console-log?task=:plugins:discovery-ec2:thirdPartyAudit

:plugins:discovery-ec2:thirdPartyAudit FAILED
--
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Unnecessary exclusions, following classes are not missing:
* com.amazonaws.jmespath.JmesPathIdentity
* com.amazonaws.jmespath.JmesPathLengthFunction
* com.amazonaws.jmespath.JmesPathLiteral
* com.amazonaws.jmespath.OpGreaterThan
* javax.servlet.ServletContextEvent
* javax.servlet.ServletContextListener
* org.apache.avalon.framework.logger.Logger
* org.apache.log.Hierarchy
* org.apache.log.Logger
* software.amazon.ion.IonSystem
* software.amazon.ion.IonWriter
* software.amazon.ion.system.IonBinaryWriterBuilder
* software.amazon.ion.system.IonSystemBuilder
* software.amazon.ion.system.IonTextWriterBuilder
* software.amazon.ion.system.IonWriterBuilder

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.2+multijob-unix-compatibility/os=redhat-8/18/console
https://scans.gradle.com/s/ct7bykcnl2ut2/console-log?task=:plugins:repository-hdfs:thirdPartyAudit

:plugins:repository-hdfs:thirdPartyAudit FAILED
--
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Unnecessary exclusions, following classes have no violations:
* com.google.common.primitives.UnsignedBytes$LexicographicalComparatorHolder$UnsafeComparator
* com.google.common.primitives.UnsignedBytes$LexicographicalComparatorHolder$UnsafeComparator$1
* org.apache.hadoop.crypto.CryptoStreamUtils
* org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm
* org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm$Slot
* org.apache.hadoop.io.FastByteComparisons$LexicographicalComparerHolder$UnsafeComparer
* org.apache.hadoop.io.FastByteComparisons$LexicographicalComparerHolder$UnsafeComparer$1
* org.apache.hadoop.io.nativeio.NativeIO
* org.apache.hadoop.io.nativeio.NativeIO$POSIX
* org.apache.hadoop.security.SecurityUtil$QualifiedHostResolver
* org.apache.hadoop.util.SignalLogger$Handler

[tvernum]

@elastic/es-core-infra can we disable RHEL8 from the matrix build? We're getting dozens of failures per day, and no successes.

Right now it's all noise, no signal.

[mark-vieira]

The Redhat 8 CI job has been disabled until we sort this out.

[alpar-t]

This turned out to be due to the worker. I'll close this for now. We'll take on testing again once we have a rhel or centos 8 ephemeral worker