Merge branch '8.1' of github.com:elastic/beats into mergify/bp/8.1/pr-29710

* '8.1' of github.com:elastic/beats: (51 commits)
  refactor pushDockerImages (#30414) (#30624)
  ci: add windows-2022 in the extended meta-stage (#30528) (#30630)
  Curate k8s testing versions to only keep the actively maintained (#30619) (#30625)
  [8.1](backport #30355) Add Beats upgrade docs for 8.0 (#30612)
  Remove references to gcp from the Functionbeat docs (#30579) (#30609)
  x-pack/auditbeat/module/system/socket: defend against exec with zero arguments (#30586) (#30597)
  [MySQL Enterprise] Adding default paths values to manifest.yml (#30598) (#30604)
  metricbeat - fix elasticsearch and kibana integration tests failures in 8.0 (#30566) (#30594)
  Install gawk as a replacement for mawk in Docker containers. (#30452) (#30465)
  [Filebeat] Remove RecordedFuture dataset from Threat Intel module (#30564) (#30568)
  Adjust the documentation of `backoff` options in filestream input (#30552) (#30557)
  packetbeat/beater: help the GC clean up the Npcap installer if it's not used (#30513) (#30546)
  Osquerybeat: Add install verification for osquerybeat (#30388) (#30404)
  Update docker/distribution to 2.8.0 (#30462) (#30540)
  Add `parsers` examples to `filestream` reference configuration (#30529) (#30537)
  [8.1](backport #30068) ZooKeeper module: Adapt to ZooKeeper 3.6+ `mntr` response fields' changes. (#30360)
  [8.1](backport #30512) Switch skip to use `CI` (#30525)
  Forward-port 8.0.1 changelog to 8.1 (#30517)
  packetbeat/beater: don't attempt to install npcap when already installed (#30509) (#30511)
  Add drop and explicit tests to avoid duplicate ingest of elasticsearch logs (#30440) (#30488)
  ...

Author: v1v

.ci/packaging.groovy

@@ -268,100 +268,47 @@ pipeline {
 def pushCIDockerImages(Map args = [:]) {
   def arch = args.get('arch', 'amd64')
   catchError(buildResult: 'UNSTABLE', message: 'Unable to push Docker images', stageResult: 'FAILURE') {
+    def defaultVariants = [ '' : 'beats', '-oss' : 'beats', '-ubi8' : 'beats' ]
+    def completeVariant = ['-complete' : 'beats']
+    // Cloud is not public available, therefore it should use the beats-ci namespace.
+    def cloudVariant = ['-cloud' : 'beats-ci']
     if (env?.BEATS_FOLDER?.endsWith('auditbeat')) {
-      tagAndPush(beatName: 'auditbeat', arch: arch)
+      tagAndPush(beatName: 'auditbeat', arch: arch, variants: defaultVariants)
     } else if (env?.BEATS_FOLDER?.endsWith('filebeat')) {
-      tagAndPush(beatName: 'filebeat', arch: arch)
+      tagAndPush(beatName: 'filebeat', arch: arch, variants: defaultVariants)
     } else if (env?.BEATS_FOLDER?.endsWith('heartbeat')) {
-      tagAndPush(beatName: 'heartbeat', arch: arch)
+      tagAndPush(beatName: 'heartbeat', arch: arch, variants: defaultVariants)
     } else if (env?.BEATS_FOLDER?.endsWith('metricbeat')) {
-      tagAndPush(beatName: 'metricbeat', arch: arch)
+      tagAndPush(beatName: 'metricbeat', arch: arch, variants: defaultVariants)
     } else if (env?.BEATS_FOLDER?.endsWith('osquerybeat')) {
-      tagAndPush(beatName: 'osquerybeat', arch: arch)
+      tagAndPush(beatName: 'osquerybeat', arch: arch, variants: defaultVariants)
     } else if ("${env.BEATS_FOLDER}" == "packetbeat"){
-      tagAndPush(beatName: 'packetbeat', arch: arch)
+      tagAndPush(beatName: 'packetbeat', arch: arch, variants: defaultVariants)
     } else if ("${env.BEATS_FOLDER}" == "x-pack/elastic-agent") {
-      tagAndPush(beatName: 'elastic-agent', arch: arch)
+      tagAndPush(beatName: 'elastic-agent', arch: arch, variants: defaultVariants + completeVariant + cloudVariant)
     }
   }
 }
 
 /**
 * @param beatName name of the Beat
 * @param arch what architecture
+* @param variants list of docker variants
 */
 def tagAndPush(Map args = [:]) {
-  def beatName = args.beatName
-  def arch = args.get('arch', 'amd64')
-  def libbetaVer = env.BEAT_VERSION
-  def aliasVersion = ""
-  if("${env.SNAPSHOT}" == "true"){
-    aliasVersion = libbetaVer.substring(0, libbetaVer.lastIndexOf(".")) // remove third number in version
-
-    libbetaVer += "-SNAPSHOT"
-    aliasVersion += "-SNAPSHOT"
-  }
-
-  def tagName = "${libbetaVer}"
-  if (isPR()) {
-    tagName = "pr-${env.CHANGE_ID}"
-  }
-
-  // supported tags
-  def tags = [tagName, "${env.GIT_BASE_COMMIT}"]
-  if (!isPR() && aliasVersion != "") {
-    tags << aliasVersion
-  }
-  // supported image flavours
-  def variants = ["", "-oss", "-ubi8"]
-
-  if(beatName == 'elastic-agent'){
-      variants.add("-complete")
-      variants.add("-cloud")
-  }
-
-  variants.each { variant ->
-    // cloud docker images are stored in the private docker namespace.
-    def sourceNamespace = variant.equals('-cloud') ? 'beats-ci' : 'beats'
-    tags.each { tag ->
-      // TODO:
-      // For backward compatibility let's ensure we tag only for amd64, then E2E can benefit from until
-      // they support the versioning with the architecture
-      if ("${arch}" == "amd64") {
-        doTagAndPush(beatName: beatName, variant: variant, sourceTag: libbetaVer, targetTag: "${tag}", sourceNamespace: sourceNamespace)
-      }
-      doTagAndPush(beatName: beatName, variant: variant, sourceTag: libbetaVer, targetTag: "${tag}-${arch}", sourceNamespace: sourceNamespace)
-    }
-  }
-}
-
-/**
-* @param beatName name of the Beat
-* @param variant name of the variant used to build the docker image name
-* @param sourceNamespace namespace to be used as source for the docker tag command
-* @param sourceTag tag to be used as source for the docker tag command, usually under the 'beats' namespace
-* @param targetTag tag to be used as target for the docker tag command, usually under the 'observability-ci' namespace
-*/
-def doTagAndPush(Map args = [:]) {
-  def beatName = args.beatName
-  def variant = args.variant
-  def sourceTag = args.sourceTag
-  def targetTag = args.targetTag
-  def sourceNamespace = args.sourceNamespace
-  def sourceName = "${DOCKER_REGISTRY}/${sourceNamespace}/${beatName}${variant}:${sourceTag}"
-  def targetName = "${DOCKER_REGISTRY}/observability-ci/${beatName}${variant}:${targetTag}"
-  def iterations = 0
-  retryWithSleep(retries: 3, seconds: 5, backoff: true) {
-    iterations++
-    def status = sh(label: "Change tag and push ${targetName}",
-                    script: ".ci/scripts/docker-tag-push.sh ${sourceName} ${targetName}",
-                    returnStatus: true)
-    if ( status > 0 && iterations < 3) {
-      error("tag and push failed for ${beatName}, retry")
-    } else if ( status > 0 ) {
-      log(level: 'WARN', text: "${beatName} doesn't have ${variant} docker images. See https://github.com/elastic/beats/pull/21621")
-    }
+  def images = [ ]
+  args.variants.each { variant, sourceNamespace ->
+    images += [ source: "${sourceNamespace}/${args.beatName}${variant}",
+                target: "observability-ci/${args.beatName}",
+                arch: args.arch ]
   }
+  pushDockerImages(
+    registry: env.DOCKER_REGISTRY,
+    secret: env.DOCKERELASTIC_SECRET,
+    snapshot: env.SNAPSHOT,
+    version: env.BEAT_VERSION,
+    images: images
+  )
 }
 
 def prepareE2ETestForPackage(String beat){

.ci/scripts/docker-tag-push.sh

@@ -3,6 +3,32 @@
 :issue: https://github.com/elastic/beats/issues/
 :pull: https://github.com/elastic/beats/pull/
 
+[[release-notes-8.0.1]]
+=== Beats version 8.0.1
+https://github.com/elastic/beats/compare/v8.0.0...v8.0.1[View commits]
+
+==== Bugfixes
+
+*Filebeat*
+
+- tcp/unix input: Stop accepting connections after socket is closed. {pull}29712[29712]
+- Fix using log_group_name_prefix in aws-cloudwatch input. {pull}29695[29695]
+- Fix multiple instances of the same module configured within `filebeat.modules` in filebeat.yml. {issue}29649[29649] {pull}29952[29952]
+- aws-s3: fix race condition in states used by s3-poller. {issue}30123[30123] {pull}30131[30131]
+
+*Filebeat*
+- Fix broken Kafka input {issue}29746[29746] {pull}30277[30277]
+- cisco module: Fix change the broke ASA and FTD configs that used `var.input: syslog`. {pull}30072[30072]
+- aws-s3: fix race condition in states used by s3-poller. {issue}30123[30123] {pull}30131[30131]
+
+*Heartbeat*
+- Fix missing mapping for `service.name`. {pull}30324[30324]
+
+*Winlogbeat*
+
+- Fix run loop when reading from evtx file {pull}30006[30006]
+
+
 [[release-notes-8.0.0]]
 === Beats version 8.0.0
 https://github.com/elastic/beats/compare/v7.17.0...v8.0.0[View commits]

CHANGELOG.asciidoc

@@ -18,6 +18,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...main[Check the HEAD dif
 
 *Filebeat*
 
+- Remove Recorded Future integration from threatintel module. {pull}30564[30564]
 
 *Heartbeat*
 
@@ -39,39 +40,44 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...main[Check the HEAD dif
 *Affecting all Beats*
 
 - Fix field names with `add_network_direction` processor. {issue}29747[29747] {pull}29751[29751]
+- Fix a logging bug when `ssl.verification_mode` was set to `full` or `certificate`, the command `test output` incorrectly logged that TLS was disabled.
+- Fix the ability for subcommands to be ran properly from the beats containers. {pull}30452[30452]
+- Update docker/distribution dependency library to fix a security issues concerning OCI Manifest Type Confusion Issue. {pull}30462[30462]
 
 *Auditbeat*
 
 - auditd: Add error.message to events when processing fails. {pull}30009[30009]
+- Fix handling of execve call events which have no argument. {issue}30585[30585] {pull}30586[30586]
 
 *Filebeat*
 
-- tcp/unix input: Stop accepting connections after socket is closed. {pull}29712[29712]
-- aws-s3: fix race condition in states used by s3-poller. {issue}30123[30123] {pull}30131[30131]
-
-*Heartbeat*
-
 
 *Filebeat*
+- auditd: Prevent mapping explosion when truncated EXECVE records are ingested. {pull}30382[30382]
+- elasticsearch: fix duplicate ingest when using a common appender configuration {issue}30428[30428] {pull}30440[30440]
 
+- Fix ECS version string in threatintel to be consistent with other modules and add event.timezone. {issue}30499[30499] {pull}30570[30570]
+- Add default paths value to MySQL Enterprise module to prevent issues with pipeline installations {pull}30598[30598]
 
 *Heartbeat*
 
-
 *Metricbeat*
 
+- Fixed missing ZooKeeper metrics due compatibility issues with versions >= 3.6.0 {pull}30068[30068]
+- Enhance metricbeat on openshift documentation {pull}30054[30054]
+- Fixed missing ZooKeeper metrics due compatibility issues with versions >= 3.6.0 {pull}30068[30068]
+- Fix Docker module: rename fields on dashboards. {pull}30500[30500]
 
 *Packetbeat*
 
 
 *Winlogbeat*
 
 - Add provider names to Security pipeline conditional check in routing pipeline. {issue}27288[27288] {pull}29781[29781]
-- Fix run loop when reading from evtx file {pull}30006[30006]
 
 *Functionbeat*
 
-- Pass AWS region configuration correctly. {issue}28520[28520] {pull}30238[30238] 
+- Pass AWS region configuration correctly. {issue}28520[28520] {pull}30238[30238]
 
 
 *Elastic Logging Plugin*
@@ -106,7 +112,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...main[Check the HEAD dif
 - Add support in httpjson input for oAuth2ProviderDefault of password grant_type. {pull}29087[29087]
 - Add support for filtering in journald input with `unit`, `kernel`, `identifiers` and `include_matches`. {pull}29294[29294]
 - Add new `userAgent` and `beatInfo` template functions for httpjson input {pull}29528[29528]
-- threatintel module: Add new Recorded Future integration. {pull}30030[30030]
 - Add pipeline in FB's supported hints. {pull}30212[30212]
 
 *Auditbeat*
@@ -132,7 +137,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...main[Check the HEAD dif
 
 *Packetbeat*
 
-- Add automated OEM Npcap installation handling. {pull}29112[29112]
+- Add automated OEM Npcap installation handling. {pull}29112[29112] {pull}30438[30438] {pull}30493[30493]
 - Add support for capturing TLS random number and OCSP status request details. {issue}29962[29962] {pull}30102[30102]
 
 *Functionbeat*
@@ -165,4 +170,3 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...main[Check the HEAD dif
 
 ==== Known Issue
 
-*Journalbeat*