[Agent] Elastic-Agent-Packaging-Linux is failed (there for e2e-testing pr support) which blocks merges - seems to be a Heartbeat container build problem

[EricDavisX]

This is from a dev via slack:
PR 'tests' are failing...
"The E2E agent linux tests are super flaky at the moment and preventing us from making imports. Is that something currently being looked at?"
citing this PR as example:
#28517

Manu helped bisect the pr failures and seemed to point towards the Heartbeat container not being available, so maybe a build problem there.

Victor noted that the e2e-testing support had been turned on (requiring that packaging) somewhat recently. PM me for slack convo link or team thread on the packaging change

image from slack:

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[EricDavisX]

@andresrc @jlind23 @KseniaElastic @v1v can can we do to help resolve this quick? I don't know the Beats packaging.

[jlind23]

@andrewvc seems that this is on Synthetics side.

[mdelapenya]

I'm currently bisecting this issue with the e2e-testing framework. For that, I'm basically passing the commit to the tests with this command:

For master (current released artifacts):

TAGS="heartbeat" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true make -C e2e/_suites/kubernetes-autodiscover functional-test

Starting the bisect with latest commit on heartbeat:

TAGS="heartbeat" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true GITHUB_CHECK_SHA1=af602c2b0df38bfc3fb5cfcfabcab1145b558022 ELASTIC_APM_ACTIVE=false make -C e2e/_suites/kubernetes-autodiscover functional-test

Will post results here.

[mdelapenya]

OK, taking this commit 99ebf3e as GOOD, I'm able to see the tests passing for that commit. Starting the bisect from that:

And the result is GOOD ✅

kind delete clusters --all && \
TAGS="heartbeat" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true \
GITHUB_CHECK_SHA1=99ebf3e4375c4dbee0ee281889e804b13a62a463 
ELASTIC_APM_ACTIVE=false make -C e2e/_suites/kubernetes-autodiscover functional-test

List of commits, starting from GOOD:

• 🔴 5186feb - (HEAD -> master, upstream/master, origin/master, origin/HEAD) Remove unneeded mergify config (20 minutes ago)
• 78fedf3 - backport: Add 7.16 branch (backport: Add 7.16 branch #28560) (25 minutes ago)
• 15abbe6 - Add proxy_url support to threatintel module's malwarebazaar fileset (Add proxy_url support to threatintel module's malwarebazaar fileset #28533) (5 hours ago)
• ec8e051 - Osquerybeat: Implement host_users, host_groups, host_processes tables as a part of our osquery_extension. (Osquerybeat: Implement host_users, host_groups, host_processes tables as a part of our osquery_extension. #28434) (13 hours ago)
• d31cae9 - [Heartbeat] Make run_once syntax a boolean ([Heartbeat] Make run_once syntax a boolean #28548) (14 hours ago)
• 48c9d79 - Improve stats api (Improve stats api #27963) (15 hours ago)
• af602c2 - Heartbeat: control network throttling in synthetics (Heartbeat: control network throttling in synthetics #28462) (17 hours ago)
• d2569fd - Add linux/rapl metricset (Add linux/rapl metricset #28238) (17 hours ago)
• 🔴 298d786 - Get metricbeat to compile on AIX (Get metricbeat to compile on AIX #27954) (18 hours ago)
• 45cea45 - [azure] signinlogs - support additional category types ([azure] signinlogs - support additional category types #28511) (19 hours ago)
• 6431e41 - Add a fully-featured dashboard for Enterprise Search (Add a fully-featured dashboard for Enterprise Search #28415) (19 hours ago)
• 🔴 53a618b - docs: link to new APM book (docs: link to new APM book #28519) (20 hours ago)
• 🔴 81c38fc - [Heartbeat][Agent] Seccomp / synthetics bugfix improvements ([Heartbeat][Agent] Seccomp / synthetics bugfix improvements #28514) (21 hours ago)
• ✅ 0a24250 - [pre-commit] for linting merge-conflict, pipelines and JJBB ([pre-commit] for linting merge-conflict, pipelines and JJBB #28501) (22 hours ago)
• ✅ 99ebf3e - Add diagnostics collect command to elastic-agent. (Add diagnostics collect command to elastic-agent. #28461) (24 hours ago)

[mdelapenya]

Trying with 298d786: Get metricbeat to compile on AIX

kind delete clusters --all && \
TAGS="heartbeat" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true \
GITHUB_CHECK_SHA1=298d786fc67301f429c0fe619fa06787093a6751 \
ELASTIC_APM_ACTIVE=false make -C e2e/_suites/kubernetes-autodiscover functional-test

And the result is BAD 🔴

[mdelapenya]

Trying with 53a618b: docs: link to new APM book

kind delete clusters --all && \
TAGS="heartbeat" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true \
GITHUB_CHECK_SHA1=53a618b36135db5c2940e1df48ffad164349b28c \
ELASTIC_APM_ACTIVE=false make -C e2e/_suites/kubernetes-autodiscover functional-test

And the result is BAD 🔴

[mdelapenya]

Trying with 0a24250: [pre-commit] for linting merge-conflict, pipelines and JJBB

kind delete clusters --all && \
TAGS="heartbeat" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true \
GITHUB_CHECK_SHA1=0a2425021bfb488cf21927443bbfafc0ec450bb7 \
ELASTIC_APM_ACTIVE=false make -C e2e/_suites/kubernetes-autodiscover functional-test

And the result is GOOD ✅

[mdelapenya]

The only commit that is left in this bisect is 81c38fc: [Heartbeat][Agent] Seccomp / synthetics bugfix improvements, which should fail the tests, being the culprit commit 🤞

kind delete clusters --all && \
TAGS="heartbeat" TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE DEVELOPER_MODE=true \
GITHUB_CHECK_SHA1=81c38fc4c009348d57c92ae85920aed35297a89e \
ELASTIC_APM_ACTIVE=false make -C e2e/_suites/kubernetes-autodiscover functional-test

And, effectively, the result is BAD 🔴, which means that we found the root cause of the issue.

[mdelapenya]

Let me explain what that test does: it uses the k8s-autodiscover test suite (@jsoriano and @ChrsMark can provide more context):

Scenario: Monitor pod availability using hints with named ports
  Given "heartbeat" is running with "hints enabled for pods"
   When "redis" is deployed with "monitor annotations with named port"
   Then "heartbeat" collects events with "kubernetes.pod.name:redis"
    And "heartbeat" collects events with "monitor.status:up"

The scenario always fails in the same step, the Then clause: Then "heartbeat" collects events with "kubernetes.pod.name:redis"

To understand the internals, the scenario creates a Kubernetes cluster with Kind, and starts a pod from Hearbeat in the version specified by the GITHUB_CHECK_SHA1 variable. If not set, the project uses released artifacts, otherwise it will go to our GCP bucket, will look up that commit and will download the artifacts needed by the test, in this case the TAR file representing heartbeat in the Docker image format. It will load the TAR file into the local Docker engine and will import that image into Kind, so that it's available inside the k8s cluster.

As described in the Then clause, the heartbeat pod is not able to collect events after 81c38fc. Tests logs are:

DEBU[2021-10-20T17:36:17+02:00] Failed to copy events from test-e18ad60b-5d71-48da-83cf-e91bb329c0fb/heartbeat-84b998df5d-tdx8x:/tmp/beats-events to /var/folders/8h/pk8n63tn3px_tbs6_l862s_w0000gn/T/test-106333045/events: exit status 1
TRAC[2021-10-20T17:36:18+02:00] Validating required tools: [kubectl]
TRAC[2021-10-20T17:36:18+02:00] Binary is present binary=kubectl path=/usr/local/bin/kubectl
TRAC[2021-10-20T17:36:18+02:00] Executing command args="[--kubeconfig /var/folders/8h/pk8n63tn3px_tbs6_l862s_w0000gn/T/test-190619484/kubeconfig --namespace test-e18ad60b-5d71-48da-83cf-e91bb329c0fb cp --no-preserve test-e18ad60b-5d71-48da-83cf-e91bb329c0fb/heartbeat-84b998df5d-tdx8x:/tmp/beats-events /var/folders/8h/pk8n63tn3px_tbs6_l862s_w0000gn/T/test-106333045/events]" command=kubectl env="map[]"
ERRO[2021-10-20T17:36:18+02:00] Error executing command args="[--kubeconfig /var/folders/8h/pk8n63tn3px_tbs6_l862s_w0000gn/T/test-190619484/kubeconfig --namespace test-e18ad60b-5d71-48da-83cf-e91bb329c0fb cp --no-preserve test-e18ad60b-5d71-48da-83cf-e91bb329c0fb/heartbeat-84b998df5d-tdx8x:/tmp/beats-events /var/folders/8h/pk8n63tn3px_tbs6_l862s_w0000gn/T/test-106333045/events]" baseDir=. command=kubectl env="map[]" error="exit status 1" stderr="error: unable to upgrade connection: container not found ("heartbeat")\n"

Where kubectl returns error: unable to upgrade connection: container not found ("heartbeat")