[Beats] Build darwin/arm64 and universal binary

[AndersonQ]

What does this PR do?

It enables the Elastic-Agent and Beats to be built for darwin/arm64 and generates a universal binary (darwin/universal).

The universal binary is the merge of the darwin/arm64 with darwin/amd64. Therefore it requires to first build the darwin/arm64 and darwin/amd64 and them "merge" them together into the darwin/universal. Also darwin/universal isn't a valid platform or OS/ARCH pair, therefore it isn't recognised by the Go compiler. Given that, the choice was to add darwin/arm64 as a new platform for the Elastic-Agent and when ever both platforms (darwin/arm64 and darwin/amd64) are built, also generate and package the universal binary.
Even though it's quite an implicit behaviour, the alternative would also require a lot of changes and exceptions to have our build process recognising darwin/universal, but compiling darwin/arm64 and darwin/amd64 and merging them. Also the current process makes available all three of them, allowing the user to choose which fits better their needs. The drawback of the universal binary is its size, the double of the single binaries.

A new target, BuildDarwinUniversal was added on the x-packs magefiles required by the Elastic-Agent.

Also fixes some small typos, redundant information on the help command and remove the use of some deprecated functions.

Why is it important?

We need to support Mac with M1 chips

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• build darwin/arm64 binary
• build universal 2 binary
• use mage to automatically build the universal binary

How to test this PR locally

1. Build the elastic-agent for the new platforms:

   • darwin/arm64: PLATFORMS="darwin/arm64" PACKAGES="tar.gz" mage package
   • darwin/universal: PLATFORMS="darwin/arm64,darwin/amd64" PACKAGES="tar.gz" mage package

2. Run the elastic-agent

####Watch out:
The elastic-agent tries to download any binary it might need to comply with the assigned policy. However none of the new binaries from this PR are present on the artifact API and other applications (e.g. fleet-server and elastic-endpoint) also aren't publishing darwin/arm64 versions. Therefore if the elastic-agent tries to download them, it'll fail. This is the expected behavior and this PR does not address it, see Out of Scope.

Out of scope:

• Handle downloading binaries that does not publish darwin-universal artifacts on the artifact API
• Handle upgrading a darwin/amd64 agent running under Rosetta2 on a Mac with M1 chip. This will be addressed on another PR.

Related issues

• Relates [Meta] Mac M1 Support  elastic-agent#151
• Closes Beats to compile natively on a Mac M1 machine #29582

[mergify]

This pull request does not have a backport label. Could you fix it @AndersonQ? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-03-21T18:06:52.389+0000

• Duration: 89 min 48 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22523
Skipped	1940
Total	24463

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[v1v]

This feature will be enabled no just in ElasticAgent but in all the other Beats, is that correct?

[ph]

@AndersonQ I've tried to build all the artifacts with make release in the top level directory and it fails on my side

>> Building using: cmd='build/mage-linux-amd64 buildDarwinUniversal', env=[CC=oa64-clang, CXX=oa64-clang++, GOARCH=arm64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-arm64]
Unknown target specified: "buildDarwinUniversal"
Error: failed building for darwin/arm64: exit status 2
failed building for darwin/arm64: exit status 2
package ran for 14m30.180628207s
Error: failed merging darwin/amd64 and darwin/arm64 into darwin/universal target=buildDarwinUniversal for platform=darwin/arm64: running "docker run --env EXEC_UID=501 --env EXEC_GID=20 -v /Users/ph/go/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=false --env DEV=false -v /Users/ph/src/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/auditbeat docker.elastic.co/beats-dev/golang-crossbuild:1.17.5-darwin-arm64-debian10 --build-cmd build/mage-linux-amd64 buildDarwinUniversal -p darwin/arm64" failed with exit code 1
failed merging darwin/amd64 and darwin/arm64 into darwin/universal target=buildDarwinUniversal for platform=darwin/arm64: running "docker run --env EXEC_UID=501 --env EXEC_GID=20 -v /Users/ph/go/pkg/mod:/go/pkg/mod:ro --rm --env GOFLAGS=-mod=readonly --env MAGEFILE_VERBOSE= --env MAGEFILE_TIMEOUT= --env SNAPSHOT=false --env DEV=false -v /Users/ph/src/beats:/go/src/github.com/elastic/beats -w /go/src/github.com/elastic/beats/auditbeat docker.elastic.co/beats-dev/golang-crossbuild:1.17.5-darwin-arm64-debian10 --build-cmd build/mage-linux-amd64 buildDarwinUniversal -p darwin/arm64" failed with exit code 1
make[1]: *** [release] Error 1
make: *** [release] Error 1

I also try to use the mage buildDarwinUniversal in the x-pack/elastic-agent without luck?
Am I missing something ?

[ph]

Should have used request changes before for my last comment.

[AndersonQ]

@v1v

This feature will be enabled no just in ElasticAgent but in all the other Beats, is that correct?

Perhaps, I'll double check with the data plane

[AndersonQ]

@ph

@AndersonQ I've tried to build all the artifacts with make release in the top level directory and it fails on my side

That's probably because not all beats have the buildDarwinUniversal target. I'll check and add the missing ones

I also try to use the mage buildDarwinUniversal in the x-pack/elastic-agent without luck? Am I missing something ?

Probably, that's the catch of the approach I took. buildDarwinUniversal assumes that the darwin/amd64 and darwin/arm64 were built as this rule only merges them 2 into the universal binary. Perhaps another name instead of build... would be better.
I'm open to suggestions...

• createDarwinUniversal,
• mergeDarwinUniversal (I don't like merge, seems that DarwinUniversal will be merged into something),
• assembleDarwinUniversal (I think this is the best so far...)

[ph]

I think assemble is also more appropriate, I was using make release because this is what the release manager will do, and Agent and his subprocess Filebeat, metricbeat, etc need to be compiled as universal.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26683-agent-arm64 upstream/26683-agent-arm64
git merge upstream/master
git push upstream 26683-agent-arm64

[v1v]

I've got some questions:

1. Is this particular universal binary something to be released as part of the Unified Release process?
2. Is it required to use macosx with `docker?

Then, it's worth to involve the CI System and Release teams as they are the ones who provide:

• MacOS workers in the CI
• Configure docker in those workers
• Run the release process as part of the unified release process.

On the other hand, I tried to solve -> #29585 (comment) in #30505 but I'm now facing issues with the docker-machine not being available at runtime. But before doing anything else, we might need to get some clarity about the above questions. MacOSX workers are not bullet proof

[amolnater-qasource]

Hi @AndersonQ
There was some issue in copying the logs.

• We have revalidated adding endpoint security on M1 universal artifact agent.
• Agent directly goes to unhealthy state when endpoint security is added.

Please find below logs for the same using diagnostics command :
elastic-agent-diagnostics-2022-02-24T04-51-13Z-00.zip

Please let us know if anything else is required from our end.
Thanks

[AndersonQ]

@amolnater-qasource @dikshachauhan-qasource it's ready for another QA round, the new packages are on the same Google drive as before.

[dikshachauhan-qasource]

Hi @AndersonQ

We have revalidated all the artifacts shared with us on 8.1 kibana build and found that:

• With only system integration all the agents were shipping data to Kibana and were in healthy state, with filebeat and metricbeat binaries in running state. Test using :
  • .tar artifact on Mac 12
  • universal artifact on Mac 12
  • universal artifact on M1
  • aarch artifact on M1
• However, when endpoint security is added to agent it goes to unhealthy state.

Screenshot:

logs for M1 agent with aarch artifact : elastic-agent-diagnostics-2022-03-10T10-40-48Z-00.zip
error-logsfor M1withUniversal-artifact.txt

Build details:
BUILD: 50485

Thanks
QAS

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26683-agent-arm64 upstream/26683-agent-arm64
git merge upstream/main
git push upstream 26683-agent-arm64

[jasonrhodes]

@AndersonQ I see that @elastic/infra-monitoring-ui is added a codeowner reviewer here, but I don't see any files owned by us there. Can you help me understand why that might be the case? (Did it at one point include changes to the kibana, elasticsearch, logstash, or beats modules, for example?)

[AndersonQ]

Hi @jasonrhodes, you're correct, there is nothing you folks would need to review. I don't know why you were asked to review.

[belimawr]

LGTM, but I have a couple of questions.

[belimawr]

What is the plan here? Comment it in once CI is green and merge or do it in another PR?

[AndersonQ]

I ended up testing it on the elastic-agent repo, didn't do what I needed. Besides in order to unblock the release team, better to merge it now and adjust the CI later. But good catch anyway :)

[belimawr]

Why not use mage dependencies to ensure the binaries are built instead of failing?

[AndersonQ]

Mage dependencies will ensure a function was called, here I need more than just that. I need the build to have been invoked with the right environment variables, it has happened 2 times (once for each architecture). Thus it isn't really a solution here.

[ph]

I think @belimawr comment is good concerning the mage dependencies to ensure it work.
Other than that, I've tested this the associated beats branch and I was able to get the artifacts built. Correctly