Fixes to support Go 1.18

[cmacknz]

This PR is branched from the Go 1.18 update branch: #32493

This includes the fixes we'll need to support Go 1.18 in beats. There were some test failures in #32493 caused by the Kafka certificates being signed with SHA-1, which is deprecated in Go 1.18 per the release notes.

Rejecting SHA-1 certificates
crypto/x509 will now reject certificates signed with the SHA-1 hash function. This doesn't apply to self-signed root certificates. Practical attacks against SHA-1 have been demonstrated since 2017 and publicly trusted Certificate Authorities have not issued SHA-1 certificates since 2015.

This can be temporarily reverted by setting the GODEBUG=x509sha1=1 environment variable. This option will be removed in a future release

I've updated the certificates used by our Kafka Docker image to use SHA-256 signatures instead of SHA-1. I also had to update the base image to Debian buster to get a version of the JDK that was compatible with the version of the Java keytool I used on my local machine to generate the certs. Otherwise Kafka will fail with java.io.IOException: Invalid keystore format.

There are still a few TODO items before I can merge this, I want to see if the tests pass first:

TODO

• Add a changelog entry about SHA-1 signed certs being deprecated by the Go 1.18 upgrade, with the temporary workaround being running beats with GODEBUG=x509sha1=1.
• Try to get a better and obvious error message into the logs when certificate verification fails with Go 1.18. Right now we get the infamous "client has run out of brokers to talk to" from Sarama" when this error occurs.
• Upgrade the linter (and elastic-agent-libs) to make them compatible with Go 1.18

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-08-09T16:36:46.554+0000

• Duration: 125 min 16 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22536
Skipped	1937
Total	24473

Steps errors

Expand to view the steps failures

filebeat-unitTest - mage build unitTest

• Took 6 min 20 sec . View more details here
• Description: mage build unitTest

metricbeat-goIntegTest - mage goIntegTest

• Took 30 min 55 sec . View more details here
• Description: mage goIntegTest

Building Beats » Beats Packaging » PR-32611

• Took 36 min 41 sec . View more details here
• Description: null

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b update-go-1.18-fixes upstream/update-go-1.18-fixes
git merge upstream/update-go-version-20220725210948-main
git push upstream update-go-1.18-fixes