[e2e] Add tests for self monitoring agent logs in otel mode

[khushijain21]

What does this PR do?

It adds tests for comparing docs ingested by agent.monitoring vs otel.yml

Notes:
Some changes made to otel.yml produced by hand here:

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

How to test this PR locally

Related issues

• Part of #4877

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b monitoringdocs upstream/monitoringdocs
git merge upstream/main
git push upstream monitoringdocs

[mergify]

This pull request does not have a backport label. Could you fix it @khushijain21? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[khushijain21]

The check-ci fails because integration tests here are running inside t.Run() blocks for readability and keeping sub tests isolated. Can we redefine this check to accommodate this?

The failure -

Running dependency: main.Integration.Check
>> check: Checking for define.Require in integration tests
Error: test TestAgentMonitoring first statement must be a function call to define.Require
make: *** [Makefile:29: check-ci] Error 1

cc: @elastic/ingest-eng-prod

[pchila]

The check-ci fails because integration tests here are running inside t.Run() blocks for readability and keeping sub tests isolated. Can we redefine this check to accommodate this?

The failure -

Running dependency: main.Integration.Check
>> check: Checking for define.Require in integration tests
Error: test TestAgentMonitoring first statement must be a function call to define.Require
make: *** [Makefile:29: check-ci] Error 1

The check for define.Require() at the beginning of the outer test function has been introduced because the initial integration test framework rely on it to skip the whole test during the discovery phase while dumping the requirements without any extra logs.
See

elastic-agent/pkg/testing/define/define_define.go

Line 15 in b011c25

func defineAction(t *testing.T, req Requirements) *Info {

and

elastic-agent/pkg/testing/define/batch.go

Line 81 in b011c25

func DetermineBatches(dir string, testFlags string, buildTags ...string) ([]Batch, error) {

The check itself is defined here

elastic-agent/magefile.go

Line 1951 in b011c25

func (Integration) Check() error {

and it parses the integration tests directory for *_test.go files containing Test* functions and validating that the first statement in the ast is a define.Require call

Your PR is not passing this check since the define.Require call has been wrapped into a t.Run() in TestAgentMonitoring, just move it out of t.Run and the check will pass. This is a limitation of the initial integration framework implementation that does not have full support for subtests with different requirements. Since you have different requirements across the subtests (like here and here ) you may need to split the TestAgentMonitoring test function into multiple ones and extract the define.Require at the Test* function level.

Maybe @blakerouse as the original implementer of the check and the batching may have another solution.

[pchila]

Add-on to my previous comment: after chatting with @khushijain21 it seems that the only difference between the 2 sub-tests is that this subtest must not be run on windows.

The same thing can be achieved by using a single define.Require() call as it was defined before this change and add a t.Skip() on the second sub-test if the OS is windows. It's not very elegant but it should achieve what you need without triggering the integration test mage check.

[cla-checker-service]

💚 CLA has been signed

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 92a728b

Failed CI Steps

• check-ci

History

• 💛 Build #18808 was flaky b4be3d9
• 💔 Build #18799 failed 7893827
• 💔 Build #18797 failed 49f85b8
• 💔 Build #18796 failed 54b9020
• 💔 Build #18794 failed c612e32
• 💔 Build #18500 failed f4dbd33

cc @khushijain21

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube