integrationservertest: APM server not ready in standalone test

[ericywl]

This issue has been observed on a few upgrade paths:

• 7.17 standalone -> 7.17 managed -> 8.19 managed (issue here)
• 7.17 standalone -> 8.19 standalone -> 8.19 managed -> 9.1 managed (issue here)
• 7.17 standalone -> 8.19 standalone -> 8.19 managed -> 9.2 managed (issue here)

During ingestion, the APM server cannot be contacted.

steps.go:179: 
            Error Trace:    /Users/ericyap/Repos/elastic/apm/apm-server/integrationservertest/steps.go:179
                                    /Users/ericyap/Repos/elastic/apm/apm-server/integrationservertest/steps.go:78
                                    /Users/ericyap/Repos/elastic/apm/apm-server/integrationservertest/standalone_test.go:49
            Error:          Received unexpected error:
                            cannot run generator: apm server not yet ready after 30s
            Test:           TestStandaloneManaged_7_17_to_8_x_to_9_x_Snapshot/Managed7

When trying to CURL the APM server, we get:

{"ok":false,"message":"The requested resource is currently unavailable."}

The Elastic APM agent policy looks to be active, but the APM server is not available.

[ericywl]

I have just tested the following and it seems to work fine. Issue most likely stems from the newer versions e.g. 8.19.

• 7.17 standalone -> 8.18 standalone -> 8.18 managed -> 9.0 managed
• 7.17 standalone -> 7.17 managed -> 8.18 managed -> 9.0 managed

[ericywl]

8.19

It seems this issue happens even just one 8.19.0-SNAPSHOT. I started a ECH deployment and APM server was immediately unavailable and stays that way until restart / reapply policy.

9.1

I also managed to reproduce this issue on 9.1.0-SNAPSHOT, but less consistently as compared to 8.19.

9.2

I created 4 9.2.0-SNAPSHOT deployments, but did not see the issue. Could also be the low sample size?

[ericywl]

To try to reproduce the issue, simply go to cloud.elastic.co CFT and create hosted deployment in either of those versions, and see that the APM server will be missing an endpoint.

Using GET .fleet-policies/_search on a good deployment and bad deployment, we can see that there are some differences in the policies. The bad deployment is missing elastic-cloud-apm which is crucial for the APM server Fleet integration. The issue can be "resolved" by either simply restarting the Integrations Server or re-applying the Elastic APM policy (e.g. updating the description and saving).

I have talked to the Elastic Agent and Fleet team to see if they know anything, pending replies.

[simitt]

Problem reported as likely to be a Fleet issue elastic/kibana#228374.

[ericywl]

Closing since it should be fixed via #17872. See elastic/fleet-server#5170 (comment).