APM component gets removed and APM integration fails, causing Hosted Fleet Server to become unhealthy inconsistently.

[amolnater-qasource]

Deployment Links:

• https://cloud.elastic.co/
• Los Angeles (us-west2)

Kibana Build details:

VERSION: 9.1.0 BC5
BUILD: 88126
COMMIT: 7f1cd025e139e0f76e4d67cadfa6c5cf9826d65f

Description:

• The APM component is intermittently removed and the Hosted Fleet Server becomes unhealthy with inconsistent failures of the APM integration.

Steps to Reproduce:

1. Setup 9.1.0 BC5 Kibana cloud build.
2. Observe that the APM component is sometimes removed.
3. Notice that the Hosted Fleet Server enters an unhealthy state with failed APM integration.

Expected Behavior:

• The APM component should remain available.
• Hosted Fleet Server should remain healthy and APM integration shouldn't fail.

Screen Capture:

[amolnater-qasource]

@muskangulati-qasource Please review.

[simitt]

Seems to be a duplicate of elastic/kibana#228374 and likely fixed by elastic/kibana#228211
cc @juliaElastic

[muskangulati-qasource]

Secondary review is Done for this ticket!

[amolnater-qasource]

Hi Team,
While testing on 9.1.0 BC6 kibana cloud environment we have observed this issue inconsistently still reproducible.

Observations:

• APM component gets removed and APM integration fails, causing Hosted Fleet Server to become unhealthy inconsistently.

Screen Capture:

Build details:
VERSION: 9.1.0 BC6
BUILD: 88250
COMMIT: 9f30374092edd41719399f7ef81cb7ae78d8a3ab

Thanks!

[juliaElastic]

Hey @amolnater-qasource , could you share the agent diagnostics for the unhealthy fleet-server agent?
I'm not sure if the unhealthy agent is related to the APM server issue.

[juliaElastic]

I noticed something strange in Fleet Server logs, it seems like fleet-server acked an older policy revision after acking the newest one (revision 2 doesn't have the apm input yet), which might explain the apm server stopping.

https://platform-logging.kb.us-west2.gcp.elastic-cloud.com/app/r/s/Jh6aL

[amolnater-qasource]

Hi @juliaElastic

Please find below fleet server logs for the same:
elastic-agent-diagnostics-2025-07-28T11-51-36Z-00.zip

Thanks!

[juliaElastic]

Thanks, I see in the logs that fleet-server goes temporarily stopped state and the revision:2 handling is retried, maybe the retry is causing the handling in incorrect order.

{"log.level":"warn","@timestamp":"2025-07-28T04:29:09.082Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker/lazy.(*Acker).Commit","file.name":"lazy/lazy_acker.go","file.line":90},"message":"lazy acker: failed ack batch, enqueue for retry: [id: policy:policy-elastic-agent-on-cloud:2, type: POLICY_CHANGE]","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2025-07-28T04:29:07.612Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/gateway/fleet.(*FleetGateway).Run","file.name":"fleet/fleet_gateway.go","file.line":154},"message":"Fleet gateway started","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:08.222Z","message":"applying new components data","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","http.request.id":"01K17MN8FBQ5SHCAN9H1DABH0W","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","service.name":"fleet-server","service.type":"fleet-server","server.address":"127.0.0.1:8221","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:08.222Z","message":"force load on unknown policyId","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.type":"fleet-server","fleet.policy.id":"policy-elastic-agent-on-cloud","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","service.name":"fleet-server","ctx":"policy agent monitor","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:08.481Z","message":"New revision of policy received and added to the queue","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","ctx":"policy agent monitor","ecs.version":"1.6.0","fleet.policy.id":"policy-elastic-agent-on-cloud","fleet.revision_idx":2,"old_revision_idx":0,"nSubs":1,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:08.739Z","message":"generating output API key b29b81eb-0222-4dd2-835f-d8e960738b94:es-containerhost","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"fleet.policy.output.name":"es-containerhost","service.name":"fleet-server","http.request.id":"01K17MN8FBQ5SHCAN9H1DABH0W","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","fleet.policy.id":"policy-elastic-agent-on-cloud","ecs.version":"1.6.0","service.type":"fleet-server","server.address":"127.0.0.1:8221","fleet.ctx":"processPolicy","fleet.revision_idx":2,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:08.756Z","message":"Updating agent record to pick up default output key.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"server.address":"127.0.0.1:8221","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","fleet.revision_idx":2,"fleet.policy.id":"policy-elastic-agent-on-cloud","service.type":"fleet-server","http.request.id":"01K17MN8FBQ5SHCAN9H1DABH0W","fleet.policy.role.hash.sha256":"23744d0c57ba438f592489519f1515f2525f5d62cfea6108c11e56bafe2e6678","fleet.default.apikey.id":"rDBKT5gB-5nIJ01cpZUF","fleet.ctx":"processPolicy","fleet.policy.output.name":"es-containerhost","ecs.version":"1.6.0","service.name":"fleet-server","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.045Z","message":"Action delivered to agent on checkin","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"createdAt":"2025-07-28T04:28:45.732Z","fleet.action.id":"policy:policy-elastic-agent-on-cloud:2","inputType":"","service.type":"fleet-server","http.request.id":"01K17MN8FBQ5SHCAN9H1DABH0W","fleet.action.type":"POLICY_CHANGE","timeout":0,"ecs.version":"1.6.0","ackToken":"","service.name":"fleet-server","server.address":"127.0.0.1:8221","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.060Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/actions/handlers.(*PolicyChangeHandler).applyLoggingConfig","file.name":"handlers/handler_action_policy_change.go","file.line":405},"message":"Setting fallback log level <nil> from policy","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.067Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).filterByCapabilities","file.name":"coordinator/coordinator.go","file.line":1844},"message":"Component 'filestream-monitoring' with input type 'filestream' filtered by capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.067Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).filterByCapabilities","file.name":"coordinator/coordinator.go","file.line":1844},"message":"Component 'beat/metrics-monitoring' with input type 'beat/metrics' filtered by capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.067Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).filterByCapabilities","file.name":"coordinator/coordinator.go","file.line":1844},"message":"Component 'http/metrics-monitoring' with input type 'http/metrics' filtered by capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.068Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1831},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"count":1},"outputs":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.068Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1533},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.071Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":135},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.071Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Reload","file.name":"reload/reload.go","file.line":107},"message":"http monitoring server is enabled in hard-coded config, but HTTP config is unset. Leaving enabled.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.071Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.071Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint ([::]:6791) finished: accept tcp [::]:6791: use of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.072Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:false, MonitorMetrics:false, MetricsPeriod:\"\", FailureThreshold:(*uint)(nil), LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc0011986f0), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:true, APM:config.APMConfig{Environment:\"production\", APIKey:\"\", SecretToken:\"m4KjsiPiBAz0w7wW81\", Hosts:[]string{\"https://7831d021b36b4d53b849aae988d3b6db.apm.us-west2.gcp.elastic-cloud.com:443\"}, GlobalLabels:map[string]string{\"deploymentId\":\"688a1b5f9fff4af38aba5ec10493f743\", \"isElasticUser\":\"true\", \"organizationId\":\"1688146396\", \"subscriptionLevel\":\"standard\"}, TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(0xc00160e3fc)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}, RuntimeManager:\"process\"}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.072Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":71},"message":"process monitoring is enabled, creating monitoring endpoints","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.072Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://0.0.0.0:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.072Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.072Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: [::]:6791 (configured: http://0.0.0.0:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.073Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1831},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"added":["fleet-server-es-containerhost"],"removed":["fleet-server-default"],"count":1},"outputs":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.073Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1533},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.073Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","sig":"terminated","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.073Z","message":"Stopping","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","state":"STOPPING","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-07-28T04:29:09.073Z","message":"failed to fetch elasticsearch version","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.type":"fleet-server","error.message":"context canceled","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-07-28T04:29:09.074Z","message":"server.Run: error while closing listener.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"close tcp [::]:8220: use of closed network connection","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.074Z","message":"context closed waiting for global checkpoints advance","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-policies","ctx":"index monitor","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.074Z","message":"index monitor exited","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-policies","ctx":"index monitor","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.074Z","message":"context closed waiting for global checkpoints advance","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-actions","ctx":"index monitor","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.074Z","message":"index monitor exited","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-actions","ctx":"index monitor","ecs.version":"1.6.0"}

{"log.level":"warn","@timestamp":"2025-07-28T04:29:09.074Z","message":"server.Run: error while closing listener.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","error.message":"close tcp 127.0.0.1:8221: use of closed network connection","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.074Z","message":"Fleet Server exited","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.074Z","message":"Stats endpoint (/app/elastic-agent/data/tmp/UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock) finished: accept unix /app/elastic-agent/data/tmp/UwGGXFL1il700DVAc6q-T-1Z9J1UjGMU.sock: use of closed network connection","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","log.logger":"fleet-metrics.api","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-07-28T04:29:09.077Z","message":"Agent wrapper received error.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","error.message":"rpc error: code = Canceled desc = context canceled","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-07-28T04:29:09.077Z","message":"Agent wrapper received error.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","error.message":"rpc error: code = Canceled desc = context canceled","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.080Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":771},"message":"Component state changed fleet-server-default (HEALTHY->STOPPED): Stopped: pid '145' exited with code '0'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STOPPED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.080Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-default (HEALTHY->STOPPED): Stopped: pid '145' exited with code '0'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STOPPED"},"unit":{"id":"fleet-server-default","type":"output","state":"STOPPED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.080Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-default-fleet-server (HEALTHY->STOPPED): Stopped: pid '145' exited with code '0'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STOPPED"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STOPPED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-07-28T04:29:09.082Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/fleetapi/acker/lazy.(*Acker).Commit","file.name":"lazy/lazy_acker.go","file.line":90},"message":"lazy acker: failed ack batch, enqueue for retry: [id: policy:policy-elastic-agent-on-cloud:2, type: POLICY_CHANGE]","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2025-07-28T04:29:09.436Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":750},"message":"Spawned new component fleet-server-es-containerhost: Starting: spawned pid '167'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.436Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":758},"message":"Spawned new unit fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server: Starting: spawned pid '167'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"STARTING"},"unit":{"id":"fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.436Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":758},"message":"Spawned new unit fleet-server-es-containerhost: Starting: spawned pid '167'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"STARTING"},"unit":{"id":"fleet-server-es-containerhost","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.456Z","message":"Boot fleet-server","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"args":["--agent-mode","-E","logging.level=debug","-E","logging.to_stderr=true","-E","http.enabled=true","-E","http.host=unix:///app/elastic-agent/data/tmp/4JIKo-kle_EWpXrqRJaRmFDMOzx6aIg9.sock","-E","path.data=/app/elastic-agent/data/run/fleet-server-es-containerhost"],"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","version":"9.1.0","commit":"2501f452","fips-distribution":false,"pid":167,"ppid":18,"exe":"/usr/share/elastic-agent/data/elastic-agent-2bcf7b/components/fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.457Z","message":"starting communication connection back to Elastic Agent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.463Z","log.logger":"component.runtime.fleet-server-es-containerhost","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":693},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.463Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":771},"message":"Component state changed fleet-server-es-containerhost (STARTING->HEALTHY): Healthy: communicating with pid '167'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.471Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"9.1.0","elasticsearch_version":"9.1.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.473Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","memory_mb":68420,"recommended_mb":32384,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.473Z","message":"Setting cache config options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","cfg":{"APIKeyJitter":300000000000,"APIKeyTTL":900000000000,"ActionTTL":300000000000,"ArtifactTTL":86400000000000,"EnrollKeyTTL":60000000000,"MaxCost":536870912,"NumCounters":6400000},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.477Z","message":"Starting","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.477Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","memory_mb":68420,"recommended_mb":32384,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.477Z","message":"initial server configuration","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.477Z","message":"starting server on configuration change","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.477Z","message":"SetGCPercent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","old":20,"new":20,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.477Z","message":"fleet-server instrumentation is enabled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.479Z","message":"Starting stats endpoint","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","log.logger":"fleet-metrics.api","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.479Z","message":"Metrics endpoint listening on: /app/elastic-agent/data/tmp/4JIKo-kle_EWpXrqRJaRmFDMOzx6aIg9.sock (configured: unix:///app/elastic-agent/data/tmp/4JIKo-kle_EWpXrqRJaRmFDMOzx6aIg9.sock)","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"log.logger":"fleet-metrics.api","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.480Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.type":"fleet-server","opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"@timestamp":"2025-07-28T04:29:09.48Z","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.486Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"9.1.0","elasticsearch_version":"9.1.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.492Z","message":"migration AgentMetadata done","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"fleet.migration.total.count":0,"fleet.migration.noops":0,"fleet.migration.deleted":0,"fleet.migration.total.duration":5.949031,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet.migration.es.timed_out":false,"fleet.migration.updated":0,"fleet.migration.version_conflicts":0,"fleet.migration.retries.bulk":0,"fleet.migration.name":"AgentMetadata","fleet.migration.es.took":2,"fleet.migration.batches":0,"fleet.migration.retries.search":0,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.496Z","message":"migration AgentOutputs done","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.name":"fleet-server","service.type":"fleet-server","fleet.migration.name":"AgentOutputs","fleet.migration.es.timed_out":false,"fleet.migration.deleted":0,"fleet.migration.total.duration":3.892576,"ecs.version":"1.6.0","fleet.migration.updated":0,"fleet.migration.batches":0,"fleet.migration.total.count":0,"fleet.migration.es.took":1,"fleet.migration.noops":0,"fleet.migration.version_conflicts":0,"fleet.migration.retries.bulk":0,"fleet.migration.retries.search":0,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.498Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","index":".fleet-policies","ctx":"index monitor","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.498Z","message":"run policy monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.name":"fleet-server","ctx":"policy agent monitor","burst":1,"event_rate":4000,"ecs.version":"1.6.0","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.498Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.name":"fleet-server","service.type":"fleet-server","index":".fleet-actions","ctx":"index monitor","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.499Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.499Z","message":"Listening on localhost:8221","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.758Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"state":"HEALTHY","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.759Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server (STARTING->HEALTHY): Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server","type":"input","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:09.759Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost (STARTING->HEALTHY): Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost","type":"output","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:10.467Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"9.1.0","elasticsearch_version":"9.1.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:10.468Z","message":"Server configuration update","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.type":"fleet-server","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:10.468Z","message":"Re-configuring","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"CONFIGURING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:10.468Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"memory_mb":68420,"recommended_mb":32384,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:10.468Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server (HEALTHY->CONFIGURING): Re-configuring","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server","type":"input","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:10.469Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost (HEALTHY->CONFIGURING): Re-configuring","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost","type":"output","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:10.686Z","message":"applying new components data","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"trace.id":"0619209ffa61a84d0cc663fd5286e761","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","http.request.id":"01K17MNAWM3VY8NVVYEJ8AWRDB","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","transaction.id":"0619209ffa61a84d","server.address":"127.0.0.1:8221","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:10.686Z","message":"force load on unknown policyId","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ctx":"policy agent monitor","fleet.policy.id":"policy-elastic-agent-on-cloud","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:10.945Z","message":"New revision of policy received and added to the queue","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.name":"fleet-server","fleet.revision_idx":2,"ecs.version":"1.6.0","service.type":"fleet-server","ctx":"policy agent monitor","fleet.policy.id":"policy-elastic-agent-on-cloud","old_revision_idx":0,"nSubs":1,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.201Z","message":"Action delivered to agent on checkin","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"createdAt":"2025-07-28T04:28:45.732Z","fleet.action.id":"policy:policy-elastic-agent-on-cloud:2","fleet.action.type":"POLICY_CHANGE","http.request.id":"01K17MNAWM3VY8NVVYEJ8AWRDB","server.address":"127.0.0.1:8221","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","inputType":"","service.type":"fleet-server","trace.id":"0619209ffa61a84d0cc663fd5286e761","transaction.id":"0619209ffa61a84d","timeout":0,"ecs.version":"1.6.0","service.name":"fleet-server","ackToken":"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.210Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/actions/handlers.(*PolicyChangeHandler).applyLoggingConfig","file.name":"handlers/handler_action_policy_change.go","file.line":405},"message":"Setting fallback log level <nil> from policy","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.213Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1831},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"count":1},"outputs":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.213Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1533},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.221Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":135},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.221Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Reload","file.name":"reload/reload.go","file.line":107},"message":"http monitoring server is enabled in hard-coded config, but HTTP config is unset. Leaving enabled.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.221Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.221Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint ([::]:6791) finished: accept tcp [::]:6791: use of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.221Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:false, MonitorMetrics:false, MetricsPeriod:\"\", FailureThreshold:(*uint)(nil), LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc0011986f0), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:true, APM:config.APMConfig{Environment:\"production\", APIKey:\"\", SecretToken:\"m4KjsiPiBAz0w7wW81\", Hosts:[]string{\"https://7831d021b36b4d53b849aae988d3b6db.apm.us-west2.gcp.elastic-cloud.com:443\"}, GlobalLabels:map[string]string{\"deploymentId\":\"688a1b5f9fff4af38aba5ec10493f743\", \"isElasticUser\":\"true\", \"organizationId\":\"1688146396\", \"subscriptionLevel\":\"standard\"}, TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(0xc00155318c)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}, RuntimeManager:\"process\"}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.221Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":71},"message":"process monitoring is enabled, creating monitoring endpoints","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.221Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://0.0.0.0:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.221Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.221Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: [::]:6791 (configured: http://0.0.0.0:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.222Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1831},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"count":1},"outputs":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.222Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1533},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.478Z","message":"ack event","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.name":"fleet-server","service.type":"fleet-server","http.request.id":"01K17MNBNQPZS0Y7PCADTVYC5H","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","transaction.id":"c79a2ce241ec9287","ecs.version":"1.6.0","server.address":"127.0.0.1:8221","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","fleet.action.id":"policy:policy-elastic-agent-on-cloud:2","n":0,"trace.id":"c79a2ce241ec9287c98b8fed6e85e135","nEvents":1,"timestamp":"2025-07-28T04:29:11.223Z","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:11.768Z","message":"ack policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"http.request.id":"01K17MNBNQPZS0Y7PCADTVYC5H","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","service.name":"fleet-server","fleet.policy.id":"policy-elastic-agent-on-cloud","policyRevision":2,"fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","nEvents":1,"trace.id":"c79a2ce241ec9287c98b8fed6e85e135","transaction.id":"c79a2ce241ec9287","service.type":"fleet-server","server.address":"127.0.0.1:8221","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:13.027Z","message":"applying new components data","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"transaction.id":"6eddc8dd1ba01c71","service.name":"fleet-server","http.request.id":"01K17MND62MGTD4XJMAKTC4W1D","server.address":"127.0.0.1:8221","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","ecs.version":"1.6.0","service.type":"fleet-server","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","trace.id":"6eddc8dd1ba01c714a260f18359d6872","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:15.027Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:15.027Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:15.027Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost","type":"output","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:20.284Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:25.544Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:30.814Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:36.084Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:41.365Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:46.625Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:51.883Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"state":"HEALTHY","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:56.746Z","message":"received notification","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ctx":"subscription monitor","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:56.746Z","message":"received notification","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ctx":"subscription monitor","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:56.747Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:56.747Z","message":"New revision of policy received and added to the queue","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.name":"fleet-server","ctx":"policy agent monitor","ecs.version":"1.6.0","service.type":"fleet-server","fleet.policy.id":"policy-elastic-agent-on-cloud","fleet.revision_idx":3,"old_revision_idx":2,"nSubs":1,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:57.018Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.520Z","message":"Action delivered to agent on checkin","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","trace.id":"6eddc8dd1ba01c714a260f18359d6872","ecs.version":"1.6.0","service.name":"fleet-server","ackToken":"","transaction.id":"6eddc8dd1ba01c71","http.request.id":"01K17MND62MGTD4XJMAKTC4W1D","server.address":"127.0.0.1:8221","createdAt":"2025-07-28T04:29:55.865Z","fleet.action.type":"POLICY_CHANGE","timeout":0,"service.type":"fleet-server","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","fleet.action.id":"policy:policy-elastic-agent-on-cloud:3","inputType":"","@timestamp":"2025-07-28T04:29:58.52Z","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.530Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/actions/handlers.(*PolicyChangeHandler).applyLoggingConfig","file.name":"handlers/handler_action_policy_change.go","file.line":405},"message":"Setting fallback log level <nil> from policy","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.531Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1831},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"count":1},"outputs":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.531Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1533},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.539Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":135},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.539Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Reload","file.name":"reload/reload.go","file.line":107},"message":"http monitoring server is enabled in hard-coded config, but HTTP config is unset. Leaving enabled.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.539Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.539Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint ([::]:6791) finished: accept tcp [::]:6791: use of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.539Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:false, MonitorMetrics:false, MetricsPeriod:\"\", FailureThreshold:(*uint)(nil), LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc0011986f0), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:true, APM:config.APMConfig{Environment:\"production\", APIKey:\"\", SecretToken:\"m4KjsiPiBAz0w7wW81\", Hosts:[]string{\"https://7831d021b36b4d53b849aae988d3b6db.apm.us-west2.gcp.elastic-cloud.com:443\"}, GlobalLabels:map[string]string{\"deploymentId\":\"688a1b5f9fff4af38aba5ec10493f743\", \"isElasticUser\":\"true\", \"organizationId\":\"1688146396\", \"subscriptionLevel\":\"standard\"}, TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(0xc0012dba3c)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}, RuntimeManager:\"process\"}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.539Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":71},"message":"process monitoring is enabled, creating monitoring endpoints","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.539Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://0.0.0.0:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.539Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.539Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: [::]:6791 (configured: http://0.0.0.0:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.543Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1831},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"added":["apm-es-containerhost"],"updated":["fleet-server-es-containerhost: [(fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server: updated)]"],"count":2},"outputs":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.543Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1533},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.552Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.name":"fleet-server","service.type":"fleet-server","fleet_version":"9.1.0","elasticsearch_version":"9.1.0","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.554Z","message":"Server configuration update","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.554Z","message":"Re-configuring","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"state":"CONFIGURING","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.554Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"memory_mb":68420,"recommended_mb":32384,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.554Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost (HEALTHY->CONFIGURING): Re-configuring","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost","type":"output","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.554Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server (HEALTHY->CONFIGURING): Re-configuring","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server","type":"input","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.682Z","message":"Home path: [/usr/share/elastic-agent/data/elastic-agent-2bcf7b/components] Config path: [/usr/share/elastic-agent/data/elastic-agent-2bcf7b/components] Data path: [/app/elastic-agent/data/run/apm-es-containerhost] Logs path: [/usr/share/elastic-agent/data/elastic-agent-2bcf7b/components/logs]","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":176,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.683Z","message":"Beat ID: 0ff8ccd5-a6ad-47fc-9ea9-fa2e70139c06","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":183,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.684Z","message":"Set gc percentage to: 70","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.origin":{"file.line":197,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.686Z","message":"Starting stats endpoint","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":65,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.686Z","message":"Metrics endpoint listening on: /app/elastic-agent/data/tmp/pIyhBDI2yYjI1X9DivNwGWlM9zvlDDb9.sock (configured: unix:///app/elastic-agent/data/tmp/pIyhBDI2yYjI1X9DivNwGWlM9zvlDDb9.sock)","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"api","log.origin":{"file.line":67,"file.name":"api/server.go","function":"github.com/elastic/beats/v7/libbeat/api.(*Server).Start.func1"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.687Z","message":"GOMEMLIMIT is updated","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","service.name":"apm-server","package":"github.com/KimMachineGun/automemlimit/memlimit","GOMEMLIMIT":966367641,"previous":9223372036854776000,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.687Z","message":"maxprocs: Updating GOMAXPROCS=8: determined from CPU quota","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":68,"file.name":"beatcmd/maxprocs.go","function":"github.com/elastic/apm-server/internal/beatcmd.adjustMaxProcs.diffInfof.func2"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.687Z","message":"Beat info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/elastic-agent/data/elastic-agent-2bcf7b/components","data":"/app/elastic-agent/data/run/apm-es-containerhost","home":"/usr/share/elastic-agent/data/elastic-agent-2bcf7b/components","logs":"/usr/share/elastic-agent/data/elastic-agent-2bcf7b/components/logs"},"type":"apm-server","uuid":"0ff8ccd5-a6ad-47fc-9ea9-fa2e70139c06"},"ecs.version":"1.6.0"},"log.logger":"beat","log.origin":{"file.line":861,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.687Z","message":"Build info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beat","log.origin":{"file.line":869,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"service.name":"apm-server","system_info":{"build":{"commit":"9f6812983335fb3d84e0f7100f96a50ff98914b9","time":"2025-07-23T15:54:36.000Z","version":"9.1.0"},"ecs.version":"1.6.0"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.687Z","message":"Go runtime info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beat","log.origin":{"file.line":872,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"service.name":"apm-server","system_info":{"ecs.version":"1.6.0","go":{"arch":"amd64","max_procs":8,"os":"linux","version":"go1.24.5"}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.688Z","message":"Host info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","system_info":{"ecs.version":"1.6.0","host":{"architecture":"x86_64","boot_time":"2024-11-28T00:02:36Z","containerized":true,"ip":["127.0.0.1/8","172.17.0.33/16"],"kernel_version":"5.15.0-1060-gcp","mac":["02:42:ac:11:00:21"],"name":"d738efc8a16f","native_architecture":"amd64","os":{"family":"","major":20230201,"minor":0,"name":"Wolfi","patch":0,"platform":"wolfi","type":"linux","version":"20230201"},"timezone":"UTC","timezone_offset_sec":0}},"log.logger":"beat","log.origin":{"file.line":876,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.688Z","message":"Process info","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"system_info":{"ecs.version":"1.6.0","process":{"capabilities":{"ambient":null,"bounding":["chown","dac_override","kill","setgid","setuid"],"effective":null,"inheritable":null,"permitted":null},"cwd":"/app/elastic-agent/data/run/apm-es-containerhost","exe":"/usr/share/elastic-agent/data/elastic-agent-2bcf7b/components/apm-server","name":"apm-server","pid":190,"ppid":18,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2025-07-28T04:29:58.200Z"}},"log.logger":"beat","log.origin":{"file.line":905,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.692Z","message":"apm-server started.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.origin":{"file.line":432,"file.name":"beatcmd/beat.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).Run"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.698Z","log.logger":"component.runtime.apm-es-containerhost","log.origin":{"function":"github.com/elastic/elastic-agent/pkg/component/runtime.(*Manager).CheckinV2","file.name":"runtime/manager.go","file.line":693},"message":"control checkin v2 protocol has chunking enabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.700Z","message":"BeatV2Manager.unitListen UnitChanged.ID(apm-es-containerhost-elastic-cloud-apm), UnitChanged.Type(added), UnitChanged.Trigger(36): added/feature_change_triggered, apm_config_change_triggered","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":507,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.700Z","message":"BeatV2Manager.unitListen UnitChanged.ID(apm-es-containerhost), UnitChanged.Type(added), UnitChanged.Trigger(36): added/feature_change_triggered, apm_config_change_triggered","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"centralmgmt.V2-manager","log.origin":{"file.line":507,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.774Z","message":"received notification","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ctx":"subscription monitor","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.774Z","message":"received notification","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ctx":"subscription monitor","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.774Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.774Z","message":"New revision of policy received and added to the queue","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ctx":"policy agent monitor","fleet.policy.id":"policy-elastic-agent-on-cloud","old_revision_idx":3,"fleet.revision_idx":4,"nSubs":0,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:58.805Z","message":"ack event","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"fleet.action.id":"policy:policy-elastic-agent-on-cloud:3","transaction.id":"a1d82359b1efbbc8","ecs.version":"1.6.0","service.type":"fleet-server","http.request.id":"01K17MPSWQQSJ7878BZNAV2M59","server.address":"127.0.0.1:8221","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","n":0,"service.name":"fleet-server","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","nEvents":1,"timestamp":"2025-07-28T04:29:58.551Z","trace.id":"a1d82359b1efbbc8e8d914892d307508","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.700Z","message":"loaded output config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.origin":{"file.line":200,"file.name":"beatcmd/reloader.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reloadOutput"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.700Z","message":"loaded apm tracing config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":216,"file.name":"beatcmd/reloader.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reloadAPMTracing"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.700Z","message":"received input from elastic-agent","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.origin":{"file.line":59,"file.name":"apm-server/main.go","function":"main.init.0.func1"},"service.name":"apm-server","unit.id":"elastic-cloud-apm","agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","agent.version":"9.1.0","unit.revision":2,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.705Z","message":"using output.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"config","log.origin":{"file.line":67,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.705Z","message":"using apm-server.agent.config.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"config","log.origin":{"file.line":73,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.705Z","message":"Falling back to elasticsearch output for API Key usage","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"config","log.origin":{"file.line":75,"file.name":"config/auth.go","function":"github.com/elastic/apm-server/internal/beater/config.(*APIKeyAgentAuth).setup"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.705Z","message":"Listening on: [::]:8200","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":142,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.listen"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.705Z","message":"loaded input config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":186,"file.name":"beatcmd/reloader.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reloadInputs"},"service.name":"apm-server","revision":2,"ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.707Z","message":"MaxConcurrentDecoders set to 102 based on 80 percent of 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":221,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.707Z","message":"Aggregation.MaxServices set to 1000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":228,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.707Z","message":"Aggregation.ServiceTransactions.MaxGroups for service aggregation set to 1000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":235,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.707Z","message":"Aggregation.Transactions.MaxGroups set to 5000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":242,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.707Z","message":"Aggregation.ServiceDestinations.MaxGroups set to 10000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":249,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.707Z","message":"APM instrumentation is enabled","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater.tracing","log.origin":{"file.line":171,"file.name":"instrumentation/instrumentation.go","function":"github.com/elastic/beats/v7/libbeat/instrumentation.initTracer"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.710Z","message":"blocking ingestion until all preconditions are satisfied","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":40,"file.name":"beater/waitready.go","function":"github.com/elastic/apm-server/internal/beater.waitReady"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.710Z","message":"docappender.DocumentBufferSize set to 819 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":828,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.docappenderConfig"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.710Z","message":"docappender.MaxRequests set to 11 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":845,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.docappenderConfig"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.715Z","message":"Path / added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater.handler","log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.715Z","message":"Path /config/v1/agents added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater.handler","log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.715Z","message":"Path /config/v1/rum/agents added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater.handler","log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.716Z","message":"Path /intake/v2/rum/events added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater.handler","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.716Z","message":"Path /intake/v3/rum/events added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater.handler","log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.716Z","message":"Path /intake/v2/events added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater.handler","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.716Z","message":"Path /v1/traces added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater.handler","log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.716Z","message":"Path /v1/metrics added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater.handler","log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.716Z","message":"Path /v1/logs added to request handler","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater.handler","log.origin":{"file.line":146,"file.name":"api/mux.go","function":"github.com/elastic/apm-server/internal/beater/api.NewMux"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.716Z","message":"Starting apm-server [9f6812983335fb3d84e0f7100f96a50ff98914b9 built 2025-07-23 17:54:36 +0200 +0200]. Hit CTRL-C to stop it.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":224,"file.name":"beater/server.go","function":"github.com/elastic/apm-server/internal/beater.server.run"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.716Z","message":"RUM endpoints enabled!","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":86,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-07-28T04:29:59.716Z","message":"CORS related setting `apm-server.rum.allow_origins` allows all origins. Consider more restrictive setting for production use.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":89,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.716Z","message":"SSL enabled.","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":98,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.(*httpServer).start"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.749Z","message":"no longer blocking ingestion as all precondition checks are now satisfied","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":68,"file.name":"beater/waitready.go","function":"github.com/elastic/apm-server/internal/beater.waitReady"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.820Z","message":"ack policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","@timestamp":"2025-07-28T04:29:59.82Z","http.request.id":"01K17MPSWQQSJ7878BZNAV2M59","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","fleet.policy.id":"policy-elastic-agent-on-cloud","policyRevision":3,"service.type":"fleet-server","server.address":"127.0.0.1:8221","transaction.id":"a1d82359b1efbbc8","service.name":"fleet-server","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","nEvents":1,"trace.id":"a1d82359b1efbbc8e8d914892d307508","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":750},"message":"Spawned new component apm-es-containerhost: Starting: spawned pid '190'","log":{"source":"elastic-agent"},"component":{"id":"apm-es-containerhost","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":758},"message":"Spawned new unit apm-es-containerhost: Starting: spawned pid '190'","log":{"source":"elastic-agent"},"component":{"id":"apm-es-containerhost","state":"STARTING"},"unit":{"id":"apm-es-containerhost","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":758},"message":"Spawned new unit apm-es-containerhost-elastic-cloud-apm: Starting: spawned pid '190'","log":{"source":"elastic-agent"},"component":{"id":"apm-es-containerhost","state":"STARTING"},"unit":{"id":"apm-es-containerhost-elastic-cloud-apm","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost","type":"output","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":771},"message":"Component state changed apm-es-containerhost (STARTING->HEALTHY): Healthy: communicating with pid '190'","log":{"source":"elastic-agent"},"component":{"id":"apm-es-containerhost","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed apm-es-containerhost (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"apm-es-containerhost","state":"HEALTHY"},"unit":{"id":"apm-es-containerhost","type":"output","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:29:59.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed apm-es-containerhost-elastic-cloud-apm (STARTING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"apm-es-containerhost","state":"HEALTHY"},"unit":{"id":"apm-es-containerhost-elastic-cloud-apm","type":"input","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.177Z","message":"applying new components data","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"service.type":"fleet-server","server.address":"127.0.0.1:8221","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","trace.id":"5fb46ea118b91d29ddaf2a378c37fdb5","transaction.id":"5fb46ea118b91d29","ecs.version":"1.6.0","service.name":"fleet-server","http.request.id":"01K17MPV7HEA8453BTGH14VBCF","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.807Z","message":"Action delivered to agent on checkin","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","fleet.action.type":"POLICY_CHANGE","service.type":"fleet-server","http.request.id":"01K17MPV7HEA8453BTGH14VBCF","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","inputType":"","timeout":0,"trace.id":"5fb46ea118b91d29ddaf2a378c37fdb5","server.address":"127.0.0.1:8221","ackToken":"","createdAt":"2025-07-28T04:29:58.115Z","service.name":"fleet-server","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","fleet.action.id":"policy:policy-elastic-agent-on-cloud:4","transaction.id":"5fb46ea118b91d29","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.816Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/actions/handlers.(*PolicyChangeHandler).applyLoggingConfig","file.name":"handlers/handler_action_policy_change.go","file.line":405},"message":"Setting fallback log level <nil> from policy","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1831},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"count":2},"outputs":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.820Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1533},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.826Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/upgrade.(*Upgrader).Reload","file.name":"upgrade/upgrade.go","file.line":135},"message":"Source URI changed from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.827Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Reload","file.name":"reload/reload.go","file.line":107},"message":"http monitoring server is enabled in hard-coded config, but HTTP config is unset. Leaving enabled.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.827Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Stop","file.name":"reload/reload.go","file.line":74},"message":"Stopping monitoring server","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.827Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":90},"message":"Stats endpoint ([::]:6791) finished: accept tcp [::]:6791: use of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.827Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring/reload.(*ServerReloader).Start","file.name":"reload/reload.go","file.line":54},"message":"Starting monitoring server with cfg &config.MonitoringConfig{Enabled:true, MonitorLogs:false, MonitorMetrics:false, MetricsPeriod:\"\", FailureThreshold:(*uint)(nil), LogMetrics:true, HTTP:(*config.MonitoringHTTPConfig)(0xc0011986f0), Namespace:\"default\", Pprof:(*config.PprofConfig)(nil), MonitorTraces:true, APM:config.APMConfig{Environment:\"production\", APIKey:\"\", SecretToken:\"m4KjsiPiBAz0w7wW81\", Hosts:[]string{\"https://7831d021b36b4d53b849aae988d3b6db.apm.us-west2.gcp.elastic-cloud.com:443\"}, GlobalLabels:map[string]string{\"deploymentId\":\"688a1b5f9fff4af38aba5ec10493f743\", \"isElasticUser\":\"true\", \"organizationId\":\"1688146396\", \"subscriptionLevel\":\"standard\"}, TLS:config.APMTLS{SkipVerify:false, ServerCertificate:\"\", ServerCA:\"\"}, SamplingRate:(*float32)(0xc00155310c)}, Diagnostics:config.Diagnostics{Uploader:config.Uploader{MaxRetries:10, InitDur:1000000000, MaxDur:600000000000}, Limit:config.Limit{Interval:60000000000, Burst:1}}, RuntimeManager:\"process\"}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.827Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":71},"message":"process monitoring is enabled, creating monitoring endpoints","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.827Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/monitoring.NewServer.exposeMetricsEndpoint.func1","file.name":"monitoring/server.go","file.line":96},"message":"creating monitoring API with cfg api.Config{Enabled:true, Host:\"http://0.0.0.0:6791\", Port:6791, User:\"\", SecurityDescriptor:\"\", Timeout:5000000000}","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.827Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start","file.name":"api/server.go","file.line":85},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.827Z","log.logger":"api","log.origin":{"function":"github.com/elastic/elastic-agent-libs/api.(*Server).Start.func1","file.name":"api/server.go","file.line":87},"message":"Metrics endpoint listening on: [::]:6791 (configured: http://0.0.0.0:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.832Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).checkAndLogUpdate","file.name":"coordinator/coordinator.go","file.line":1831},"message":"component model updated","log":{"source":"elastic-agent"},"changes":{"components":{"updated":["fleet-server-es-containerhost: [(fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server: updated)]","apm-es-containerhost: [(apm-es-containerhost-elastic-cloud-apm: updated)]"],"count":2},"outputs":{}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.832Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).refreshComponentModel","file.name":"coordinator/coordinator.go","file.line":1533},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.833Z","message":"BeatV2Manager.unitListen UnitChanged.ID(apm-es-containerhost-elastic-cloud-apm), UnitChanged.Type(modified), UnitChanged.Trigger(2): modified/config_change_triggered","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.origin":{"file.line":507,"file.name":"management/managerV2.go","function":"github.com/elastic/beats/v7/x-pack/libbeat/management.(*BeatV2Manager).unitListen"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"centralmgmt.V2-manager","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.833Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed apm-es-containerhost-elastic-cloud-apm (HEALTHY->CONFIGURING): Configuring","log":{"source":"elastic-agent"},"component":{"id":"apm-es-containerhost","state":"HEALTHY"},"unit":{"id":"apm-es-containerhost-elastic-cloud-apm","type":"input","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.861Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","fleet_version":"9.1.0","elasticsearch_version":"9.1.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.864Z","message":"Server configuration update","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.864Z","message":"Re-configuring","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"CONFIGURING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:00.864Z","message":"Found settings with recommended ram.","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"recommended_mb":32384,"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","memory_mb":68420,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.055Z","message":"request ok","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","url.original":"/","http.request.body.bytes":0,"log.logger":"beater.handler.request","http.request.method":"GET","user_agent.original":"pekko-http/1.1.0","source.address":"10.47.192.73","event.duration":187527,"http.response.status_code":200,"log.origin":{"file.line":63,"file.name":"middleware/log_middleware.go","function":"github.com/elastic/apm-server/internal/beater/api.apmMiddleware.LogMiddleware.func1.1"},"http.request.id":"fa2b73f3-107d-481f-9642-d1fba028e97f","ecs.version":"1.6.0","ecs.version":"1.6.0"}


{"log.level":"info","@timestamp":"2025-07-28T04:30:01.159Z","message":"ack event","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"n":0,"ecs.version":"1.6.0","service.name":"fleet-server","http.request.id":"01K17MPW48G6NABYTJT7BJ994R","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","nEvents":1,"fleet.action.id":"policy:policy-elastic-agent-on-cloud:4","service.type":"fleet-server","transaction.id":"2f23ff151429d4ad","trace.id":"2f23ff151429d4adb9a5390842b88afe","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","server.address":"127.0.0.1:8221","timestamp":"2025-07-28T04:30:00.84Z","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.659Z","message":"ack policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"http.request.id":"01K17MPW48G6NABYTJT7BJ994R","server.address":"127.0.0.1:8221","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","trace.id":"2f23ff151429d4adb9a5390842b88afe","service.name":"fleet-server","service.type":"fleet-server","fleet.policy.id":"policy-elastic-agent-on-cloud","transaction.id":"2f23ff151429d4ad","ecs.version":"1.6.0","nEvents":1,"policyRevision":4,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.659Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost (HEALTHY->CONFIGURING): Re-configuring","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost","type":"output","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.659Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server (HEALTHY->CONFIGURING): Re-configuring","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server","type":"input","state":"CONFIGURING","old_state":"HEALTHY"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.833Z","message":"received input from elastic-agent","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"unit.id":"elastic-cloud-apm","service.name":"apm-server","agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","agent.version":"9.1.0","unit.revision":2,"ecs.version":"1.6.0","log.origin":{"file.line":59,"file.name":"apm-server/main.go","function":"main.init.0.func1"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"using output.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"config","log.origin":{"file.line":67,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"using apm-server.agent.config.elasticsearch for fetching agent config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"config","log.origin":{"file.line":73,"file.name":"config/agentconfig.go","function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"Falling back to elasticsearch output for API Key usage","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.logger":"config","log.origin":{"file.line":75,"file.name":"config/auth.go","function":"github.com/elastic/apm-server/internal/beater/config.(*APIKeyAgentAuth).setup"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"Listening on: [::]:8200","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":142,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.listen"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"Stop listening on: 0.0.0.0:8200","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":110,"file.name":"beater/http.go","function":"github.com/elastic/apm-server/internal/beater.(*httpServer).stop"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"stopping apm-server... waiting maximum of 30s for queues to drain","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":197,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run.func1"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"Server stopped","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":243,"file.name":"beater/server.go","function":"github.com/elastic/apm-server/internal/beater.server.run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"stopping aggregator","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"aggregator","log.origin":{"file.line":292,"file.name":"aggregators/aggregator.go","function":"github.com/elastic/apm-aggregation/aggregators.(*Aggregator).Close"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"LSM aggregator stopped","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":194,"file.name":"apm-server/main.go","function":"main.runServerWithProcessors.func1"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.834Z","message":"running final aggregation","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"aggregator","log.origin":{"file.line":304,"file.name":"aggregators/aggregator.go","function":"github.com/elastic/apm-aggregation/aggregators.(*Aggregator).Close"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.835Z","message":"update routine done","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"service.name":"apm-server","ecs.version":"1.6.0","log.logger":"beater.sourcemap","log.origin":{"file.line":130,"file.name":"sourcemap/metadata_fetcher.go","function":"github.com/elastic/apm-server/internal/sourcemap.(*MetadataESFetcher).startBackgroundSync.func1"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.835Z","message":"loaded input config","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"revision":2,"ecs.version":"1.6.0","log.origin":{"file.line":186,"file.name":"beatcmd/reloader.go","function":"github.com/elastic/apm-server/internal/beatcmd.(*Reloader).reloadInputs"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.835Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed apm-es-containerhost-elastic-cloud-apm (CONFIGURING->HEALTHY): Healthy","log":{"source":"elastic-agent"},"component":{"id":"apm-es-containerhost","state":"HEALTHY"},"unit":{"id":"apm-es-containerhost-elastic-cloud-apm","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.836Z","message":"MaxConcurrentDecoders set to 102 based on 80 percent of 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":221,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.837Z","message":"Aggregation.MaxServices set to 1000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"ecs.version":"1.6.0","log.logger":"beater","log.origin":{"file.line":228,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.837Z","message":"Aggregation.ServiceTransactions.MaxGroups for service aggregation set to 1000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":235,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.837Z","message":"Aggregation.Transactions.MaxGroups set to 5000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":242,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:01.837Z","message":"Aggregation.ServiceDestinations.MaxGroups set to 10000 based on 1.0gb of memory","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-es-containerhost","type":"apm"},"log":{"source":"apm-es-containerhost"},"log.logger":"beater","log.origin":{"file.line":249,"file.name":"beater/beater.go","function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run"},"service.name":"apm-server","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:02.274Z","message":"applying new components data","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"trace.id":"cfb8839b0cfbcebfed87ce4a80296ce4","ecs.version":"1.6.0","service.name":"fleet-server","http.request.id":"01K17MPXF25M9TTN49MMP6DHYF","fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","transaction.id":"cfb8839b0cfbcebf","service.type":"fleet-server","server.address":"127.0.0.1:8221","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:02.279Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:02.280Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost-fleet-server-fleet_server-elastic-cloud-fleet-server","type":"input","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:02.280Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.logComponentStateChange","file.name":"coordinator/coordinator.go","file.line":794},"message":"Unit state changed fleet-server-es-containerhost (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-es-containerhost","state":"HEALTHY"},"unit":{"id":"fleet-server-es-containerhost","type":"output","state":"HEALTHY","old_state":"CONFIGURING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:07.537Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:12.793Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:18.063Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"state":"HEALTHY","ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:23.320Z","message":"Running on policy with Fleet Server integration: policy-elastic-agent-on-cloud","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"ecs.version":"1.6.0","service.name":"fleet-server","service.type":"fleet-server","state":"HEALTHY","@timestamp":"2025-07-28T04:30:23.32Z","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-28T04:30:27.785Z","message":"ack event","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-es-containerhost","type":"fleet-server"},"log":{"source":"fleet-server-es-containerhost"},"fleet.access.apikey.id":"qjBKT5gB-5nIJ01cdpVO","nEvents":1,"timestamp":"2025-07-28T04:30:27.523Z","ecs.version":"1.6.0","service.type":"fleet-server","fleet.agent.id":"b29b81eb-0222-4dd2-835f-d8e960738b94","n":0,"fleet.action.id":"policy:policy-elastic-agent-on-cloud:2","trace.id":"e6f949af108a5dbf3eb5287ea6894620","transaction.id":"e6f949af108a5dbf","service.name":"fleet-server","http.request.id":"01K17MQP630D8PW4C2XNXZN8PX","server.address":"127.0.0.1:8221","ecs.version":"1.6.0"}

[cmacknz]

We think the fix here is going to be elastic/elastic-agent#6446 most likely

[juliaElastic]

Any idea why this issue started happening in 8.19 and 9.1?
Do we consider this as a blocker @simitt?
The linked issue looks longer to complete, it needs tech definition.