[Fleet]: Filebeat and Metricbeat running agent logs are not available under Logs tab.

[ghost]

Kibana version: 8.6 BC3 Kibana cloud environment

Host OS and Browser version: All, All

Build details:

VERSION: 8.6 BC3
Build: 58485
COMMIT: b52b34c2ff5216c395bd49c5fbc97744b646f34d
https://staging.elastic.co/8.6.0-94db6bc6/summary-8.6.0.html

Preconditions:

1. 8.6 BC3 Kibana cloud environment should be available.
2. Agent should be installed.

Steps to reproduce:

1. Login to Kibana environment.
2. Install Agent.
3. Navigate to Logs tab.
4. Observe Filebeat and Metricbeat running agent logs are not available under Logs tab.

Expected Result:

• Filebeat and Metricbeat running agent logs should be available under Logs tab.

Screenshot:

Elastic-agent logs:
ubuntu_logs.zip

[dikshachauhan-qasource]

Secondary Review is done

[cmacknz]

Tracked in #1814

[jlind23]

Closing this as we will use #1814 to track the fix.

[cmacknz]

Should be fixed in 8.6. The next BC is currently scheduled for December 6th. The fix should also be available to test in the next snapshot.

[ghost]

Hi @cmacknz
We have revalidated this issue on latest 8.6 BC6 kibana cloud environment and found this issue still reproducible.

Observations:

• Filebeat and Metricbeat running logs are still not available for elastic-agent datasets.

Build details:
BUILD: 58740
COMMIT: f329a77595950244361736dff7208a810299fd69

Screenshots:

Agent Logs:

[Healthy]elastic-agent-diagnostics-2022-12-08T06-42-27Z-00.zip

Hence, we are reopening this issue.

Please let us know if anything else is required from our end.
Thanks

[cmacknz]

I tried the latest 8.6.0 snapshot and I can see logs for Filebeat and Metricbeat if I explicitly opt into displaying their datasets:

Is the expectation that these logs are visible by default? It's not clear from the issue what the success criteria is here. I am wondering if we have just changed the behaviour such that you need to explicitly opt-in to the filebeat+metricbeat datasets rather than seeing them displayed by default.

[ghost]

Hi @cmacknz
The logs here are explicitly fetched for the filebeat and metricbeat by selecting the dataset .

However, earlier the filebeat and metricbeat running logs were visible by default. If we even opt for datasets also, the runnning logs are not present.

Expected behaviour:

Please let us know if we are missing anything.

Thanks.

[cmacknz]

Ah, the log messages have changed with the new agent V2 architecture. The agent state is now reported in a way that more closely matches the way an agent policy is written, with state reported per input+output combination instead of just listing the Beat processes.

In the agent logs you will instead see a message like this for each input listed in an agent policy:

{"log.level":"info","@timestamp":"2022-12-08T06:37:11.675Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":325},"message":"Existing component state changed","component":{"id":"log-default","state":"HEALTHY","message":"Healthy: communicating with pid '7044'","inputs":[{"id":"log-default-logfile-system-153ccfeb-91a3-4b7b-8e54-3bcff0f43581","state":"HEALTHY","message":"beat reloaded"}],"output":{"id":"log-default","state":"HEALTHY","message":"reloaded output component"}},"ecs.version":"1.6.0"}

Specifically you will see "component":{"id":"log-default","state":"HEALTHY" where the id field is the input type plus the output name. In this case this message is for a log input using the default output.

In the agent policy this maps back to a logfile input defined like:

inputs:
  id: logfile-system-153ccfeb-91a3-4b7b-8e54-3bcff0f43581
  meta:
    package:
      name: system
      version: 1.20.4
  name: system-3
  type: logfile
  use_output: default

[dikshachauhan-qasource]

Hi @cmacknz

Thanks for the feedback.

However, could you please confirm if we need to check logs as mentioned above using agent policy input ids and their statuses after agent installation from now.

Further, also, will logs make a new entries each time as per agent policy input ids, whenever we perform actions like, changing agent policy or changing agent log level, deleting/adding any integrations in policy.

Please let us know if you need anything more regarding our queries.

[jlind23]

Closing this as fixed following a conversation I had with @cmacknz.
Craig will also raise this topic during your next weekly sync.

[dikshachauhan-qasource]

Hi @cmacknz

We have checked agent logs on latest build, and updated logs are not available having agent policy input ids.

Screenshot:

Tested on:
Version: 8.6.0 BC7
Artifact link: https://staging.elastic.co/8.6.0-75d87829/downloads/beats/elastic-agent/elastic-agent-8.6.0-linux-x86_64.tar.gz

As discussed in call, we will be waiting for further feedback, so that we can update our testcases.

Thanks!

[dikshachauhan-qasource]

Hi @cmacknz

Update: Today, while performing testing on 8.6 BC7 staging env, I have found required logs under debug logging level.

Screenshot:

[cmacknz]

Thanks. I think this only works for endpoint-security. I'm going to open a follow up issue to figure out why we can't see any of the component details in these logs.

[cmacknz]

#1954

[ghost]

Hi @cmacknz
We have observed the the current logs on the latest 8.6 Snapshot build with different pid status.
Further, we have validate these pid's on the VM.

Screenshots:
Metricbeat services pid

Filebeat services pid:

Query1:
Could you please confirm that the Agent logs will be in running status like this only in the future or else some more changes will be there?

Query2:
There are multiple pid's for filebeat and metricbeat. Could you please confirm upon this behavior?

Please let us know if anything else is required from our end.
Thanks

[cmacknz]

Query1:
Could you please confirm that the Agent logs will be in running status like this only in the future or else some more changes will be there?

Yes this is the last planned change to the agent log format.

Query2:
There are multiple pid's for filebeat and metricbeat. Could you please confirm upon this behavior?

This is correct, in 8.6 there will be one Beat process per input type in the Elastic agent policy. For example, an input of type logfile and an input of type filestream in the agent policy would lead two Filebeat processed being created.

When metrics or logs monitoring is enabled the agent will also spawn an additional Metricbeat process to collect process metrics and a Filebeat process to collect logs.

[ghost]

Hi @cmacknz

Thank you for looking into the queries. As per your feedback we have revalidated this issue on latest 8.6 Bc10 kibana cloud environment also, and found this issue fixed now.

• Filebeat and Metricbeat pid's are running in Agent logs

Build Details:

Version: 8.6.0 BC10
BUILD: 58852
COMMIT: d3a625ef4a6e611a5b3233a1ce5cbe8ef429eb47
Artifacts: https://staging.elastic.co/8.6.0-b6c773f9/summary-8.6.0.html

Screenshot:

Hence, marking this issue as QA:Validated.
Thanks