Set service.name and event.dataset in logs

[felixbarny]

Currently, the APM Server logs don't contain a service.name field which makes it hard to find the server logs in the logs UI as users can't filter for service.name: "apm-server".
The event.dataset field should also be set to apm-server.log. See the ecs logging spec for more details.

This issue depends on elastic/ecs-logging-go-zap#24

[axw]

I'll need to double check this later, but I'm pretty sure that Elastic Agent (by way of Filebeat) adds event.dataset for APM Server when running under Elastic Agent/Fleet. Seeing as that's the path forward, do you think it's sufficient to rely on Filebeat setting that field?

We can/should probably set service.name for all Beats as well as APM Server.

[felixbarny]

I think the application emitting the logs has the best knowledge on what the service.name and the event.dataset should be. The idea with ECS loggers is that there's no need to configure Filebeat to add specific fields (besides add metadata). The content of the log files is more or less exactly what gets stored in ES. This lets users have the same Filebeat configuration for all of their apps.

We can/should probably set service.name for all Beats as well as APM Server.

👍
I think the best way would be to configure that in the logging setup of each beat.
The event.dataset would then be automatically set to ${service.name}.log by ecs-logging-go-zap, according to the ECS logging spec.

[axw]

I think the application emitting the logs has the best knowledge on what the service.name and the event.dataset should be.

Yes, fair enough. We'll have to see what happens when we do that -- whether Filebeat overrides the specified value, or if it doesn't, whether doing so has any negative effects on Fleet's agent logs UI.

[jalvz]

As I just learnt, service name = beat name. Shouldn't this issue be on Beats repo then?

[axw]

I think the apm-server team will be implementing it either way so ultimately it shouldn't matter. Feel free to transfer it though.