[docs] Update Logstash output documentation

[bmorelli25]

This PR updates the output config documentation for Logstash. Closes #1631. These changes will also need to be made in elastic/beats.

A preview of the following changes to logstash-output.html is available here.

• Replaces "The Beat" with "APM Server"
• Updates @metadata information
• Adds two sample logstash configs for APM users

Here is the logstash config I got to work with APM. It'd be great if someone with more logstash experience could take a look and recommend any improvements.

input {
    beats {
        port => 5044
    }
}

filter {
    if [@metadata][beat] == "apm" {
        if [processor][event] == "sourcemap" {
            mutate {
                add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" } <1>
            }
        } else {
            mutate {
                add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}-%{+yyyy.MM.dd}" } <2>
            }
        }
    }
}

output {
    elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "%{[@metadata][index]}"
    }
}

In addition, the following configs work with manual ILM. Is one preferred over the others?

input {
    beats {
        port => 5044
    }
}

output {
    elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}"
    }
}

vs

input {
    beats {
        port => 5044
    }
}

filter {
    if [@metadata][beat] == "apm" {
        mutate {
            add_field => { "[@metadata][index]" => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[processor][event]}" }
        }
    }
}

output {
    elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "%{[@metadata][index]}"
    }
}

[bmorelli25]

Hey @simitt. Thanks for the review and sorry this is confusing. It was late and I didn't do a good job of asking my questions. Here's what I'm really getting at:

Our current docs only mention four metadata fields that you can use for Logstash filtering and indexing:

However, when I look at the metadata object in 7.3, this is what I see:

   {  
      "beat":"apm",
      "pipeline":"apm",
      "ip_address":"127.0.0.1",
      "type":"_doc",
      "version":"7.3.1"
   }

So I have three questions:

• type has been deprecated and hardcoded to _doc for a long time. Does it make sense to still document it?
• What is ip_address and should we document it?
• Pipeline it sounds like it makes sense to document.

[simitt]

type has been deprecated and hardcoded to _doc for a long time. Does it make sense to still document it?

I agree not to document it any more.

What is ip_address and should we document it?

I am honestly not aware of where the IP address is coming from..

Pipeline it sounds like it makes sense to document.

Fully agree, it is set to apm by default. See logstash#use-ingest-pipelines for reference of logstash config. If the users want to use another pipeline, they should directly configure it in logstash config, as we don't have a way of configuring it in the apm-server.yml for output different than ES.

[bmorelli25]

Wonderful, thank you! I've updated the PR