Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Logstash-compatible formatter #612

Merged
merged 1 commit into from
Oct 15, 2018

Conversation

springerigor
Copy link
Contributor

ELK Stack (Elasticsearch, Logstash & Kibana trio) is commonly used today for collecting and presenting logs.

Logstash, to parse logs correctly, needs to have them delivered in a compatible JSON format:

{
  "message"    => "hello world",
  "@version"   => "1",
  "@timestamp" => "2018-10-09T23:03:14.111Z",
  "type"       => "stdin",
  "host"       => "hello.local"
}

Only @timestamp & @version are required.

  • @timestamp is the ISO8601 high-precision timestamp for the event.
  • @version is the version number of this json schema

Every other field is valid and fine.

To support Logstash format by default I introduced a brand new LogstashFormatter. It generates a compatible JSON message with the following fields:

{
  "@timestamp" => "2018-10-09 17:54:58 +0200",
  "@version" => 1,
  "content_length" => "512",
  "http_method" => "GET",
  "message" => "[HTTParty] 200 \"GET http://my.domain.com/my_path\" - ",
  "path" => "http://my.domain.com/my_path",
  "response_code" => 200,
  "severity" => "info",
  "tags" => ["HTTParty"]
}

The required fields are present + some additional ones which I found useful to have in Kibana for filtering & debugging purpose. Looking forward to your comments.

ELK Stack (Elasticsearch, Logstash & Kibana trio) is commonly used today for collecting and presenting logs.

Logstash, to parse logs correctly, needs to have them delivered in a compatible JSON format:

```
{
  "message"    => "hello world",
  "@Version"   => "1",
  "@timestamp" => "2018-10-09T23:03:14.111Z",
  "type"       => "stdin",
  "host"       => "hello.local"
}
```

Only `@timestamp` & `@version` are required.

* `@timestamp` is the ISO8601 high-precision timestamp for the event.
* `@version` is the version number of this json schema

Every other field is valid and fine.

To support Logstash format by default I introduced a brand new `LogstashFormatter`. It generates a compatible JSON message with the following fields:

```
{
  "@timestamp" => "2018-10-09 17:54:58 +0200",
  "@Version" => 1,
  "content_length" => "512",
  "http_method" => "GET",
  "message" => "[HTTParty] 200 \"GET http://my.domain.com/my_path\" - ",
  "path" => "http://my.domain.com/my_path",
  "response_code" => 200,
  "severity" => "info",
  "tags" => ["HTTParty"]
}
```

The required fields are present + some additional ones which I found useful to have in Kibana for filtering & debugging purpose. Looking forward to your comments.
@springerigor springerigor changed the title Add Logstash-compatible logger Add Logstash-compatible formatter Oct 11, 2018
@jnunemaker jnunemaker merged commit 4917a5f into jnunemaker:master Oct 15, 2018
@springerigor springerigor deleted the logstash-formatter branch October 15, 2018 17:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants